Hi,
I have been trying to get a rather large CSV into a Drupal content type using Feeds for over a month now. The CSV has about 80,000 records, but each record has only 5 short fields. In every attempt I have tried, Feeds runs for a while and then crashes the server. The last run brought in 5,000 records but has also done damage to the database. If I try and import this database now I get the following error:
SQL query:
-- -- Dumping data for table `feeds_source` -- INSERT INTO `feeds_source` (`id`, `feed_nid`, `config`, `source`, `batch`) VALUES ('new_site_part_importer', 0, 'a:2:{s:16:"FeedsHTTPFetcher";a:1:{s:6:"source";s:40:"http://www.site.com/dev/siteParts2.csv";}s:14:"FeedsCSVParser";a:2:{s:9:"delimiter";s:1:",";s:10:"no_headers";i:0;}}', 'http://www.site.com/dev/siteParts2.csv', 'O:14:"FeedsHTTPBatch":13:{s:6:"\0*\0url";s:40:"http://www.site.com/dev/siteParts2.csv";s:12:"\0*\0file_path";s:84:"sites/default/files/feeds/FeedsHTTPBatch_e4887f3e227df8a27d51a0f08bfb8c4c_1314329688";s:15:"\0*\0current_item";a:5:{s:7:"part_id";s:6:"0-2310";s:11:"description";s:11:"FELT WASHER";s:8:"licensor";s:20:"site Owned (company)";s:2:"fn";s:1:"1";s:3:"mro";s:1:"0";}s:6:"\0*\0raw";s:0:"";s:5:"title";s:0:"";s:11:"description";s:0:"";s:4:"link";s:0:"";s:5:"items";a:59950:{i:0;a:5:{s:7:"part_id";s:6:"0-2311";s:11:"description";s:11:"FELT WASHER";s:8:"licensor";s:20:"site Owned (company)";s:2:"fn";s:1:"1";[...]
MySQL said:
#2006 - MySQL server has gone away
I should mention that this database is currently running okay on the server. However if I try and bring it down, either via backup and migrate OR a phpMyAdmin database export, I get the exact same error at the exact same place, right when the import tries to bring in the feeds_source table.
What would you suggest I do?
Do I have any hope of ever importing this CSV?
Thank you.
Comments
Comment #1
mcfilms commentedIt's been a couple weeks and I thought I might bump this issue to see if anyone has an "aha moment."
Is it possible for me to scrub this section from the database and then bring it into my local environment? I can't really use the database if I can't read it in MAMP.
Comment #2
adbo commentedsubscribe
Comment #3
colanSome folks I work with say that Feeds runs out of RAM in these situations. Writing new plug-ins that store import data on disk instead of keeping it in memory would be a solution here.
Comment #4
dooug commentedUtilizing the batch process api to break the feeds imports into smaller chunks might be better solution:
#1363088: Feeds batch process not fully implemented - needed for large imports
Comment #5
mcfilms commentedAs a note, I was able to use Node Import with the Job Scheduler module and imported all my data successfully. The maintainers of Feeds might want to look at how Node import can (successfully) import a huge amount of data.
Comment #6
ayesh commentedRelated: #744660: Expand batch support to fetchers and parsers (D6 tho)
Comment #7
emackn commentedFeeds is not a data migration tool. Have you looked at Migrate?
Comment #8
mcfilms commentedMigrate does sound like a good possible solution. I chose Feeds because it was pitched as the successor to Node Import and I had successfully used Node Import to create nodes from a CVS previously. Ultimately Feeds was not up to the job. What DID work for me was to go back to the Node Import module. Using this and the Job Scheduler module allowed me to import this large database. It took hundreds of cron runs but after a number of days the task was complete.
But looking at the Migrate module I see this would have probably been the best solution. In fact it looks like you can import the data right out of another SQL database. I only have one question: Where were you four months ago?
;)
Comment #9
colanMakes sense to me.