Parsing big xml file (250 mo /15.000 nodes) [#1213324]

Hi,

New with parsing xml file ( thank you Drupal) . So i have a big file to parse.

First problem, i can't parse it with http method, and can't upload a local file (200 mo !). Need a third method, like upload the file via ftp inside files folder, and call the xml. Any ideas.

Is Feed Xpath parser done for this kind of big import ( more than 15.000 nodes) ?

Thanks,
Florent

Comments

Comment #1

twistor commented 15 July 2011 at 22:51

As far as your first question, select the "File upload" fetcher. In the settings section for the fetcher, check "Supply path to file or directory directly." This will allow you to specify the location of the file on your server.

For your second question, it depends on the amount of memory provided to PHP, the allowed execution time, and the size of each feed item.

I would recommend splitting up your file into multiple files(still using the method above). That will allow Feeds to batch over each file, providing more manageable chunks.