The 2026 Drupal Business Survey is open. Your response will help shape the future of the Drupal business ecosystem.I'm dealing with Ning exports, which involve massive JSON files (e.g., for 37K members we have a 30MB file to content with, and Ning allows exporting up to 100K members). All PHP gives us to work with is json_decode(), which takes an entire string at once, so we need to find a way to incrementally parse the JSON file, akin to MigrateSourceXML which makes use of the XMLReader class.
Comments
Comment #1
mikeryanI've committed the code, need to document it.
Comment #2
mikeryanTagging feature requests I'd like to get into Migrate 2.6.
Comment #3
luksakMy migration script with a list of 14000 entries creating commerce products and a lot of field collections is getting stuck at some point. It is using highwater marks even though it does not really help performence when not using a SQL source, right? So, after migrating 700 products at a rate of 10/min the speed drops even more to one product per 2500 seconds. Something is really slow. Is this the source of this problem?
Comment #4
mikeryanThat doesn't sound like the big-JSON-handling source would be the bottleneck, most likely there's some unindexed query that's getting really slow as the table it's querying is getting big. I suggest profiling with xhprof, or the poorman's migration profiler, drush mi --instrument=timer
Comment #5
mikeryanOK, documentation finally done at http://drupal.org/node/1152160#MigrateSourceJSON.