I'm dealing with Ning exports, which involve massive JSON files (e.g., for 37K members we have a 30MB file to content with, and Ning allows exporting up to 100K members). All PHP gives us to work with is json_decode(), which takes an entire string at once, so we need to find a way to incrementally parse the JSON file, akin to MigrateSourceXML which makes use of the XMLReader class.

Comments

mikeryan’s picture

mikeryan
he/him
Primary language English
Location Pittsfield, MA, USA
commented
Component: Code » Documentation
Status: Active » Needs work

I've committed the code, need to document it.

mikeryan’s picture

mikeryan
he/him
Primary language English
Location Pittsfield, MA, USA
commented
Issue tags: +Migrate 2.6

Tagging feature requests I'd like to get into Migrate 2.6.

luksak’s picture

luksak
he/him
commented

My migration script with a list of 14000 entries creating commerce products and a lot of field collections is getting stuck at some point. It is using highwater marks even though it does not really help performence when not using a SQL source, right? So, after migrating 700 products at a rate of 10/min the speed drops even more to one product per 2500 seconds. Something is really slow. Is this the source of this problem?

mikeryan’s picture

mikeryan
he/him
Primary language English
Location Pittsfield, MA, USA
commented

That doesn't sound like the big-JSON-handling source would be the bottleneck, most likely there's some unindexed query that's getting really slow as the table it's querying is getting big. I suggest profiling with xhprof, or the poorman's migration profiler, drush mi --instrument=timer

mikeryan’s picture

mikeryan
he/him
Primary language English
Location Pittsfield, MA, USA
commented
Status: Needs work » Fixed

OK, documentation finally done at http://drupal.org/node/1152160#MigrateSourceJSON.

Automatically closed -- issue fixed for 2 weeks with no activity.