The import process right now works like this:

1. Read every file in the content directory and build a graph of dependencies based on hard-coded assumptions (look for target_type and target_id and assume it's a content reference).
2. Sort the graph.
3. Chunk the graph into an array of arrays which can be processed concurrently.
4. Spawn multiple processes to process batches of entities from each chunk.

This is really fast, but that step (1) has me worried - what if you had so much content that PHP ran out of memory loading all the files? What happens when we want to detect a dependency (create a graph edge) that isn't from an entity reference field? It seems like something we should fix now.

My plan:

1. On _export_, write an index file in content/ that stores all edges. This lets us use the real entity when calculating dependencies which has a lot of benefits.
2. On _import_, read the index file into a graph object, sort it, and chunk it. This could still be slow at scale but is way less data to process.
3. Spawn multiple processes to process batches of entities from each chunk.

I'll need to provide a script for people already on Tome to generate the index if it does not exist, as well.

Comments

samuel.mortenson created an issue. See original summary.

samuel.mortenson’s picture

Status: Active » Needs review
StatusFileSize
new9.54 KB

My brain hurts! This should do the trick though.

For my few (if any) users on existing Tome builds, I've added a temporary command to build a new index file. Once this is committed, you'll need to update and run drush tome:temp-build-index to get a new index. From that point on the index file should be updated automatically.

Status: Needs review » Needs work

The last submitted patch, 2: 2981422-2.patch, failed testing. View results

samuel.mortenson’s picture

Status: Needs work » Needs review
StatusFileSize
new16.07 KB

Bad patch.

  • samuel.mortenson authored 12c5a83 on 8.x-1.x
    Issue #2981422 by samuel.mortenson: Store content dependencies in...
samuel.mortenson’s picture

Status: Needs review » Fixed

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.