The static site generator has race conditions when dealing with assets associated with a page. When a page is processed by the static generator, any assets (CSS files, images, etc.) are immediately processed by either performing a file copy, or making another request to Drupal (i.e. in the case of image styles, Drupal needs to generate a derivative).
If you've seen any errors with touch() or files not existing, this issue is for you.
Something like flock _could_ help, but I think you would still end up with race conditions for image styles, or at the worst extra processing time as each process runs the same logic.
I think the best solution for this is to process every path initially collected by the generator, then do a new set of concurrent processes for their assets. It might look something like:
1. Collect paths - ex: /, /node/1, /home
2. Process paths concurrently
3. Each process returns assets it found in a pipe
4. Parent process filters duplicate paths
5. Parent process copies all files that are found on disk
6. Start the event loop again at (1) for paths that need a new Drupal request
7. Continue 1-6 until no new paths are found
Right now each process spawns even more processes for new paths Drupal needs to handle, which means that if you pass --process-count=5 you could actually end up with 25 concurrent processes, which is pretty bad.
| Comment | File | Size | Author |
|---|---|---|---|
| #3 | 2980526.patch | 5.4 KB | samuel.mortenson |
Comments
Comment #2
samuel.mortensonComment #3
samuel.mortensonThis patch adds flock, as well as stops processes from spawning even more processes. This is a bit more annoying because you have multiple progress bars for each run, but is safer and should prevent race conditions.
Comment #5
samuel.mortenson