The static site generator has race conditions when dealing with assets associated with a page. When a page is processed by the static generator, any assets (CSS files, images, etc.) are immediately processed by either performing a file copy, or making another request to Drupal (i.e. in the case of image styles, Drupal needs to generate a derivative).

If you've seen any errors with touch() or files not existing, this issue is for you.

Something like flock _could_ help, but I think you would still end up with race conditions for image styles, or at the worst extra processing time as each process runs the same logic.

I think the best solution for this is to process every path initially collected by the generator, then do a new set of concurrent processes for their assets. It might look something like:

1. Collect paths - ex: /, /node/1, /home
2. Process paths concurrently
3. Each process returns assets it found in a pipe
4. Parent process filters duplicate paths
5. Parent process copies all files that are found on disk
6. Start the event loop again at (1) for paths that need a new Drupal request
7. Continue 1-6 until no new paths are found

Right now each process spawns even more processes for new paths Drupal needs to handle, which means that if you pass --process-count=5 you could actually end up with 25 concurrent processes, which is pretty bad.

CommentFileSizeAuthor
#3 2980526.patch5.4 KBsamuel.mortenson

Comments

samuel.mortenson created an issue. See original summary.

samuel.mortenson’s picture

Issue summary: View changes
samuel.mortenson’s picture

Status: Active » Needs review
StatusFileSize
new5.4 KB

This patch adds flock, as well as stops processes from spawning even more processes. This is a bit more annoying because you have multiple progress bars for each run, but is safer and should prevent race conditions.

  • samuel.mortenson authored 0595103 on 8.x-1.x
    Issue #2980526 by samuel.mortenson: Fix race conditions in the static...
samuel.mortenson’s picture

Status: Needs review » Fixed

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.