We need locking around the css and js aggregation. We added locking to other subsystems in Drupal 7, and this one was just missed. Without locking, multiple processes try to build and rebuild the css and js. On our site, this puts a high load on the file system, which causes apache to throw 503 errors.
Also, if you simply update the css or js, the files will not rebuild. Say you upgrade a module, and it adds 5 lines to it's css, or say you modify your theme and change things around, if the bundle is already built, and if no new files were added to the page, then the bundle will not rebuild. I've introduced a "version" for this. The intent of the "version" is that some contrib module can be written to allow incrementing the version either from an admin page or via drush, and that the site developers will know when this needs to be done, and will simply increment the version, forcing a rebuild.
An additional problem surfaced while trying to solve this, which is that if the creation of the css file fails, we currently return FALSE; however the function that calls drupal_build_css_cache() doesn't interprete the result, and simply adds the empty string as a css file. This is an obvious bug. We need to handle the FALSE return in css, just like we do in js.
So my solution is to use locks so that we generate each bundle in only one process (allowing multiple bundles to build simultaneously, but making sure only one process builds each bundle), to use a global lock when saving the map, returning stale bundles during the rebuild or on error.
Anyone new to this issue can problem skip ahead to #30 or so.
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] Unable to apply patch css-stampede-886488-100.patch. Unable to apply patch. See the log in the details link for more information. View
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch css-stampede-886488-86_0.patch. Unable to apply patch. See the log in the details link for more information. View
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch css-stampede-886488-86.patch. View
PASSED: [[SimpleTest]]: [MySQL] 31,527 pass(es). View
PASSED: [[SimpleTest]]: [MySQL] 29,849 pass(es). View