Automatic updates: For MVP, instead of an a/b bootloader, implement a semi-atomic composer update with a brief maintenance-mode window

Problem/Motivation

See #2940731: Automatic Updates Initiative overview and roadmap for the larger context of the Automatic Updates initiative.

• For this issue, the goal is to allow automatic updates to be performed on a production site, meaning a site that is serving requests while being updated. This is for the benefit of the majority of Drupal sites, which are on a single server (whether shared hosting, VPS, or dedicated) with only a production environment and no existing deployment process other than manually copying new Drupal releases (whether via FTP or git pull or whatever) onto the production server. This is not for sites with advanced hosting, such as sites with multiple web nodes, dev/stage/prod environments, or required deployment processes, because those sites will either not run Drupal core's automatic updates at all, or will run them in a non-production environment and then trigger their existing deployment processes to deploy to production.
• For the case where we're performing the update directly on the production environment, we need to prevent the situation of production requests being served (other than ones deemed sufficiently safe, such as page cache hits or the "your site is in maintenance mode" message) while the site's codebase is in an in-between state (e.g., while a composer update is in progress).
• The most robust way to do this is with an a/b bootloader. #3093093: Replace single index.php with a frontend controller that supports A/B updating (automatic updates) is the issue for that. The idea of that is that instead of a single docroot directory and vendor directory, you have an "a" copy and a "b" copy of each, and an index.php file that routes to one of them while an update is performed on the other one, and then the index.php file switches which one it routes to after the update is finished. However, there are a number of challenges with implementing that, which I won't comment on here, but we should update that issue's summary with a list of what those challenges are.
• In this issue, we (the Automatic Updates initiative team) are proposing an alternative solution that's slightly less robust, but easier to implement, which we recommend doing as an MVP. See the Proposed Resolution for details.

Proposed resolution

• Create a library (Composer Stager) that implements 3 Symfony console commands: begin, stage, and commit (the "begin" and "commit" nomenclature is borrowed from database transactions). As a separate library, this can be used by, and hopefully co-maintained by, other PHP projects in addition to Drupal.
  • begin: Copies the working directory (the directory where the root composer.json file is) to a staging directory.
  • stage: Runs whatever follows it as a Composer command within the staging directory. For example, composer-stager stage update or composer-stager stage require foo would run composer update or composer require foo within the staging directory. Multiple composer-stager stage ... commands could be run prior to calling commit.
  • commit: Efficiently copies (e.g., with rsync or similar) all differences between the staging directory and the original working directory back to the working directory.

  This reduces the amount of time that the working directory is in an in-between state to the duration of the rsync (or similar) operation, rather than the duration of a composer update.

• Although an rsync is fast, it's not perfectly atomic. Therefore, the Drupal code within the (not yet written) autoupdates module that interacts with this library can put the site into maintenance mode prior to calling commit and take it out of maintenance mode after that command finishes. The key here is that the maintenance mode window is only for the duration of the commit command. The earlier commands can happen without the site being in maintenance mode.
• We need to exclude from the rsync operation directories that contain content files rather than code files. In other words, files that are written as part of the normal operation of the production site, such as the sites/default/files directory. This is because the contents of these directories can both be very large and can change on the production site at the same time as composer operations are getting applied to the staging directory, and we don't want to revert those production site changes when we rsync the staging directory back to the production directory.

Remaining tasks

• An earlier proof of concept of this was Staged Composer Update, which had these open issues. We need to move the ones that are still relevant to the new Composer Stager project.
• Meanwhile, in this drupal.org issue, please comment with whether you agree with or have concerns about the approach in general.

User interface changes

API changes

Data model changes

Release notes snippet