Augment then Replace current Update Manager URL download based updates with Staged-Composer workflow

TLDR

We should add an option to use a Composer aware backend using the Composer Staged Update plugin to the current Update module update functionality, first supporting modules and themes and then core updates. The UI could remain largely the same.

We should do this before we support unattended Automatic Updates because this does not open new security concerns compared to the current Update module and it allows us to implement a portion of the functionality eventually needed for unattended Automatic Updates.

Problem/Motivation

This issue is meant to address a few problems:

Current Update Manager is not Composer aware

Currently the Update module provides the ability to update extensions via downloading tar files, extracting them into a temporary location, and then replacing the current extension directory.

Current steps

1. Step 1: select from available updates and download.
   
2. Step 2: Perform Updates(step downloaded into temp folder)
   
3. Step 3: Prompt to run database updates
   

This process does not take Composer into account. This could lead to various problems

1. The extension might already be in the sites composer.json/.lock files but updating will not update the version in these files
2. The new version of the extensions may require updates to dependencies specified in it's composer.json file and these updates will not be performed.
3. It might require new dependencies in it's composer.json file and these dependencies will not be installed
4. This could lead to a mix of some extensions being update via the command line using Composer and some being updated via the browser with composer being unaware of the changes
5. Once Drupal core provides Composer based automatic update functionality this would likely mean that any site that used our current Update functionality would be compatible with Automatic Updates as the composer.lock file would not be up-to-date

The current Update manager does not support updating Drupal core.

In the Automatic Updates initiative we are planning on providing functionality to update Drupal core patches releases(as an MVP). Part of the work of doing this will be creating a Drupal agnostic(probably) Staged Update Composer Plugin.

This plugin is a work in progress but the imagined workflow would be:

1. Copy over the existing composer installation including /vendor, core, and extensions but excluding directories like /sites/*/files into a temp directory
2. Run composer update(or require(tbd) in the temp directory
3. Check the update
4. In maintenance mode copy back just the new/updated files

see #3199171: Automatic updates: For MVP, instead of an a/b bootloader, implement a semi-atomic composer update with a brief maintenance-mode window for more specifics

We are also working on providing extra security for core updates by implementing The Update Framework. This is needed because these would be unattended updates so we want to be extra cautious and we don't want to trust the server's TSL implementation.

Currently though we are trusting the server's TSL to provide updates. Whether we update via Core or extensions via TSL we are not opening to different risk because extensions could pretty much do anything(even replace core?).

Currently the Automatic Updates Initiative has been focused on making this possible for unattended updates but presumably some users will not want to do this in an unattended fashion but will want to do this at a specific time whether this is to make sure someone is there to check the update or so that update does not happen during a busy time for their site or so that Drupal vulnerabilities cannot be exploited to make Drupal overwrite its own code. So we will probably have to build a "Update Now" button to do the same thing the unattended version does.

If we have to build the unattended update functionality should be pretty identical just triggered by a user interaction instead of cron.

Either way at the end we will have a Composer aware way to update core and way to update extensions that is not aware of Composer.

Automatic Update functionality will not be rolled out gradually

Currently we are working on many different parts of the Automatic Update Initiative inside and outside core. While some of the parts will be finished before others the plan for the first production use of this system is to be for patch level core unattended updates. This will not be experimental functionality but likely in a stable module.

This will require these parts to work together in an unattended fashion

1. PHP-TUF
2. PHP-TUF composer integration plugin
3. Drupal.org TUF repo which will also mirror some or all of packagist. // @todo find write up
4. Composer Stage Update plugin
5. The core module to validate and trigger the update

Since this will likely be in a new Automatic Updates core module the first use of the module to actually update core will be when the next patch release(or security release if we only support that) after the patch release containing the new functionality is published. Since these will be unattended updates all sites that have the functionality enabled will be triggered within a very short time frame(or the next time they have page a request if they are using automated_cron. This could be 10's of thousands of site in a short period of time.

The proposal in this issue would allow parts 4 & 5 to be used and hardened before all the parts have to work together. There may also be a way to use #1 first by making our Update XML targets of a new TUF repo hosted served Drupal.org. This TUF repo would be much simpler that the TUF repo that would have to mirror packagist.

Proposed resolution

We should consider augmenting and eventually replacing our functionality to Update extensions through the browser with the same process we will use for Automatic Updates. The Automatic Updates workflow should be able handle updating extensions with little changes. We probably can't remove the current URL download based updating until Drupal 10.

Updating via Composer would have to be available as an additional method at first because not allowing the current method would stop sites that have used the current method not from updating(because their composer.json is probably not valid).

Most of the readiness checkers we make after #3162655: Create Automatic Updates Readiness Checks in new experimental module should be applicable to replacing our current update functionality with Composer. Currently \Drupal\update\Form\UpdateManagerUpdate currently just has a simple flag making Drupal core updates not available through the UI. This flag could be removed in the case that Composer is being used.

Remaining tasks

1. Determine in which module the new Composer update functionality should live. Probably either current Update module or a new Automatic Updates module.
2. Determine in what order new functionality should be added:
   A proposed order could be
   • #3162655: Create Automatic Updates Readiness Checks in new experimental module but possibly in the Update module itself
   • Add all update checkers needed for updates. These will likely be the same as needed for core's eventual unattended updates.
   • Add a Composer version of current update functionality which is only extensions not core.
   • Add ability to update core using the method above removing flag in \Drupal\update\Form\UpdateManagerUpdate when using the Composer method
   • Add unattended patch level updates for core once PHP-TUF composer workflow is possible including drupal.org providing signatures of all vendor packages during updates

User interface changes

A way to opt-in to the new Composer functionality for Updating extensions. We may be able to use the current UI that is used for selecting updates.

API changes

A new Composer Updater that uses Staged Composer Update Plugin

Data model changes

None

Release notes snippet

TBD