Drupal Core Roadmap for Package Manager and Update Manager

Removed #314137: Userpoints no negative not working properly since that's an issue of another module. I think there was a typo there :P

Added #3314137: Make Automatic Updates Drupal 10-compatible and #3319045: Assert that all expected package manager events are fired during build tests..

Adding #3318587: Research alternatives to curl request on post apply event during cron updates.

Oh it already was in there, it just had not yet been tagged…

Added #3311200: Cron updater should delete the existing stage if not available and the site is currently on an insecure version to the "security" gate.

I think #3318964: automated_cron should not run cron when visiting update.php would be a beta or stable blocker, even though it's technically a bug in Drupal core, not in Automatic Updates. It represents too much risk when AU is installed for it to remain unsolved.

I think #3252299: Reliably support cweagans/composer-patches in Package Manager & Automatic Updates: validate stage should be an alpha blocker but could be convinced it should be a beta blocker. Stable blocker is too late/risky IMHO. See #3252299-16: Reliably support cweagans/composer-patches in Package Manager & Automatic Updates: validate stage.

#3321905: Add colinodell/psr-test-logger to core's dev dependencies added.

We'll need similar issues for

• php-tuf/composer-stager
• @todo symfony/config
• @todo symfony/finder

… unless we can get release managers to pre-approve these. The latter 2 should be trivial, since it's a pre-existing upstream vendor.

Either way, we need to do this before the actual core merge request can get merged, because dependencies need to be individually vetted first anyway. There's no reason we need to wait to do this.

Furthermore, this has consequences for the alpha experimental state AFAICT: just excluding the module from packaged releases is insufficient; it'd still include these dependencies. We need to surface that to release managers ASAP.

Adding #3321933: Remove dependency on symfony/finder (addresses part of #17).

Added #3322917: Create a test (or tests) to prove Package Manager works with submodules as implemented by packages.drupal.org to Reliability.

Adding #3323461 to Reliability because #3323461-5: Hosting environment (e.g. cPanel) may add additional files (including symlinks) to the project, which breaks AU.

Adding #3311229 to Reliability because of #3311229-5: Validate compliance with composer minimum stability during PreRequireEvent.

Added #3321972: Make readiness check failure messages clear, consistent, and actionable to post-alpha, under Usability.

Discovered one more alpha blocker unfortunately: #3316368: Remove our runtime dependency on composer/composer: remove ComposerUtility.

Added #3325522: Automatic Updates & Package Manager should use DependencySerializationTrait when needed to Reliability.

Added #3267646 to Reliability per #3267646-7: Refine multisite detection: many aliases for a single site is fine!.

Added #3329002: \Drupal\Tests\package_manager\Kernel\TestStageTrait::dispatch() should stop using a deprecated PHP language feature to core mergeability.

Replaced @todo php-tuf/composer-stager with the actual issue I just created: #3331078: Add php-tuf/composer-stager to core dependencies — for experimental Automatic Updates & Project Browser modules.

Added new Public API section for #3331355: Refactor exception architecture.

Got this issue in sync with the actual reality for the currently 180 core-mvp-tagged issues.

Added the following from the 3rd and 4th page of issues (80 issues total) tagged core-mvp (which means I still have to check the 100 most recently updated issues with those tags!)

1. #3314787: StagedDBUpdateValidator will throw an exception during status check if the stage exists, but is unclaimed
2. #3314143: Add documentation for testing minor updates
3. #3315798: Post-apply always fails during cron
4. #3304365: Do not check excluded folders for symlinks
5. #3280403: Show all updates for all supported branches in the current major on the update form
6. #3316611: If unattended updates are enabled, send an email when status checks start failing
7. #3320487: fake-site fixtures has invalid packages in composer files
8. #3281340: [Plan] Send emails
9. #3316668: ComposerSettingsValidator should run `composer config` to determine if HTTPS is enabled
10. #3326486: Rename Stage to StageBase to clarify its relationship to its subclasses, and add "Stage" suffix to the Updater classes
11. #3320824: Fix PHP Warning: serialize() in tests on PHP 8
12. #3327229: Remove `@requires PHP >= 8.0` annotation from all tests
13. #3323706: Split up UpdaterFormTest to speed up test runs: from 13.5 to 10.5 minutes
14. #3323003: Mark FailureMarker @internal and document ComposerUtility, PathLocator and ValidationResult
15. #3322931: OverwriteExistingPackagesValidator does not handle packages with no install path
16. #3321386: Add missing test coverage in SupportedReleaseValidatorTest
17. #3322203: Make sure packages in our fake-site fixture match in both installed.json and installed.php
18. #3322918: Mark everything under */tests/src @internal
19. #3323211: Remove duplicate aaa_update_test.1.1.xml
20. #3323037: Add comment in every release history XML fixture file
21. #3322404: Remove active composer fixture files used for SupportedReleaseValidatorTest
22. #3322589: Assert 'type' key is set in FixtureUtilityTrait::addPackage
23. #3316484: Check if the message "Your site has not recently run an update readiness check." would appear on admin pages after cron is run.
24. #3322150: Rename remaining instances of 'test_theme' to 'automatic_updates_extensions_test_theme'
25. #3318625: Remove active composer fixture files in UpdaterFormTest
26. #3275991: Write test coverage for the bug fix in #3275311
27. #3318846: Remove staged fixture files used for SupportedReleaseValidatorTest
28. #3320638: Give more meaningful error results in CoreUpdateTest::testCron
29. #3318313: Make OverwriteExistingPackagesValidatorTest fixture-less
30. #3319044: Refactor assertUpdateSuccessful() to get more meaningful error results.
31. #3320815: Make validation result comparison test messages more helpful
32. #3319497: Build test failure on local in 8.x-2.x due to drupal core packages being added to vendor.json
33. #3310666: UpdaterForm throws an exception if you try to update to the next minor beta
34. #3316895: Create a test to prove our fake-site is error free starting point
35. #3317988: Make ComposerExecutableValidatorTest easier to understand
36. #3317996: FixtureUtilityTrait's package manipulation methods need to take installation path into account
37. #3317385: Remove staged fixtures from StagedProjectsValidatorTest
38. #3317599: Log an error if protected Stage constants are overridden
39. #3317267: Deprecate the overridability of Stage::TEMPSTORE_LOCK_KEY and Stage::TEMPSTORE_METADATA_KEY
40. #3317278: Remove info files from stub modules in StagedProjectsValidatorTest fixtures
41. #3317409: Update Composer Stager to v1.2.0
42. #3317220: UpdatePathTest fails because expirable values have expired
43. #3298889: Make \Drupal\package_manager_test_api\ApiController more easily extendable
44. #3315700: Remove Environment::setTimeLimit() call from build tests
45. #3316115: Change "readiness check" to "status check" everywhere except user-facing strings
46. #3316318: Ensure release history fixture files are readable in build tests
47. #3257134: Don't stage node_modules directories
48. #3310696: if only newer releases are pre-releases Updating from one minor to latest version Drupal version, module does not show any other updates
49. #3316131: Add array return type hint to all implementations of getSubscribedEvents()
50. #3314764: Stop listening to ReadinessCheckEvent
51. #3302527: Updater::dispatch() doesn't need to handle a StageValidationException with no message
52. #3317232: Stages can be owned by someone other than their creator
53. #3314734: automatic_updates_extensions\BatchProcessor does not ensure that PostApply runs in another web request
54. #3315449: Don't bother testing core updates via the UI or API with the LegacyProject template
55. #3314805: Create a base class for UpdaterForm and UpdateReady to help run status checks and display the results
56. #3313349: Possible DB updates are displayed twice on UpdateReady form
57. #3314946: UpdateReady mistakenly assigns a special CSS class to all validation results
58. #3315139: package_manager.update_data_subscriber does not have to be declared dynamically
59. #3312937: Fix spelling mistakes
60. #3313717: Improve the wording of UpdateException when re-throwing an ApplyFailedException
61. #3310946: Improve the wording of the email notifications about a failed unattended update
62. #3312669: Add help text explaining how to set an alternate port for cron

After that: splitting in done vs remaining and in automatic_updates vs package_manager.

I untagged two issues.

As promised, the 2nd page, good for another 50 issues …

Untagged: #3308843: Automatic Updates Extensions' forms should check for the failure marker, #3303900: Remove PreApply check in PackagesInstalledWithComposerValidator, #3307168: \Drupal\Tests\automatic_updates_extensions\Kernel\AutomaticUpdatesExtensionsKernelTestBase::setReleaseMetadataForProjects is a duplicate of \Drupal\Tests\automatic_updates\Kernel\AutomaticUpdatesKernelTestBase::setReleaseMetadata

Added:

1. #3313630: Automatic Updates help has the wrong route name and errors
2. #3109082: Allow hosting platforms to declare that they don't support Package Manager
3. #3310914: LockFileValidator should listen to StatusCheckEvent
4. #3312981: Move XdebugValidator into Package Manager
5. #3302524: Support Composer 2.2
6. #3313346: UpdateReady is not displaying all the messages for statusCheck validation results
7. #3303124: [meta] Package Manager should add validation to try avoid conflicts with non-Composer installed code
8. #3299087: Override the core Update module's routes to redirect to Automatic Updates' form
9. #3312963: Package Manager's requirements titles should be translated
10. #3312779: Improve Composer package name validation
11. #3293146: Don't run cron updates with PHP's built-in web server without an alternate port
12. #3310901: Stage::require() should validate the incoming package names
13. #3307611: Create a validator to add a warning if updated extensions have database updates
14. #3311534: package_manager_requirements() should check for the presence of the failure marker
15. #3312085: UpdateReady hides the Continue button if StatusCheckEvent only returns warnings
16. #3311020: Rename staged/9.8.1 to something clearer
17. #3308828: Handle the case where the marker file has been detected but can't be decoded
18. #3305167: Warn if apply and post-apply are done in same request
19. #3311436: Stage::require() should override the default process timeout to 300 seconds
20. #3308711: Dispatch StatusCheckEvent in UpdateReady forms and do not allow the update to continue if there are errors
21. #3293417: If an update failed to apply don't allow more use of the module
22. #3306163: Also skip info.yml files in more test directories in DuplicateInfoFileValidator
23. #3298444: Symlink validator should delegate to Composer Stager's precondition
24. #3307369: Validate all changed or updated Drupal projects with Update XML
25. #3303185: Add unit test coverage of PathLocator::getWebRoot()
26. #3299612: Send an email when an unattended update fails
27. #3310702: Stage::__construct() should require $path_factory
28. #3310000: RequireEventTrait should default unspecified version constraints to *
29. #3309025: Document that Stage::require() does not need version numbers
30. #3308686: Logic error in XdebugValidator
31. #3309676: \Drupal\Tests\automatic_updates\Kernel\ReadinessValidation\StagedProjectsValidatorTest::testEventConsumesExceptionResults should not disable validators
32. #3305564: Create a validator to stop newly installed packages from overwriting existing directories on apply
33. #3308404: Make PackageManagerUpdateProcessor internal and final
34. #3305874: \Drupal\package_manager\ComposerUtility::getPackageForProject() assumes direct project to package conversion when it should not
35. #3305240: Add a link to the the symlink validation message in package_manager to the updated help page
36. #3304617: Display readiness checks failure at admin/reports/updates/automatic-update
37. #3304367: Add StatusCheckEvent to report errors and warnings in the staging error to the user
38. #3307163: Add the ability to ProjectInfo handle projects not in the active codebase
39. #3303902: Move ProjectInfo class into package_manager
40. #3304640: Logic error in \Drupal\automatic_updates\Controller\UpdateController::onFinish
41. #3305994: We don't want to run the XdebugValidator for non-Automatic Updates stage.
42. #3305568: Create a validator that detects duplicate info.yml files in the stage on apply
43. #3305998: Ensure AutoUpdate validator only apply where the stage is an Updater instance
44. #3303727: Document in README how to add paths to composer.json:extra.drupal-core-vendor-hardening to avoid symlink errors
45. #3309602: Error: Composer could not find the config file

As promised, the 1st page, good for the last 50 out of 180 issues!

Untagged: none.

Added:

1. #3321256: Fix race condition in \Drupal\automatic_updates_test_cron\Enabler
2. #3325869: Improve documentation for package_manager_bypass test module
3. #3248975: Use "stage directory" instead of "staging area" or "staging directory" everywhere
4. #3326934: Tests are failing for php 7.4
5. #3325716: Deprecate config factory in Stage
6. #3322913: Create an easy way for functional tests to simulate an update (and update kernel tests to use the same)
7. #3323565: Add 'declare(strict_types = 1)' in tests + recheck test modules + check *.module files.
8. #3321236: Add actual project folders in \Drupal\Tests\package_manager\Traits\FixtureUtilityTrait::addPackage
9. #3320836: Handle runtime error correctly in ComposerExecutableValidator
10. #3321206: The same xdebug warning shows on the status report 2 times
11. #3326334: Remove unneeded fixtures folders
12. #3321994: Fail early in all Automatic Updates tests if there's a failure marker
13. #3328516: Remove AutomaticUpdatesFunctionalTestBase::setCoreUpdate() in favor of StageFixtureManipulator
14. #3320755: Remove unnecessarily disabled automatic_updates.composer_executable_validator
15. #3328742: phpcs stopped working since the switch to testing on Drupal 10.0.x by default
16. #3321904: Remove automatic_updates_extensions/tests/fixtures/stage_composer in favor of using StageFixtureManipulator
17. #3321684: Most validators that subscribe to PreCreate to check readiness to update should also subscribe to PreApply
18. #3331471: Add documentation for SymlinkValidatorTest
19. #3318065: SymlinkValidatorTest doesn't test PreApplyEvent
20. #3331310: Exclude unknown paths in project base: only allow vendor + web root + whatever drupal/core-composer-scaffold allows
21. #3327391: Improve FixtureManipulator DX: validate package name + ensure StageFixtureManipulator is committed + ensure `package_manager_bypass_composer_stager` is not set to FALSE
22. #3325654: Improve the user experience of having your staged update deleted before it was applied
23. #3331168: Limit trusted Composer plugins to a known list, allow user to add more
24. #3330365: Rename AutomaticUpdatesFunctionalTestBase::setCoreVersion() as it is very confusing: does it apply to active or stage?
25. #3332256: package_manager_bypass should *minimally* bypass php-tuf/composer-stager, not maximally
26. #3321474: Adopt PHP 8.1-only capabilities such as constructor property promotion + drop BC layers
27. #3299094: Prevent staging areas that nested in the active Composer project directory
28. #3322546: Update initiative roadmap issue
29. #3330139: Harden PathExclusionsTrait::excludeInProjectRoot() for paths outside of project root.

Untagging this issue core-mvp though, because it's not an actionable issue with work associated with it 😅

At the bottom of #33 I wrote:

After that: splitting in done vs remaining and in automatic_updates vs package_manager.

The latter is nearly impossible because the Package Manager component is not being used consistently to signal that an issue affects the package_manager base module too, and not just the automatic_updates module. So … for now, not doing that, and here's hoping that the d.o PHP-TUF infrastructure will be ready in time for Automatic Updates to go into core before Project Browser. If that doesn't happen, we'll need to split this roadmap up, but that will require re-checking every single of these issues… 😳

Will now do the "done vs remaining"…

Done vs remaining split: done!

Spotted one mistake in #37.

Tweaked the anchors to avoid duplicates. e.g. #security and #completed-security.

#3327391: Improve FixtureManipulator DX: validate package name + ensure StageFixtureManipulator is committed + ensure `package_manager_bypass_composer_stager` is not set to FALSE landed.

#3320782: xdebug being enabled causes tests to fail without clear indication that it is the problem landed. 🥳

#3330140: Update StatusCheckTrait::runStatusCheck() to reorder/avoid dispatching CollectIgnoredPathsEvent earlier than StatusCheckEvent landed and unblocked #3315834: GitExcluder should not ignore .git directories that belong to packages installed by Composer.

#3299094: Prevent staging areas that nested in the active Composer project directory landed.

Added #3334906: Improve UX for trusting additional composer plugins to post-MVP.

Added #3334054: Add error handling for \Drupal\package_manager\Stage::getIgnoredPaths() to Reliability.

#3323706: Split up UpdaterFormTest to speed up test runs: from 13.5 to 10.5 minutes landed!

(And moved #319679: Taxonomy question + #3320792: Make build tests fail 1) more explicitly, 2) earlier when possible (failing StatusCheckEvent subscribers) + #317815: cannot change php or .htaccess file, cannot find code mentioned from the Testing bucket to the Maintainability bucket.)

#3331471: Add documentation for SymlinkValidatorTest landed (and moved it from the Documentation bucket to the Testing bucket).

Added #3335766: Harden LockFileValidator, add stopPropagation() at failures (discovered by @omkar.podey) to Reliability.

Added #3335802: Add addDotGitFolder functionality to \Drupal\fixture_manipulator\FixtureManipulator to Maintainability, which is the new blocker for #3315834: GitExcluder should not ignore .git directories that belong to packages installed by Composer that @tedbow identified.

#3325522: Automatic Updates & Package Manager should use DependencySerializationTrait when needed is now obsolete thanks to other issues having landed ⇒ #3320824: Fix PHP Warning: serialize() in tests on PHP 8 is unblocked.

Added #3335908: The 'fake_site' fixture cannot be using with `composer show` because the packages are not installed to Testing.

Landed:

• #3334054: Add error handling for \Drupal\package_manager\Stage::getIgnoredPaths()
• #3332256: package_manager_bypass should *minimally* bypass php-tuf/composer-stager, not maximally

Added #3336243: Update Package Manager event documentation in package_manager.api.php to Documentation.

#3315834: GitExcluder should not ignore .git directories that belong to packages installed by Composer landed 🥳

Added #3336867: Identify & vet commonly used composer plugins in the Drupal ecosystem to post-MVP.

#3328742: phpcs stopped working since the switch to testing on Drupal 10.0.x by default landed 🎆

Landed:

• #3320792: Make build tests fail 1) more explicitly, 2) earlier when possible (failing StatusCheckEvent subscribers)
• #3316668: ComposerSettingsValidator should run `composer config` to determine if HTTPS is enabled

Improving consistency of the buckets that are done 🥳

New MVP issue: #3338666: Add functional test that proves there is reasonable UX whenever a stage event subscriber has an exception.

New post-MVP issues:

• #3338651: Drupal core's coding standards forbid translated exceptions, but does that anyway — f.e. FileValidationException
• #3338667: Add build test to test cweagans/composer-patches end-to-end

#3317815: Remove all fixtures except for one: `fake_site` landed and triggered a new post-MVP issue: #3338392: Document that StageFixtureManipulator should not be used in build tests.

Added #3338346: Do not allow drupal/core-composer-scaffold to be used by packages other than core.

#3325654: Improve the user experience of having your staged update deleted before it was applied landed, which means we have no further known MVP UX improvement issues on the roadmap! 🎉

Added #3339016: "Composer not found" does provide link to help page.

#3331168: Limit trusted Composer plugins to a known list, allow user to add more and #3335908: The 'fake_site' fixture cannot be using with `composer show` because the packages are not installed landed!

Added many MVP issues created in the past 10 days that were still missing from the roadmap:

1. #3339714: Dynamically copy scaffold file mapping when create fake_site fixture
2. #3338789: Random failure: "PHP temp directory (/tmp) does not exist or is not writable to Composer."
3. #3339657: Always show summary of validation result if exists
4. #3337760: ComposerPatchesValidator should use ComposerInspector instead of ComposerUtility
5. #3337068: PackageManagerKernelTestBase::assertResults does not show actual results if none were expected. (already solved)
6. #3337062: PackageManagerKernelTestBase::assertResults does not give helpful error message if asserting results are empty fails (already solved)
7. #3337049: Assert no errors after creating the test project in ModuleUpdateTest

Added #3340022: Tighten ComposerPluginsValidator: support only specified version constraint.

#3311229: Validate compliance with composer minimum stability during PreRequireEvent landed.

#3339657: Always show summary of validation result if exists landed.

#3335766: Harden LockFileValidator, add stopPropagation() at failures was closed.

Landed:

1. #3336243: Update Package Manager event documentation in package_manager.api.php
2. #3325869: Improve documentation for package_manager_bypass test module
3. #3339016: "Composer not found" does provide link to help page

#3331310: Exclude unknown paths in project base: only allow vendor + web root + whatever drupal/core-composer-scaffold allows landed.

#3252299: Reliably support cweagans/composer-patches in Package Manager & Automatic Updates: validate stage shipped 🚢

New MVP issues created since #67 that I was not aware of previously:

1. #3340284: anonymous ProcessOutputCallbackInterface class ComposerInspector::getConfig() assumes __invoke does receive any data in error buffer
2. #3340892: Tag 8.x-2.7, and move development to new 3.x branch to focus on Core inclusion — already finished
3. #3341406: Document when PostApplyEvent should be used instead of hook_update_n() or post update
4. #3341708: Update requirements for 3.x

Added #3341224: Always catch \Throwable, not \Exception, and pass the old exception when re-throwing..

#3316368: Remove our runtime dependency on composer/composer: remove ComposerUtility was listed twice! 🙈

Adding #3341974: Finalize \Drupal\automatic_updates\Development\Converter script to update core MR.

Adding #3319030: Drupal Core Roadmap for Package Manager and Update Manager.

#3341708: Update requirements for 3.x and #3321474: Adopt PHP 8.1-only capabilities such as constructor property promotion + drop BC layers landed.

Adding

1. #3342120: Fix PHPStan violations (happened because PHPStan code checks stopped running…)
2. #3342137: Rename validator methods to `validate` unless there different methods for different events
3. #3342227: ScaffoldFilePermissionsValidator should use ComposerInspector instead of ComposerUtility

#3303167: Stage no longer needs the config factory happened in #3321474: Adopt PHP 8.1-only capabilities such as constructor property promotion + drop BC layers 👋

(Also: #80 was wrong 😅 — got it right and all up-to-date in #82, but still have to undo what I did in #80.)

29 remain (34 actually remain, but 5 of these are automatic_updates-specific and hence postponed because we want to land package_manager first)

#3320824: Fix PHP Warning: serialize() in tests on PHP 8 is finally fixed!

28 remain (33 actually remain, but 5 of these are automatic_updates-specific and hence postponed because we want to land package_manager first)

Adding

1. #3342430: Hard failure after module install if composer is not found
2. #3342364: Make StageEvent::$stage a public readonly property, and remove getStage()
3. #3342460: $tempStore property of stage was removed — already fixed

30 remain (35 actually remain, but 5 of these are automatic_updates-specific and hence postponed because we want to land package_manager first)

#3342364: Make StageEvent::$stage a public readonly property, and remove getStage() also landed.

Adding #3342726: UnknownPathExcluder doesn’t consider hidden files and #3342817: Decide which classes should be internal and/or final — delete ExcludedPathsTrait, make CollectPathsToExcludeEvent richer.

30 remain (35 actually remain, but 5 of these are automatic_updates-specific and hence postponed because we want to land package_manager first)

Moved one issue to the right place.

#3327229: Remove `@requires PHP >= 8.0` annotation from all tests and #3342726: UnknownPathExcluder doesn’t consider hidden files landed!

28 remain (33 actually remain, but 5 of these are automatic_updates-specific and hence postponed because we want to land package_manager first)

Landed:

1. #3342137: Rename validator methods to `validate` unless there different methods for different events
2. #3334994: Add new InstalledPackagesList which does not rely on Composer API to get package info — Which was somehow missing?!?! 😱 → unblocked several issues: #3340022: Tighten ComposerPluginsValidator: support only specified version constraint and #3342227: ScaffoldFilePermissionsValidator should use ComposerInspector instead of ComposerUtility (which in turn blocks another issue).

Closed: #3340284: anonymous ProcessOutputCallbackInterface class ComposerInspector::getConfig() assumes __invoke does receive any data in error buffer.

26 remain (31 actually remain, but 5 of these are automatic_updates-specific and hence postponed because we want to land package_manager first)

New: #3312960: Create an API for base requirement validators which run before other validators and stop event propagation if they fail, blocking #3342430: Hard failure after module install if composer is not found.

27 remain (32 actually remain, but 5 of these are automatic_updates-specific and hence postponed because we want to land package_manager first)

A lot has happened in the past 48 hours!

Landed:

1. #3339714: Dynamically copy scaffold file mapping when create fake_site fixture
2. #3341224: Always catch \Throwable, not \Exception, and pass the old exception when re-throwing.
3. #3267646: Refine multisite detection: many aliases for a single site is fine!
4. #3342120: Fix PHPStan violations (happened because PHPStan code checks stopped running…)

New:

1. #3343430: Stop reusing core's commit-code-check.sh in favor of 4 simple commands — already landed
2. #3344124: Validate fake_site with composer — already landed
3. #3343768: Benchmark how slow using composer require would be in kernel tests
4. #3343827: Update FixtureManipulator to work with InstalledPackagesList, real composer show command
5. #3343889: Drop support for end-of-life versions of Composer
6. #3344039: Add a validate() method to ComposerInspector to ensure that Composer is usable
7. #3344127: Run `composer validate` after FixtureManipulator commits its changes

28 remain (33 actually remain, but 5 of these are automatic_updates-specific and hence postponed because we want to land package_manager first)

I learned from @Wim Leers and @tedbow that some of the issues in the alpha blockers list are a consequence of trying to remove the dependency on composer/composer within tests. I just want to point out that the scope of #3316368: Remove our runtime dependency on composer/composer: remove ComposerUtility is to remove the runtime dependency on that. It's okay to leave composer/composer as a dev dependency, since it already is that for core anyway.

So if it helps to move some of the issues for refactoring tests to not use composer/composer to post-alpha, I'd be +1 to that, so long as we think that the tests are still sufficiently reliable without that refactoring.

Landed #3344039: Add a validate() method to ComposerInspector to ensure that Composer is usable.

Added #3344556: Make ComposerInspector::getVersion() private.

Landed: #3344556: Make ComposerInspector::getVersion() private

Added:

• #3344583: ComposerInspector::validate() should throw ComposerNotReadyException instead of \Exception
• #3344595: ComposerInspector::validate() should run `composer validate`
• #3344689: Do not disable package_manager.validator.composer in tests — already landed

28 remain (33 actually remain, but 5 of these are automatic_updates-specific and hence postponed because we want to land package_manager first)

#3344127: Run `composer validate` after FixtureManipulator commits its changes landed.

@tedbow, @Wim Leers, @lauriii, and I met and discussed the alpha vs beta classification of many of the issues.
The issues that were moved to a new Beta section are no less important to the codebase or the end user, but they could happen after an initial commit to a dev branch of core.

See https://www.drupal.org/about/core/policies/core-change-policies/experime... for more distinction on alpha vs beta.

Thanks — #96 + #98 look great!

And thanks for fixing what I got wrong in #97! 🙈😄

Credited @lauriii because @tim.plunkett did not have the necessary permissions.

Landed: #3344595: ComposerInspector::validate() should run `composer validate`

• package_manager alpha blockers: 12 remain
• package_manager + automatic_updates alpha + beta blockers: 27 remain (32 actually remain, but 5 of these are automatic_updates-specific and hence postponed because we want to land package_manager first)

#3321933: Remove dependency on symfony/finder was blocked for months. It's now unblocked. That's why it's now also clear that we won't need an issue to add symfony/finder as a new core dependency. But we will need that for symfony/config. I already reported that on Dec 9.

@longwave already responded favorably. I just posted a patch there, and am adding it to this issue.

I'm also making it more clear which issues are core issues already — of the 12 remaining (now 13), there are 3 core issues. Finally: this did not yet link to the issue where this module will be added to core: #3253158: Add Alpha level Experimental Update Manager module. Fixed that too.

• package_manager alpha blockers: 13 remain, 10 are under our control, 3 are issues to add core dependencies
• package_manager + automatic_updates alpha + beta blockers: 28 remain (33 actually remain, but 5 of these are automatic_updates-specific and hence postponed because we want to land package_manager first)

#3343889: Drop support for end-of-life versions of Composer landed while I was writing #102 🚀

• package_manager alpha blockers: 12 remain, 9 are under our control, 3 are issues to add core dependencies
• package_manager + automatic_updates alpha + beta blockers: 27 remain (32 actually remain, but 5 of these are automatic_updates-specific and hence postponed because we want to land package_manager first)

#3321933: Remove dependency on symfony/finder landed.

Removing #3326246: Add symfony/config to core's dependencies for package_manager in favor of #3345039: Remove dependency on symfony/config, which sidesteps the problem completely and doesn't require core (or Package Manager!) to add additional dependencies.

#3345039: Remove dependency on symfony/config landed!

Indeed! 😊

• package_manager alpha blockers: 10 remain, 8 are under our control, 2 are issues to add core dependencies
• package_manager + automatic_updates alpha + beta blockers: 25 remain (30 actually remain, but 5 of these are automatic_updates-specific and hence postponed because we want to land package_manager first)

#3331355: Refactor exception architecture is done!

Landed:

• #3343827: Update FixtureManipulator to work with InstalledPackagesList, real composer show command → unblocked #3342227: ScaffoldFilePermissionsValidator should use ComposerInspector instead of ComposerUtility + #3337760: ComposerPatchesValidator should use ComposerInspector instead of ComposerUtility.
• #3312960: Create an API for base requirement validators which run before other validators and stop event propagation if they fail

New beta blockers:

• #3345549: PackageManagerKernelTestBase::assertResults ignores event class — already landed
• #3345646: InstalledPackage::$path for metapackages should be NULL
• #3345649: Update every ComposerValidator-dependent validator to have explicit test coverage that that dependency works

New beta blockers which are blocking #3316368: Remove our runtime dependency on composer/composer: remove ComposerUtility (see #3316368-23: Remove our runtime dependency on composer/composer: remove ComposerUtility):

• #3345754: Updater should use ComposerInspector instead of ComposerUtility
• #3345755: ExtensionUpdater should use ComposerInspector instead of ComposerUtility
• #3345757: \Drupal\automatic_updates_extensions\Form\UpdateReady should use ComposerInspector instead of ComposerUtility
• #3345760: \Drupal\automatic_updates_extensions\Form\UpdaterForm should use ComposerInspector instead of ComposerUtility
• #3345761: UpdateReleaseValidator should use ComposerInspector instead of ComposerUtility
• #3345762: GitExcluder should use ComposerInspector instead of ComposerUtility
• #3345763: UnknownPathExcluder should use ComposerInspector instead of ComposerUtility
• #3345764: OverwriteExistingPackagesValidator should use ComposerInspector instead of ComposerUtility
• #3345765: SupportedReleaseValidator should use ComposerInspector instead of ComposerUtility
• #3345766: CollectIgnoredPathsFailValidator should use ComposerInspector instead of ComposerUtility
• #3345767: FakeSiteFixtureTest should use ComposerInspector instead of ComposerUtility
• #3345768: StatusCheckTraitTest should use ComposerInspector instead of ComposerUtility
• #3345769: RequestedUpdateValidator should use ComposerInspector instead of ComposerUtility
• #3345771: VersionPolicyValidator should use ComposerInspector instead of ComposerUtility
• #3345772: \Drupal\automatic_updates\Form\UpdateReady should use ComposerInspector instead of ComposerUtility

Forgot to move #3343827: Update FixtureManipulator to work with InstalledPackagesList, real composer show command!

• package_manager alpha blockers: 9 remain, 8 are under our control, 3 are issues to add core dependencies
• package_manager + automatic_updates alpha + beta blockers: 39 remain (44 actually remain, but 5 of these are automatic_updates-specific and hence postponed because we want to land package_manager first)

#3316368: Remove our runtime dependency on composer/composer: remove ComposerUtility is listed as an alpha blocker, which would make the 15 new beta blockers at the bottom of #109 actually alpha blockers. But @tim.plunkett, @lauriii, @tedbow and I kept #3316368 as an alpha blocker — I think because proving that it can in fact be removed is an alpha blocker, but actually finishing it is not? We need to get clarity on this.

From #110:

#3316368: Remove our runtime dependency on composer/composer: remove ComposerUtility is listed as an alpha blocker, which would make the 15 new beta blockers at the bottom of #109 actually alpha blockers. But @tim.plunkett, @lauriii, @tedbow and I kept #3316368 as an alpha blocker — I think because proving that it can in fact be removed is an alpha blocker, but actually finishing it is not? We need to get clarity on this.

→ @effulgentsia confirmed that they are alpha blockers. Moved them. Because they all block the Dependencies bucket primarily, moved them all in there. That means we now have zero things left under Reliability! 😊

Landed:

• #3345765: SupportedReleaseValidator should use ComposerInspector instead of ComposerUtility
• #3345764: OverwriteExistingPackagesValidator should use ComposerInspector instead of ComposerUtility
• #3345768: StatusCheckTraitTest should use ComposerInspector instead of ComposerUtility

New & already landed: #3345881: Remove fake-site fixture for automatic_updates_extensions.

• package_manager alpha blockers: 22 remain, 20 are under our control, 2 are issues to add core dependencies
• package_manager + automatic_updates alpha + beta blockers: 36 remain (41 actually remain, but 5 of these are automatic_updates-specific and hence postponed because we want to land package_manager first)

• Landed: #3345766: CollectIgnoredPathsFailValidator should use ComposerInspector instead of ComposerUtility.
• I see @tedbow tagged one more issue core-mvp: #3345633: Remove FixtureManipulator::modifyPackage() last usage.

Landed:

• #3345772: \Drupal\automatic_updates\Form\UpdateReady should use ComposerInspector instead of ComposerUtility
• #3345754: Updater should use ComposerInspector instead of ComposerUtility
• #3345767: FakeSiteFixtureTest should use ComposerInspector instead of ComposerUtility

⇒

• package_manager alpha blockers: 19 remain, 17 are under our control, 2 are issues to add core dependencies
• package_manager + automatic_updates alpha + beta blockers: 33 remain (38 actually remain, but 5 of these are automatic_updates-specific and hence postponed because we want to land package_manager first)

Landed:

• #3345771: VersionPolicyValidator should use ComposerInspector instead of ComposerUtility
• #3345757: \Drupal\automatic_updates_extensions\Form\UpdateReady should use ComposerInspector instead of ComposerUtility

⇒

• package_manager alpha blockers: 17 remain, 15 are under our control, 2 are issues to add core dependencies
• package_manager + automatic_updates alpha + beta blockers: 31 remain (36 actually remain, but 5 of these are automatic_updates-specific and hence postponed because we want to land package_manager first)

Landed:

• #3345769: RequestedUpdateValidator should use ComposerInspector instead of ComposerUtility
• #3345755: ExtensionUpdater should use ComposerInspector instead of ComposerUtility

⇒

• package_manager alpha blockers: 15 remain, 13 are under our control, 2 are issues to add core dependencies
• package_manager + automatic_updates alpha + beta blockers: 29 remain (34 actually remain, but 5 of these are automatic_updates-specific and hence postponed because we want to land package_manager first)

I see I failed to mention that my proposal to close #3342430 as outdated in #3342430-19: Hard failure after module install if composer is not found has been accepted at #3342430-27: Hard failure after module install if composer is not found — one less! 👍

⇒

• package_manager alpha blockers: 14 remain, 12 are under our control, 2 are issues to add core dependencies
• package_manager + automatic_updates alpha + beta blockers: 28 remain (33 actually remain, but 5 of these are automatic_updates-specific and hence postponed because we want to land package_manager first)

#3345761: UpdateReleaseValidator should use ComposerInspector instead of ComposerUtility landed.

⇒

• package_manager alpha blockers: 13 remain, 11 are under our control, 2 are issues to add core dependencies
• package_manager + automatic_updates alpha + beta blockers: 27 remain (32 actually remain, but 5 of these are automatic_updates-specific and hence postponed because we want to land package_manager first)

#3345760: \Drupal\automatic_updates_extensions\Form\UpdaterForm should use ComposerInspector instead of ComposerUtility landed.

⇒

• package_manager alpha blockers: 12 remain, 10 are under our control, 2 are issues to add core dependencies
• package_manager + automatic_updates alpha + beta blockers: 26 remain (31 actually remain, but 5 of these are automatic_updates-specific and hence postponed because we want to land package_manager first)

#3337760: ComposerPatchesValidator should use ComposerInspector instead of ComposerUtility landed.

⇒

• package_manager alpha blockers: 11 remain, 9 are under our control, 2 are issues to add core dependencies
• package_manager + automatic_updates alpha + beta blockers: 25 remain (30 actually remain, but 5 of these are automatic_updates-specific and hence postponed because we want to land package_manager first)

#3345762: GitExcluder should use ComposerInspector instead of ComposerUtility landed.

⇒

• package_manager alpha blockers: 10 remain, 8 are under our control, 2 are issues to add core dependencies
• package_manager + automatic_updates alpha + beta blockers: 24 remain (29 actually remain, but 5 of these are automatic_updates-specific and hence postponed because we want to land package_manager first)

@tedbow discovered a new alpha blocker: #3346628: FixtureManipulator should only use Composer commands, rather than manipulating JSON directly.

😬

#3346628: FixtureManipulator should only use Composer commands, rather than manipulating JSON directly is done!

And so is #3345763: UnknownPathExcluder should use ComposerInspector instead of ComposerUtility.

Also clarifying the the alpha roadmap is for #3346707: Add Alpha level Experimental Package Manager module since we first want to land Package Manager in 10.1 to enable Project Browser to also land in 10.1 because Automatic updates is blocked on d.o infrastructure and will definitely not make 10.1. #3253158: Add Alpha level Experimental Update Manager module is slated for 10.2 at this point.

⇒

• package_manager alpha blockers: 9 remain, 7 are under our control, 2 are issues to add core dependencies
• package_manager + automatic_updates alpha + beta blockers: 22 remain (27 actually remain, but 5 of these are automatic_updates-specific and hence postponed because we want to land package_manager first)

Landed:

1. #3345633: Remove FixtureManipulator::modifyPackage() last usage
2. #3342227: ScaffoldFilePermissionsValidator should use ComposerInspector instead of ComposerUtility

@phenaproxima opened a new blocker for #3316368: #3347031: Stage::validatePackageNames() should not use the Composer API.

⇒

• package_manager alpha blockers: 8 remain, 6 are under our control, 2 are issues to add core dependencies
• package_manager + automatic_updates alpha + beta blockers: 21 remain (26 actually remain, but 5 of these are automatic_updates-specific and hence postponed because we want to land package_manager first)

Landed:

• #3347031: Stage::validatePackageNames() should not use the Composer API
• #3316368: Remove our runtime dependency on composer/composer: remove ComposerUtility

⇒

• package_manager alpha blockers: 6 remain, 4 are under our control, 2 are issues to add core dependencies
• package_manager + automatic_updates alpha + beta blockers: 19 remain (24 actually remain, but 5 of these are automatic_updates-specific and hence postponed because we want to land package_manager first)

#3338789: Random failure: "PHP temp directory (/tmp) does not exist or is not writable to Composer." landed.

⇒

• package_manager alpha blockers: 5 remain, 3 are under our control, 2 are issues to add core dependencies
• package_manager + automatic_updates alpha + beta blockers: 19 remain (24 actually remain, but 5 of these are automatic_updates-specific and hence postponed because we want to land package_manager first)

#3318306: Define the Package Manager API (package_manager.api.php is outdated) is in; docs gate is cleared!

#3318306: Define the Package Manager API (package_manager.api.php is outdated) landed!

⇒

• package_manager alpha blockers: 4 remain, 1 is under our control, 1 is blocked on upstream (Composer Stager), 2 are issues to add core dependencies
• package_manager + automatic_updates alpha + beta blockers: 18 remain (23 actually remain, but 5 of these are automatic_updates-specific and hence postponed because we want to land package_manager first)

#3344583: ComposerInspector::validate() should throw ComposerNotReadyException instead of \Exception landed and was a beta blocker.

The issue summary had an Alpha blocker section for Package Manager, but was missing an Alpha blocker section for Automatic Updates. So I created that missing section and moved the two issues for Unattended Updates from the Beta blockers section to the Automatic Updates Alpha blockers section.

Adding #3347959: ComposerPluginsValidator uses Composer's internal Package class as an alpha blocker, which should have been fixed as part of removing our dependency on composer/composer, but slipped through the cracks.

I think you meant #3348276: In CoreUpdateTest::testUi, confirm that the UI says no update is available after updating successfully? 😁

#3341974: Finalize \Drupal\automatic_updates\Development\Converter script to update core MR landed.

⇒

• package_manager alpha blockers: 3 remain: 1 is blocked on upstream (Composer Stager), 2 are issues to add core dependencies
• package_manager + automatic_updates alpha + beta blockers: 15 remain (20 actually remain, but 5 of these are automatic_updates-specific and hence postponed because we want to land package_manager first)

#3321905: Add colinodell/psr-test-logger to core's dev dependencies is in.

That leaves two remaining alpha blockers, both of which are blocked on a new tag of Composer Stager. 😎

I just merged #3340022: Tighten ComposerPluginsValidator: support only specified version constraint. Nice to get that beta blocker in!

#3277034: Unhandled Composer Stager exceptions leave the update process in an indeterminate state and #3319507: Add symlink support to Composer Stager 2.0, require that version, and simplify UX & tests accordingly are in.

That means we are down to our last remaining alpha blocker...and it's something that requires review from core committers. 😎

#3345646: InstalledPackage::$path for metapackages should be NULL landed.

⇒

• package_manager alpha blockers: 1 remains: an issues to add a core dependency
• package_manager + automatic_updates alpha + beta blockers: 10 remain (15 actually remain, but 5 of these are automatic_updates-specific and hence postponed because we want to land package_manager first)

#3338346: Do not allow drupal/core-composer-scaffold to be used by packages other than core is in!

Removing a duplicate item from the roadmap.

Whoops, removed the wrong thing.

#3348129: Autowire everything everywhere all at once … in *Test.php files is done too.

#3348122: Autowire everything everywhere all at once … in *.services.yml files landed.

#3347267: ComposerMinimumStabilityValidator doesn't check dev packages is in.

#3352198: Rip out the update path is in.

#3337667: Allow JsonProcessOutputCallback and other Composer runner callbacks to gracefully handle deprecated command options is done! 🎉

#3348162: Ensure all remaining @todos have a link to an open issue is in too! Yeah!!!

The monumental #3326486: Rename Stage to StageBase to clarify its relationship to its subclasses, and add "Stage" suffix to the Updater classes landed!

#3348159: Fix remaining @todos in ComposerPluginsValidatorTest is done.

Removing #3338666: Add functional test that proves there is reasonable UX whenever a stage event subscriber has an exception and adding #3354099: Add functional test that proves there is reasonable UX whenever Composer Stager operations have a hard failure in its place.

Hey, look at that: alpha-blocking security-related #3351247: Harden our HTTPS requirement is in! 🎉

New alpha-blocking security-related issue: #3356804: Flag a warning during status check if the OpenSSL extension is not enabled — related to #3351247: Harden our HTTPS requirement that landed previously.

#3356804: Flag a warning during status check if the OpenSSL extension is not enabled is in.

#3357578: str_starts_with($path, '/') does not correctly detect absolute paths on Windows and #3342817: Decide which classes should be internal and/or final — delete ExcludedPathsTrait, make CollectPathsToExcludeEvent richer are in!

#3316617: Add a validator to check that PHP-TUF's Composer integration is present and configured correctly is done too.

This roadmap has not been updated in almost a month.

#3284443: Enable unattended updates landed, and so did #3354099: Add functional test that proves there is reasonable UX whenever Composer Stager operations have a hard failure and #3357657: ComposerValidator's hook_help() integration is imprecise and incomplete. #3345222: Optimize Composer calls in FixtureManipulator was closed as was #3338667: Add build test to test cweagans/composer-patches end-to-end.

AFAICT missing:

1. #3362143: Use the rsync file syncer by default
2. #3360485: Add Symfony Console command to allow running cron updates via console and by a separate user, for defense-in-depth
3. #3357969: For web server dependent unattended updates run the entire life cycle in a separate process that will not be affected by hosting time limits

AFAICT obsolete:

• #3318587: Research alternatives to curl request on post apply event during cron updates
• #3318964: automated_cron should not run cron when visiting update.php

Moved #3284443: Enable unattended updates to completed.

Added a nice green checkmark to the Core Alpha Experimental blockers for Automatic Updates section :)

Removed the green checkmark from the Security section of package manager alpha blockers, since there's an open issue in that section.

Added some more core and infrastructure issues to the package manager alpha blockers list.

Moved the following to Completed:

• #3354099: Add functional test that proves there is reasonable UX whenever Composer Stager operations have a hard failure
• #3355628: Package Manager should keep an audit log of changes it applied to the active codebase
• #3357657: ComposerValidator's hook_help() integration is imprecise and incomplete

Removed the following, but did not move to Completed, as they were closed without fixing.

• #3345222: Optimize Composer calls in FixtureManipulator
• #3338667: Add build test to test cweagans/composer-patches end-to-end

Added:

• #3362143: Use the rsync file syncer by default
• #3360485: Add Symfony Console command to allow running cron updates via console and by a separate user, for defense-in-depth
• #3357969: For web server dependent unattended updates run the entire life cycle in a separate process that will not be affected by hosting time limits

For the two possibly obsolete issues mentioned in #171, I added notes by them that they might be obsoleted by #3357969: For web server dependent unattended updates run the entire life cycle in a separate process that will not be affected by hosting time limits, but I didn't remove them from the list in case #3357969: For web server dependent unattended updates run the entire life cycle in a separate process that will not be affected by hosting time limits doesn't work out for whatever reason (though I think it will work out).

I think all points from #171 were resolved, so removing the assignment to @tedbow that that comment added. @tedbow: please feel free to reassign to yourself if there's something else you want to update here.

Added:

• #3368808: Override Composer Stager's TranslatableFactory to return Drupal's TranslatableMarkup
• #3358504: Require PHP-TUF's Composer integration plugin

Issues tagged core-mvp that aren't in the issue summary. Either they should be added to the issue summary or their core-mvp tag should be removed:

• #3366271: In Core build tests have core module change a service class constructor
• #3366217: Limit non-stable releases to RCs on updater form
• #3349004: Update ProcessOutputCallback to fail on Composer deprecations, to get early notifications of Composer changes
• #3364135: Determine if the PHP syncer can and should be the default file copier
• #3361244: Document PACKAGE_MANAGER_FAILURE.yml in https://www.drupal.org/docs/8/update/automatic-updates
• #3362110: Since #3361983 was committed to Drupal core, psr/http-message needed to be explicitly required for build tests
• #3349351: Is it an error if drupal/core has no scaffold file mappings?
• #3351594: As of Composer 2.5.5, `composer config` JSON-encodes boolean values
• #3352654: PHPStan error in core merge request conversion

Added #3364565: [policy, no patch] Make PHP's OpenSSL extension a requirement for installing and using Package Manager (and therefore, Automatic Updates and Project Browser) and #3351190: [policy, no patch] Should Package Manager require Composer HTTPS? as alpha-blocking policy questions in the core queue.