This issue is part of the composer initiative: #2958021: Proposal: Composer Support in Core initiative

Problem/Motivation

The vendor cleanup scripts were moved into a Composer plugin in #3057094: Add Composer vendor/ hardening plugin to core. This is a follow-on issue to that task.

The original sanitization scripts still exist in the repository at Drupal\Core\Composer\Composer::vendorTestCodeCleanup(), and are called as an event script in drupal/drupal's composer.json like this:

    "scripts": {
        "post-package-install": "Drupal\\Core\\Composer\\Composer::vendorTestCodeCleanup",
        "post-package-update": "Drupal\\Core\\Composer\\Composer::vendorTestCodeCleanup",

These scripts are now deprecated, and should be removed.

Proposed resolution

Once the deprecated scripts are no longer being called, they should be removed. There are three potential places where these scripts may be called:

  1. From sites installed from a tarball download of Drupal
  2. From git clones of the Drupal repository (core development)
  3. External sources

The first set of calls will be removed by #2982680: Add composer-ready project templates to Drupal core. When the Drupal template projects are created, they will use the new Vendor Hardening plugin rather than the cleanup scripts. Once tarballs are generated from the template projects, the scripts will be unused in tarball-based sites (managed via `drush pm-update` etc.).

The second use-case is Drupal core development. To stop using the scripts during core development, the Vendor Hardening plugin will need to be added as a "path" repository of the root-level composer.json file of drupal/drupal, and the references to the scripts in the "scripts" section will need to be removed. This work can be done as part of this issue.

The final place where these scripts may be called is from external sources, e.g. community-managed Drupal template projects. These cleanup scripts are not needed for Drupal sites that utilize a relocated document root, so drupal-composer/drupal-project, for example, does not call these scripts. At the moment, there are no known examples of external projects that use these scripts. If there are any that exist, then this work cannot be done until Drupal 9.

Remaining tasks

Decide on the timing of when to do this work.

User interface changes

None.

API changes

None.

Data model changes

None.

Release notes snippet

tbd

Issue fork drupal-3076684

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

greg.1.anderson created an issue. See original summary.

mile23’s picture

Mixologic’s picture

Do these count as an api and thus need to wait for 9.x to be removed? asking for a friend: #3082866: composer complains "Skipped installation of bin bin/composer for package composer/composer: file not found in package"

greg.1.anderson’s picture

So, we originally left these scripts in place exactly for this reason: to remove them in Drupal 9, so that they would continue to exist to preserve backwards compatibility through the end of the 8 y.8.x series.

Backwards compatibility is needed if there is any remaining use-case where the drupal/drupal composer.json is used in production (or perhaps I should say "live", to cover test / staging environments that are exposed on the internet without necessarily being production -- although in theory, these should mirror production layout anyway). To answer that question (can we remove the scripts), we need to enumerate all of the use cases.

  1. git clone + composer install: We previously discussed that this workflow is for the most part used for core development, not for deployment. We also discussed that for the most part, it should be rare for anyone to install this way in production. Simplytest.me might install this way, although if it does, perhaps it is not calling `composer install` anyway, in which case it would not be using these scripts.
  2. New Composer-managed sites will install with the new Composer template projects, which do not contain drupal/drupal's composer.json.
  3. Most existing Composer-managed sites install with drupal-composer/drupal-project template projects, which do not contain drupal/drupal's composer.json.
  4. There are a couple other templates like webflo/kickstart; I think these pretty much all have to use drupal/core, just like drupal-composer/drupal-project, so perhaps we can say that none of these contain drupal/drupal's composer.json without enumerating them
  5. Existing sites that install with a tarball download will eventually be upgraded (e.g. via `drush pm-update) to use a newer archive produced from drupal/legacy-project. If anyone runs `composer install` on one of these sites, it will use the Vendor Hardening plugin, and will not need these scripts.

So, in short, I think that only the first of these is a concern. We should find out about simplytest.me; if it is not an issue, then perhaps we could just make a patch here and get a core committer to decide whether it's okay to commit it.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

mile23’s picture

Status: Active » Needs review
StatusFileSize
new3.13 KB

So I think we're a little behind the times here. If we had caught this a bit earlier we could have formally deprecated Drupal\Core\Composer\Composer::vendorTestCodeCleanup() and other methods that aren't needed before Drupal 9.

But it might be that we can't deprecate for 9, and would have to go for 10.

Nevertheless, here's a patch which deprecates for removal before 9.0.x, in hopes that maybe core maintainers don't want to maintain this code any longer than they have to.

Also explicitly marking protected methods as @internal.

There are two methods worth singling out:

removeTimeout() was added in #2911319: Provide a single command to install & run Drupal, I think by mistake. It seems like the idea was to allow you to type composer install-a-quick-start, but then that idea was dropped in favor of a CLI implementation. Marking as deprecated.

upgradePHPUnit() is left over from the good old days of PHP 5 compatibility. It might be in use now as a convenient way to upgrade to a version of PHPUnit ahead of current Drupal. Also has a friend: upgradePHPUnitCheck(). Not marking as deprecated, but as @internal.

Please push back on any of these decisions as needed. :-)

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

andypost’s picture

StatusFileSize
new3.13 KB

re-roll for 9.1.x deprecation, guess it needs deprecation test and CR

longwave’s picture

Should we add @trigger_error deprecations inside each of the deprecated methods, or output a warning if the relevant Composer commands are used?

I agree upgradePHPUnit() should stay and just be marked internal, it is about to be revived in #3127141: Use PHPUnit 9 for PHP 7.4+, while keeping support for PHPUnit 8.4 in PHP 7.3

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

longwave’s picture

Status: Needs review » Needs work

Let's add some kind of messaging to the user. #3162228: Composer 2 Fatal error Call to undefined method Composer\DependencyResolver\Operation\UpdateOperation::getJobType() in /home/mysite/public_html/core/lib/Drupal/Core/Composer/Composer.php:170 shows that these scripts are still in use in the wild and so we need to tell people to stop using them and switch to alternatives before we actually can remove them.

kapilv’s picture

Status: Needs work » Needs review
StatusFileSize
new3.13 KB

Here a Reroll patch.

vikashsoni’s picture

StatusFileSize
new32.69 KB
new95.6 KB

Patch #13 is applied successfully and patch #6 giving me error

bnjmnm’s picture

Patch #13 is applied successfully and patch #6 giving me error

It's expected that patch #6 would not apply to Drupal 9.3. created for Drupal 8.9 and was rerolled in #13 so it would apply to 9.3.

In other words, the patch in #13 only exists because we're all already aware that patch #6 does not apply. There's no need to provide further evidence of that.

longwave’s picture

Status: Needs review » Needs work
Issue tags: +Needs change record

#12 still applies. We need some kind of messaging to existing users that these will be removed, and a change record that we can point users to for documentation of the removal and what they should do instead.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

andypost’s picture

StatusFileSize
new2.99 KB

reroll for 9.4

andypost’s picture

Status: Needs work » Needs review
Issue tags: -Needs change record
StatusFileSize
new3.17 KB

Added CR, looks phpunit upgrade also could be deprecated, but not sure about removal from "scripts" section, probably it needs follow-up

andypost’s picture

StatusFileSize
new2.12 KB
new3.2 KB

Fix CS

longwave’s picture

@andypost any thoughts about #12/#16 - marking these as deprecated is a start but I think we should show a message somewhere to anyone who is still using these?

andypost’s picture

Status: Needs review » Needs work

++ to #21

longwave’s picture

Status: Needs work » Needs review
StatusFileSize
new8.49 KB

This patch adds unsilenced deprecation notices to the three script methods and corresponding tests.

It also includes the fix from #3162228: Composer 2 Fatal error Call to undefined method Composer\DependencyResolver\Operation\UpdateOperation::getJobType() in /home/mysite/public_html/core/lib/Drupal/Core/Composer/Composer.php:170 as ::vendorTestCodeCleanup doesn't work with Composer 2 and it's not possible to execute it from a test case without the fix. As that's the case I do wonder how anyone is actually using these any more.

longwave’s picture

The root composer.json also includes

        "pre-autoload-dump": "Drupal\\Core\\Composer\\Composer::preAutoloadDump",

which we are deprecating here - I guess this needs removing?!

paulocs’s picture

StatusFileSize
new1.51 KB
new8.45 KB

Fixing custom commands failure.

longwave’s picture

StatusFileSize
new8.47 KB
new427 bytes

Adding @group.

longwave’s picture

I'm still not sure preAutoloadDump should be deprecated or removed here. It is used by both the root composer.json and core/composer.json to ensure that bootstrap classes are added to the autoloader classmap, even for users that don't dump an optimized autoloader.

andypost’s picture

I find it weird that some plugin takes over composer functionality.

User's build system should be configured to use -o or whatever required, or at least there should be ability to disable(skip) this plugin.

Other option is to populate autoload - classmap at d-org packaging system (at release build for example).

Anyway it requires change record to be filed

andypost’s picture

At this point I think we should split deprecation(9.4.0) and removal(10.0.x) into separate issues

longwave’s picture

Status: Needs review » Needs work

@andypost this script adds ~15 classes to the classmap that are useful for bootstrap, that is minor compared to the 4800+ that are added if I dump an optimized autoloader.

Having said that we do need an issue to update the vendor classmap as symfony/http-kernel is duplicated and laminas/laminas-zendframework-bridge is no longer a dependency - we should check which classes are needed for bootstrap in Symfony 6 and only list those.

mile23’s picture

Version: 9.4.x-dev » 9.5.x-dev

I'm not sure what the deprecation rules are here, but Drupal\Core\Composer\Composer is not marked @internal, and vendorTestCodeCleanup() is not marked @internal or @deprecated.

I'm visiting from #3272110: Drupal 9 and 10's Drupal\Component composer.json files are totally out of date where I noticed that vendorTestCodeCleanup() is still present despite the existence of the vendor hardening plugin.

I guess the next step is to find out if we can deprecate stuff in 9.5.x for removal in 10.x.

andypost’s picture

Issue tags: +Deprecation Removal
mile23’s picture

Version: 9.5.x-dev » 10.0.x-dev
Status: Needs work » Postponed

Added #3280589: Deprecate Composer Vendor Cleanup Scripts

Postponing this issue on that one, per #30.

mile23’s picture

catch’s picture

Title: Remove deprecated vendor cleanup scripts » [PP-1] Remove deprecated vendor cleanup scripts
catch’s picture

Title: [PP-1] Remove deprecated vendor cleanup scripts » Remove deprecated vendor cleanup scripts
andypost’s picture

Status: Postponed » Needs work
spokje’s picture

Assigned: Unassigned » spokje
spokje’s picture

Hiding outdated patches.

spokje’s picture

Assigned: spokje » Unassigned
Status: Needs work » Needs review
longwave’s picture

Status: Needs review » Needs work

Nice work! One more simplification we can do then I think this is RTBC.

spokje’s picture

Status: Needs work » Needs review
longwave’s picture

Status: Needs review » Reviewed & tested by the community

Thank you @Spokje.

  • catch committed c042f34 on 10.1.x
    Issue #3076684 by Spokje, andypost, longwave, paulocs, Mile23, KapilV,...
catch’s picture

Status: Reviewed & tested by the community » Fixed

Committed/pushed to 10.1.x and cherry-picked to 10.0.x, thanks!

  • catch committed 3527513 on 10.0.x
    Issue #3076684 by Spokje, andypost, longwave, paulocs, Mile23, KapilV,...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.