Problem/Motivation

Drupal\Core\Composer\Composer::vendorTestCodeCleanup() needs to be able to run on the whole install and not just one package at a time. Refactoring that to allow for post-create-project hooks would be useful.

This is to support #2990257: Determine how to handle composer scripts in drupal/drupal and be able to perform the vendor test cleanup once the library which supports it is available, which in turn supports creating the drupal/drupal-project-legacy project template in #2982680: Add composer-ready project templates to Drupal core

Regardless of how #2990257: Determine how to handle composer scripts in drupal/drupal is implemented, we will likely need to be able to clean up the vendor directory in one step at the end of the installation process, since the library will have become available and autoloadable by that point.

Proposed resolution

Remaining tasks

User interface changes

API changes

Data model changes

Comments

Mixologic created an issue. See original summary.

mile23’s picture

The reason that vendorTestCodeCleanup() runs per-package is because that's a Composer limitation.

Composer event scripts receive different event classes based on what event is being handled. Different event classes allow for discovery of different information. A post-create-project event does not include easy access to a list of packages that were installed; it has to be deduced from the lock file.

I managed to implement this as part of drupal-project-legacy, right about here: https://github.com/paul-m/drupal-project-legacy/blob/f3f806eb8de548d8513...

That's the do-it-once-at-the-end solution, but it's fragile because it uses the lock file to determine all the packages to clean up.

A better solution is to do it the way we do it in drupal/drupal, per package during install, which is what that branch of drupal-project-legacy ended up looking like: https://github.com/paul-m/drupal-project-legacy/blob/passing/composer.js...

As it turns out this also meets our needs better in terms of generating .htaccess files and so forth for drupal-project-legacy, because it's located in the same Composer class.

mile23’s picture

Updating IS to reflect conversations w/ @mixologic.

mile23’s picture

Issue summary: View changes
mile23’s picture

Status: Active » Needs review
StatusFileSize
new5.03 KB

This patch turns vendorTestCodeCleanup() into a method that can run during post-install-cmd, post-update-cmd, and post-create-package-cmd.

You can use it locally to see it work with composer install -vv.

You can see that it has the same equivalent functionality in https://travis-ci.org/paul-m/drupal-project-legacy which is a test of https://github.com/paul-m/drupal-project-legacy/tree/passing

That test is related to #2982680: Add composer-ready project templates to Drupal core, trying to make a Composer-based template that will install core to be exactly the same as the current tarball.

The patch might also break a few things related to case-sensitive file systems. Being on a non-sensitive file system it's difficult to know.

mile23’s picture

StatusFileSize
new6.65 KB
new5.37 KB

Chatted with @mixologic about this in slack and we realized that we need BC for anyone who's already installed using drupal/drupal.

Mixologic’s picture

+++ b/composer.json
@@ -50,8 +50,8 @@
-        "post-package-install": "Drupal\\Core\\Composer\\Composer::vendorTestCodeCleanup",
-        "post-package-update": "Drupal\\Core\\Composer\\Composer::vendorTestCodeCleanup",
+        "post-install-cmd": "Drupal\\Core\\Composer\\Composer::vendorTestCodeCleanup",
+        "post-update-cmd": "Drupal\\Core\\Composer\\Composer::vendorTestCodeCleanup",

I think we should leave these as is, and not switch these to commands in drupal/drupal.

+++ b/core/lib/Drupal/Core/Composer/Composer.php
@@ -187,57 +188,84 @@ public static function upgradePHPUnitCheck($phpunit_version) {
+      return static::doTestCodeCleanup(
+        $event->getComposer()->getConfig()->get('vendor-dir'),
+        $package_key,
+        static::$packageToCleanup[$package_key],
+        $event->getIO()
+      );

We can't return the result of doTestCodeCleanup, because it doesnt return anything.
I dont think we need to add a return value as the original function doesnt have a return value.

+++ b/core/lib/Drupal/Core/Composer/Composer.php
@@ -187,57 +188,84 @@ public static function upgradePHPUnitCheck($phpunit_version) {
-    $package_key = static::findPackageKey($package->getName());

+    if ($package_key = static::findPackageKey($package->getName())) {
...
-    if ($package_key) {

Do we need to collapse variables like this? I find code that has been 'made more efficient' like this is harder to parse, harder to debug (with xdebug). Also, we should name this var $package_path to be consistent with the function call and usage in the other method.

+++ b/core/lib/Drupal/Core/Composer/Composer.php
@@ -187,57 +188,84 @@ public static function upgradePHPUnitCheck($phpunit_version) {
-    $vendor_dir = $event->getComposer()->getConfig()->get('vendor-dir');
-    $io = $event->getIO();
...
+      return static::doTestCodeCleanup(
+        $event->getComposer()->getConfig()->get('vendor-dir'),
+        $package_key,
+        static::$packageToCleanup[$package_key],
+        $event->getIO()
+      );

This call is hard to understand without looking at what we're passing to doTestCodeCleanup. Can we leave in the shorthand variables ($vendor_dir/$io) and add one more for static::$packageToCleanup[$package_key], (like $cleanup_paths).

+++ b/core/lib/Drupal/Core/Composer/Composer.php
@@ -187,57 +188,84 @@ public static function upgradePHPUnitCheck($phpunit_version) {
-      if ($io->isVeryVerbose()) {
-        // Add a new line to separate this output from the next package.
-        $io->write("");

Seems like we should preserve this behavior ?.

Otherwise I think this should work.

mile23’s picture

StatusFileSize
new126.97 KB
new3.3 KB

Addressing #7.

Mixologic’s picture

I think that might be the wrong patch, but everything else looks great, except theres still some returning the return of a void function

+++ b/core/lib/Drupal/Core/Composer/Composer.php
@@ -187,57 +188,89 @@ public static function upgradePHPUnitCheck($phpunit_version) {
+      return static::doTestCodeCleanup($vendor_dir, $package_name, $cleanup_paths, $io);
mile23’s picture

StatusFileSize
new5.79 KB

Woops. Forgot to rebase.

This patch is what should be in #8.

phenaproxima’s picture

I think this patch makes sense. Can't find much to complain about.

  1. +++ b/core/lib/Drupal/Core/Composer/Composer.php
    @@ -187,57 +188,89 @@ public static function upgradePHPUnitCheck($phpunit_version) {
    +   * @param \Composer\Script\Event $event
    +   *   A Composer Event object to get the configured composer vendor directories
    

    The type hint in the method did not change to match this.

  2. +++ b/core/lib/Drupal/Core/Composer/Composer.php
    @@ -187,57 +188,89 @@ public static function upgradePHPUnitCheck($phpunit_version) {
    +      $cleanup_paths = static::$packageToCleanup[$package_name];
    +      return static::doTestCodeCleanup($vendor_dir, $package_name, $cleanup_paths, $io);
    

    What if $package_name is not set in $packageToCleanup?

  3. +++ b/core/lib/Drupal/Core/Composer/Composer.php
    @@ -187,57 +188,89 @@ public static function upgradePHPUnitCheck($phpunit_version) {
    +  protected static function doTestCodeCleanup($vendor_dir, $package_path, $cleanup_paths, IOInterface $io) {
    

    Can $cleanup_paths be type hinted as an array?

Mixologic’s picture

StatusFileSize
new5.73 KB
new1.76 KB

Here's a patch addressing #9, #11-1, and #11-3.

#11-2 is already handled by the check surrounding the call to $packageToCleanup:

    $package_name = static::findPackageKey($package->getName());
    if ($package_name) {
      $cleanup_paths = static::$packageToCleanup[$package_name];
      static::doTestCodeCleanup($vendor_dir, $package_name, $cleanup_paths, $io);
    }

I have this patch applied to a 'fork' of drupal/core over at https://github.com/ryanaslett/core/blob/8.6.x/lib/Drupal/Core/Composer/C..., and I am using it in the build process for a travis build that utilizes it.

It works exactly how we'd like it to, at least for

https://travis-ci.org/drupal/drupal-project-legacy/builds/443325811#L558...

So. one problem I see is that if we have the post-create-project-cmd as a script, then this runs fine the first time we do a composer create-project, but we also want to ensure that those test dirs are stripped out whenever we update to newer versions of those vendor libs.

So, if we add the post-package-install and post-package-update scripts to our template's composer.json, then we end up with a bunch of

Package operations: 47 installs, 0 updates, 0 removals
  - Installing composer/installers (v1.5.0): Downloading (100%)
Class Drupal\Core\Composer\Composer is not autoloadable, can not call post-package-install script
  - Installing drupal/drupal-scaffold (8.6.x-dev 7bd18ee): Downloading (100%)
Class Drupal\Core\Composer\Composer is not autoloadable, can not call post-package-install script
  - Installing wikimedia/composer-merge-plugin (v1.4.1): Downloading (100%)
Class Drupal\Core\Composer\Composer is not autoloadable, can not call post-package-install script
  - Installing doctrine/cache (v1.6.2): Downloading (100%)
Class Drupal\Core\Composer\Composer is not autoloadable, can not call post-package-install script
  - Installing doctrine/collections (v1.3.0): Downloading (100%)
Class Drupal\Core\Composer\Composer is not autoloadable, can not call post-package-install script

type of error messages on our create project, because that script does not exist until drupal/core gets installed.

Which is making me start to think that this shouldnt be in core's composer.php class at all, and instead should be part of the drupal-scaffold plugin, and be part of the composer plugin framework, instead of being executed as a script. Nonwithstanding the fact that drupal-scaffold's responsibility is to 'add files to the filesystem to setup the product' - i.e. the corrollary of "remove files from the filesystem that we dont want" fits right into its role.

I think I might see about migrating this cleanup code to the drupal scaffold plugin to see if that works better.

mile23’s picture

+1 on #12. I was thinking of making it another library altogether, but it could live in drupal-scaffold.

Mixologic’s picture

As an experiment I moved vendorTestCodeCleanup and vendorTestCodeCleanupCommand to drupal/drupal-scaffold, and it works great. We no longer get the post-package-install script errors.

In fact, the post-create-project-cmd ends up doing essentially nothing now that the post-package-install hook works again. So, Im wondering if maybe we don't need to introduce the "make it work on both command and package events" thing.

I also got rid of the script execution in composer.json and moved the functionality into the plugin.

Im working on incorporating all the feedback from https://www.drupal.org/project/drupal/issues/2982684#comment-12769425 and https://www.drupal.org/project/drupal/issues/2982684#comment-12769839, and I think I'll post on that thread.

andypost’s picture

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

mile23’s picture

Status: Needs review » Closed (duplicate)

Tentatively closing this as duplicate because we're implementing vendorTestCodeCleanup() as part of the drupal-scaffold plugin here: #2982684: Add a composer scaffolding plugin to core

Please re-open if we decide on another strategy there.