Problem/Motivation

When attempting to update a project from core 9.4.7 to core 9.5.0 (along with the drupal/core-composer-scaffold) plugin, I have issues where all of the Drupal folders remain in vendor/drupal. The files do properly get placed in my webroot (for me, that's docroot) but they do not get properly removed from vendor/drupal when install concludes. This is a super wonky problem, but a few things I've observed:

  1. Pinning to drupal/core-composer-scaffold 9.4.7 or removing it entirely resolves this problem. So, something in the (VERY MINOR) change that occurred https://github.com/drupal/core-composer-scaffold/compare/9.4.7...9.5.0 is likely to blame? Somehow?
  2. I didn't run into this on one project, but did on a second. At further examination, the project that "worked" has multiple custom repositories defined and if I add additional repositories to the broken project, it starts working too.
  3. There are no clear errors / problems, the only reason I discovered this was my includes for DRUPAL_ROOT started throwing PHP Fatals after the update

Steps to reproduce

1. Clone https://github.com/Drupal4Gov/drupal4gov.us
2. Run a composer install
3. check vendor/drupal (should be ok)
4. run composer update "drupal/core*" -W (should update to 9.5.x)
5. check vendor/drupal (will now contain core and all the contrib modules)

after the above, if you remove the core/core-composer-scaffold, nuke the vendor folder, and re-run the composer update it should be fine. if you re-add core-composer-scaffold but pin to 9.4.7, nuke the vendor directory, and re-run composer update it should be fine. there's something about the 9.5.0 version of this that is problematic.

Proposed resolution

Either roll back the code change in https://github.com/drupal/core-composer-scaffold/compare/9.4.7...9.5.0 or iterate on the proposed approach to address this edge case.

Remaining tasks

User interface changes

API changes

Data model changes

Release notes snippet

CommentFileSizeAuthor
#17 3328893-16.diff1.1 KBclaudiu.cristea

Issue fork drupal-3328893

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

About issue forks

Comments

mike.madison@acquia.com created an issue. See original summary.

mikemadison’s picture

mikemadison commented
Issue summary: View changes
mikemadison’s picture

mikemadison commented
Issue summary: View changes
effulgentsia’s picture

effulgentsia commented
Related issues: +#3283498: Scaffold ReplaceOp::copyScaffold() throws an error for empty files

So, something in the (VERY MINOR) change that occurred https://github.com/drupal/core-composer-scaffold/compare/9.4.7...9.5.0 is likely to blame?

If it's correct that that diff is the culprit, then #3283498: Scaffold ReplaceOp::copyScaffold() throws an error for empty files is the issue that introduced that, so linking it as the related issue.

I guess a way to debug this would be to change the === FALSE back to a looser falsy (e.g., !()) check, and identify what is the case where a file is being written with 0 bytes, and why is treating that as an error condition (as 9.4.7 did) producing the desired result whereas treating it as an okay condition (as 9.5.0 does) produces this bug.

mglaman’s picture

mglaman
Primary language English
Location WI, USA
commented

Chiming in: I was able to reproduce using the provided commands

mile23’s picture

mile23
Primary language English
Location Seattle, WA
commented

I wasn't able to repro. I had to set up under ddev, using PHP 8.0 because that's what the locked dependencies require.

% ddev composer update "drupal/core*" -W

[ stuff happens ]

% ls -al vendor/drupal
total 0
drwxr-xr-x   6 paul  staff   192 Dec 23 02:26 .
drwxr-xr-x  63 paul  staff  2016 Dec 23 02:26 ..
drwxr-xr-x  14 paul  staff   448 Dec 23 02:24 coder
drwxr-xr-x  21 paul  staff   672 Jun 19  2022 core-composer-scaffold
drwxr-xr-x  14 paul  staff   448 May  7  2021 drupal-driver
drwxr-xr-x  20 paul  staff   640 Jun  1  2020 drupal-extension
mikemadison’s picture

mikemadison commented

FWIW I'm running composer 2.5.1 and running PHP 8.1 on my system.

mile23’s picture

mile23
Primary language English
Location Seattle, WA
commented

The code that would move the modules around would be composer/installers or oomphinc/composer-installers-extender plugins.

Those plugins aren't explicitly included in https://github.com/Drupal4Gov/drupal4gov.us/blob/main/composer.json so maybe a newer version of some dependency doesn't include them either...? Some version that only loads when you're using PHP 8.1 but not 8.0?

reg’s picture

reg commented

Having had this same issue I went looking at the scaffolding documentation and this part got my attention:

The purpose of scaffolding files is to allow Drupal sites to be fully managed by Composer, and still allow individual asset files to be placed in arbitrary locations. The goal of doing this is to enable a properly configured composer template to produce a file layout that exactly matches the file layout of a Drupal 8.7.x and earlier tarball distribution.

Since it's simple to change a web servers root and I can easily change my web servers root to /web so is there any reason for scaffolding if I simply make my web root /web? That way if it's not working one can just disable it and I have no reason to be concerned about whether the structure matches 8.7 and earlier. As long as it installs and works the same I am fine with the structure and it certainly installs if I just change the web server's root to /web. However, I haven't extensively tested it working after installation as of yet.

Also, is there some reason why it wouldn't be fully managed by composer if left as is, i.e., scaffolding disabled?

mile23’s picture

mile23
Primary language English
Location Seattle, WA
commented
Status: Active » Postponed (maintainer needs more info)

Here's what the scaffolding plugin does in a fresh install of core 10.0.0. This is extracted from a log of running composer create-project drupal/recommended-project:

Scaffolding files for drupal/core:
  - Copy [project-root]/.editorconfig from assets/scaffold/files/editorconfig
  - Copy [project-root]/.gitattributes from assets/scaffold/files/gitattributes
  - Copy [web-root]/.csslintrc from assets/scaffold/files/csslintrc
  - Copy [web-root]/.eslintignore from assets/scaffold/files/eslintignore
  - Copy [web-root]/.eslintrc.json from assets/scaffold/files/eslintrc.json
  - Copy [web-root]/.ht.router.php from assets/scaffold/files/ht.router.php
  - Copy [web-root]/.htaccess from assets/scaffold/files/htaccess
  - Copy [web-root]/example.gitignore from assets/scaffold/files/example.gitignore
  - Copy [web-root]/index.php from assets/scaffold/files/index.php
  - Copy [web-root]/INSTALL.txt from assets/scaffold/files/drupal.INSTALL.txt
  - Copy [web-root]/README.md from assets/scaffold/files/drupal.README.md
  - Copy [web-root]/robots.txt from assets/scaffold/files/robots.txt
  - Copy [web-root]/update.php from assets/scaffold/files/update.php
  - Copy [web-root]/web.config from assets/scaffold/files/web.config
  - Copy [web-root]/sites/README.txt from assets/scaffold/files/sites.README.txt
  - Copy [web-root]/sites/development.services.yml from assets/scaffold/files/development.services.yml
  - Copy [web-root]/sites/example.settings.local.php from assets/scaffold/files/example.settings.local.php
  - Copy [web-root]/sites/example.sites.php from assets/scaffold/files/example.sites.php
  - Copy [web-root]/sites/default/default.services.yml from assets/scaffold/files/default.services.yml
  - Copy [web-root]/sites/default/default.settings.php from assets/scaffold/files/default.settings.php
  - Copy [web-root]/modules/README.txt from assets/scaffold/files/modules.README.txt
  - Copy [web-root]/profiles/README.txt from assets/scaffold/files/profiles.README.txt
  - Copy [web-root]/themes/README.txt from assets/scaffold/files/themes.README.txt

These files are placed by the drupal/core configuration. Note that one of them is index.php. You might need that, and if you disable the scaffold plugin you won't get it. :-)

These files are placed in *either* the root of the project for the old-school vendor-in-docroot style installation (aka drupal/legacy-project) or the best-practice vendor-outside-docroot style (drupal/recommended-project).

The remaining file system requirements and best practices are handled by the composer/installers or oomphinc/composer-installers-extender plugins. That's how Drupal Core puts itself into web/core/ instead of vendor/drupal/, and puts modules in modules/contrib/ and so forth.

If you can configure your web server's docroot to be different from /www/html/, such as /www/html/web/, then the best practice is to: a) Configure the installers plugins to use that, and b) tell the scaffold to place the scaffolded files in your configured web-root. The easiest way to do that is to look at drupal/recommended-project and either copy it or use it as the template for your new site.

It's entirely unclear why this issue with vendor duplication is occurring, and again I can't repro it.

One thing that normally happens is: During early phases, everything is installed in vendor/. It isn't until the end that these items are moved out of vendor/ and into modules/contrib or whatever. It could be that there is something about the toolchains in use that error this out or somehow prevent it from happening.

Is this process expected to happen under an Acquia dev environment or something? I noticed that BLT and other Acquia tools are requirements, which is why I ask.

reg’s picture

reg commented

@Mile23: Thank you for that rundown and it confirms what I was thinking for the web root. One possibility, at least in my case, I know that index.php file is in /web and naturally put my web root to /web before ever running the site so maybe there's some permissions thing that can happen on some servers in that circumstance that allows the copying to doc root but not the subsequent removal from the sources inside /web.

I personally can't see how that can be but perhaps not everybody sets their web root to /web from the get go so that's a possibility for a cause in it's change of behavior and like someone above said it also seems to be specific to 9.5.0 (and I guess possibly above).

reg’s picture

reg commented

If your setup is such that you are not going to use scaffolding I think the most unobtrusive way to do so is just to run the line:

composer config allow-plugins.drupal/core-composer-scaffold false

This should disable scaffolding but leave all the settings (other than disabling it) intact. I'm testing this now and will add more if I find out anything different.

claudiu.cristea’s picture

claudiu.cristea
Primary language Romanian
Location Arad 🇷🇴
commented
Status: Postponed (maintainer needs more info) » Active

Same here but with different symptoms: The web/core directory is polluted with non-Drupal packages from vendor/

claudiu.cristea’s picture

claudiu.cristea
Primary language Romanian
Location Arad 🇷🇴
commented
Priority: Normal » Major

When I set the "drupal/core-composer-scaffold" plugin to false, everything looks good. Setting to Major as this blocks me to update to 9.5.x

claudiu.cristea’s picture

claudiu.cristea
Primary language Romanian
Location Arad 🇷🇴
commented

Reverting https://github.com/drupal/core-composer-scaffold/compare/9.4.7...9.5.0 is fixing my issue. But how to satisfy this and also the source issue?

claudiu.cristea opened merge request !3229

claudiu.cristea’s picture

claudiu.cristea
Primary language Romanian
Location Arad 🇷🇴
commented
Priority: Major » Critical
Status: Active » Needs review
StatusFileSize
new 3328893-16.diff1.1 KB

The MR if fixing my issue. Curious if it's resolving also the original report issue.

Posting as patch because of https://drupal.slack.com/archives/C51GNJG91/p1673438297534429

mikemadison’s picture

mikemadison commented

I imagine it does, I was pinning to the previous version of the scaffold which is effectively the same as the patch to revert the change made in 9.5.0.

claudiu.cristea’s picture

claudiu.cristea
Primary language Romanian
Location Arad 🇷🇴
commented

No, sorry, in fact it doesn't solve my issue.

mile23’s picture

mile23
Primary language English
Location Seattle, WA
commented
Issue tags: +Needs steps to reproduce

It would be super-great if someone could provide repro steps... What environments is this happening under? Are there symlinks? If you remove all the plugins except the ones that come with core, does it do the right thing? If you update all your plugins to the latest stable, what happens? Etc.

smustgrave’s picture

smustgrave commented
Status: Needs review » Needs work

Additional info was requested in #20

For what it's worth when we upgraded all our 9.4 sites to 9.5 no one reported this issue.

mile23’s picture

mile23
Primary language English
Location Seattle, WA
commented
Status: Needs work » Postponed (maintainer needs more info)

Based on #20 and #21.

Version: 9.5.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

pameeela’s picture

pameeela commented
Status: Postponed (maintainer needs more info) » Closed (cannot reproduce)
Issue tags: +Bug Smash Initiative

Closing this since there hasn't been any additional info provided after it was requested more than a year ago.