Come together with the global Drupal community in Rotterdam, 28 Sept – 1 Oct 2026. Sessions, contribution, connection, and Early Bird savings until 8 June.
By fkelly12054@gmail.com on
I'm making progress on getting composer based install and update to work on shared hosting. Install works, update of the drupal core works. Beautiful thing.
My directory structure is
public_html
... web
... vendor
the composer.json file is in the public_html directory and I run composer right from that directory. Earlier today 9/16/2021 I ran:
composer update drupal/core "drupal/core-*" --with-all-dependencies
All seemed fine. After running update php my status report showed I was at 9.2.6. However I noticed in my PHP error log a series of Symfony related error messages. I compared the vendor directory in the 9.2.6 tar.gz file versus the one on my site. Obvious that the vendor directory is not getting updated by composer. Not obvious (to me) how I get this to happen.
IF I switch into my vendor directory and run composer outdated I get a long list of files that have updates. But if I run composer update --dry-run (to be safe) I am told there is nothing to update. I guess I could copy the tar.gz's vendor directory to the one in public_html but that seems to be defeating the purpose of having using composer to update my site.
Suggestions welcome.
Comments
What makes you think vendor is not updated?
There is no need to traverse to vendor, you should run outdated from the folder above, where the composer.json is. You also should use -m on updated or even better start with -Dm.
You'll always have outdated due to your project requirements. Especially when using drupal/core-recommended as it is pinning to specific versions, a sinn in composer world!
The drupal community is trying to keep up with newer versions but it is not there yet. So, the requirements will keep you on earlier version of basic frameworks like symfony. Hence my suggestion for a Drupal project to start with -Dm on outdated. Those are the primary dependencies you need to keep current. Replace drupal/core-recommended with either drupal/core or your own metapackage at least allowing patch releases (~#.#.#). For selected packages extend the constraint to also allow minor or even major releases.
Once primary dependencies are taken care of you can investigate lower level of dependencies. Suggest to start with -m to figure out patch or minor releases. You need to filter as there is no easy way I know of to only consider patch releases (bug fixes). With minor updates available it doesn't inform you about patch releases.
After patch releases applied investigate minor and major releases. Use prohibits to find out newer releases being block by version constraints.
the composer.json file is in
If you're using the Drupal Composer template, that location is wrong, it should be one directory above that.
Contact me to contract me for D7 -> D10/11 migrations.
Nothing wrong with that
What makes you think there is anything wrong with that? The composer.json should alway be in the same folder als web and vendor.
The composer.json should
The poster doesn’t have it in that directory. They have it one below that in public_html.
Contact me to contract me for D7 -> D10/11 migrations.
No, he doesn't!
No, he doesn't!
First, as always, thanks to
First, as always, thanks to Jaypan and MMJVB for responding.
Second, composer.json is in the public_html directory which is at the same level as /web and /vendor. Composer.json follows the recommended-project convention of sticking the Drupal files in the /web directory. That part is working fine.
On my shared hosting (InmotionHosting) the web root is public_html. While you can override this Inmotion strongly discourages this. All their software is set up assuming your web root is public_html and software features, such as restoring backups, that Inmotion does for you will not work properly if you change the web root. Also, Inmotion allows you some liberties such as picking a version of PHP (7.4x in my case) within the public_html directory. Their standard PHP outside that is 7.2, which won't work with Drupal. I suspect that they probably have other customers using older software that won't work with 7.4 and are constrained by that in how fast they can move forward. Example Drupal 7 is still listed on their Softaculous installation list without any mention of Drupal 8 (which is just as well, I suppose).
An advantage of using public_html is that I can then install and manage my own version of Composer within that and do self-update. For a long time Inmotion only supported composer 1.x which was a complete memory hog and wouldn't run under shared hosting memory constraints. I now have composer based installation and update (with the exception of the vendor issues which I'll get to) working in public_html. For Drupal this works beautifully, again with the exception of finding a way to get vendor updated appropriately.
There are some deficiencies with this approach. As I understand it, one of the goals of the recommended project was that the vendor directory would be at the same level as the web root (and thus outside the web root). I've thought running Composer with the no-install option, then modifying the resulting composer.json to point the installer_paths to ... well ... where it says:
but that would put vendor outside of the public_html directory where I have control over what version of PHP runs. Since my current approach works I will just live with having vendor in web root for now (the htaccess in my public_html directory directs everything to the web directory where Drupal lives already).
@mmjvb I will have to experiment with the suggestions in your third paragraph on my test sites on Wampserver. In the interim I may just copy the vendor directory from the tar.gz distribution of 9.2.6 since the core committers at least know what versions of the various software (symfony in particular) work with Drupal. I posted a support request over in the issue queue for Composer documentation asking for help there. Seems like if we are going to be using Composer for updates we ought to have a way for users (i.e. site admins) to reliably get the appropriate updates for all the files in the vendor directory. I suspect that, since the software in vendor comes from many sources and is constantly under development, there will be many cases where they may "get ahead" of what Drupal works properly with. So we need a reliable way to get the right files. I'll try your suggestions, but locally first. @alexpotts and @catch and other core committers responsible for generating the tar.gz know this stuff like the back of their hands, but most of us don't and need help.
and @mmjvb thanks for the note you sent while I was typing this. You are right.
Suggest to use subdomain
Can you use a subdomain. No problem the domain points to public_html as long as the .htaccess gets you to the subdomain pointing to public_html/web
You can configure composer to have vendor in a different place and different name, I believe, haven't done that myself. That is not related to drupal-scaffold, two different things. See vendor-dir and COMPOSER_VENDOR_DIR too use vendor at different location.
The suggestion for a subdomain is to prevent the domain to be available via http by redirecting to subdomain. To prevent access through http to vendor. The definition of domain is relevant for the provider, doubt that it is relevant through http. Suspect it to use other protocols than http.
Also believe we are correct, but wonder what makes Jaypan think we are wrong. Do appreciate his contribution, even more when it turns out we are wrong.
.htaccess in public_html
.htaccess in public_html simply directs everything coming to my site to the web directory:
The subdomain I have simply points to a menu item in the top horizontal menu on my site. It's my wife's area for her sculpture. She directs people to her domain name (sculptandprint.com) and those people never have to bother with the rest of the site, or even be aware of it. It's like Multi-site without multi-site baggage.
Tomorrow I'm going to build a local site from scratch on Wampserver with composer. I've done this dozens of times now. Then I'm going to compare what gets put into the vendor directory to what was put in there 6 weeks ago when I did a composer build. We've gone through 3 Drupal updates and/or security fixes since and I want to see how the vendor directories have changed.
And yeah I appreciate anything Jaypan posts as well as the articles he's written.
I have some work to do on a
I have some work to do on a test site but I'll just add a link to this:
https://www.drupal.org/docs/develop/using-composer/using-drupals-vendor-...
I'm pretty sure the core developers are using some version of vendor hardening to generate the vendor directory they include in the tar.gz. This is based on a file comparison between the latest 9.2.6 release (the vendor directory in the tar.gz) and what is generated by composer without vendor hardening. I will be interested to see how a "virgin" 9.2.6 install using composer (on a local system of course) goes compared to the vendor directory that was generated a month of two on earlier installs. Then what happens when I include the vendor hardening plugin in composer.
It seems like a shame to go to the work to get composer install and update working on shared hosting, then go back to copying the latest version of the vendor directory from the 9.2.6 tar.gz over what composer created on my site.
More to come.
Content of vendor
Doesn't sound like the right expectation. The content of vendor is the result of the earlier steps. Different steps result in different result. You shouldn't expect the result to be the same. The tar.gz is meant as a start for a new project. It is not intended to be used to update existing projects.
Could you be specific in the differences you see?
In the file replacement
In the file replacement approach to updating Drupal you replace old version core with tar.gz version of core, vendor with vendor, and what I call base files (those directly in the "web" directory ... excepting .htaccess) with tar.gz base files. In this approach the tar.gz is what you use to update. This is how I've done it since Drupal 8.0 ... probably 75 times. I'm trying to transition to the composer approach. The vendor directory issues are causing questions.
This morning, on a local 9.2.6 system I did:
and also modified my composer json in line with the instructions in:
https://www.drupal.org/docs/develop/using-composer/using-drupals-vendor-...
It appears to work, somewhat. What I mean is that before the composer require I had 5853 files and 1003 folders in the vendor directory, now I have 5608 files and 968 folders. The plugin also added a .htaccess in the vendor directory. Also, a "good thing" and a positive step to see that the developers recognize that some folks can't easily keep vendor outside of web root. +1 for that.
However the tar.gz vendor directory has 2639 files and 468 folders. So something different is going on there.
I thought the goal of all this was that, when the monthly (or more with security fixes) version of Drupal comes out you could go to your command line and do a composer update and your update would be taken care of.
Don't have time to pursue this further today. I don't see why the tar.gz should diverge from what a composer update would create.
Indeed, that is what you would do when manually updating
In that case you would use the vendor of the tar.gz to replace what you have installed. But, that assumes you haven't used composer. As soon as you have, you can't use the replacing from tar.gz anymore. That is because what the use of composer has as effect on your installation.
I wrote a 5-part blog on
I wrote a 5-part blog on composer for those first using it with Drupal. It’s a bit outdated in that it was written before the current Drupal template that the docs on Drupal refer to, but other than that it’s still applicable in Dupal 8 and 9. https://www.morpht.com/blog/drupal-and-composer-part-1-understanding-com...
Contact me to contract me for D7 -> D10/11 migrations.
Thanks again @jaypan and
Thanks again @jaypan and @mmjvb. I just spent an hour rereading your articles @jaypan. Yes, I essentially attempted to "composerize" my site. At the time you (jaypan) wrote the article drupal-project was recommended as the template for running composer. Now it's recommended-project. I'm using the latter. I'm not using git. I am trying to run composer directly on my shared hosting server. I have identical sites (with the same Drupal code and an exported (from the production server) database running on my Wampserver. So, I can try everything out (as many times as necessary) to make sure it runs locally, then run composer update on production. Of course I have backup.
For everything except /vendor this seems to work. In fact after a couple of updates this way (using Composer) my site seems fine except for some /vendor (primarily symfony) related errors in my PHP error log on the production server. Everything that a user or admin would see on the site works normally. Now I'm not saying that's good enough ... we want the /vendor directory to be correct and no errors generated.
This is why I'm investigating what @mmjvb said with regard to /vendor.
"As soon as you have, you can't use the replacing from tar.gz anymore. That is because what the use of composer has as effect on your installation."
I'm trying to understand that fully. The vendor hardening plug-in is supposed to help by filtering out /vendor files that are not needed. However, it is not included in the recommended-project composer setup. So, doing an composer install with recommended project you are going to get a very bloated /vendor directory. As I mentioned earlier there are twice as many files and twice as many files and folders if you just do a composer install without vendor hardening as there are in the tar.gz. Seems to me that the vendor hardening should be included in recommended-project. No?
It also appears that the vendor directory is not getting updated when you go from say 9.2.5 to 9.2.6 by doing composer update using
Now, if it's not possible to manage your site with composer instead of copying tar.gz files around, I'm not sure what the point of using composer at all is. (Actually I am sure: there are some modules and other features that can't reasonably be installed or updated without composer, but that's another issue).
As time permits I'm going to explore:
1. What happens if I do a "from scratch" Drupal install using recommended project but use the no-install option. Then edit the composer.json to include in the vendor hardening code. Then run the install. Will I come close to the tar.gz version of /vendor?
2. When you have contrib modules listed (required) in composer.json (say for example from the one I am using)
are these modules pulling files into the vendor directory?
The "extra" section of my composer.json now includes vendor hardening as per:
This results in slightly fewer files and folders in /vendor, but not nearly the reduction that the tar.gz would imply. What's more it is not updating the /symfony directory in /vendor where most of the PHP errors I'm seeing in production seem to stem from.
@mmjvb I have been experimenting locally with the "outdated" command and options you recommended three days ago. Running
composer outdated -m
shows a list of packages that seems to correspond to areas where my production system has not been updated. Would it make sense to try to run a separate update for these? And are we headed towards a situation where you have to do separate steps: one to update drupal/core and second to update various dependencies? I don't see where any of the documentation on drupal.org addresses this.
Also went through the articles of @jaypan
Unfortunately, didn't see anything relevant to your situation. To composerize an existing site is only the first step. You want to mix composer and manual work. It requires you to know a lot more of how composer works and what was done how in the past.
Scenario 1
Would expect scenario 1 you described above to come very close to the tar.gz vendor. but I haven't tried it jet. Should be:
composer create-project drupal/recommended-project myproject --no-install
cd myproject
composer require drupal/core-vendor-hardening --no-update
composer install
Not sure that you need to provide the extra setting for hardening, isn't that the default? The tar.gz is based on drupal/legacy-project but created at a certain moment in time. The `composer install` ran at a different time. That means things could have changed since then. So, if the world stopped turning you would have the same thing. Very unlikely to be the case, so your requirements might resolve differently. The metapackage drupal/core-recommended does pin dependencies to a fixed version, but their dependencies are not pinned. That is why a `composer install` is only producing the same result with a composer.lock file present. Without it composer resolves your requirements based on what is available at the moment you run. So, the contents of vendor is time based. As soon as you allow composer to update it resolves certain requirements depending on its scope.
When composerized you NEED to update with composer, you can no longer use the manual approach. Unless you want to composerize every time.
composer outdated -m
Unfortunately, it shows what is available. Not what is usable in your project. Use prohibits to find out what blocks it or --dry-run to see what it is going to do to verify that it is what you want.
Indeed, once the project is extended you need to have it in your workflow to check for updates. That is why I suggest to start with -D to consider primary dependencies. The -m is for updates within the current major, that includes minor and patch releases. You should be able to apply patch releases trouble free because they are bug fixes only (provided they follow proper development according to semver.
Obviously, it needs to be according to your specifications. You may need to change requirements or even request support request as the requirement are out of your control.
So, as a partial report on approaching the vendor directory
So, as a partial report on progress. I wiped out one of my test (Wampserver) directories and started fresh with a
"composer create-project --no-install drupal/recommended-project:9.2.6 web"
then I edited the composer.json to insert vendor hardening. Then I ran a composer install. I'm not even sure I will install a site in this directory, I was primarily interested in what would happen with the /vendor directory.
1. tar.gz vendor from the official 9.2.6 release = 2639 files and 468 folders
2. vendor on a local site before vendor hardening = 5853 files and 1003 folders
3. vendor on same local site after vendor hardening = 5608 files and 968 folders
4. vendor on local site where vendor hardening was inserted into composer.json before install = 5307 files and 885 folders
Lesson: the Drupal tar.gz creators are doing something else besides (in addition) to vendor hardening to reduce the size of the vendor directory.
Now to compare the two vendor directories and see what the difference (in detail) is between #4 and #3. Obviously (to me) the vendor directory is a moving target, one that moves independently of Drupal. So one created a month later than another will have many differences ... and this will continue. The question in a composer approach to updating Drupal is how to keep vendor in synch with what is required by drupal core.
The question in a composer
The question in a composer approach to updating Drupal is how to keep vendor in synch with what is required by drupal core.
I think the answer is by running
composer install, but I'm not sure if I' missing something.Contact me to contract me for D7 -> D10/11 migrations.
@jaypan. It must be me who
@jaypan. It must be me who is missing something. I don't see where a composer install is going to do anything helpful about generating a valid /vendor directory much less a current one that has vendor hardening baked in.
A quick scan of composer initiative issues
revealed https://www.drupal.org/project/infrastructure/issues/3092402
No time to read it now, but it might provide info on what is done. You may put in a support request and ask the people involved.
Core composer issues: https://www.drupal.org/project/issues/search/drupal?text=&assigned=&subm...
Change filter to composer initiative for the work on legacy-project
Filter on closed issues as it was done a while back.
thanks mmjvb. I spent an
thanks mmjvb. I spent an hour this morning reading. Obviously the core developers and committers have been working on issues surrounding the topic in this thread for years. I don't see any silver bullets in anything they have posted and may have to tailor my own question in one of their issue queues. Which is fine, that's what they are there for. I tend to be a bit reticent over in the issue queue area to avoid being a distraction from the important work they are doing. That said, how we generate a valid /vendor directory for an updated version of Drupal is kind of important as Drupal moves further into requiring Composer for updates.
Indeed, they have put a lot of effort into this
I am sure they would be willing to share their knowledge. Suggest to give it a try. When you refer to this issue, they should realize you are putting some effort in as well.
I did my best to post an
I did my best to post an issue over at:
https://www.drupal.org/project/drupal/issues/3238567
We'll see what falls out. I have looked further at vendor hardening and even gone over into the Drupal git repository. From all I can determine, the core committers and developers are applying additional changes to the vendor directory in the tar.gz beyond what should be generated by the vendor hardening code. The vendor hardening code itself is in the vendor directory under the /drupal directory.
I think that the "recommended-project" template should automatically apply vendor hardening to the composer.json as an initial step in installation. And I think the composer generated /vendor has to be identical to the tar.gz one. We'll see.