[meta] Only support Drupal core installs managed by composer

On the d.o side, de-emphasising/hiding tarballs on project pages seems like the main thing.

In core, we'd need to remove the tarball download/update feature from update manager. Can we deprecate/remove that before automatic updates is done?

We can't stop people installing via tarball - i.e. if they wanted to manually download tarballs for core and all of our dependencies from github they still could, but that is 'manual composer' not what we have now.

If we keep tarballs for composer to download which I assume we'd have to, then keeping the version information in packaging also seems fine. If we remove that, we'd need to get it from elsewhere for update status - seems like something for much later if at all.

…switching from a tarball installation to a Composer installation on an existing site is a difficult process for many users

…especially the category of user who shied away from Composer in the first place.

+PHP_INT_MAX for this issue and for doing it ASAP.

I wanted to confirm that this discussion is about D9 and higher versions, not for D7 - is that correct?

@irinaz yes that's correct, just Drupal 9/10 and higher.

+1 for continuing to push more towards composer. While I think removing tarballs entirely will cause problems for people right now, once automated updates are available it'll be significantly easier to get beginners (people not comfortable with the command line) to do it.

Thinking through it a little more, I think this is 100% worth doing for D9+ with the stated goal that tarballs will be removed when Automated Updates are available.

There needs to be a way to convert from a tarball install to a composer install.

One way would be to limit it to new installs, where installing the site will also run the necessary scripts to make it a composer site and then any module install would also add that module to composer.json.

It would however be dificult to know if drupal/core-recommended could be used here - it will be "safer" to convert to drupal/drupal (because how would you determine if the parent directory to the current web directory is web accessible or not? eg if site is at example.com/drupal)

On the update module UI, it is likely the core equivalent of automatic_updates will be in core before this is ready. It may be useful to have one UI to cover both cases or have that module conflict with the standard update module.

There needs to be a way to convert from a tarball install to a composer install.

Drupal 9 and 10 tarballs are already generated by running Composer in packaging. So they are already composer-ready. You can download a core tarball, then run composer commands.

Tarballs will always effectively pin dependencies in some way, packaged releases on Drupal.org never change.

In that case when #3253158: Add Alpha level Experimental Update Manager module is in, it should replace that page for all users.

Just found usages of tarballs in the quickstart instructions.

Should we open a drupalorg/project issue to remove this section from release nodes?

Should we open a drupalorg/project issue to remove this section from release nodes?

Except for D7 (just stating the obvious...)

As a start for #5. The wiki page is updated with a warning that tarballs are being deprecated.

What about combining 1 and 2 and move both

Tarball downloads are currently featured prominently on https://www.drupal.org/download

Made issue for Step 6.

Also the evaluator guide needs to rewritten since it uses the tar-ball for quick setup.

@hansfn, Thanks!

I've update the IS with a new step in the remaining tasks where links can be added to any other pages that needed to be changed.

Maybe I'm missing something, but composer doesn't magically remove the need for tarballs. It's just a different (mostly better) way to install tarballs for you. But that's what a "dist" installation is. Composer has to get your code from somewhere. Either it's "prefer source" (Git checkout) or "prefer dist" (a "tarball").

1. Yes, we can hide them from the UI of release nodes. #3419966: De-emphasize tar.gz and zip files on release pages and change language around installation/Drupal versions Not sure why that's postponed, seems it could proceed any time.
2. Yes, we can deprecate the parts of Update Manager that let you install "directly from tarball" via the UI once we have project browser. I believe that's #3417136: Remove adding an extension via a URL
3. Yes, we can deprecate the parts of Update Manager that update your currently installed codebase once we have autoupdates. Is that #3201968: Augment then Replace current Update Manager URL download based updates with Staged-Composer workflow? The summary here lists #3417136 twice. 3201968 has its own ancestry via autoupdates, not sure we should re-parent it.

But none of that means we can "Deprecate tarballs", because all of it still depends on a dist release artifact under the hood. E.g. if you run composer require drupal/slim_select, you'll eventually see:

  - Installing drupal/slim_select (1.0.0-alpha4): Extracting archive

"Extracting archive" means tar -zxf slim_select-1.0.0-alpha4.tar.gz in this case. 😅

We can't actually do #3417136 or #3201968 (or whatever issue we should use for point 2) until we have a stable replacement. #3419966 could happen anytime.

@dww this is only about the core packaged tarballs like https://ftp.drupal.org/files/projects/drupal-10.2.3.tar.gz that include /vendor code and are generated by the packaging script. drupal/core, drupal/core-recommended etc. would still be in .tar.gz format but they're only pulled via composer and not exposed for direct download and install. Updated the title to make this a bit more explicit.

This in the issue summary hasn't been true

Tarball installations are completely incompatible with modules that declare their own Composer dependencies. Furthermore, switching from a tarball installation to a Composer installation on an existing site is a difficult process for many users.

The tarballs are made with composer create-project legacy-project.

It is somewhat good to have tarballs as a test in packaging, to make sure composer create-project works.

@drumm I think that statement means: if you download core and Commerce as tarballs, Commerce won't work because it has its own Composer dependencies that never get merged into the top-level vendor directory. And if you do have a working tarball setup, to switch to Composer you need to manually add each of your dependencies to composer.json, there is no automatic way of doing this.

We do have build tests in core to ensure that create-project works, but obviously they can't test a released version until the version is actually released :)

➜  ~/Downloads tar xf tar.gz
➜  ~/Downloads cd drupal-10.2.5
➜  ~/Downloads/drupal-10.2.5 composer require 'drupal/commerce:^2.38'
./composer.json has been updated
Running composer update drupal/commerce
Loading composer repositories with package information
Updating dependencies
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Root composer.json requires drupal/commerce ^2.38 -> satisfiable by drupal/commerce[2.38.0].
    - drupal/commerce 2.38.0 requires drupal/inline_entity_form ^1.0@RC || ^3.0@RC -> found drupal/inline_entity_form[dev-1.x, dev-3.x, 1.0.0-alpha1, ..., 1.x-dev (alias of dev-1.x), 3.0.0-rc15, ..., 3.x-dev (alias of dev-3.x)] but it does not match your minimum-stability.


Installation failed, reverting ./composer.json and ./composer.lock to their original content.
➜  ~/Downloads/drupal-10.2.5 composer require drupal/inline_entity_form ^1.0@RC
./composer.json has been updated
Running composer update drupal/inline_entity_form
Loading composer repositories with package information
Updating dependencies
Lock file operations: 1 install, 0 updates, 0 removals
  - Locking drupal/inline_entity_form (1.0.0-rc17)
Writing lock file
Installing dependencies from lock file (including require-dev)
Package operations: 1 install, 0 updates, 0 removals
  - Downloading drupal/inline_entity_form (1.0.0-rc17)
  - Installing drupal/inline_entity_form (1.0.0-rc17): Extracting archive
Generating autoload files
Hardening vendor directory with .htaccess and web.config files.
42 packages you are using are looking for funding.
Use the `composer fund` command to find out more!
Cleaning installed packages.
No security vulnerability advisories found.
➜  ~/Downloads/drupal-10.2.5 composer require 'drupal/commerce:^2.38'
./composer.json has been updated
Running composer update drupal/commerce
Loading composer repositories with package information
Updating dependencies
Lock file operations: 10 installs, 0 updates, 0 removals
  - Locking commerceguys/addressing (v2.2.0)
  - Locking commerceguys/intl (v2.0.5)
  - Locking doctrine/collections (2.2.1)
  - Locking drupal/address (2.0.1)
  - Locking drupal/commerce (2.38.0)
  - Locking drupal/entity (1.4.0)
  - Locking drupal/entity_reference_revisions (1.11.0)
  - Locking drupal/profile (1.10.0)
  - Locking drupal/state_machine (1.11.0)
  - Locking drupal/token (1.13.0)
Writing lock file
Installing dependencies from lock file (including require-dev)
Package operations: 10 installs, 0 updates, 0 removals
  - Downloading doctrine/collections (2.2.1)
  - Downloading commerceguys/addressing (v2.2.0)
  - Downloading drupal/token (1.13.0)
  - Downloading drupal/state_machine (1.11.0)
  - Downloading drupal/entity (1.4.0)
  - Downloading drupal/profile (1.10.0)
  - Downloading drupal/address (2.0.1)
  - Downloading commerceguys/intl (v2.0.5)
  - Downloading drupal/commerce (2.38.0)
 0/9 [>---------------------------]   0%
 5/9 [===============>------------]  55%
 6/9 [==================>---------]  66%
 8/9 [========================>---]  88%
 9/9 [============================] 100%
  - Installing doctrine/collections (2.2.1): Extracting archive
  - Installing commerceguys/addressing (v2.2.0): Extracting archive
  - Installing drupal/token (1.13.0): Extracting archive
  - Installing drupal/state_machine (1.11.0): Extracting archive
  - Installing drupal/entity (1.4.0): Extracting archive
  - Installing drupal/profile (1.10.0): Extracting archive
  - Installing drupal/entity_reference_revisions (1.11.0): Extracting archive
  - Installing drupal/address (2.0.1): Extracting archive
  - Installing commerceguys/intl (v2.0.5): Extracting archive
  - Installing drupal/commerce (2.38.0): Extracting archive
  0/10 [>---------------------------]   0%
  7/10 [===================>--------]  70%
  9/10 [=========================>--]  90%
 10/10 [============================] 100%
Generating autoload files
Hardening vendor directory with .htaccess and web.config files.
43 packages you are using are looking for funding.
Use the `composer fund` command to find out more!
Cleaning installed packages.
No security vulnerability advisories found.

Sorry, I missed this part:

and Commerce as tarballs

Tarballs are indeed not good for contrib, and https://www.drupal.org/project/commerce/releases/8.x-2.38 says to use Composer over downloads.

Core tarballs are composer-compatible, you get the same thing as composer create-project drupal/legacy-project

I'm not sure why you would do #26 - if you have Composer then you can run create-project yourself already? But given that example maybe core tarballs aren't really a problem, it's contrib ones that can be problematic. Maybe we can start on the remaining tasks like removing the Updater UI, and start removing more references to tarballs from d.o without waiting for Automatic Updates?

Is there any way of telling from access logs how many people still download tarballs manually compared to via Composer? Unsure also how we will achieve point 3 of notifying existing tarball users that they should switch to Composer.

Yes, we should go ahead and continue discouraging using contrib tarballs directly. They are still totally used by Composer internally of course.

We prominently feature core tarballs on https://www.drupal.org/download and various other places mentioned in this issue. Those need auditing & updating before we think about removing core tarballs. Otherwise, I'm mostly mentioning that core tarballs do work with composer, and the issue summary needs updating.

Is there any way of telling from access logs how many people still download tarballs manually compared to via Composer?

We could probably get the access logs from S3, filter to tar/zip release downloads, and see how many have Composer as a user agent. That would be good numbers for contrib, since Composer and manual downloads are the same URL. We would not get a comparison for core, since the downloads via Composer don’t hit our servers, we’d only see the volume of non-composer downloads.

Unsure also how we will achieve point 3 of notifying existing tarball users that they should switch to Composer.

Could core check for the codebase mismatching composer's installed.json? I think automatic updates already has a check for that.

Can an alternative installer be provided instead that simply (potentially bundles and) runs a composer command to download and install drupal?

That would also solve the problem (when Automatic updates and Project Browser are in core) of not requiring people to be comfortable with the command line to use drupal.

Would it help to remove this before D7 EOL, if we still need to keep all the code for 7.x tarballs?

I do not know the usage stats, but if these are relatively high, would it make sense to remove these tarballs before Project Browser / Auto Updates?

D7 core actually has the same packaging as contrib. Only core D8+ is special, and will remain special for subtree splits, which are part of packaging.

What we do get to simplify around D7 EOL is special cases for displaying tarballs. At some point https://www.drupal.org/project/webform/releases/7.x-3.31 could look the same as https://www.drupal.org/project/webform/releases/6.2.2, probably not for a few months after D7 EOL since there will inevitably be stragglers. And D7 packages.drupal.org support will have to remain until usage is sufficiently low.

Tried to add clarification to the first item in the issue summary. Rearranged the remaining task to be more like the order in which the items can be done.

Do we have a date when sites using tarballs will no longer be available and site must switch to composer?

@quietone

Do we have a date when sites using tarballs will no longer be available and site must switch to composer?

No as I understand it, this issue is for deciding when that date would be amongst other details of deprecating tarballs.

What I think might make sense is to pick a date that is a few months in the future and announce 'Drupal core tarball installs will no longer be supported after x date' so people have plenty of warning, and then we can figure out all the steps between now and then. It wouldn't be a deadline, but it would mean we don't end up stuck here because we didn't do an announcement in time.

Given we have a convenient future date of 5 January 2025 for Drupal 7 EOL should we announce the same date for EOL of tarballs as well?

That works for me - even if d.o doesn't actually drop support for a few months afterwards it still makes sense to announce as the policy change.

I agree with setting the date to 5.1.2025 to match the D7 EOL.

#3417136: Remove adding an extension via a URL was listed for both point 3 and 4. I replaced the mention in point 3 with #3201968: Augment then Replace current Update Manager URL download based updates with Staged-Composer workflow which I think is closer. Remember, the current "Update Manager" has 2 main aspects:

- Update your currently installed stuff (point 3) -> to be replaced with Auto Updates
- Install new stuff (point 4) -> to be replaced with Project Browser

They really need to be handled separately, so let's not conflate them and use the same issue to track both.

Meanwhile, I still think point #6 (the d.o UI) could happen any time, I don't think that needs to be blocked on everything else.

All that said, I think we need to be careful how we word this effort and the related communications. As I wrote at #22, we're still going to have tarballs, even "packaged" ones. We're just telling people they have to use composer to install them. Maybe I'm hung up on pedantic semantics, but I believe the title here is misleading and confusing.

Updated the Issue Summary and title to reflect the recent comments.

As for the wording, should we start a google doc for that?

For #6 It doesn't seem right to remove the ability to download until the deprecation date. In the meantime a warning could be added.

Maybe this for the re-title?

Thanks for addressing my concerns. I believe @quietone's version was closer to reality. 😅 This issue started as (paraphrase) "Release managers hate tarballs of core that include vendor", but the title from #41 is going in the wrong direction in a few ways:

1. The Update Manager doesn't do anything with core, tarballs or otherwise. 😂
2. There are many other things to deal with if we want to stop letting folks directly install from tarball than just Update Manager (d.o changes, communications, etc). These are spelled out in the summary, but the title from #41 only covers 2 of 6 points.

Therefore, restoring the title from #40 (until something better comes along).

Thanks again,
-Derek

whoops, just notice the grammar bug, sorry for the noise

p.s. I just opened #3439737: RFC for Ludwig maintainers: Core wants to require composer for everything to alert the Ludwig maintainers about this issue, since I figured those folks might care and want to chime in...

Arggh I thought I was on another issue, which I now can't even find, sorry for the noise.

I thought this issue was originally about Drupal core. Remove tar/zip download links from https://www.drupal.org/download, https://www.drupal.org/project/drupal/releases/10.2.5, and elsewhere, focusing on only composer for installing Drupa. Then dismantle as much of the supporting infrastructure as we want.

Some of the reasons in the issue summary are not true, and others are new, like #3294241: Remove drupal/legacy-project as a starting point for Drupal 12.

For contrib, packaging and tarballs remain, download links may be more-hidden or removed from release pages, and #3417136: Remove adding an extension via a URL moves forward.

A possibly better title might be "Only support Core installs managed by composer" so that we get away from the argument about the existence of tarballs in the backed.

After retitling this and reading #46 it seems to me that this is not dependent on Auto Updates and Project Browser (as I have though previously), as we are talking mostly about core installs. So if I understand it correctly, after we remove these download links, it would not be possible to install Drupal for anyone without the knowledge of composer (if they do not know about the github repository)? So all users currently doing download - extract - install will be cut out, if they do not understand composer?

Then I think it would be great to have a look at the stats mentioned in #28/#29. Because this change could increase the level of knowledge needed for even installing the basic Drupal installation for newcommers (mostly for those without deeper technical knowledge). I think it will be good to take this factor into account as well.

Considering sites will have already designed and implemented their D10 deployment processes, should this be reserved for a major release onward so that site owners can implement this at the same time they typically plan to refactor their deployments? Possibly change this to "In the release of D11 onward"

Positive:

• Moves the first implementation of this forward
• Allows sites a more delineated changeover that they can choose to adopt when ready (either in the D10 release cycle before upgrading to D11, or when they choose to upgrade to D11)
• Helps encourage sites upgrading from D7 to implement the the new workflow.

Negative:
D10 is now slated to survive until D12 release in 2026 meaning full removal would be considerably longer possibly decreasing the overall gain to Release Managers.

Re Statsticis:
Would those be biased towards composer due to CI runs? Since they do not have core releases do they tell us much about the actual ecosystem?

We’re plowing ahead with at least removing the UI to install new stuff. #3417136: Remove adding an extension via a URL now has 2 MRs:

• Full removal for 11.x branch: https://git.drupalcode.org/project/drupal/-/merge_requests/8781 (removes routes, links, forms, dead code, adds deprecations, etc).
• Backport version for 11.0.x (and 10.*.x branches?) https://git.drupalcode.org/project/drupal/-/merge_requests/8810 (only removes routes, action links, and tests, leaves all the dead code in place).

I’m not interested in moving all this now broken stuff to a contrib module and deprecating it. It served a need for a long time, and I’m glad I wrote it when I did, but it should go, even without project browser in core.

I’m a little more torn about the updater parts. I think we really need a working replacement before we remove it all. But perhaps we could at least formally deprecate all that (not exactly sure how we deprecate a form and route, honestly).

P.s. as Drumm has pointed out multiple times, parts of the summary aren’t true.

Latest composer 2.8.1 became stricter on tags so it may affect split https://github.com/composer/composer/issues/12150#issuecomment-2423790429

#3491731: [META] Remove the ability to update modules and themes via authorize.php is now the correct dedicated issue for deprecating / removing the authorize.php code and the UI from update.module.