[policy, no patch] Should Package Manager require Composer HTTPS?

Discussed this with @tedbow and @phenaproxima yesterday evening in slack.

First of all I think we need to be clear on what 'require https' actually means - it's not just using an HTTPS URL, we need to be sure that the client is actually verifying certificates, rather than accepting invalid ones etc.

An example would be the -k --insecure argument to curl: https://curl.se/docs/manpage.html#-k or chrome's 'advanced mode do you really want to visit this insecure site that's going to steal your information?' button.

From the slack conversation and looking at a few things, I think what is necessary to require HTTPS in composer is the following:

1. A hard requirement on the php_openssl extension in package_manager

2. Composer has to be configured without disable-tls

3. Composer has to be configured with secure-https enabled.

#1 duplicates #2 but means that we could prevent install of the module with a clear message rather than allowing install and then it not working.

If you have both secure-https and disable-tls options enabled, then you always use https URLs but never verify certificates.

Docs for #2 and #3 are at https://getcomposer.org/doc/06-config.md#disable-tls

It's also my understanding that contrib package_manager currently only requires https urls, not #1 and #2 at the moment. It would be good if people could double check the logic of the above, but if that's right then it's not preventing you from running an insecure configuration and this would need to be implemented both in the contrib module as it currently is and before any core inclusion (if we decided to keep that logic, or to have it in a hook_requirements() warning if we moved it to that).

I don't know exactly where the https vs. TUF discussion originally took place, but I do have a vague memory of it. One of the reasons not to require https was that we wanted automatic_updates to run on shared hosting etc. and those users won't be able to enable php_openssl if it's not already enabled.

That discussion was some time ago though whenever it was, and SSL is a lot more widespread than it was even 3-4 years ago (let's encrypt etc.), however looking for details on composer ssl support, I did find a lot of evidence that even people's local environments often don't have php_openssl enabled. I have no idea what the level of support is on common web hosts etc. though and not sure if there's an easy way to find out.

https://stackoverflow.com/questions/67613751/the-openssl-extension-is-re...
https://stackoverflow.com/questions/35249620/the-openssl-extension-is-re...
(many more examples if you search for 'composer disable ssl' or 'composer disable tls')

I'm adding option #6 and #7 to the issue summary:

#6: Require both HTTPS and TUF for Drupal.org and packagist, only HTTPS for anything else. This is probably the most security we can do, but then rules out some hosting.

#7. Require TUF for Drupal.org and packagist, encourage HTTPS via a hook_requirements() warning.

It's also my understanding that contrib package_manager currently only requires https urls, not #1 and #2 at the moment. It would be good if people could double check the logic of the above, but if that's right then it's not preventing you from running an insecure configuration and this would need to be implemented both in the contrib module as it currently is and before any core inclusion (if we decided to keep that logic, or to have it in a hook_requirements() warning if we moved it to that).

Per #3351247: Harden our HTTPS requirement this isn't true, which is great - because composer specially handles the two config flags and doesn't allow you to have contradictory config, so if you have secure-http = true and disable-lts = true it will change the value of secure-https to false which is what automatic_updates checks for. However also per that issue we should do our own explicit checks on the things we depend on both for better error handling and because that composer behaviour is as far as I know undocumented.

that composer behaviour is as far as I know undocumented

https://getcomposer.org/doc/06-config.md#disable-tls says:

Enabling this will implicitly disable the secure-http option.

But yeah, they don't explicitly detail what "implicitly disabled" means.

I learned from @drumm that packages.drupal.org currently only supports https. That means anyone currently who has installed Drupal with Composer with at least one contrib module or theme successfully has done so without using disable-tls. That said, it's possible that there are people without openssl installed who failed at installing Drupal via Composer for this reason and switched to using tarballs. If part of the goal of Project Browser and Automatic Updates is to make Composer accessible to people for whom it currently isn't, then that might include people without openssl installed.

In my opinion, the only options that are secure enough are (these numbers do not correspond to the issue summary's numbers):

Option 1

Enforce https (disallow disable-tls) in order to use Project Browser and Automatic Updates. Assume that in 2023 that it's a negligible amount of sites that will be prevented from using these modules as a result of that. Basically, treat http (without TLS) as a relic of the past, like Internet Explorer, that we no longer have to accommodate.

Option 2

If option 1 isn't acceptable and we must support http (disable-tls), then I think that would mean that we'll have to do the following:

1. Change packages.drupal.org to allow http access.
2. Ensure non-Drupal vendor dependencies are signed via one of:
   • Wait for packagist.org to deploy TUF/Rugged (assuming they're willing to), or
   • Implement a partial mirror of packagist on drupal.org that includes some reasonable subset of packages. For example, perhaps all of Drupal core's dependencies plus the dependencies of the 100 most popular Drupal contrib projects. And then for people using disable-tls, to fail any install/update that needs a vendor package that's not in this partial mirror (i.e., enforce tuf: true for the packagist.org repo itself, which means anything that hits that repo will fail until packagist.org deploys TUF). However, implementing this partial mirror for non-Drupal packages would include a fair amount of complexity, so I don't think we should do this without compelling data that there's enough Drupal sites unable to make https requests to justify that work.

Personally, I'm biased to option 1, and only doing option 2 if/when we get reports of a large number of sites unable to use PB/AU due to an inability to make https requests.

It would be good to make a decision on this prior to the security audit referenced in #3352210: Security review of secure signing components for package manager that the DA is working on arranging, so that how we handle disable-tls (whether entirely disallowing it or supporting it in one of the ways in 6.2) is part of what gets audited.

It's worth noting that, in addition to sites that might have given up on Composer and shifted to tarballs, there are potentially also evaluators who tried ton install Drupal for the first time using Composer, failed because of the SSL requirement, and gave up entirely on Drupal because they couldn't install it. Folks hit by that would probably never even report it to us.

• Is it possible to check the packages.drupal.org server log for failed HTTP calls? And, while we're at it, check the update server to check how many requests for data are HTTP vs. HTTPS?
   

• On the other hand, that might not even help, because it could be that Composer spits out an error message rather than falling back to HTTP if HTTPS is unavailable or has cert issues. It's worth manually testing what happens on such an environment, also. Is there a meaningful error message? Do outgoing HTTP requests even happen? Etc.
   

• Should we audit known shared hosting providers to check whether any of them have HTTPS limitations or old versions of TLS?
   

We could also possibly make a decision to simply not support auotupdates without Composer HTTPS for defense-in-depth, and accept that we might lose some potential long-tail users because of it. I'm hesitant about that because we did explicitly decide that we had to support HTTP earlier in the initiative, so I think changes to that would need:

1. A writeup with actual data on TLS issues five years ago vs. now.
    

2. Prominent handbook documentation (e.g. in the server requirements pages) about HTTPS support requirements with all the various errors someone might receive (from Drupal, Composer, etc.), and clear installer, status report, and update status documentation and error messaging about it.
    

3. Discussion of whether this might be a barrier for the nearly 50% of sites that still have to upgrade from Drupal 7.
    

4. The above info on HTTP requests.
    

5. Discussion among the committer team.
    

6. Signoff from Dries (once we have a clear case to present the pros and cons), since it could affect who can adopt Drupal and Drupal 7 upgrades.

This isn't necessarily the case:

That means anyone currently who has installed Drupal with Composer with at least one contrib module or theme successfully has done so without using disable-tls.

disable-tls can be set with the openssl extension for example to ignore SSL warnings with self-signed certificates. Whether we need to cater for that is a different question but we can't assume no-one is relying on it just from the URLs. There might be other examples but this is afaik why it's a different setting to secure-http,

Agree with everything @xjm and @catch wrote in #8 and #9.

Re: #8.2: FYI: we already have:

• https://www.drupal.org/docs/getting-started/system-requirements/php-requ...
• https://www.drupal.org/node/3170647 (weird that has no path alias?) "PHP OpenSSL requirements"

I added all that when we had Update Manager start "requiring" HTTPS to phone home, how to enable the fallback if needed, etc.

Currently if you try to use Composer with Drupal from command and you have disable-tls enabled you would get a failure.

No this isn't the case. disable-tls doesn't mean 'only use http:// URLs' it means 'don't bother with SSL certificates' which is not equivalent.

$composer config -g -- disable-tls true
You are now running Composer with SSL/TLS protection disabled.
$ composer outdated
You are running Composer with SSL/TLS protection disabled.
Color legend:
- patch or minor release available - update recommended
- major release available - update possible

You get a warning, not a failure, composer operations run just fine (insecurely).

What I didn't test, but I think someone should, is:

1. PHP with no openssl extension
2. Composer with disable-tls = true

Then can composer actually not handle the https URL at all? And what does it say when that happens?

Another situation that it seems likely people could run into:

openssl is enabled

certificate config is outdated/wrong.

In this case they have the extension, but it can't deal with the certificate - this is another case that disable-tls would 'help' with, but again I don't know what the composer error message looks like when you're in that situation (or even how to simulate it tbh).

#3351247: Harden our HTTPS requirement landed, AFAICT that means we can close this?

Yay on #3351247: Harden our HTTPS requirement, but this issue is about deciding whether once TUF is working if we want to intentionally allow automatic updates in core to be usable by sites unable to make secure https requests (whether that's due to PHP's openssl extension not being installed, certificates being wrong, etc.), and if so, how to handle non-Drupal vendor packages on packagist per #6.2.

Trying to break down who might run into issues and why:

1. Local with openssl + disable-tls:

User has the openssl extension installed, can make https requests, but for whatever reason has manually switched composer's disable-tls setting on. An example could be someone with a local or institutional packagist front-end with self-signed certificates. This user can make https requests to Drupal.org, but they'd ignore invalid certificates. If we require disable-lts to be off, then they'd have trouble with the self-signed certificates elsewhere, and would either have to start using proper certificates, or not use automatic updates.

For me personally, I think this could be a real use-case/person, but I don't think we need to support people intentionally making their composer install less secure because they're doing strange things with self-signed certificates.

2. Local with no openssl + disable-tls

User doesn't have the openssl extension installed. I am personally not clear on whether it's possible to make a request to an https URL without the openssl extension or not - i.e. can you insecurely request the URL (similar to #1), or are you just unable to parse it at all?

If clients are unable to parse without the openssl extension at all, and Drupal.org packagist already requires https, then we can be pretty certain that this user doesn't actually exist.*

* except for possibly someone not yet using composer at all who's still on tarball installs, but again, not sure those people still exist either.

3. Hosting with no openssl

This would be a case where you have a new or existing Drupal site, and you want that site to start using automatic updates on production. However, the environment that the site is hosted on doesn't have openssl. Because you were never running composer on production before, you never ran into this before. Any stats we have about existing composer https support are somewhat irrelevant because this is environments that currently don't run it. Update status doesn't provide us useful statistics because lots of sites are still requesting http even if they could request https. To know whether this is a likely use case or not, we probably need phpinfo data from some of the bigger webhosts, especially shared hosting. If openssl is widely installed, then this might not be an issue.

For me I don't think we should cater for #1, and probably not #2. I think #3 was the original reason not to require https, and that would be the one to try to get some data on too.

3. Hosting with no openssl

As far as I can tell, this flat out can't work with packages on github.com, because:

• FileDownloader::processUrl() throws an exception when you don't have OpenSSL and a package's dist URL starts with "https".
• If I do curl -v http://repo.packagist.org/p2/masterminds/html5.json, even though I accessed that with http, the response uses "https" for all of the zipballs. That means Composer would trigger the exception above when it tries to download any of those zipballs.
• Even if Composer had code somewhere that converted those https dist URLs to http (which I don't think it does), if I do curl -v http://api.github.com/repos/Masterminds/html5-php/zipball/897eb517a343a2281f11bc5556d6548db7d93947, the response is a 301 redirect to the https URL. In other words, GitHub doesn't support fetching these over http.

In addition to the 3 cases in #16, I think another reason someone might think they need to disable-tls is if their system doesn't have a CA root certificate bundle, or has an outdated one. However, Composer already includes a solution for this, which is that they ship an up to date one from Mozilla inside composer.phar, which is used as a fallback if it can't find a usable one on the system. So I don't think we need to accommodate this case as being a legitimate need to use disable-tls.

To summarize #16 through #18:

• Sites without openssl extension can't download zip files from GitHub, and since Drupal core depends on a bunch of packages that are on GitHub, I think this is a non-starter unless we want to add a proxy on drupal.org for sites to make http requests to for downloading those zip files.
• Sites with openssl extension but without a usable root CA bundle on their host system are covered by the bundled CA file in composer.phar.
• Sites with openssl extension but wanting to use self-signed certificates for some packages, we could either decide we don't need to support that use-case, or could those sites just configure their project's cafile to include themselves as an authority?

Given #17 I think it would essentially be a usability improvement to require SSL, since that way people will get a nice message in the UI when they don't have support, instead of an exception in the UI or on cron.

So does that mean that #3356804: Flag a warning during status check if the OpenSSL extension is not enabled was not strict enough, and we want to change that from a warning to an error?

(Because an informative error from Package Manager is better than an obscure/very techy one from low-level HTTP fetching logic.)

Because per #16 this issue's scope includes needing to make decisions for both the openssl extension and also other reasons why disable-tls might be wanted, I opened a child issue, #3364565: [policy, no patch] Make PHP's OpenSSL extension a requirement for installing and using Package Manager (and therefore, Automatic Updates and Project Browser), to start with a decision on just that piece. Later, I'll also open a separate child issue for the "have openssl but want to disable certificate verification" case.

I opened #3385409: [policy, no patch] Disallow using Package Manager (and therefore Automatic Updates and Project Browser) when Composer's disable-tls setting is true as a stub issue. Will fill in the details there later.

I added the two followups from #3364565-28: [policy, no patch] Make PHP's OpenSSL extension a requirement for installing and using Package Manager (and therefore, Automatic Updates and Project Browser) as child issues of this one.

Once #3385409: [policy, no patch] Disallow using Package Manager (and therefore Automatic Updates and Project Browser) when Composer's disable-tls setting is true is marked fixed, then I think the answer to this issue's title is: yes for the initial commit of Package Manager to core, but with #3398988: Collect data on openssl and tls support across hosting environments and #3398995: [PP-1] [policy] Allow package manager to work without openssl and/or a valid certificate bundle still open to decide on if/when to loosen that requirement.

Per [# #3364565] and #3385409: [policy, no patch] Disallow using Package Manager (and therefore Automatic Updates and Project Browser) when Composer's disable-tls setting is true I think this essentially done now. The last related issue still open is #3394754: [policy, no patch] Use Update XML in Package manager to determine release support status, but that might be fine to discuss and resolve in its own right?

Moving to RTBC since I think this mostly needs confirmation (or not) from other committers on whether it's really done or not.

Can the issue summary be updated to include which of the 7 options was selected?

I think we ended up with #6.

Update the IS with the resolution

Fix formating and update credit.

Looks good, marking this one fixed.

Changing to latest version when this was fixed and updating credit.