Should security releases occur during an outage at a hosting provider?

It's my opinion that it should fall on enterprise hosting providers to make every effort to ensure the most platform stability within security release windows. I'm not necessarily saying that this already isn't the case as customers aren't privy to platform implementation details, but having official internal SOP's in place could go a long way here. Things like not making risky changes that could overlap with release windows for example.

This seems a lot more feasible to me than the other way around. Is this something that the security team could possibly influence?

From my point of view (agency owner): I would hope that the Drupal security team might speak to the effected hosting provider and ask their opinion. It would seem strange to me if a bunch of security updates were released when Acquia or similar sized Drupal hosting provider is known to have a big outage (unless the parties talked to each other and provide a solution or reason for going ahead anyway).

I really appreciate that the team brings up this question now and trust you'll make the right decision.

Drawing a line where which provider are "big" enough to hold on a security release seems quite opinionated. Should this target only Drupal-dedicated hosting, or generic hosting too ? And with how many sites/clients ?

I do feel that outside a cloud provider being down (AWS, Azure, GCP, OVH, etc) where multiple Drupal or generic hosting provider chould be affected, the security release window should stay. It's easier for hosting provider to align themselves with the community than ask for the whole community to wait until a provider that is not their partner fix their stuff.

But then, I'm sure (I don't wish it) that one security vulnerability will actually prove that sometime, waiting for a majority of hosting providers to be ready will be a good thing to do. Security team should have the final call on this (as they already do, but adding clarity around the process never hurt).

I agree with arakwar that you need clear guidelines if you are going to make this a policy. I think it makes sense to withhold a security release if a large number of sites may not be able to roll out those changes. I think where I would have an issue is if the security team is put in a position of picking who qualifies in the moment. For example what if a campus that self hosts goes down and has 100's of Drupal sites, would that qualify for skipping a release window? Would Linnode having issues count as much as Acquia? I noticed Acquia reached out in slack today, will it be on the hosting provider to ask for an exemption or would it be on the security team to go through a list of companies and make sure they are up and running. This doesn't appear to be an issue most of the time, but I think a clear standard would be helpful to the community, perhaps a list of which companies/organizations qualify.

I don't think a provider being down should have any impact on security releases. Everyone waits on security releases and they have a specific window. The danger here is delaying an important release and then pushing it outside the release window. If the alternative is just see you next week then that would be acceptable as people expect to be doing security updates in that time window.

Thanks for the clarification on #6, greggles. I am in no way affiliated with Acquia (other than having sites hosted with them).

This does seem like a tricky policy to implement while keeping things unbiased and fair. Holding up a release for one provider (or a specific list of providers) doesn't seem like a very inclusive policy to implement.

It also doesn't feel right that the security team, as volunteers, would take on any additional overhead to accommodate for things that are provider problems. Release windows are common knowledge and the security team is already admirably doing their part. If a provider is degraded during an important release, it's just that much more pressure on them to restore service in a timely manner.

I regret dropping the note in the slack channel in retrospect. It wasn't considerate of the bigger picture.

I'd expect any hosting provider that's clearly big enough to be contacted directly would have someone on the Drupal Security team already.

To support hosting providers that are not-quite-big-enough, it might be enough to set up a Security Notification group or discussion listserv that gets pre-release notices of security updates. (I'm taking this discussion to be evidence that such a thing doesn't already exist.) The notices would include a list of modules and their CVE severity scores, but no details of the patch itself, and expected release dates. If a hosting provider knows there is a Core 20/25 release and a Views 22/25 coming on Wednesday a week from now, they'll take that into consideration for their own updates, either delaying a PHP patch for a day or maybe excluding customers they know run Drupal from getting patched that day. It's also a good way to let a vetted group know about zero-days that are imminent, but not yet announced publicly.

I think it should be a full discussion list so hosting providers can request the Drupal Security team delay a release if there is something else major happening at the same time affecting multiple hosting providers, like a critical RHEL security update, or a wide-spread attack in progress.

"Big enough" can be self-selecting by putting the onus on the hosting providers to join the group, not the Drupal Security team to identify and reach out to them directly.

This is a good business opportunity for DA, and for Drupal general.

"Drupal security partner"
Something like: If a company has more than 1000 commits (in the last 365 days) to Drupal core, or paying a monthly (put a big number here) security subscription fee, or has a permanent member in the Drupal security team, then the company can negotiate the security release time. All companies and the security team must agree on the time change. Disagreement must be for a serious reason. Security team always have the final call, in case of disagreement.
Time change request must be delivered at least X hours before the security release window (before the security team starts preparing the release).
The approved time change request would be an extra cost of course (time is money)..

The partnership must be a legal paper, containing stuff about the security disclosure or something..

The security team can deny/stop the partnership with a company for any reason.
Also that the company can get info about the security release beforehand, is a good idea. Only the "outsiders" will remain on the #security-questions Slack channel on Wednesdays :)

Seems to me that the burden should fall to major hosts seeking special accomodation. Let them contact the security team. From there the decision should be entirely Drupal-centric. If this host's issues could hurt the Drupal project, which I suspect would be true of several big Drupal players like Pantheon, Platform SH, Acquia, etc, then it could be in the community's best interests to delay the security release. Assuming of course that the negative impact would be notably greater than the negative impact of delaying the security release.