Problem/Motivation

Must-haves prior to tagging Drupal 9.0.0-beta1

8.9.x and 9.0.x will enter beta at the same time and will have the same beta stability requirements as a normal minor release. Additionally:

  1. There must be no outstanding issues to update PHP and JavaScript dependencies (i.e. they should be on the latest release of the latest branch we intend to ship with). Also unused dependencies should be removed prior to beta.

     

  2. ALL BETA SCOPE DONE: All deprecated code usages and backwards compatibility layers must be removed and deprecations properly documented - we can't remove deprecated code that is called by Drupal core. (see the @deprecated tag). This includes deprecated JavaScript libraries etc.
    See:

     

  3. ALL BETA SCOPE DONE: Any critical Drupal 8 upgrade path bugs must be triaged / resolved in 8.x so that sites are not stranded on pre-8.9.x versions due to known core bugs.
     
  4. ALL BETA SCOPE DONE: Also in 9.0.x, updates from versions prior to Drupal 8.8.x must be removed and hook_update_last_removed() implementations added: #3108254: [META] Drupal 8-9 upgrade path clean-up This is to minimise the surface area for 8-9 upgrade path bugs.

    #3095333: Extend filled database dump with new stable modules and content for them is still a nice to have for beta

     

  5. ALL BETA SCOPE DONE: There must be a Drupal 9-specific version of the Stable theme, so that contributed and custom themes are able to port to it. #3050374: Create Drupal 9 stable theme
     
  6. NO OUTSTANDING ISSUES: Any known 9.0.x-only security or data integrity criticals must be resolved.
     
  7. ALL BETA SCOPE DONE: Simpletest module must be either successfully moved to contrib, or retained in core and undeprecated #3110983: Verify that simpletest tests can be run in contrib, or add simpletest back to core. (done)

Should-haves

These issues are not hard blockers to releasing Drupal 9, but there would be significant negative consequences of them not being completed in time for Drupal 9's release. Outstanding should-haves may cause 9.0.0 to be deferred from the June release window to the August window.

  1. NO OUTSTANDING ISSUES: Any critical API changes, API additions, or data model changes should be completed, as these changes will no longer be allowed during the beta.
     
  2. ALL BETA SCOPE DONE: All Drupal 6 and Drupal 7 -> Drupal 8/9 core content migrations should be stable. Only multilingual migrations are experimental at this point. So #2208401: [META] Remaining multilingual migration paths needs to be completed. If we don't do this, we can't expect people to migrate from Drupal 6 and Drupal 7. It would still be nice to land #3076447: Migrate D7 entity translation revision translations.
     

  3. ALL BETA SCOPE DONE: We should have adopted a policy for PHP 7.x version support, as this needs to be announced as far as possible before the LTS and 9.0.0 release so hosts have time to upgrade. #2917655: [9.4.x only] Drop official PHP 7.3 support in Drupal 9.4.
     
  4. ALL BETA SCOPE DONE: We should have MySQL, PostgreSQL and SQLite requirements implemented.

     

  5. ALL BETA SCOPE DONE: We should provide accurate update recommendations (9.0, not a non-existent minor version of Drupal 8) on the Status Report (/admin/reports/status). #2991207: Drupal core should inform the user of the security coverage for the site's installed minor version including final 8.x LTS releases #3111929: If no recommended update is found, Update Status recommends the latest release, even if it is unsupported.
     
  6. ALL BETA SCOPE DONE: We should have a proper policy for CSS and markup changes.
     
  7. ALL BETA SCOPE DONE: Drupal.org should be able to fully handle contributed projects that are compatible with both 8.x and 9.x and not assume core compatibility based on version/branch name: #3054433: Support multiple core branch functionality for contributed projects on drupal.org This spans multiple projects and areas such as project listing, composer façade, update.xml from drupal.org and localization files.

    &snbp;

  8. We should have dropped support for Node.js 8 since it's EOL. #3107918: Require Node.js 12

Nice to haves

  1. #3111409: Add new Olivero frontend theme to Drupal 9.1 core as beta only if done enough in time, otherwise moves to Drupal 9.1.0 earliest.

Must have and should have issues (and their children) have been tagged "Drupal 9.0.0-beta1 requirement" on a best effort basis for an easier cut-through view of all issues regardless of their hierarchy or project affected.

Comments

Gábor Hojtsy created an issue. See original summary.

Gábor Hojtsy’s picture

Title: [META] Requirements for tagging the Drupal 9 beta » [META] Requirements for tagging Drupal 9.0.0-beta1
catch’s picture

Issue summary: View changes
xjm’s picture

Issue summary: View changes

Added a few additional notes, as everything that's required to ship 9.0.0 needs to be in beta1.

catch’s picture

Issue summary: View changes

Added Drupal 8 upgrade path bugs to the requirements, so that people aren't stranded on pre-8.9.x known critical upgrade path bugs when 9.0.x comes out. This will also help us to triage any 8-9 upgrade path issues

catch’s picture

Issue summary: View changes
catch’s picture

Issue summary: View changes
catch’s picture

Gábor Hojtsy’s picture

I would contest number 1 right away :)

All dependencies (PHP, JS, CSS, etc.) must be updated to their latest versions, or the dependency removed from the 9.0.x branch

I would qualify this with something like "where appropriate". There may be a PHP dependency that when updated to its latest version has bugs earlier versions don't have or incompatibilities with other dependency versions. So I can see this as a worthy goal but not as a *requirement* flat out IF fulfilling this goal causes problems. Eg. we would not delay our release for dependency A to fix a bug in its latest version but downgrade to a working version instead even though it will not be the latest.

Any critical Drupal 8 upgrade path bugs must be resolved in 8.x so that sites are not stranded on pre-8.9.x versions due to known core bugs.

This apparently includes a 3+ year old issue. The people experiencing this would already be stranded on a 3+ year old Drupal, so how is keeping being stranded there worse by going to Drupal 9 as opposed to just going on with 8.x+ which is in no way limited by these issues? I would support wholeheartedly the resolving of critical upgrade path issues, not against that at all. But I have a sense that these may be among the last things we'll be trying to piece apart at the end.

catch’s picture

Issue summary: View changes

Changed point #1 to be no outstanding dependency updates - does that look better? We obviously don't want to block beta updates on updates we don't actually want to do. But we do want to make sure versions are as close to what 9.0.x will release on as possible so that beta testing is meaningful and we don't fall too far behind.

On updates:

This apparently includes a 3+ year old issue.

Just untagged #2738879: system.schema can end up with missing schema information for some modules, resulting in hook_update_N() not getting called because it's an update system bug, as opposed to a bug with a Drupal 8 update as such, so affects contrib rather than core. Still critical but regular critical rather than beta blocking critical.

There are two other issues in there which are getting old, but everything else was introduced with 8.7.x I think. Partly because it's the most recent minor release, partly because it unfortunately introduced a lot of new upgrade path issues compared to some older minors.

so how is keeping being stranded there worse by going to Drupal 9 as opposed to just going on with 8.x+ which is in no way limited by these issues?

A few reasons that the new major branch makes it worse:

1. Drupal 9 is going to remove all hook_update_N() prior to 8.8.0, this means that it will be impossible to fix the upgrade path bugs in 9.x and they can only be fixed in 8.x. This reduces the variables for potential upgrade scenarios- i.e. no-one trying to go from 8.3.1 to 9.4.2, but it also means that once 8.x is EOL there will be no branch to commit fixes for them against any more (unlike other critical bugs which might persist between majors).

2. Some of the upgrade path bugs may require API additions/deprecations, or further updates to be added to correct previous ones (for example re-loading and re-saving configuration). We tend to commit upgrade path fixes to patch releases but this may not be possible for every single one - and 8.9.x should be the last minor.

3. Some sites are stranded because they tried to update to say 8.7.x from 8.6.x but hit critical bugs and rolled back. They already know they're stranded and are hopefully testing patch releases which may fix old updates.

However, other sites are sitting on 8.6.x out of choice or inaction, and haven't even tried to update yet. Once we release 9.x, I would expect at least some sites that have been slow to apply core updates (there are more than 15,000 sites still on 8.5.x for example) to try to catch up to a supported branch again, and that's exactly the point they'll start to hit unfixed critical upgrade paths if we still have them. 8.6.x will be 3-4 months out of security support when we're tagging the 8.9.x betas, so easily 15,000+ sites still on it when we get to that point that will suddenly have an extra impetus to get back on a supported branch.

4. While I don't have a specific example, it's possible that an upgrade path/data integrity issue which causes a notice or deprecation error on 8.x will result in a fatal error on 9.x (stricter validation? bc layer removed?). So for people who are not stranded as such but have a 'wrongly upgraded' site, they may only find out about this when they update to 9.x (and it will probably be too late for them to roll back to a backup). There are some concerning examples on #3052464: Cannot update to 8.7.0 because of taxonomy_post_update_make_taxonomy_term_revisionable where people have updated and only later realised they can't save terms for example.

catch’s picture

Status: Postponed » Active

Don't think this needs to be postponed given it's a policy issue.

xjm’s picture

Issue summary: View changes
tedbow’s picture

xjm’s picture

Issue summary: View changes

Just adding some whitespace for readability.

xjm’s picture

Issue summary: View changes
effulgentsia’s picture

The list looks good to me. I'd just recommend adding a separate heading for the Should-have items to make it easier to distinguish them from the Must-have items.

webchick’s picture

Some late night rambling thoughts...

I think the suggestions outlined in the issue summary make sense to me, on the surface, at least.

One thing that would help in the "sign-off" realm is if we had some sense of how close/far away these various things are (not necessarily "precise" figures, but even a guesstimate would help). Like I have no idea how to track #1. (Is there a tag? If not, should there be one?) How far away is #2 from completion, and is it generally trending up or down? (I'm less concerned about that one being a MUST vs. SHOULD since we certainly shipped D8 itself with deprecated, procedural code still sitting around, and laxing this requirement allowed us to ship N years sooner). One workaround there too is always "welp, we didn't get rid of it in time, so it's deprecated for Drupal 10 now." And pushing off deprecations only makes the upgrade path easier, which is basically the single most important thing we can possibly do for Drupal's future: https://dri.es/making-drupal-upgrades-easy-forever

But I digress.

I think to some extent too, signing off on this requires knowledge of what the fallback plan is if we hit March or whatever and decide to push the date back by 6 months. (There might already be an issue for discussing this; if so, happy to read/comment there instead.) Like does this date slip also extend D8 and D7 EOL by 6 months? If so, I'm a lot less fussed. If not, then hitting that date becomes absolutely mission critical, and then would recommend limiting the "musts" to only the MUST MUST MUST (there's been good justification for why existing core upgrade path criticals should be in that MUST MUST MUST list, for example).

catch’s picture

Issue summary: View changes
xjm’s picture

Issue summary: View changes
xjm’s picture

Issue summary: View changes
xjm’s picture

Issue summary: View changes

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

catch’s picture

Issue summary: View changes
catch’s picture

@webchick for #1 (PHP/JS dependency updates and removals) we have #3009213: [META] Update / reconsider PHP dependencies for Drupal 9 - although it will need fleshing out and/or a lot of the details like Symfony are in the sub-issues. While we've done as much ground work as we can in Drupal 8 for those updates, the actual updates themselves can only happen in 9.x, so that specific chunk of work has not started yet.

For #2 it's a similar situation. We are down to nearly zero deprecated code usages in core, but removing the deprecated code itself and the bc layers is 9.x-only work that also hasn't started yet.

With #1, if we haven't done our PHP dependency updates and kept up with them in 5 months, we're in serious trouble.

For #2 we could indeed move some deprecations to Drupal 10 as last resort, but we'd still need to make the decision to do that prior to tagging the beta.

On the fallback I don't think there's an issue for this yet. Our position has been that if we miss June 3rd, we slip to December. Drupal 7 and Drupal 8 LTS would be unaffected as long as we hit December because that still gives sites a year to update.

However I am increasingly unsure about the strict 6 months slippage. For example let's say we get to March, we have a couple of nasty upgrade path critical bugs left that really should block the release, these are fixed on May 16th, everything else was finished February. 8.9.x could still come up June 3rd, we could also start the beta/rc phase for 9.0.x on June 3rd and release in September or whenever, i.e. g to beta on the first window after blockers are fixed, rather than artificially holding the beta/rc phase until the autumn.

Gábor Hojtsy’s picture

Issue summary: View changes

Broke out removing update functions to its own item.

Gábor Hojtsy’s picture

Issue summary: View changes
catch’s picture

Issue summary: View changes
xjm’s picture

Issue summary: View changes

Adding the meta for deprecated code removal directly to the IS so contributors can more readily locate it. I also re-parented the issue.

webchick’s picture

I still would prefer to see us set a deadline (maybe mid-February?) for removing all the deprecated code, and whatever's not done, gets deprecated for Drupal 10 instead, versus holding up Drupal 9 on crossing all Ts and dotting all Is. The other items in this list seem a lot less negotiable, but not as sure about this one.

webchick’s picture

And/or, move to a "strong should have" and extend the deadline to whenever the "must-haves" are completed.

Berdir’s picture

We've already made good progress on that: https://twitter.com/DropIsMoving/status/1199734552060669954. And it's not even really the focus yet.

Most of these issues are trivial, we do have a somewhat of a blocker atm with the update functions, as we need to remove these first before we can remove some deprecated things. But people really enjoy cleaning that up (I do, and judging from the number of issues that are being opened, I'm not the only one).

Maybe we will run into some problematic cases, like REQUEST_TIME that still has a ton of usages in core, but I believe those will be exceptions.

I think the upgrade path issues are much more likely to be a problem, because they've historically been some of the most problematic issues (remember trying to get Drupal 7.0 out? ;)) and unlike the deprecation removal issues, they're not fun. not even a little bit :)

webchick’s picture

Agreed that we're progressing well so far: https://dev.acquia.com/drupal9/deprecation_status/graphs As long as we don't hold up D9 on more "nice to have" stuff like replacing REQUEST_TIME, I'm happy. ;) That's not how the issue summary currently reads, however.

catch’s picture

@webchick That paragraph was initially written before we had properly understood how much we were coming up against Symfony EOLs, and agreed it's less important than other things prompting Drupal 9's release now. When it was originally written we were only thinking about removing deprecations in 9.x and not a lot else so it was kind of tautological.

Having said that, 99.9% of deprecation removals are just removing lines of code from core (because we've made a tonne of progress the past couple of years removing deprecation usages). If we update a deprecation from Drupal 9 to 10, it will mean changing the deprecation message (and test coverage if there is any) and updating change records; this is likely to be more work than just removing the code.

So unless we did that work after the beta, it probably wouldn't help us release the beta faster and may even slow it down (for example a patch that was removing code but not RTBC/committed yet, now needs to be redone to update the deprecation message instead).

If there is something that hasn't properly had its usages removed when we're approaching the deadline then we may just need to move to Drupal 10 though, constants like REQUEST_TIME are the most likely to end up in that category. However that is somewhat covered by we can't remove deprecated code that is called by Drupal core.

xjm’s picture

Issue summary: View changes

Clarifying what "should-have" means for the 9.0 beta, based on discussions between myself and @catch last month.

Mile23’s picture

Per @catch: #2608062-109: [META] Requirements for tagging Drupal 9.0.0-alpha1

Also this doesn't affect the Drupal 9.0.x alpha - we can remove simpletest from Drupal 9 any time up until beta.

I'd really hate to see D9 with a functional simpletest module in core.

catch’s picture

Issue summary: View changes
xjm’s picture

Issue summary: View changes

Removing #3075954: Remove duplicate scaffold files based on discussion on that issue.

xjm’s picture

Issue summary: View changes
Gábor Hojtsy’s picture

Issue summary: View changes

Update the update path section from policy issue to the actionable issue.

Gábor Hojtsy’s picture

Issue summary: View changes

Adding actionable PHP 7.3 issue to the PHP requirement section.

Gábor Hojtsy’s picture

Issue summary: View changes

Created an issue for MySQL/MariaDB/Percona and linked in the right policy issue for PostgreSQL. Do we need one for SQLite?

daffie’s picture

Do we need one for SQLite?

I think thank would be a good idea. The current minimum version for SQLite is 3.6.8. Which was released on 12th january 2009. That is over 11 years ago.

daffie’s picture

lauriii’s picture

Issue summary: View changes

Added a should have to drop support for Node.js 8 which is EOL #3107918: Require Node.js 12.

xjm’s picture

Issue summary: View changes

Updated the remove-old-updates item in the IS to the meta.

catch’s picture

Issue summary: View changes
catch’s picture

Issue summary: View changes
Gábor Hojtsy’s picture

Issue summary: View changes

Adding a nice to have

#3111409: Add new Olivero frontend theme to Drupal 9.1 core as beta only if done enough in time, otherwise moves to Drupal 9.1.0 earliest

(As it currently stands very likely it would go to 9.1.0).

Gábor Hojtsy’s picture

Issue summary: View changes

Added link to alpha release that is out now.

Gábor Hojtsy’s picture

Issue summary: View changes
Gábor Hojtsy’s picture

dww’s picture

I tagged #3113992: The 'Update' page has no idea that some updates are incompatible as another beta1 requirement, but I'm not sure if/where to put it into this summary. According to the UX team, that's another release-blocking critical (since we've currently got a UI in core that makes it extremely easy to break your site in unrecoverable ways so long as #2917600: update_fix_compatibility() puts sites into unrecoverable state isn't resolved). So for now, just commenting here about it, but it should probably go into the summary, too. I just don't feel qualified to make decisions on should-have vs. must-have, etc...

Thanks/sorry,
-Derek

catch’s picture

Issue summary: View changes
catch’s picture

Issue summary: View changes
catch’s picture

Issue summary: View changes
Gábor Hojtsy’s picture

catch’s picture

Issue summary: View changes
catch’s picture

Issue summary: View changes
Gábor Hojtsy’s picture

Issue summary: View changes

Swapping out #2659890: [Policy] [Plan] Drupal 9 and 10 markup and CSS backwards compatibility with #3050374: Create Drupal 9 stable theme which is the only remaining beta requirement as per suggestion from @catch, verified with @lauriii.

catch’s picture

Issue summary: View changes

Move the stable theme from 'should' to 'must' have on the basis of discussions with @xjm and @lauriii. Basically this needs to be included so that themes relying on the (now deprecated) Drupal 8 version of stable can port to it.

catch’s picture

Issue summary: View changes
catch’s picture

Issue summary: View changes
Gábor Hojtsy’s picture

Issue summary: View changes

Accurate update recommendations are done and backported all the way to Drupal 8.8. One more should have down.

Gábor Hojtsy’s picture

Issue summary: View changes

CSS and markup policy is also done for beta.

Gábor Hojtsy’s picture

Issue summary: View changes

Multi-core compatibility support on drupal.org is also considered done for beta based on discussions with @xjm and @catch. Marking that should have like so.

Gábor Hojtsy’s picture

Issue summary: View changes

PHP policy is also done for beta. Also making done markers more visual.

catch’s picture

Issue summary: View changes
catch’s picture

Issue summary: View changes
effulgentsia’s picture

We should have a policy for MySQL, PostgreSQL and SQLite version support.

What does it mean that this is a should-have rather than a must-have? In other words, what happens if we release a beta1 without this completed? Are we then still able to set this policy after beta? Or are we then stuck with supporting whichever versions are still listed in core/INSTALL.txt at the time that beta1 is released?

catch’s picture

@effulgentsia discussed this with @xjm a bit and something like the following:

1. MySQL policy + implementation should block beta.

2. sqlite implementation could happen after beta, since it's mostly a dev dependency.

3. PostgreSQL policy + implementation should be beta-deadline - i.e. either we raise the requirement before then or we're stuck with what we've got.

For me personally I still think policy should block beta but implementation would not have to (and could be done between beta and rc), but obviously it's better the more lands the quicker.

I do not think we should be hitting or missing the beta deadline due to raising sqlite or postgresql requirements either way.

catch’s picture

Issue summary: View changes
Gábor Hojtsy’s picture

Issue summary: View changes

Renamed "DONE FOR BETA" to "ALL BETA SCOPE DONE" for clarity. @xjm pointed out "DONE FOR BETA" may sound like its all done, while it was meant to mean the scope for beta is done.

catch’s picture

Issue summary: View changes
Gábor Hojtsy’s picture

Issue summary: View changes

#3108416: Remove workspace_update_8803() landed.

Also updated the database requirements text to say "we should have them implemented" rather than "we should have a policy". Two of three of those already landed.

Gábor Hojtsy’s picture

Issue summary: View changes

Broke down the database issues to policies and implementation, then we an mark them each done as appropriate.

catch’s picture

Issue summary: View changes
Gábor Hojtsy’s picture

Issue summary: View changes

"Denormalized" the migrate multilingual meta as well for clarity / exposure.

catch’s picture

Issue summary: View changes
xjm’s picture

Issue summary: View changes

Adding the database environment blockers.

xjm’s picture

Issue summary: View changes
xjm’s picture

Issue summary: View changes
catch’s picture

Issue summary: View changes
Gábor Hojtsy’s picture

catch’s picture

Issue summary: View changes
Gábor Hojtsy’s picture

Issue summary: View changes
catch’s picture

Issue summary: View changes
xjm’s picture

Issue summary: View changes

Ditto #2917600: update_fix_compatibility() puts sites into unrecoverable state; all these issues are just open to discuss 8.8.x backport strategy.

catch’s picture

Issue summary: View changes

That's all the critical blocking upgrade path bugs committed, the closest we've been to a clean sheet for well over a year.

#2989745: views_update_8500() inlines configuration changes instead of this being done on config save for bc is still a should-have - but can only be committed to 8.9/8.8 and only affects configuration for one view, so not permanent data loss or fatal errors on update.php like the various issues we just cleared.

Gábor Hojtsy’s picture

Gábor Hojtsy’s picture

Gábor Hojtsy’s picture

Gábor Hojtsy’s picture

Issue summary: View changes

Fix formatting of upgrade path bugs

Martijn de Wit’s picture

Is #474684: Allow themes to declare dependencies on modules needed to be resolved before the Beta release to come with D 9.0, or wil it be postponed to a later version then?
It seems that only a frame work manager needs to review it.

Gábor Hojtsy’s picture

#3050374: Create Drupal 9 stable theme landed, so we are down to #2550717: [JS] Replace jQuery.cookie with JS-cookie and provide a BC layer being the last must have

@Martijn de Wit: I don't think that would be required for beta1. At least a frontend framework manager and a release manager was on the issue earlier and neither identified it as such.

catch’s picture

Martijn de Wit’s picture

Clear answers. Thank you both!

Gábor Hojtsy’s picture

Gábor Hojtsy’s picture

Issue summary: View changes

#3050374: Create Drupal 9 stable theme is done now. Marking as such.

Also #2966856: Hide and disable Drupal Migrate Multilingual landed.

These complete two points in the list.

The only remaining must have beta blocker is #2550717: [JS] Replace jQuery.cookie with JS-cookie and provide a BC layer.

catch’s picture

AND THEN THERE WERE NONE!

We should be on track to release the beta next week.

Thanks to everyone who helped with this!

Gábor Hojtsy’s picture

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.