Problem/Motivation

In #2942591: Start reporting specific releases as insecure in update status XML and #2804155: If the next minor version of core has a security release, status still says "Security update required!" even if the site is on an equivalent, secure release already, Drupal.org and core are being changed to allow the Drupal security team to automatically mark old releases as insecure, but to manually mark an older release as secure. This is useful when (for example) the same security advisory is fixed in two different minor branches, which we already have to do about 30% of the time for Drupal 8 core and which we want to do for every release when we adopt #2909665: [plan] Extend security support to cover the previous minor version of Drupal.

Following the Drupal.org change that has already been made, the Drupal status report only links the latest security release. So, for example, in this scenario:

  • SA-CORE-2018-002 creates the following core security releases, all of which are considered equally secure: 8.3.9, 8.4.6, and 8.5.1.
  • The site is on 8.4.5.

The status report will link to 8.5.1, but the site owner also actually has the option to update to 8.4.6.

Proposed resolution

If the site is on an insecure version of an old minor and there is a secure version of that old minor available, link the latest secure release of the old minor branch on the update status report.

Continue to also provide a link to the latest version.

Testing results:

For 10.0 the displayed options are:
Recommended: 10.2.4 (The latest version)
Security Update: 10.2.2 (The latest security version)
Security Update: 10.1.8 (The latest security version for the earlier supported version of core)

For Drupal core 10.1.2 the displayed options are:
Recommended: 10.2.4 (The latest version)
Security Update: 10.2.2 (The latest security version)
Security Update: 10.1.8 (The latest security version for this version of core)

Remaining tasks

Review

User interface changes

Before

After

API changes

N/A

Data model changes

N/A

Issue fork drupal-2990476

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

xjm created an issue. See original summary.

xjm’s picture

Issue summary: View changes
StatusFileSize
new186.5 KB
xjm’s picture

Issue tags: +Usability
gábor hojtsy’s picture

Issue summary: View changes

@xjm asked me to review. I think displaying both versions is fine, it largely depends on the design. The less disruptive update (with same minor) should be displayed with higher prominence, since that is sufficient to resolve the same security issues (IF it is sufficient) and is much lower risk.

We can also display the later minor security update, but that would be in my mind informative if they are working on a site update to that anyway, so they are aware that the bugs are fixed there as well and not just this minor.

The info about the next minor's security fix becomes important when support lengths are taken into account and support is going to be over soon for this minor. That is not in scope here though.

dyannenova’s picture

I think between Security Update linking to the latest secure release of the old minor branch and displaying the Latest Version underneath, that covers the important use cases without adding in a third link.

In that case, the simplest solution is to switch the top link to the latest secure release of the old minor branch and leave everything else exactly as it is now.

xjm’s picture

Thanks @Gábor Hojtsy and @DyanneNova

The problem is that the update status report does not display a link to the security update for the old minor at all after #2804155: If the next minor version of core has a security release, status still says "Security update required!" even if the site is on an equivalent, secure release already. It only shows the latest security release. In the screenshot in the summary, 8.4.6 should be recommended as an option for the site owner for their security update, but it is not.

I'm not sure what the third link referred to is.

So what we need is a design that lets the site owner know that 8.4.6 is a security update that they can apply immediately.

tedbow’s picture

Issue summary: View changes
StatusFileSize
new123.53 KB

Here is what the updates page would list now

This a screenshot from the test \Drupal\Tests\update\Functional\UpdateCoreTest::testSecurityUpdateAvailability()
In this case the site is on 8.0.0

It shows both the security release of the sites installed minor and the next minor. But it shows the next minor first. Actually before it shows 8.2.0-rc2 which in the test case is a security release.

So should we reverse the orders of th security releases to start with the current minor and the go up?

xjm’s picture

Status: Postponed » Active

Ah, thanks @tedbow. So at this point the site owner has an actual choice, but the user sees five choices instead of two. What we need to do is communicate to the user with design/text/etc. what the consequences of the different choices are.

#2992631: Update report incorrectly recommends security releases for old minors when a security update is needed and a secure version of the old minor is also available will need to be fixed too, but #7 is not affected by that since the site is on 8.0.0 when there is a theoretical SA issued as 8.0.2, 8.1.2, and 8.2.0-rc2.

This can be active now that the blocker is in!

xjm’s picture

Hm, it doesn't help that both 8.2.0-rc2 and 8.1.2 are listed twice in the example.

xjm’s picture

So in the example, the site owner has three good options if they are on a previous minor:

  1. Update to the security update for their minor version. This is 8.0.2 in the example, bur updating to 8.0.2 is their best bet for a quick security fix.
  2. Update to the "Recommended release" which is 8.1.2 -- but if they do, we need to inform them about minor coverage and updates
  3. Update to the latest version (a dev branch)
tedbow’s picture

Should we make a follow up or new issue to deal with the fact that releases are listed 2x.

I think what happens is the recommended, the latest, and all security releases are listed. And if any of these are the exact same releases there is not check for that.

So you will often get releases that fall into 2 of these categories. Of course if we listed them once we need to have wording to convey that a release is both the recommended and a security release or recommended and latest etc. Or maybe all three in 1 release

xjm’s picture

Re. #11 We might want to split it out, but let's see what designs we end up with first and make scoping decisions based on them.

benjifisher’s picture

At the UX meeting last week, we discussed this issue. We agreed that there is not enough room on the "available updates" report to explain fully the advantages of updating to one version or another. I suggested adding a documentation page with a full discussion and a few short phrases describing the options. Then the "available updates" report can link to that page and use the same short phrases.

I just added a documentation page: https://www.drupal.org/docs/8/update/which-update-should-i-apply. This page is published, and it is part of the "Upgrading Drupal 8" guide, but it has not been added to the menu of that guide.

The "short phrases" I chose are

  • least disruptive
  • latest and greatest
  • bleeding edge

I guess that "latest and greatest" is equivalent to the version labeled "recommended" in the current "available updates" report. I do not like the term "recommended": in some cases, I might recommend the "least disruptive" update.

As the comments above show, we currently list pre-release versions. I added the "bleeding edge" option specifically so that I could say that it is not recommended for a production site.

dyannenova’s picture

Status: Active » Postponed
StatusFileSize
new32.9 KB
new45.03 KB

Here are mockups incorporating the security update. Does everyone feel that we need to show the bleeding edge release here? My impression has been that users who are interested in and able to run the bleeding edge release are not discovering and downloading it through the UI.

Security Update and Recommended Release Mockup

Security Update, Recommended Release, and Development Release Mockup

dyannenova’s picture

Status: Postponed » Needs review

Fixing the status

benjifisher’s picture

Copying here since Slack history will not last long: the documentation page I added and mentioned in #13 above duplicates content on some other pages. Look at these: most need updating, and we can probably use one of them as a link in the updated "available updates" report:

Quoting @xjm on that last one: "which has the actual current definition of what alpha/beta/RCs are, as well as the technical specifics"

xjm’s picture

Issue summary: View changes
StatusFileSize
new128 KB

Thanks @DyanneNova and @benjifisher! I think we're on a good track here.

I think it's out of scope to remove the latest a.k.a. "bleeding edge" release from the page, but simply adding action buttons for the other two options while leaving that latest version as text only should accomplish what we need. That will also mimic the download pages, where the supported versions have the action button but the dev version is just text:

So I think there are a few issues here we can split out:

  1. Making "Download recommended release" a button rather than a link is something we can create/review/test separately from this issue. We should review what that change by itself looks like on various different combinations of both core and contrib updates available.
  2. To not repeat releases. For example, something like:
    • if the latest version is a security update and is the only available security update, it should be presented as the security update, and not repeated as the latest version. The same probably for the latest version.
    • Conversely, pre-release versions probably should not be presented as security updates unless you are on a pre-release version before them.
  3. Once that is done, then we could look into presenting the special buttons for "security update required!". The reason to do this last is that the recommended release will of necessity be a secure release, so we aren't regressing anything by fixing those others first. If we adjust the security buttons first without fixing the other issues then we'll make a currently overwhelming page even more overwhelming.

Other notes:

  • I think the text for the recommended release, when it's different from another security update displayed, should include something like,

    The recommended release also includes the latest security fixes and is ready for use on production sites.

    It's safe to say that because an insecure release will not ever be shown as a recommended release. (Without something like this, the site owner might think they need to download the security update first.)

  • The recommended release might not always include a minor update even when it's different from the security update -- for example, the site might be on 8.6.2 which might a bugfix release, 8.6.3 might be a security update, and 8.6.4 might be the latest release.
  • We'll need to think more about whether we include special text when the site's minor version is lower (and what that text is).
  • I'll think more about the handbook docs too.

    Thanks again!

dyannenova’s picture

Status: Needs review » Needs work

I've created the follow up issue for the buttons here: #3000742: Make "Download recommended release" a button rather than a link on update status report

It sounds like we need work on this issue to not repeat releases, so I'm updating the status here.

webchick’s picture

Priority: Major » Critical

While we made the Extended Security Support policy change last year, we still haven’t completed this issue, which is critical to to ensure users know about security coverage for their minor and make the right choice in response to security releases. After talking to @xjm, escalating to critical.

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

dww’s picture

This at least seems related to #3100115: The update module should recommend updates based on supported_branches rather than major versions majors. Don't yet fully understand the scope of each issue, but they seem to be talking about very similar things...

catch’s picture

Just to say I think the proposed solution here is good - show the latest version in the current branch and also the next branch(es).

Potentially a site could be on 8.8.7, with 8.8.8, 8.9.1 and 9.0.3 all available at the same time.

Like dww, I also think #3100115: The update module should recommend updates based on supported_branches rather than major versions majors is probably a duplicate.

catch’s picture

tedbow’s picture

@catch I think we can fold #3100115: The update module should recommend updates based on supported_branches rather than major versions majors into this issue but the current proposed

If the site is on an insecure version of an old minor and there is a secure version of that old minor available, link the latest secure release of the old minor branch on the update status report.

Continue to also provide a link to the latest version.

Only covers insecure installed version and security updates. If we want handle the problem in #3100115 we should always show the latest release for given minor and also the latest release for the major. This would be regardless of whether we were dealing security releases or not.

Also when this issue was created the problem only affected core but since then drupal.org has updated support semantic versioning release for contrib projects. So we would have this same problem in contrib.
So if we want to handle all of the issues in this 1 we will need to have contrib act the same way. Always show the latest release for given minor and also the latest release for the major. This only applies to contrib that have semver releases.

catch’s picture

If we want handle the problem in #3100115 we should always show the latest release for given minor and also the latest release for the major.

I think it probably makes sense to show the latest release on the current branch and also the latest releases on any later branches.

This would help people to see exactly how far behind they are on updates, which might give some people a chance to avoid running into #3098475: Add more strict checking of hook_update_last_removed() and better explanation.

dww’s picture

FYI: Marked #3127247: Add ability to update to "Also available" module versions related to this since it's asking to change the Update Manager GUI to provide options to update to an 'Also available' releases, not just 'Recommended'. So if we're expanding the choices for also available (based on #26 from @catch), we might want to consider #3127247 in our plans.

xjm’s picture

Wrote some thoughts in #3111767-5: API to provide a recommended next version for project that are relevant to this issue as well.

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

xjm credited shaal.

xjm’s picture

Issue summary: View changes
StatusFileSize
new69.21 KB

@shaal pointed out today that our designs should also cover the report when no security update is required, since it can be confusing when you update to (e.g.) 8.8.8 and after that it's still telling you about 8.9.1 and 9.0.1 on a yellow row. (#14 also should cover that, but I dunno that we discussed it explicitly before.)

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

pasqualle’s picture

Version: 9.2.x-dev » 9.1.x-dev
xjm’s picture

Version: 9.1.x-dev » 9.2.x-dev
Issue tags: -beta target

This was only a beta target for 9.0.x. 9.2.x is the correct branch.

pasqualle’s picture

As I see this is a security improvement. Why it can't be backported to 9.1.x?

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

tedbow’s picture

Pretty sure this Automatic Updates because so far we are relying on update_calculate_project_update_status() to determine which module we will update to.

catch’s picture

Category: Task » Bug report

Had a report from @adamps that the 9.2.8 release wasn't marked as insecure when 9.2.9 came out, this is likely because we already have 9.3.0-beta1 etc. and it's hitting this bug.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

xjm’s picture

Re: #38, I thought it had just been a case of one of the "insecure" checkboxes on the release nodes getting ticked incorrectly for that particular release? It's very easy for it to happen by mistake (fortunately also easy to fix).

stephencamilo’s picture

Status: Needs work » Closed (won't fix)
xem8vfdh’s picture

poker10’s picture

Status: Closed (won't fix) » Needs work

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

quietone’s picture

Issue summary: View changes
StatusFileSize
new79.79 KB
new79.93 KB
new14.46 KB

I tried to make some progress here. I tested the patch on 10.0.4 and used those results for the before and after screenshots. It is probably best just to look at those too see the changes.

quietone’s picture

Status: Needs work » Needs review
quietone’s picture

StatusFileSize
new4.55 KB
new19.02 KB

Fix errors caught by PHPStan

quietone’s picture

StatusFileSize
new563 bytes
new19.21 KB

Sigh, and again.

Status: Needs review » Needs work

The last submitted patch, 50: 2990476-51.patch, failed testing. View results

quietone’s picture

Status: Needs work » Needs review
StatusFileSize
new18.68 KB

The failure is about themes which I have not worked on yet. This restores drupalci so all tests runs. No interdiff.

Status: Needs review » Needs work

The last submitted patch, 52: 2990476-52.patch, failed testing. View results

joachim’s picture

This looks like a good change -- list the immediate and simple patch version update first, before the more offputting minor version update.

I notice from the screenshots we've no longer got one of them in bold.

Is there a spec somewhere for what a bold means on the update status report?

quietone’s picture

Issue summary: View changes
Status: Needs work » Needs review
StatusFileSize
new3.98 KB
new19.14 KB
new65.79 KB

A few more fixes:

  • The failing test is fixed by finding the element with new label, "Recommended security update".
  • I don't know of any spec for the use of bold. It looks like it is simply used to highlight the recommended version.
  • This also restore some accidental formatting changes
  • Change watchdog_exception to Error::logException(
  • New after screenshot
smustgrave’s picture

Status: Needs review » Needs work

Wondering if this could be answered in the Issue summary

Decide whether to continue linking the latest security release if it is not the latest release overall and the user is on a different minor branch.

If there is a 10.0.9 would that be the recommended security update version?

quietone’s picture

Issue summary: View changes
Status: Needs work » Needs review

If there is a 10.0.9 would that be the recommended security update version?

Yes. The screenshot is for a site on Drupal 10.0.4 and there are two security releases after that. It is the latest one that is displayed.

Still to do #56.2. I am setting back to NR because the patch doesn't need work right now. We need to make sure this change fits with the overall plan for what is show on the page.

smustgrave’s picture

@quietone who's needed to make that call?

needs-review-queue-bot’s picture

Status: Needs review » Needs work
StatusFileSize
new85 bytes

The Needs Review Queue Bot tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".

This does not mean that the patch needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

quietone’s picture

Status: Needs work » Needs review
Issue tags: +no-needs-review-bot
StatusFileSize
new23.65 KB
new21.66 KB

Convert to MR and hide patches.
And here are some more after images

quietone’s picture

Issue summary: View changes
smustgrave’s picture

Same question as before who can make the call. This just sat there before.

smustgrave’s picture

Lets see if a submaintainer can chime in.

quietone’s picture

Issue summary: View changes
Status: Needs review » Needs work

This screenshot shows that this is still not quite right. For 10.0, the latest security release should be displayed and it is not.

quietone’s picture

Issue summary: View changes
Status: Needs work » Needs review
StatusFileSize
new21.99 KB
new21.77 KB

Testing with a Drupal version of 10.1.0 shows the changes made here.

smustgrave’s picture

Status: Needs review » Postponed

Postponing on #3100110: Convert update_calculate_project_update_status() into a class

This should be done before #2990476: If the site is on an insecure version of an old minor and there is a secure version of that old minor available, the update status report should link that release to avoid making update_calculate_project_update_status() even more complicated

quietone’s picture

Status: Postponed » Needs review

That issue will take much longer to resolve. I think it is more important to fix this Critical bug than wait for an architectural change.

smustgrave’s picture

Then can the other issue summary be updated

dww’s picture

Issue tags: +Bug Smash Initiative

As usual, this fell off my radar. Tagging to be smashed. I can't do a full subsys maintainer review anytime soon, but trying to remind myself to get back to this ASAP.

dww’s picture

Status: Needs review » Needs work

@smustgrave (bless his soul!) traded me a review here for his reviews of #2640994: Fix label token replacement for views entity reference arguments (which just landed for 10.3.0 🎉). Finally circling back to uphold my end of the bargain. 😂

In principle, +1 to the general approach of marking the recommended update as "Recommended security update" if it's a security update from the same branch as you're currently running. We want folks to get off known-insecure versions ASAP, and a security release from the same branch is the least disruptive thing to do, so that's what we should "Recommend".

However, I left a detailed review on the MR and opened 9 threads, some of them fairly easy suggestions to review, but a couple of important ones.

I've only looked at the code, haven't tried any manual testing.

One of my key review points is that I don't think the test changes are testing anything about the new functionality, so we need to fix that before we can have confidence from the GitLab pipeline. Probably would be good to do some manual testing, too.

Thanks for keeping this alive and moving forward, and apologies for the long lapses in my availability to review this...
-Derek

p.s. Crediting @smustgrave for all the efforts here, both reviewing and tracking subsystem maintainers, and myself for the detailed review.

dww’s picture

So long as the threads are addressed and the tests are actually testing this, I'm happy. So signing off as subsystem maintainer.

quietone’s picture

Issue summary: View changes
Status: Needs work » Needs review
StatusFileSize
new50.84 KB
new42.89 KB

@dww and @snustgrave, thanks for working together so this issue got a review!

I have updated the MR and left comments in the MR.

quietone’s picture

Issue summary: View changes
larowlan’s picture

Status: Needs review » Needs work

Looks like this needs a reroll

quietone’s picture

Rerolled and there is a valid test failure. I don't see why yet and debugging these update tests is working as I would like.

Version: 11.x-dev » main

Drupal core is now using the main branch as the primary development branch. New developments and disruptive changes should now be targeted to the main branch.

Read more in the announcement.