[regression] Symfony dependencies downgraded in 10.5.7 and 11.2.9

git show -m -p 5c892149b638207f9e48cfb322aa35bc8efd18d2 shows the merge commit accidentally reverting the changes to composer.lock and the metapackages.

I think the fix is to run composer update symfony/* drupal/core on 10.5.x.

This may have been due to an error during the tagging process that was corrected for the other branches but possibly not for 10.5. We should double-check to ensure that 10.4 doesn't have the same issue.

Pipeline failure was:

    1) Drupal\Tests\system\Functional\System\CronRunTest::testCronUI
    Cron does not run when saving the configuration form.
    Failed asserting that 1764887928 matches expected 1764887733.

Sounds random, so I requeued it.

I just checked and while 10.4.x has not had a release since 10.4.9, it looks like it has the same lockfile merge issues that 10.5.x does. (Installing 10.4.x HEAD after 10.4.9 potentially downgrades Symfony HTTP-Foundation from 6.4.29 to 6.4.15.) This means that the next time a 10.4.x security release is changed, it will revert the Symfony update. So we will need an MR for 10.4.x as well.

I think this is due to an issue with the security release tagging script naively assuming that security updates do not change the lockfile beyond the version constant. So while we resolved the new issue that came up with the new attempted process tags themselves during the security window, we did not resolve the second, similar, existing issue with the "Back to dev" commit.

In the longer term this might require a change to the tagging script in order to create proper "Back to dev" commits, or at least a manual workaround by the release manager to amend the commit prior to pushing branch endpoints after the window.

Critical was an overreaction on my part because I was annoyed with myself; downgrading to major. While this is a serious bug from a release process standpoint, it is not a security issue. The site owner can still update the Symfony dependency easily -- it is only a patch-level update -- and Drupal itself isn't even vulnerable to the issue.

@longwave correctly pointed out that 10.4.x is end-of-life in about 6 days, so we probably don't actually need to make a 10.4.x merge request because we're unlikely to create another 10.4 release unless some highly critical core security issue comes up that we would backport outside normal policy. Hopefully that doesn't happen. 😅 If that happens, or if we backport some other upgrade path fix to 10.4.x for some reason, then we can fix this, but otherwise 10.5.x itself is enough for now. Thanks!

10.6.x should be fine because it did not have a merge commit, just a forward-port, and also did not even have an alpha yet when the advisory was released. Edit: Confirmed it installs HTTP-Foundation 6.4.29.

However, I just checked and 11.2.9 has the same issue. It installs with 7.3.0, while 11.2.8 installs with 7.3.7. We definitely need an MR for 11.2.

Draft release notes for both !0 and 11 are in the same doc.

Fix for 11.2, there might be a more efficient way but this works for me to temporarily pin Composer deps to ~7.3.0 and update before reverting:

git checkout -b 3561574-bump-versions-11.2.x origin/11.2.x
sed -i 's/\^7.3/~7.3.0/g' composer.json core/composer.json
composer update symfony/* drupal/core
git checkout -- composer.json core/composer.json
composer update drupal/core

This results in:

+----------------------------------+---------+---------+
| Production Changes               | From    | To      |
+----------------------------------+---------+---------+
| symfony/console                  | v7.3.0  | v7.3.8  |
| symfony/dependency-injection     | v7.3.0  | v7.3.8  |
| symfony/error-handler            | v7.3.0  | v7.3.6  |
| symfony/event-dispatcher         | v7.3.0  | v7.3.3  |
| symfony/filesystem               | v7.3.0  | v7.3.8  |
| symfony/finder                   | v7.3.0  | v7.3.5  |
| symfony/http-foundation          | v7.3.0  | v7.3.8  |
| symfony/http-kernel              | v7.3.0  | v7.3.8  |
| symfony/mailer                   | v7.3.0  | v7.3.5  |
| symfony/mime                     | v7.3.0  | v7.3.8  |
| symfony/polyfill-ctype           | v1.32.0 | v1.33.0 |
| symfony/polyfill-iconv           | v1.32.0 | v1.33.0 |
| symfony/polyfill-intl-grapheme   | v1.32.0 | v1.33.0 |
| symfony/polyfill-intl-idn        | v1.32.0 | v1.33.0 |
| symfony/polyfill-intl-normalizer | v1.32.0 | v1.33.0 |
| symfony/polyfill-mbstring        | v1.32.0 | v1.33.0 |
| symfony/polyfill-php84           | v1.32.0 | v1.33.0 |
| symfony/process                  | v7.3.0  | v7.3.4  |
| symfony/psr-http-message-bridge  | v7.3.0  | v7.3.8  |
| symfony/routing                  | v7.3.0  | v7.3.8  |
| symfony/serializer               | v7.3.0  | v7.3.8  |
| symfony/service-contracts        | v3.6.0  | v3.6.1  |
| symfony/string                   | v7.3.0  | v7.4.0  |
| symfony/translation-contracts    | v3.6.0  | v3.6.1  |
| symfony/validator                | v7.3.0  | v7.3.8  |
| symfony/var-dumper               | v7.3.0  | v7.3.5  |
| symfony/var-exporter             | v7.3.0  | v7.4.0  |
| symfony/yaml                     | v7.3.0  | v7.3.8  |
+----------------------------------+---------+---------+

+----------------------+--------+--------+
| Dev Changes          | From   | To     |
+----------------------+--------+--------+
| symfony/browser-kit  | v7.3.0 | v7.3.6 |
| symfony/css-selector | v7.3.0 | v7.3.6 |
| symfony/dom-crawler  | v7.3.0 | v7.3.3 |
| symfony/lock         | v7.3.0 | v7.3.4 |
+----------------------+--------+--------+

Both the 11.2.x and 10.5.x MRs look good.

One thing that's interesting is that these are lockfile-only issues. Did we not increase constraints for the Symfony updates last month?

Sorry, I think this is actually NW. There should be a corresponding increase in the constraint of http-foundation in core-recommended to the secure version. 10.6.x has this, and these branches also did prior to the merge commit eating it. See #3557393: Update Symfony to 7.3.7 / 6.4.29.

I addressed 10.4.x by cherry-picking the original fix from #3557393: Update Symfony to 7.3.7 / 6.4.29 and pushing it again. That worked since there were no commits on 10.4.x since the release and none of the other changes in the security release touched the metapackages or lockfile.

It's interesting that no tests are failing despite the dependencies in composer.lock not matching what's in the metapackages in the above MRs.

Probably we should just try to follow the documented steps for updating core Composer dependencies for at least HttpFoundation.

Sorry I have no idea what diff I was looking at that had only lockfile changes and nothing else. Please disregard above several comments (aside from the 10.4.x cherry-pick which is fine).

Reassuringly, the branch I created by cherry-picking the original commit to 10.5.x differs from the new MR that was created above by only:

diff --git a/composer.lock b/composer.lock
index d8cf1dc814e..339476ca045 100644
--- a/composer.lock
+++ b/composer.lock
@@ -10211,5 +10211,5 @@
     "platform-overrides": {
         "php": "8.1.0"
     },
-    "plugin-api-version": "2.6.0"
+    "plugin-api-version": "2.9.0"
 }

I made a similar cherry-pick on 11.2 for comparison. That diff includes different versions of numerous Symfony packages than what we had before. To some extent, this is to be expected; however, I did notice that we're upgrading the minor version from 7.3.0 to 7.4.0 of symfony-string and symfony/var-exporter. That shouldn't be happening in a patch release. In the previous issue, they were updated to 7.3.4.

I went ahead and committed the 10.5.x MR despite the normal backport policy. In some ways, the 10.5.x hotfix release is more urgent, since 10.5.x sites are more in need of security-only dependency management and less likely to want to simply update Symfony to the latest release for the sake of updating it.

NW for the 11.2.x fix to remove the minor updates of those two components which must be slipping by indirect constraints or something. (The patch-level updates are fine.)

Ah, those are transient dependencies so temporarily pinning composer.json doesn't work because they aren't there. Fixed with:

sed -i 's/\^7.3/~7.3.0/g' composer.json core/composer.json
composer update symfony/* drupal/core --with=symfony/string:~7.3.0 --with=symfony/var-exporter:~7.3.0
git checkout -- composer.json core/composer.json
composer update drupal/core

Now only 7.3 dependencies:

$ composer-lock-diff --from=origin/11.2.x --no-links
+----------------------------------+---------+---------+
| Production Changes               | From    | To      |
+----------------------------------+---------+---------+
| symfony/console                  | v7.3.0  | v7.3.8  |
| symfony/dependency-injection     | v7.3.0  | v7.3.8  |
| symfony/error-handler            | v7.3.0  | v7.3.6  |
| symfony/event-dispatcher         | v7.3.0  | v7.3.3  |
| symfony/filesystem               | v7.3.0  | v7.3.8  |
| symfony/finder                   | v7.3.0  | v7.3.5  |
| symfony/http-foundation          | v7.3.0  | v7.3.8  |
| symfony/http-kernel              | v7.3.0  | v7.3.8  |
| symfony/mailer                   | v7.3.0  | v7.3.5  |
| symfony/mime                     | v7.3.0  | v7.3.8  |
| symfony/polyfill-ctype           | v1.32.0 | v1.33.0 |
| symfony/polyfill-iconv           | v1.32.0 | v1.33.0 |
| symfony/polyfill-intl-grapheme   | v1.32.0 | v1.33.0 |
| symfony/polyfill-intl-idn        | v1.32.0 | v1.33.0 |
| symfony/polyfill-intl-normalizer | v1.32.0 | v1.33.0 |
| symfony/polyfill-mbstring        | v1.32.0 | v1.33.0 |
| symfony/polyfill-php84           | v1.32.0 | v1.33.0 |
| symfony/process                  | v7.3.0  | v7.3.4  |
| symfony/psr-http-message-bridge  | v7.3.0  | v7.3.8  |
| symfony/routing                  | v7.3.0  | v7.3.8  |
| symfony/serializer               | v7.3.0  | v7.3.8  |
| symfony/service-contracts        | v3.6.0  | v3.6.1  |
| symfony/string                   | v7.3.0  | v7.3.8  |
| symfony/translation-contracts    | v3.6.0  | v3.6.1  |
| symfony/validator                | v7.3.0  | v7.3.8  |
| symfony/var-dumper               | v7.3.0  | v7.3.5  |
| symfony/var-exporter             | v7.3.0  | v7.3.4  |
| symfony/yaml                     | v7.3.0  | v7.3.8  |
+----------------------------------+---------+---------+

+----------------------+--------+--------+
| Dev Changes          | From   | To     |
+----------------------+--------+--------+
| symfony/browser-kit  | v7.3.0 | v7.3.6 |
| symfony/css-selector | v7.3.0 | v7.3.6 |
| symfony/dom-crawler  | v7.3.0 | v7.3.3 |
| symfony/lock         | v7.3.0 | v7.3.4 |
+----------------------+--------+--------+

Compared this again to the original commit from the original issue. Confirmed the only differences between cherry-picking the commit and the MR are in the lockfile hash and core-recommended:

[ayrton:drupal | Tue 19:46:18] $ git diff 3561574-bump-versions-11.2.x --stat
 composer.lock                                      | 150 ++++++++++-----------
 composer/Metapackage/CoreRecommended/composer.json |  24 ++--
 2 files changed, 85 insertions(+), 89 deletions(-)

I then reviewed those specific changes with:

[ayrton:drupal | Tue 19:46:37] $ git diff 3561574-bump-versions-11.2.x --color-words="."

I verified that the only differences are in lockfile hashes and timestamps, patch version increases to individual Symfony packages, and in one case an addition to the release managers listed as supporting a package in the lockfile. So those are all appropriate changes in a patch release.

Finally, I ran a composer validate --check-lock just to make sure the hashes etc. were correct, and they are, so RTBCing. (I had an irrelevant diff upon generating the MR itself from scratch or doing a composer update --lock on top of the MR, but it went away when I updated Composer itself to the most recent version.)

Good find on the transient minor update, completely missed that when I read through the first time.

Committed/pushed the 11.2.x MR to 11.2x, thanks! Since the 10.5.x MR is already in, closing this one out.