Problem/Motivation

Drupal 8 is not compatible with PHP 7.1 because the version of Symfony we are using is not compatible. This is because there is a bug in the Yaml dump when discovering whether or not an array is a hash or not. This is solved in https://github.com/symfony/symfony/pull/18861.

This is a critical issue because 7.1 is a stable version of PHP an we claim to support PHP 7.

Proposed resolution

Update Symfony components by doing:

composer update Symfony/*

This outputs:

Loading composer repositories with package information
Updating dependencies (including require-dev)
  - Removing symfony/polyfill-apcu (v1.1.1)
  - Installing symfony/polyfill-apcu (v1.3.0)
    Downloading: 100%

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Removing symfony/class-loader (v2.8.4)
  - Installing symfony/class-loader (v2.8.15)
    Downloading: 100%

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Removing symfony/debug (v2.7.6)
  - Installing symfony/debug (v2.8.15)
    Downloading: 100%

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Removing symfony/polyfill-mbstring (v1.1.0)
  - Installing symfony/polyfill-mbstring (v1.3.0)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Removing symfony/console (v2.8.4)
  - Installing symfony/console (v2.8.15)
    Downloading: 100%

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Removing symfony/dependency-injection (v2.8.4)
  - Installing symfony/dependency-injection (v2.8.15)
    Downloading: 100%

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Removing symfony/polyfill-php55 (v1.1.0)
  - Installing symfony/polyfill-php55 (v1.3.0)
    Downloading: 100%

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Removing symfony/polyfill-php54 (v1.1.0)
  - Installing symfony/polyfill-php54 (v1.3.0)
    Downloading: 100%

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Removing symfony/http-foundation (v2.8.4)
  - Installing symfony/http-foundation (v2.8.15)
    Downloading: 100%

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Removing symfony/event-dispatcher (v2.8.4)
  - Installing symfony/event-dispatcher (v2.8.15)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Removing symfony/http-kernel (v2.8.4)
  - Installing symfony/http-kernel (v2.8.15)
    Downloading: 100%

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Removing symfony/routing (v2.8.4)
  - Installing symfony/routing (v2.8.15)
    Downloading: 100%

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Removing symfony/serializer (v2.8.4)
  - Installing symfony/serializer (v2.8.15)
    Downloading: 100%

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Removing symfony/translation (v2.8.4)
  - Installing symfony/translation (v2.8.15)
    Downloading: 100%

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Removing symfony/validator (v2.8.4)
  - Installing symfony/validator (v2.8.15)
    Downloading: 100%

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Removing symfony/process (v2.8.4)
  - Installing symfony/process (v2.8.15)
    Downloading: 100%

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Removing symfony/polyfill-iconv (v1.1.1)
  - Installing symfony/polyfill-iconv (v1.3.0)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Removing symfony/yaml (v2.8.4)
  - Installing symfony/yaml (v2.8.15)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Removing symfony/css-selector (v2.8.4)
  - Installing symfony/css-selector (v2.8.15)
    Downloading: 100%

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Removing symfony/dom-crawler (v2.8.13)
  - Installing symfony/dom-crawler (v2.8.15)
    Downloading: 100%

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Removing symfony/browser-kit (v2.7.6)
  - Installing symfony/browser-kit (v3.2.1)
    Downloading: 100%

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
Writing lock file
Generating autoload files
> Drupal\Core\Composer\Composer::preAutoloadDump
> Drupal\Core\Composer\Composer::ensureHtaccess

Remaining tasks

We have to 8.3.x first and the 8.2.x. Maybe for 8.2.x we might want to force the major and minor version upgrades not to occur - but personally I think we should just accept them since they are not our API.

User interface changes

None

API changes

None

Data model changes

None

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

alexpott created an issue. See original summary.

alexpott’s picture

alexpott’s picture

The PHP 7.0 failures appear to be due to new core fails introduced by DrupalCI changes :(

catch’s picture

Status: Needs review » Reviewed & tested by the community

Given those are also on the branch tests, moving this to RTBC.

We can discuss 8.2.x backport once it's in. In theory it's a good idea but there are a couple of indirect library changes that aren't patch level changes (polyfill and browser).

alexpott’s picture

Issue tags: +php7.1
alexpott’s picture

Issue tags: -php7.1 +PHP 7.1
alexpott’s picture

Looking at the minor version bumps...

https://github.com/symfony/polyfill/blob/master/CHANGELOG.md and doing a git diff v1.1.1 v1.3.0 there are:

  • There is a generic change to the polyfills that affects Iconv and MbString which is feature #51 Use plain PHP for data maps to benefit from OPcache on PHP 5.6+ (nicolas-grekas) - which whilst is says this is a feature I really don't think this API and could result in some performance improvements for Drupal8 - yay.
  • APCu: no changes
  • Mbstring: added polyfills for mb_chr(), mb_ord() and mb_scrub(), silenced iconv_strlen() in mb_strlen() polyfill, bypassed iconv for some charsets in mb_strlen, fixed mb_convert_variables() poylfill
  • PHP55: no changes
  • PHP54: bug #49 Fix hex2bin return null (fuhry, binwiederhier) - but we don't support 5.4 so not even sure why this is here :)
  • Iconv: no significant changes - there was: [Iconv] Fix wrong use in bootstrap.php
alexpott’s picture

The only reason we have browserkit is as a dependency of behat/mink-browserkit-driver and fabpot/goutte - both of these support the major versions 2 and 3 and are only used by the testing framework - so there is no real change on stock Drupal. However given is a major version upgrade we probably should not do this on the 8.2.x. So maybe we want to try to limit that to only moving to 2.8.15 in this patch and let the upgrade to Symfony 3 happen in the massive issue to do that?

alexpott’s picture

That said the only change in the changelog for browserkit is:

--- a/src/Symfony/Component/BrowserKit/CHANGELOG.md
+++ b/src/Symfony/Component/BrowserKit/CHANGELOG.md
@@ -1,6 +1,11 @@
 CHANGELOG
 =========

+3.2.0
+-----
+
+ * Client HTTP user agent has been changed to 'Symfony BrowserKit'
+
 2.3.0
 -----

Doing git diff v2.7.6 v3.2.1 src/Symfony/Component/BrowserKit

alexpott’s picture

BrowserKit also has:

     "require": {
-        "php": ">=5.3.9",
-        "symfony/dom-crawler": "~2.0,>=2.0.5"
+        "php": ">=5.5.9",
+        "symfony/dom-crawler": "~2.8|~3.0"
     },

But this is fine for us.

dawehner’s picture

I went through every change in the various components, stumbled upon the getTrustedHost() changes in the Request object, but they are converted to a 400 http error, so not an actual unhandled exception.

Another change I struggled a bit with was:

diff --git a/src/Symfony/Component/Serializer/Serializer.php b/src/Symfony/Component/Serializer/Serializer.php
index 5994355..0259bfe 100644
--- a/src/Symfony/Component/Serializer/Serializer.php
+++ b/src/Symfony/Component/Serializer/Serializer.php
@@ -250,15 +250,6 @@ class Serializer implements SerializerInterface, NormalizerInterface, Denormaliz
             return $normalizer->denormalize($data, $class, $format, $context);
         }

-        foreach ($this->normalizers as $normalizer) {
-            if (
-                $normalizer instanceof DenormalizerInterface &&
-                $normalizer->supportsDenormalization($data, $class, $format)
-            ) {
-                return $normalizer->denormalize($data, $class, $format, $context);
-            }
-        }
-

Turns out the call to getDenormalizer in

    private function denormalizeObject($data, $class, $format, array $context = array())
    {
        if (!$this->normalizers) {
            throw new LogicException('You must register at least one normalizer to be able to denormalize objects.');
        }

        if ($normalizer = $this->getDenormalizer($data, $class, $format)) {
            return $normalizer->denormalize($data, $class, $format, $context);
        }

does the same already, so this particular change is fine.

The change in

diff --git a/src/Symfony/Component/Validator/Constraint.php b/src/Symfony/Component/Validator/Constraint.php
index 61def98..fc5288d 100644
--- a/src/Symfony/Component/Validator/Constraint.php
+++ b/src/Symfony/Component/Validator/Constraint.php
@@ -129,6 +129,9 @@ abstract class Constraint
             unset($options['value']);
         }

+        if (is_array($options)) {
+            reset($options);
+        }

seems like it could cause issues. Is this just me?

Status: Reviewed & tested by the community » Needs work

The last submitted patch, 2: 2840596-2.patch, failed testing.

alexpott’s picture

Status: Needs work » Reviewed & tested by the community
catch’s picture

@dawehner The reset() looks unnecessary to me rather than potentially harmful - anything specific as to why it could cause problems?

dawehner’s picture

@dawehner The reset() looks unnecessary to me rather than potentially harmful - anything specific as to why it could cause problems?

Oh I somehow read something like $options = reset($options);. I doubt this line causes any harm

RTBC +1

  • catch committed 30de60c on 8.3.x
    Issue #2840596 by alexpott: Update Symfony components to ~2.8.15
    
catch’s picture

Status: Reviewed & tested by the community » Postponed

Moving this back to 8.2.x, but we should only even consider committing this there if we have a PHP 7.1 environment consistently green IMO, if not waiting for 8.3.x isn't the worst thing in the world.

catch’s picture

Version: 8.3.x-dev » 8.2.x-dev

Arghh should've given dawehner commit credit for the line-by-line review, added issue credit at least.

Also moving version back.

dawehner’s picture

One reason why it we should maybe better add it to 8.2 is the potential incident of a security release of symfony 2.8.x. We want to make the transition as smooth as possible in the future, so just having the security update instead of also the other patch level changes could be helpful.

catch’s picture

Version: 8.2.x-dev » 8.3.x-dev
Status: Postponed » Reviewed & tested by the community

Rolled back for now, this may have re-introduced
https://www.drupal.org/pift-ci-job/567249

  • catch committed 72bc194 on 8.3.x
    Revert "Issue #2840596 by alexpott: Update Symfony components to ~2.8.15...
catch’s picture

Pretty sure it was that commit - branch tests failed immediately following.

Another plug for #2825358: Event class constants for event names can cause fatal errors when a module is installed then...

alexpott’s picture

The PR that broke this is: https://github.com/symfony/symfony/pull/18312. This has resulted in APCu caching of class not founds which means that the additional autoloader fix is working but on the next request the class is not found even though the autoloader can find it :(

alexpott’s picture

Status: Reviewed & tested by the community » Needs review
FileSize
864 bytes
30.37 KB

So one fix is to register the composer classmap as a fallback classmap. I'm not sure how #2 passed and then failed branch tests so tagging needs manual testing since that should the fail and proved the fix for me.

https://github.com/composer/composer/pull/5559 is also quite scary for us since this will add APCu caching for missing classes to composer. Hopefully it'll be opt-in.

catch’s picture

Status: Needs review » Reviewed & tested by the community

We discussed several alternatives to this in irc:

- implementing our own APCu classloader, unfortunately this would also mean re-implementin wincache, xcache support etc.

- adding a chained class loader that encapsulates the cached and fallback classloader, so that $this->classloader is still 100% accurate. This adds both code and runtime complexity to do about the same thing as the change here - we'd be re-implementing PHP's support for multiple classloaders really.

- an upstream Symfony PR to make the negative caching configurable - might be worth it, this change, in a patch release no less, makes it not really useable for projects that allow the UI to determine what code gets loaded.

We should probably open a follow-up to discuss some of these further, but I think the patch @alexpott posted is the minimal impact change we can do.

slasher13’s picture

Status: Reviewed & tested by the community » Needs work
...
- Installing symfony/routing (v2.8.4) Loading from cache
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing symfony-cmf/routing (1.4.0) Loading from cache
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing symfony/polyfill-apcu (v1.1.1) Loading from cache
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing symfony/class-loader (v2.8.4) Loading from cache
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing symfony/console (v2.8.4) Loading from cache
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing symfony/dependency-injection (v2.8.4) Loading from cache
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing symfony/polyfill-iconv (v1.1.1) Loading from cache
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing symfony/process (v2.8.4) Loading from cache
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing symfony/psr-http-message-bridge (v0.2) Loading from cache
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing symfony/serializer (v2.8.4) Loading from cache
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing symfony/translation (v2.8.4) Loading from cache
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing symfony/validator (v2.8.4) Loading from cache
...
Generating autoload files
> Drupal\Core\Composer\Composer::preAutoloadDump
> Drupal\Core\Composer\Composer::ensureHtaccess
Loading composer repositories with package information
Installing dependencies (including require-dev) from lock file
Nothing to install or update
Generating autoload files
> Drupal\Core\Composer\Composer::preAutoloadDump
> Drupal\Core\Composer\Composer::ensureHtaccess

Testbot didn't install composer dependencies from composer.lock (https://dispatcher.drupalci.org/job/default/295565/consoleText). Same issue in https://www.drupal.org/node/2572605

alexpott’s picture

Status: Needs work » Reviewed & tested by the community

@slasher13 well that explains why it didn't fail. This is not a problem with the patch this is a problem with DrupalCI.

alexpott’s picture

catch’s picture

Version: 8.3.x-dev » 8.2.x-dev
Status: Reviewed & tested by the community » Postponed

Committed/pushed to 8.3.x, thanks!

Back to 8.2.x again.

  • catch committed 4f6dc9a on 8.3.x
    Issue #2840596 by alexpott, dawehner: Update Symfony components to ~2.8....
Mixologic’s picture

Re-arranged the steps to patch first, then composer: #2842443: Patches should be applied before running composer., drupalci should pick up those changes now.

alexpott’s picture

Status: Postponed » Needs work
Issue tags: +Triaged D8 critical

Discussed with @xjm, @catch, @effulgentsia, @Cottser and @cilefen. We decided to keep this critical as supporting the latest stable version of PHP is a must for Drupal 8.

It is unfortunate that we had to add an additional autoloader to fix a regression caused by a Symfony bug fix. However, that is indicative of the fact the Symfony is just not built for applications that can change what code is available during a request on production. We probably should consider decoupling ourselves from Symfony's autoloader implementation. There are a number of issues in this area all with different focuses - see #2504685: Create a faster autoloader, #2536610: Ensure all modules are autoloaded with PSR-4 only if enabled.

Also this issue highlights that we've not been keeping symfony updated which might lead to problems if they did a security release tomorrow. I think we need to add a step to the RC process to create an issue to do a composer update to update the composer.lock file and run tests.

alexpott’s picture

Title: Update Symfony components to ~2.8.15 » Update Symfony components to ~2.8.16
Status: Needs work » Needs review
FileSize
45.32 KB
33.59 KB

Dang there's been a new Symfony bug fix release which makes updating 8.2.x with the simple composer command in the summary not simple. I propose we update both 8.2.x and 8.3.x here to 2.18.16.

The last submitted patch, 33: 2840596-33.8.2.patch, failed testing.

Status: Needs review » Needs work

The last submitted patch, 33: 2840596-33.8.3.patch, failed testing.

alexpott’s picture

So the latest and greatest composer breaks Drupal\Tests\ComposerIntegrationTest so that needs a followup issue.

alexpott’s picture

The last submitted patch, 36: 2840596-36.8.2.patch, failed testing.

dawehner’s picture

Status: Needs review » Reviewed & tested by the community

I talked with alex about the browser-kit version number. I think updating is fine, especially we don't have a direct dependency, which means it is just use in tests.

  • catch committed f96bb93 on 8.3.x
    Issue #2840596 by alexpott, catch, dawehner: Update Symfony components...

  • catch committed f0286ae on 8.2.x
    Issue #2840596 by alexpott, catch, dawehner: Update Symfony components...
catch’s picture

Status: Reviewed & tested by the community » Fixed

OK committed both patches to 8.3.x and 8.2.x respectively.

We need to keep an eye out for unexpected regressions in 8.2.x, but otherwise this should mean full PHP 7.1 support in the next patch release of core.

alexpott’s picture

Status: Fixed » Needs work

the updates of our dependencies dependencies to 3.2.* is causing problems (via @webflo) - OG tests are failing.

alexpott’s picture

The patch attached fixes the symfony dependencies that moved to 3.2.x and moves them back to 2.8.16.

alexpott’s picture

Creating the patches is a bit of a pain because you have to do things like:

composer require symfony/browser-kit:2.*
composer require symfony/dom-crawler:2.*
// Edit the composer.json to remove the new fake dependencies...
composer update --lock
webflo’s picture

Status: Needs review » Reviewed & tested by the community

Looks great, thank you!

  • catch committed 4bf6ae4 on 8.3.x
    Issue #2840596 by alexpott, catch, dawehner: Update Symfony components...

  • catch committed 22aa450 on 8.2.x
    Issue #2840596 by alexpott, catch, dawehner: Update Symfony components...
catch’s picture

Status: Reviewed & tested by the community » Fixed

Committed/pushed both versions to both versions, thanks!

alexpott’s picture

@andypost what does the CR contain? There should be nothing for a developer or someone to do.

xjm’s picture

Issue tags: +8.2.6 release notes

  • catch committed 30de60c on 8.4.x
    Issue #2840596 by alexpott: Update Symfony components to ~2.8.15
    
  • catch committed 4bf6ae4 on 8.4.x
    Issue #2840596 by alexpott, catch, dawehner: Update Symfony components...
  • catch committed 4f6dc9a on 8.4.x
    Issue #2840596 by alexpott, dawehner: Update Symfony components to ~2.8....
  • catch committed 72bc194 on 8.4.x
    Revert "Issue #2840596 by alexpott: Update Symfony components to ~2.8.15...
  • catch committed f96bb93 on 8.4.x
    Issue #2840596 by alexpott, catch, dawehner: Update Symfony components...

  • catch committed 30de60c on 8.4.x
    Issue #2840596 by alexpott: Update Symfony components to ~2.8.15
    
  • catch committed 4bf6ae4 on 8.4.x
    Issue #2840596 by alexpott, catch, dawehner: Update Symfony components...
  • catch committed 4f6dc9a on 8.4.x
    Issue #2840596 by alexpott, dawehner: Update Symfony components to ~2.8....
  • catch committed 72bc194 on 8.4.x
    Revert "Issue #2840596 by alexpott: Update Symfony components to ~2.8.15...
  • catch committed f96bb93 on 8.4.x
    Issue #2840596 by alexpott, catch, dawehner: Update Symfony components...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

jibran’s picture