[meta] Ensure vendor (PHP) libraries are on latest stable release

Note to the folks filing sub-issues:

I talked this issue over with the other branch maintainers the other day. Normally, we would indeed make any issues that are hard blockers to solving critical issues critical as well. However, in the case of meta issues like this where every child issue is essentially a copy/paste of the other ones, it's actually polluting the list of criticals quite a bit, which has a number of down sides: it makes it hard to tell how we're progressing against release; the volume "drowns out" other issues that are actually critical blockers; it also makes it impossible to determine whether an issue involves just routine library updates or if upgrading a library actually is critical (for example, if it solves a security issue).

So please go ahead and just file these children as "normal" unless there's something about the library upgrade that would meet the standard issue priority definitions. Rest assured, we're all constantly looking at the RTBC queue for these library update issues because a) we know they help resolve this critical issue and b) they are normally pretty easy to sign-off on and commit. (If for some reason one of them sits at RTBC for longer than a couple of days feel free to ping one of us about it in #drupal-contribute.)

This is now captured as part of #2485119: [meta] The Drupal 8.0.0-rc1 Release Checklist. Downgrading.

Added a table of all libraries currently being used by core/composer.json

@MustangGB how did you generated that table? I'm not sure how to update that table while composer.json has different list than in that table?

I was able to identify four updates on composer update:

• #2493907: Update doctrine/collections to the latest stable release
• #2493909: Update doctrine/cache to the latest stable release
• #2493911: Update guzzle, goutte and mink-goutte-driver to the latest release
• #2493913: Update mikey179/vfsstream to the latest stable release

egulias/email-validator 1.2.8 should already be supported, so I don't think it needs to say there is an update.

Some of those projects bring out releases very frequently, especially guzzle (which just released Guzzle 6, we started using it with version 3 I think..).

Might make sense to wait with short before the release with updating those that are just minor updates and aren't blocking other work, otherwise we might repeat it a few times before we get 8.0.0 out?

@Berdir I agree in general. Having read the Guzzle 6 release notes, it is an example of something we have to push on.

Word from Michael Dowling of Guzzle: https://twitter.com/mtdowling/status/603415290802614272

6 should be the last stable for a while (hopefully), but requires PHP 5.5. Thus, updating to it is blocked by (and supports doing) #2296557: [policy] Require PHP 5.5

Updating PHPUnit/Symfony version information

Adding related issue where external libraries are tried to be removed from and let Composer handle it: #1475510: Remove external dependencies from the core repo and let Composer manage the dependencies instead

The future of issues like this could seriously helped by committing and deploying the patches in:
#1923582: Add ability for testbot to run 'composer install' during installation
#2315545: Install composer dependencies to D8 when packaging

There is no patch and this is a meta so we can use the new cat.

Updated Symfony and Twig versions.

I don't know of a non-manual way to do this, so here's how I'm spending my Thursday night. ;)

Haven't figured out the best way to do this yet. Here's a git diff after:

rm -rf core/vendor
rm core/composer.lock
composer install
git diff core/vendor # since for some reason Drupal core's files get overwritten to some older version..?

However, this will only tell us of point-release upgrades, not minor/major version upgrades. Still. Quite a lot in here.

Glancing at that, it looks like at least:

- Composer
- Guzzle
- Symfony
- Twig

Need an update.

I can't think of issues for others, but for Symfony, we have #2548135: Upgrade to Symfony 2.7.3. I think it's lying there since quite some time. (This update has security fixes as well.)

EDIT: Ignore above, I was clearly thinking of something else. :)

Moar table updates. PHPUnit is way behind (2 minor versions) as well.

Damn it, Guzzle.

+++ b/core/vendor/composer/installed.json
@@ -2680,32 +2789,43 @@
-        "name": "symfony/http-foundation",
-        "version": "v2.7.3",
-        "version_normalized": "2.7.3.0",
+        "name": "symfony/translation",
+        "version": "v2.7.2",
+        "version_normalized": "2.7.2.0",

I see this and many others like it. It seems it is moving it back to 2.7.2. I wonder what changed.

Better links on version numbers to packagist pages for the respective projects so it's a lot easier to check these next time.

@hussainweb: Oh, you should probably assume I have no idea what I'm doing. :)

But something I noticed was when I ran composer update/install, it basically goes and downloads the old beta14 of Drupal 8 and overwrites my existing install (which seems... suboptimal :P). So it's possible those are just remnants of that.

@webchick: Oh, I think you were running composer install/update in the root directory? The composer.json in root is for use with the entirely composer driven workflow. I think for our purposes, we should run composer in the core directory.

That also explains why you got the old Symfony 2.7.2. I don't think #2548135: Upgrade to Symfony 2.7.3 is in beta14 and it probably picked up the version from the composer.lock in beta14.

Ok, think that (&@#ing table is updated now.

We definitely have some work to do here. So while I love being at 5 criticals more than life itself, need more eyes on this one now. Escalating to critical, and an Actionable D8 Critical.

@webchick: The table lists an update for Symfony, but I believe Symfony is already updated in core.

@hussainweb: Oh, good call. Here's the same thing from that directory instead.

cd core/
rm composer.lock
composer update

That's... a big... patch... ;)

Welp. No idea what to make of all of that, but correcting the Symfony and EasyRDF rows based on that, if nothing else.

BTW, if you can get on IRC this might be easier to hash out in person. I also could always just go to bed like a sane person. ;)

For giggles, let's see what testbot makes of that.

Interesting, this does not update symfony, I guess they did not had a release in the meantime. I hope there will be one until we release, given that had
quite a DX improvement lately, see https://github.com/symfony/symfony/pull/15380

Ups I actually did not intended to upload that patch. Sorry

@webchick

PHP Warning:  include(/var/lib/drupaltestbot/sites/default/files/checkout/core/vendor/phpunit/php-timer/src/Timer.php): failed to open stream: No such file or directory in /var/lib/drupaltestbot/sites/default/files/checkout/core/vendor/composer/ClassLoader.php on line 412
PHP Warning:  include(): Failed opening '/var/lib/drupaltestbot/sites/default/files/checkout/core/vendor/phpunit/php-timer/src/Timer.php' for inclusion (include_path='.:/usr/share/php:/usr/share/pear') in /var/lib/drupaltestbot/sites/default/files/checkout/core/vendor/composer/ClassLoader.php on line 412
PHP Warning:  include(/var/lib/drupaltestbot/sites/default/files/checkout/core/vendor/phpunit/php-text-template/src/Template.php): failed to open stream: No such file or directory in /var/lib/drupaltestbot/sites/default/files/checkout/core/vendor/composer/ClassLoader.php on line 412
PHP Warning:  include(): Failed opening '/var/lib/drupaltestbot/sites/default/files/checkout/core/vendor/phpunit/php-text-template/src/Template.php' for inclusion (include_path='.:/usr/share/php:/usr/share/pear') in /var/lib/drupaltestbot/sites/default/files/checkout/core/vendor/composer/ClassLoader.php on line 412
Drupal\datetime\Tests\Views\ArgumentDateTimeTest              12 passes
PHP Warning:  include(/var/lib/drupaltestbot/sites/default/files/checkout/core/vendor/phpunit/php-text-template/src/Template.php): failed to open stream: No such file or directory in /var/lib/drupaltestbot/sites/default/files/checkout/core/vendor/composer/ClassLoader.php on line 412
PHP Warning:  include(): Failed opening '/var/lib/drupaltestbot/sites/default/files/checkout/core/vendor/phpunit/php-text-template/src/Template.php' for inclusion (include_path='.:/usr/share/php:/usr/share/pear') in /var/lib/drupaltestbot/sites/default/files/checkout/core/vendor/composer/ClassLoader.php on line 412
PHP Fatal error:  Class 'Text_Template' not found in /var/lib/drupaltestbot/sites/default/files/checkout/core/vendor/phpunit/phpunit/src/Framework/TestCase.php on line 622
PHP Warning:  Invalid argument supplied for foreach() in /var/lib/drupaltestbot/sites/default/files/checkout/core/scripts/run-tests.sh on line 593

Did you made a git diff or did you also ensured that you include new files?

@webchick: This is what I get on composer update (at the end). The total size of changes comes to 2.6 MiB (but that's with the -M flag).

@dawehner: It seems that the PR was merged immediately after 2.7.3 release. There has been no release since. It is there in 2.3.32 but that doesn't help us.

hw@d8:/var/www/d8task/core-[git 8.0.x] $ composer update
Loading composer repositories with package information
Updating dependencies (including require-dev)
  - Removing doctrine/annotations (v1.2.1)
  - Installing doctrine/annotations (v1.2.7)
    Loading from cache

  - Removing doctrine/collections (v1.2)
  - Installing doctrine/collections (v1.3.0)
    Loading from cache

  - Removing doctrine/cache (v1.3.1)
  - Installing doctrine/cache (v1.4.2)
    Loading from cache

  - Removing doctrine/inflector (v1.0)
  - Installing doctrine/inflector (v1.0.1)
    Loading from cache

  - Removing doctrine/common (v2.4.2)
  - Installing doctrine/common (v2.4.3)
    Downloading: 100%

  - Removing easyrdf/easyrdf (0.9.0)
  - Installing easyrdf/easyrdf (0.9.1)
    Downloading: 100%

  - Removing sebastian/version (1.0.5)
  - Installing sebastian/version (1.0.6)
    Loading from cache

  - Removing sebastian/recursion-context (1.0.0)
  - Installing sebastian/recursion-context (1.0.1)
    Downloading: 100%

  - Removing sebastian/exporter (1.2.0)
  - Installing sebastian/exporter (1.2.1)
    Downloading: 100%

  - Removing sebastian/environment (1.2.2)
  - Installing sebastian/environment (1.3.2)
    Downloading: 100%

  - Removing sebastian/comparator (1.1.1)
  - Installing sebastian/comparator (1.2.0)
    Downloading: 100%

  - Removing doctrine/instantiator (1.0.4)
  - Installing doctrine/instantiator (1.0.5)
    Loading from cache

  - Removing phpspec/prophecy (1.4.0)
  - Installing phpspec/prophecy (v1.5.0)
    Downloading: 100%

  - Removing phpunit/php-text-template (1.2.0)
  - Installing phpunit/php-text-template (1.2.1)
    Loading from cache

  - Removing phpunit/phpunit-mock-objects (2.3.1)
  - Installing phpunit/phpunit-mock-objects (2.3.7)
    Downloading: 100%

  - Removing phpunit/php-timer (1.0.5)
  - Installing phpunit/php-timer (1.0.7)
    Downloading: 100%

  - Removing phpunit/php-token-stream (1.4.1)
  - Installing phpunit/php-token-stream (1.4.6)
    Downloading: 100%

  - Removing phpunit/php-file-iterator (1.4.0)
  - Installing phpunit/php-file-iterator (1.4.1)
    Downloading: 100%

  - Removing phpunit/php-code-coverage (2.0.16)
  - Installing phpunit/php-code-coverage (2.2.2)
    Downloading: 100%

  - Removing phpunit/phpunit (4.6.4)
  - Installing phpunit/phpunit (4.6.10)
    Downloading: 100%

  - Removing zendframework/zend-stdlib (2.4.0)
  - Installing zendframework/zend-stdlib (2.4.7)
    Downloading: 100%

  - Removing zendframework/zend-escaper (2.4.0)
  - Installing zendframework/zend-escaper (2.4.7)
    Downloading: 100%

  - Removing zendframework/zend-feed (2.4.0)
  - Installing zendframework/zend-feed (2.4.7)
    Downloading: 100%

  - Removing stack/builder (v1.0.2)
  - Installing stack/builder (v1.0.3)
    Downloading: 100%

  - Removing fabpot/goutte (v3.1.0)
  - Installing fabpot/goutte (v3.1.1)
    Downloading: 100%

  - Removing masterminds/html5 (2.1.0)
  - Installing masterminds/html5 (2.1.2)
    Downloading: 100%

Writing lock file
Generating autoload files
> Drupal\Core\Composer\Composer::preAutoloadDump
> Drupal\Core\Composer\Composer::ensureHtaccess

I checked guzzle and MinkGoutteDriver and both commits are not in releases yet. We could update guzzle to latest commit but there is nothing substantial in MinkGoutteDriver as of now. I think we can wait for a proper release for both of them. I am continuing to look into others.

Marking fabpot/goutte, masterminds/html5, stack/builder components as has updates. I also marked that zendframework/zend-feed and phpunit also has a patch version update. Added more rows as well.

Removing zendframework components which are not directly used.

It would be useful if an option in reports/status report listed all the libraries, together with their version numbers (and links to vendor websites), in the same way that you can get more information about PHP at status/php, and perhaps an indication of the latest vendor version.

It would be useful if an option in reports/status report listed all the libraries, together with their version numbers (and links to vendor websites), in the same way that you can get more information about PHP at status/php, and perhaps an indication of the latest vendor version.

It would be great to discuss that as part of its own issue.

I hope our composer.json in core will specify dependency ranges that are the 'minimum tested version-> maximum tested version" that we support when a release is finalized. We're testing drupal against a known set of changes + a known set of dependencies, and it'd be a drag if two weeks after we release, there is a patch version of a dependency that accidentally breaks an api, and people trying to download 8.0.0 end up with broken dependencies

in other words if egulias/email-validator releases 1.2.10, but it accidentally refactors away something it should'nt and breaks an api we're using, then we cant really say that 8.0.0 works with 1.2.*

we could hope that nobody ever accidentally breaks semantic versioning, but that feels really, really risky.

@Mixologic: I think composer.lock helps there. We would follow the same process to update components in core as we do now, except that we will only ever commit composer.lock changes, nothing else. With that, the scenario you describe will be averted.

So the important changes are in composer.json and composer.lock (extra txt file attached for that changes).

The patch are all updates done as outlined in the issue summary. Let's see where this fails.

So we have the "invalid patch format" error again, although "git apply php-lib-update-2400407-55.patch" works fine for me locally.

I'm assuming that the bot has a problem with the large patch file size (over 3MB) and it also choked on the 2MB patch from dawehner.

Let's try to do smaller chunks of updates then, I'm going to try the phpunit update alone next.

I thought the problem is the --binary but yeah you are maybe right. My patch was smaller because I use a quite aggressive moving setting in my gitconfig.

Let's try to do smaller chunks of updates then, I'm going to try the phpunit update alone next.

+1
We have one issue for the doctrine stuff already as well.

The phpunit update alone is 2 MB, created with changing the phpunit entry in composer.json and:

composer update phpunit/php-timer phpunit/phpunit sebastian/environment phpunit/php-code-coverage

@catch: Yes, I was just going to create multiple sub-issues and post patches for each. It is simple and we already do that everywhere else. :)

RTBC, but note, this issue won't be done after that.

Per catch, let's get that one split off into its own dedicated issue.

I'll also be around for the next 8+ hours for anyone who wants to do the "rapid serial patches - i.e. upgrade one, commit, upgrade next, commit" dance.

Sorry! I misread. Serial patches, not issues! Great.

Ok, #2105825: Update Doctrine/common Component to 2.5.1 (security) (includes cache/annotations too) was committed, so the PHPUnit one needs a small re-roll now.

on it.

reroll for the phpunit update.

@hussainweb : It was my understanding that composer.lock only helps with locking versions at the root of your project, and there can only be one lockfile per project (no nested lock files). If a user wants to use composer for their d8 site and either add modules or have modules with dependencies, then core's composer.lock will have to be ignored, in which case they may potentially get versions of dependencies that have been released after the core release happened, which holds the possibility that a dependency broke an api somewhere. I know that its a lot of ifs, and it probably doesn't matter until we no longer ship core/vendor with the tarball. But once we do, it'll be something we need to consider implementing.

Committed and pushed #68 to 8.0.x. YAY!

Let's get subsequent library update patches in their own issues tho, for better visibility. It's fine if their issue summary is just "do it" ;)

Updating table.

And the rest.

Updating easyrdf:

hw@d8:/var/www/d8task/core-[git 8.0.x] $ composer update easyrdf/*
Loading composer repositories with package information
Updating dependencies (including require-dev)
  - Removing easyrdf/easyrdf (0.9.0)
  - Installing easyrdf/easyrdf (0.9.1)
    Loading from cache

Writing lock file
Generating autoload files
> Drupal\Core\Composer\Composer::preAutoloadDump
> Drupal\Core\Composer\Composer::ensureHtaccess

@Mixologic: You're right. I didn't think of when Drupal itself would be a dependency in the regular composer workflow. To counter that, it makes sense to lock the max versions for components that don't promise semantic versioning. Maybe this could be discussed in a different issue?

Updating the table...

Committed and pushed #76 to 8.0.x.

Next? :D

Twig is updated as well. :)

The table in the IS isn't real clear what is left and what is in. Can someone add a column or some such notation to denote what is completed?

Updating all zendframework/* components:

hw@d8:/var/www/d8task/core-[git 8.0.x] $ composer require zendframework/zend-fe
ed:"~2.4"
./composer.json has been updated
Loading composer repositories with package information
Updating dependencies (including require-dev)
Nothing to install or update
Writing lock file
Generating autoload files
> Drupal\Core\Composer\Composer::preAutoloadDump
> Drupal\Core\Composer\Composer::ensureHtaccess
hw@d8:/var/www/d8task/core-[git 8.0.x] $ composer update zendframework/*
Loading composer repositories with package information
Updating dependencies (including require-dev)
  - Removing zendframework/zend-stdlib (2.4.7)
  - Installing zendframework/zend-stdlib (2.6.0)
    Downloading: 100%

  - Removing zendframework/zend-escaper (2.4.7)
  - Installing zendframework/zend-escaper (2.5.1)
    Downloading: 100%

  - Removing zendframework/zend-feed (2.4.7)
  - Installing zendframework/zend-feed (2.5.2)
    Downloading: 100%

Writing lock file
Generating autoload files
> Drupal\Core\Composer\Composer::preAutoloadDump
> Drupal\Core\Composer\Composer::ensureHtaccess
hw@d8:/var/www/d8task/core-[git 8.0.x] $ composer require zendframework/zend-di
actoros:"~1.1"
./composer.json has been updated
Loading composer repositories with package information
Updating dependencies (including require-dev)
  - Removing zendframework/zend-diactoros (1.1.0)
  - Installing zendframework/zend-diactoros (1.1.3)
    Downloading: 100%

Writing lock file
Generating autoload files
> Drupal\Core\Composer\Composer::preAutoloadDump
> Drupal\Core\Composer\Composer::ensureHtaccess

@heddn: I thought about that but then I think that before long, all will be updated and the table will be easy to manage.

Committed and pushed to 8.0.x. Thanks!

Next up... Twig! :)

I said twig was already updated by mistake. It was to do with composer.json. Attaching the update:

hw@d8:/var/www/d8task/core-[git 8.0.x] $ composer require twig/twig:"~1.20"
./composer.json has been updated
Loading composer repositories with package information
Updating dependencies (including require-dev)
  - Removing twig/twig (v1.20.0)
  - Installing twig/twig (v1.21.1)
    Loading from cache

Writing lock file
Generating autoload files
> Drupal\Core\Composer\Composer::preAutoloadDump
> Drupal\Core\Composer\Composer::ensureHtaccess

Updating the table ...

Committed and pushed to 8.0.x. Next! :D

Oops.

I noticed that we didn't update various dependencies in PHPUnit patch. Here it is first. I think we are mostly done after this. I could just post one more patch combining the remaining components now that the mostly used ones are in.

Oh, these are the updates in the above patch. We don't use any of these directly but PHPUnit uses them.

hw@d8:/var/www/d8task/core-[git 8.0.x] $ composer update phpunit/*
Loading composer repositories with package information
Updating dependencies (including require-dev)
  - Removing phpunit/php-file-iterator (1.4.0)
  - Installing phpunit/php-file-iterator (1.4.1)
    Loading from cache

  - Removing phpunit/php-text-template (1.2.0)
  - Installing phpunit/php-text-template (1.2.1)
    Loading from cache

  - Removing phpunit/phpunit-mock-objects (2.3.1)
  - Installing phpunit/phpunit-mock-objects (2.3.7)
    Loading from cache

  - Removing phpunit/php-token-stream (1.4.1)
  - Installing phpunit/php-token-stream (1.4.6)
    Loading from cache

Writing lock file
Generating autoload files
> Drupal\Core\Composer\Composer::preAutoloadDump
> Drupal\Core\Composer\Composer::ensureHtaccess
hw@d8:/var/www/d8task/core-[git update] $ composer update sebastian/* phpspec/*
Package "sebastian/*" listed for update is not installed. Ignoring.
Package "phpspec/*" listed for update is not installed. Ignoring.
Loading composer repositories with package information
Updating dependencies (including require-dev)
  - Removing sebastian/version (1.0.5)
  - Installing sebastian/version (1.0.6)
    Loading from cache

  - Removing sebastian/recursion-context (1.0.0)
  - Installing sebastian/recursion-context (1.0.1)
    Loading from cache

  - Removing sebastian/exporter (1.2.0)
  - Installing sebastian/exporter (1.2.1)
    Loading from cache

  - Removing sebastian/comparator (1.1.1)
  - Installing sebastian/comparator (1.2.0)
    Loading from cache

  - Removing phpspec/prophecy (1.4.0)
  - Installing phpspec/prophecy (v1.5.0)
    Loading from cache

Writing lock file
Generating autoload files
> Drupal\Core\Composer\Composer::preAutoloadDump
> Drupal\Core\Composer\Composer::ensureHtaccess

Let's try this again...

cd core/
rm composer.lock
composer update
git diff -M > ~/Desktop/upgrade-all-the-phps.patch 

Hm. I did something goofy there. Definitely did not mean to delete hussainweb's patch.

Unfortunately, I can't seem to commit that patch (OSX case insensitivity strikes again :\):

$ git apply --index phpunit-deps-2400407_0.patch 
error: core/vendor/phpspec/prophecy/composer.lock: already exists in working directory
core/phpunit-deps-2400407_0.patch:6592: new blank line at EOF.
+
core/phpunit-deps-2400407_0.patch:12300: new blank line at EOF.
+

I asked @alexpott, but he wasn't comfortable committing without knowing the exact composer command that lead to the patch's generation.

Following that gist from @webchick I wrote a script that updates all dependencies in core one by one and then make a patch. If a dependency does not need to be updated, an empty patch will be created then cleaned at the end of the process.

Here are all the patches given by this script to see if the testbot is happy :)

Trying to upload that patch that combines all previous green updates to see if the testbot is still happy and help @webchick to know if it's safe to commit.

@webchick here is a "script" you can play with closed eyes. It updates all dependencies that passed the two previous tests (first alone then all together) and commit them separately.

cd core

composer update fabpot/goutte
git add -A .
git commit -m "Issue #2400407 by DuaelFr, webchick, klausi, hussainweb, dawehner: Ensure vendor fabpot/goutte is on latest stable release"

composer update phpspec/prophecy
git add -A .
git commit -m "Issue #2400407 by DuaelFr, webchick, klausi, hussainweb, dawehner: Ensure vendor phpspec/prophecy is on latest stable release"

composer update phpunit/php-file-iterator
git add -A .
git commit -m "Issue #2400407 by DuaelFr, webchick, klausi, hussainweb, dawehner: Ensure vendor phpunit/php-file-iterator is on latest stable release"

composer update phpunit/php-text-template
git add -A .
git commit -m "Issue #2400407 by DuaelFr, webchick, klausi, hussainweb, dawehner: Ensure vendor phpunit/php-text-template is on latest stable release"

composer update phpunit/php-token-stream
git add -A .
git commit -m "Issue #2400407 by DuaelFr, webchick, klausi, hussainweb, dawehner: Ensure vendor phpunit/php-token-stream is on latest stable release"

composer update phpunit/phpunit-mock-objects
git add -A .
git commit -m "Issue #2400407 by DuaelFr, webchick, klausi, hussainweb, dawehner: Ensure vendor phpunit/phpunit-mock-objects is on latest stable release"

composer update sebastian/comparator
git add -A .
git commit -m "Issue #2400407 by DuaelFr, webchick, klausi, hussainweb, dawehner: Ensure vendor sebastian/comparator is on latest stable release"

composer update sebastian/exporter
git add -A .
git commit -m "Issue #2400407 by DuaelFr, webchick, klausi, hussainweb, dawehner: Ensure vendor sebastian/exporter is on latest stable release"

composer update sebastian/recursion-context
git add -A .
git commit -m "Issue #2400407 by DuaelFr, webchick, klausi, hussainweb, dawehner: Ensure vendor sebastian/recursion-context is on latest stable release"

composer update sebastian/version
git add -A .
git commit -m "Issue #2400407 by DuaelFr, webchick, klausi, hussainweb, dawehner: Ensure vendor sebastian/version is on latest stable release"

composer update stack/builder
git add -A .
git commit -m "Issue #2400407 by DuaelFr, webchick, klausi, hussainweb, dawehner: Ensure vendor stack/builder is on latest stable release"

And finally...

git push origin 8.0.x

=> Week-end!

OMG!! YOU. ARE. THE. BEST!!! :D

Working on this now.

Ok, did that. Let's see what testbot says in a half hour or so. :)

Updating the table. I think the only one left is mastermind/html5 and that had errors. However, they seem a bit goofy, so trying a re-test on that.

We also need to eventually get guzzlehttp/guzzle and behat/mink-goutte-driver on a "real" version, not a specific Git commit.

And finally, there's composer/installers which is in the main composer.json file, not the one under core/. However, when I try and do composer update composer/installers it starts pulling in all the things again. :( So need some help on that.

Awesome. I think we missed one. Attaching the patch for masterminds/html5.

Ah, crosspost. I see it was not missed but test failure. Maybe something got missed as the patch size is different.

Regarding composer/installers. We don't actually use that composer.json at all. It is more like an example composer.json file we may use if we want to use Drupal completely using composer. In that case, you need that package to correctly install Drupal in core directory instead of composer's default location. We should actually remove that from the table.

Oh right, that won't work, because composer.lock. :P

Thanks, @hussainweb! Let's see if we have better luck with PIFT this time. Hope you got some sleep. :)

Oh, great. Done!

Also, why is it not called composer.example.json? (DUCK)

I was just posting this when three other comments flew in:

I think this little patch is all that is needed for composer/installers. We don't ship that package's files in the distro as far as I can tell.

Actually, since the current root composer.json has a minimum version of composer/installers set, we should not have to do anything.

+1 on an issue for #110

Or, not, it seems like it needs to be there: https://github.com/composer/installers

#1475510: Remove external dependencies from the core repo and let Composer manage the dependencies instead

Just confirmation that both qa.d.o and DrupalCI came back with a pass on HEAD. Go team!! :D

YAY! masterminds/html5 came back green!

Committed and pushed to 8.0.x.

Let's see what happens with composer installers, but then I think we're done here?!?!

Hm. DrupalCI didn't like #111 for some reason. Sending for a re-test.

Second time's a charm. :)

Committed and pushed to 8.0.x. And. We. Are. DONE.

Great work, everyone!! :D

Syss-MacBook-Air:8.x webchick$ git reset --hard
HEAD is now at 5823083 Issue #2400407: One more: Update composer/classloader to latest version.
Syss-MacBook-Air:8.x webchick$ git status
On branch 8.0.x
Your branch is up-to-date with 'origin/8.0.x'.
nothing to commit, working directory clean
Syss-MacBook-Air:8.x webchick$ cd core
Syss-MacBook-Air:core webchick$ composer update
Loading composer repositories with package information
Updating dependencies (including require-dev)
Generating autoload files
> Drupal\Core\Composer\Composer::preAutoloadDump
> Drupal\Core\Composer\Composer::ensureHtaccess
Syss-MacBook-Air:core webchick$ git diff
Syss-MacBook-Air:core webchick$ 
Syss-MacBook-Air:core webchick$ :)

Nice! Excellent work all!

hw@d8:/var/www/d8task/core-[git 8.0.x] $ sudo composer self-update
You are already using composer version afd98b74f13b9def1b2447f16fd0bfd4d96dff94.
hw@d8:/var/www/d8task/core-[git 8.0.x] $ composer update
Loading composer repositories with package information
Updating dependencies (including require-dev)
Generating autoload files
> Drupal\Core\Composer\Composer::preAutoloadDump
> Drupal\Core\Composer\Composer::ensureHtaccess

Awesome! But, there is a slight issue. The commit in 5823083 actually brings ClassLoader.php to an older version. It was updated in an earlier library update. I think it would be enough to just revert that one (it's good we are doing atomic commits).

Setting to Needs work for above. If you need a patch, let me know. But I think just reverting that commit is enough.

#124 seems right to me as well. Looks like Composer reverted the same change in https://github.com/composer/composer/commit/fbae6b1589ef28b77f0c38d1c9d4... and I don't see a commit there that re-introduces it, so I reverted Drupal's 5823083 to match. Thanks for catching that!

Yay! Thanks for catching that. But the reason I did that commit is because if I repeat the steps in #122:

$ git pull --rebase
remote: Counting objects: 120, done.
remote: Compressing objects: 100% (61/61), done.
remote: Total 63 (delta 46), reused 0 (delta 0)
Unpacking objects: 100% (63/63), done.
From git.drupal.org:project/drupal
   5823083..ae7ec6c  8.0.x      -> origin/8.0.x
First, rewinding head to replay your work on top of it...
Fast-forwarded 8.0.x to ae7ec6c585e400d628350acf9d4a4d9f13a10618. # various small cache fixes, committed after this one's revert...
Syss-MacBook-Air:8.x webchick$ git status
On branch 8.0.x
Your branch is up-to-date with 'origin/8.0.x'.
nothing to commit, working directory clean
Syss-MacBook-Air:8.x webchick$ cd core/
Syss-MacBook-Air:core webchick$ composer update
Loading composer repositories with package information
Updating dependencies (including require-dev)
Generating autoload files
> Drupal\Core\Composer\Composer::preAutoloadDump
> Drupal\Core\Composer\Composer::ensureHtaccess
$ git diff
diff --git a/core/vendor/composer/ClassLoader.php b/core/vendor/composer/ClassLoader.php
index 5e1469e..4e05d3b 100644
--- a/core/vendor/composer/ClassLoader.php
+++ b/core/vendor/composer/ClassLoader.php
@@ -351,7 +351,7 @@ private function findFileWithExtension($class, $ext)
             foreach ($this->prefixLengthsPsr4[$first] as $prefix => $length) {
                 if (0 === strpos($class, $prefix)) {
                     foreach ($this->prefixDirsPsr4[$prefix] as $dir) {
-                        if (file_exists($file = $dir . DIRECTORY_SEPARATOR . substr($logicalPathPsr4, $length))) {
+                        if (is_file($file = $dir . DIRECTORY_SEPARATOR . substr($logicalPathPsr4, $length))) {
                             return $file;
                         }
                     }
@@ -361,7 +361,7 @@ private function findFileWithExtension($class, $ext)
 
         // PSR-4 fallback dirs
         foreach ($this->fallbackDirsPsr4 as $dir) {
-            if (file_exists($file = $dir . DIRECTORY_SEPARATOR . $logicalPathPsr4)) {
+            if (is_file($file = $dir . DIRECTORY_SEPARATOR . $logicalPathPsr4)) {
                 return $file;
             }
         }
@@ -380,7 +380,7 @@ private function findFileWithExtension($class, $ext)
             foreach ($this->prefixesPsr0[$first] as $prefix => $dirs) {
                 if (0 === strpos($class, $prefix)) {
                     foreach ($dirs as $dir) {
-                        if (file_exists($file = $dir . DIRECTORY_SEPARATOR . $logicalPathPsr0)) {
+                        if (is_file($file = $dir . DIRECTORY_SEPARATOR . $logicalPathPsr0

:(

So something's still a bit screwy here...

OH! I get it. sudo composer self-update was the step I was missing. So somehow my local version of Composer can damage the integrity of a project that uses Composer..? Yikes. :P~

Anyway, all's well that ends well. :) Thanks!!

OH! I get it. sudo composer self-update was the step I was missing. So somehow my local version of Composer can damage the integrity of a project that uses Composer..? Yikes. :P~

@webchick, if you are concerned about this we can add a dependency on Composer itself to composer.json. Instead of using your globally installed composer you can instead do:

$ cd core/
$ ./vendor/bin/composer update fabpot/goutte

This would ensure that everybody uses the exact same version of Composer, so there would never be any differences in the rendering of the autoloader and composer.lock.

The drawbacks are that it is another library to maintain and it adds a significant amount of code to the repository.

Also we need this issue or an identical followup open to track security or other significant library updates during rc as a release checklist item. This more or less needs to be a release checklist item every patch release for security updates. We have cooperation with Symfony on security releases, but nothing in place for doctrine afaik - given we found out about their security update via this issue. So at least a monthly check seems necessary while that's the case.

Yep, my thinking was the "At RC time" portion of #2485119: [meta] The Drupal 8.0.0-rc1 Release Checklist becomes part of https://www.drupal.org/node/721106 once all's said and done (so we'd re-open this issue $time_period before a new release). I think it makes sense to wait until we try it once with RC1 and see if the steps outlined are the right ones, though.

I wonder if it's better to spin the Guzzle issue off as its own critical issue, marked postponed, with a pointer to the GitHub thread. I would hate for anyone to waste time re-doing that table again because they think this issue as a whole is still in play atm.

+1 on another issue to track Guzzle.

Re #129: https://security.sensiolabs.org/check could help with security checks for our dependencies. This is what I get when I upload beta-15's composer.lock. HEAD passes with flying colors.

Somehow... composer/installers is now set to require at least 1.0.21, but I can't see any reason or documentation why that would be the case. We require 1.0.20 or greater, but less than 2.0.0 which is what was in there, now it says that we require 1.0.21 or greater, but for no reason. :(

Also worth noting that the testbots are now putting out an error message when running the composer step:

Here is what we used to see, indicating that composer.json and composer.lock file are in sync

09:10:34 composer install --working-dir /var/www/html/core --prefer-dist
09:10:34 Command created as exec id a5b2191d
09:10:34 Loading composer repositories with package information
09:10:34 Installing dependencies (including require-dev) from lock file
09:10:34 Nothing to install or update

And here is what all the testbots have now:

19:09:17 composer install --working-dir /var/www/html/core --prefer-dist
19:09:17 Command created as exec id 99214f18
19:09:28 Loading composer repositories with package information
19:09:28 Installing dependencies (including require-dev) from lock file
19:09:28 Warning: The lock file is not up to date with the latest changes in composer.json. You may be getting outdated dependencies. Run update to update them.
19:09:28 Nothing to install or update

#139, I agree. The version ^1.0.20 would pick everything before 2.0.0 and it was enough (basically what we need). But given that this is more like an example composer.json file, I think it is fine. If we need to roll it back, I guess we could just revert this commit.

On that note, I think there are a lot of version rules we can streamline, like Mixologic has been suggesting in #53. I think that can be something to deal with in another issue. It won't make a difference to our current workflow but it could matter if we are using composer. For example, out phpunit requirement is 4.8.*, which means if another component needs 4.9, it would fail. It is okay with something closely tied in as Symfony but I think something like PHPUnit should be flexible enough. We probably don't need PHPUnit 4.8, just 4.6 (or maybe even lower) and above is fine.

#140, I believe that once #2565337: Upgrade to Symfony 2.7.4 gets in, that error should be resolved.

More updates to the IS.

Could also be #2563799: Upgrade Guzzle to a tagged release. I was a bit hasty in committing that earlier. :P

Could also be #2563799: Upgrade Guzzle to a tagged release. I was a bit hasty in committing that earlier. :P

Yeah it seemed to be that noone actually reviewed the changes, nor did I :(

#2565337: Upgrade to Symfony 2.7.4 landed.

Twig 1.22.0 released.

PHPUnit 4.8.7 is out - #2568595: Upgrade PHPUnit to latest 4.8.x.

Updated version for twig.

Adding updates for behat and related.

Updated versions and issues for zendframework/*.

Created #2581683: Update PHP dependencies pre RC using composer to do this a final time before RC.

Updating all version information.

new patch release http://symfony.com/blog/symfony-2-7-6-released

composer update pulls down a vast number of dependency updates.

Filed #2609268: Upgrade to Symfony 2.7.6

Filed #2610984: Add Archive Tar via Composer, with BC shim because we have third-party (vendor) code in Core that looks outdated and not tracked here.
Also having this code in core blocks ongoing work in a whole #2571965: [meta] Fix PHP coding standards in core, stage 1

Updated IS with latest versions

egulias/email-validator 1.2.9 => 1.2.10
fabpot/goutte 3.1.1 => 3.1.2
mikey179/vfsStream 1.5.0 => 1.6.0
zendframework/zend-diactoros 1.1.3 => 1.1.4

phpunit/phpunit 5.0.8

filed separate patch for twig #2609110-6: Update Twig to 1.23.1

Updating dependencies (including require-dev)
  - Removing composer/installers (v1.0.21)
  - Installing composer/installers (v1.0.22)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Removing wikimedia/composer-merge-plugin (dev-master 47bb338)
  - Installing wikimedia/composer-merge-plugin (v1.3.0)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Removing twig/twig (v1.22.2)
  - Installing twig/twig (v1.23.1)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Removing egulias/email-validator (1.2.9)
  - Installing egulias/email-validator (1.2.10)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Removing zendframework/zend-diactoros (1.1.3)
  - Installing zendframework/zend-diactoros (1.1.4)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Removing composer/semver (1.0.0)
  - Installing composer/semver (1.2.0)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Removing sebastian/global-state (1.0.0)
  - Installing sebastian/global-state (1.1.1)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Removing phpunit/phpunit (4.8.11)
  - Installing phpunit/phpunit (4.8.16)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Removing doctrine/cache (v1.4.2)
  - Installing doctrine/cache (v1.5.1)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Removing doctrine/inflector (v1.0.1)
  - Installing doctrine/inflector (v1.1.0)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Removing zendframework/zend-stdlib (2.7.3)
  - Installing zendframework/zend-stdlib (2.7.4)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Removing guzzlehttp/promises (1.0.2)
  - Installing guzzlehttp/promises (1.0.3)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Removing guzzlehttp/psr7 (1.2.0)
  - Installing guzzlehttp/psr7 (1.2.1)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Removing fabpot/goutte (v3.1.1)
  - Installing fabpot/goutte (v3.1.2)
    Loading from cache

Hmm, I wonder if we even need composer/installers anymore...

It appears that the patch in #160 contains updates to Twig as well, even though that is in a different issue.

andypost,

You'll probably need to specify each package you want to update:
https://getcomposer.org/doc/03-cli.md#update

@davidwbarratt suppose the issue is about all libraries and twig already commited, so we need to update all bundled

I'd say that at least for the sake of git history, it is better to update components in their own issue. Minor updates are fine though, I guess. #2608174: Update composer-merge-plugin to latest version is close to RTBC and assuming that gets in first, this patch would work on top of that.

I got the same patch as #167

@Oleksiy: Thanks for the patch. However, this patch also contains wikimedia/composer-merge-plugin which is handled in another issue. This is just FYI.

So here's a patch without wikimedia/composer-merge-plugin #2608174: Update composer-merge-plugin to latest version
Interesting that size of patch without it bigger...

composer update composer/installers egulias/email-validator zendframework/zend-diactoros composer/semver sebastian/global-state phpunit/phpunit doctrine/cache doctrine/inflector zendframework/zend-stdlib guzzlehttp/promises guzzlehttp/psr7 fabpot/goutte 
Loading composer repositories with package information
Updating dependencies (including require-dev)
  - Removing composer/installers (v1.0.21)
  - Installing composer/installers (v1.0.22)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Removing egulias/email-validator (1.2.9)
  - Installing egulias/email-validator (1.2.11)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Removing zendframework/zend-diactoros (1.1.3)
  - Installing zendframework/zend-diactoros (1.1.4)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Removing composer/semver (1.0.0)
  - Installing composer/semver (1.2.0)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Removing sebastian/global-state (1.0.0)
  - Installing sebastian/global-state (1.1.1)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Removing phpunit/phpunit (4.8.11)
  - Installing phpunit/phpunit (4.8.18)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Removing doctrine/cache (v1.4.2)
  - Installing doctrine/cache (v1.5.1)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Removing doctrine/inflector (v1.0.1)
  - Installing doctrine/inflector (v1.1.0)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Removing zendframework/zend-stdlib (2.7.3)
  - Installing zendframework/zend-stdlib (2.7.4)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Removing guzzlehttp/promises (1.0.2)
  - Installing guzzlehttp/promises (1.0.3)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Removing guzzlehttp/psr7 (1.2.0)
  - Installing guzzlehttp/psr7 (1.2.1)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Removing fabpot/goutte (v3.1.1)
  - Installing fabpot/goutte (v3.1.2)
    Loading from cache

> Drupal\Core\Composer\Composer::vendorTestCodeCleanup

Landed #2608174: Update composer-merge-plugin to latest version

Here's an update of all the things

The folder vendor got removed from git, do we need to care about this issue? I think it is outdated now. #1475510: Remove external dependencies from the core repo and let Composer manage the dependencies instead

The patch will get smaller due to the removed vendor dir, but we still have a composer.lock, so we still need to update our dependencies from time to time, specifically right before a new minor release.