It's very difficult to manage our composer.json file and third-party vendor code...

Problem/Motivation

  1. Our third-party vendor code needs some updates (Symfony, Assetic, PHPUnit, etc)
  2. Whenever we update a vendor, we need to update composer.json for each individual package that needs updates. Requires micro-management of each version string, which is very cumbersome.
  3. Whenever we interact with Composer, you need to append --prefer-dist so that it downloads the package rather than checks it out from git. That's a pain.

Proposed resolution

  1. Update to the latest minor version of each library we use
  2. Lock to specific versions in composer.lock rather than composer.json
  3. Use package versions like 2.2.* rather than a specific version. This will target the latest in the 2.2.x life cycle.
  4. When we need an alpha, or beta release, target it with @alpha or @beta, this allows the removal of minimum-stability and specific version targeting
  5. Update .gitignore with a new file paths, and make sure to also ignore any binary data we don't need
  6. Use preferred-install in composer.json to instruct Composer to retrieve distributed release packages rather than git source checkouts
  7. Move core/vendor/.gitignore to core/.gitignore so that it's possible to completely remove core/vendor and run another composer install to re-download the packages if needed

Remaining tasks

The usual review process.

User interface changes

No user interface changes.

API changes

Updating an individual vendor

Before
  1. Edit composer.json with the new version number
  2. $ composer update symfony/yaml --prefer-dist
After

$ composer update symfony/yaml

Re-installing all vendors

Before
$ mv vendor/.gitignore .gitignore
$ rm -rf vendor
$ composer install --prefer-dist
$ mv .gitignore vendor/.gitignore
After
$ rm -rf vendor
$ composer install

Update all vendor code

Before

Edit composer.json and figure out the new version of each individual package. Have fun.

After

$ composer update

CommentFileSizeAuthor
#18 1977570.patch917.06 KBRobLoach
#5 1977570.patch903.92 KBRobLoach
#3 1977570.patch897.27 KBRobLoach
vendor-code.patch851.08 KBRobLoach
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

RobLoach’s picture

RobLoach
he/him
CreditAttribution: RobLoach commented
Issue tags: +WSCCI 
+++ b/composer.jsonundefined
@@ -4,22 +4,22 @@
+    "doctrine/common": "2.3.*",
+    "guzzle/http": "3.1.*",
+    "kriswallsmith/assetic": "1.1.*@alpha",
     "symfony-cmf/routing": "dev-master#ea4a10",
-    "easyrdf/easyrdf": "0.8.0-beta.1",
-    "phpunit/phpunit": "3.7.15"

Tagging WSCCI so that the WSCCI crew can provide a bit more information as to why we need dev-master#ea4a10 here. Should we work with symfony-cmf/routing to get out a 1.1.0-alpha.1 release?

Crell’s picture

Crell CreditAttribution: Crell commented

When we pulled in Symfony-cmf/Routing, it had no usable tagged release. I thought they were going to tag a release a while ago, but looks like there isn't. Unless it's just the 1.0.0-beta1 that came out 2 weeks ago, which we should then update to. I'll try to bug dbu or lsmith for confirmation.

RobLoach’s picture

RobLoach
he/him
CreditAttribution: RobLoach commented
FileSize
1977570.patch897.27 KB

Here it is with 1.0.*, which resolves to 1.0.0, which was released two weeks ago.

ParisLiakos’s picture

ParisLiakos CreditAttribution: ParisLiakos commented

i think we need 1.1 now for routing since the pull request https://github.com/symfony-cmf/Routing/pull/53 just got merged, so that we can move on here
#1289536: Switch Watchdog to a PSR-3 logging framework

RobLoach’s picture

RobLoach
he/him
CreditAttribution: RobLoach commented
FileSize
1977570.patch903.92 KB

I ran an update and 1.0.1 came down the pipeline. There isn't a 1.1 yet:

$ drush composer update
Loading composer repositories with package information
Updating dependencies
  - Removing symfony-cmf/routing (1.0.0)
  - Installing symfony-cmf/routing (1.0.1)
    Downloading: 100%         

Writing lock file
Generating autoload files

This puts us at:

$ drush composer show --installed
doctrine/common              2.3.0         Common Library for Doctrine projects
easyrdf/easyrdf              0.8.0-beta.1  EasyRdf is a PHP library designed to make it easy to consume and produce RDF.
guzzle/common                v3.1.2        Common libraries used by Guzzle
guzzle/http                  v3.1.2        HTTP libraries used by Guzzle
guzzle/parser                v3.1.2        Interchangeable parsers used by Guzzle
guzzle/stream                v3.1.2        Guzzle stream wrapper component
kriswallsmith/assetic        v1.1.0-alpha4 Asset Management for PHP
phpunit/php-code-coverage    1.2.9         Library that provides collection, processing, and rendering functionality for PHP code coverage information.
phpunit/php-file-iterator    1.3.3         FilterIterator implementation that filters files based on a list of suffixes.
phpunit/php-text-template    1.1.4         Simple template engine.
phpunit/php-timer            1.0.4         Utility class for timing
phpunit/php-token-stream     1.1.5         Wrapper around PHP's tokenizer extension.
phpunit/phpunit              3.7.19        The PHP Unit Testing framework.
phpunit/phpunit-mock-objects 1.2.3         Mock Object library for PHPUnit
psr/log                      1.0.0         Common interface for logging libraries
symfony-cmf/routing          1.0.1         Extends the Symfony2 routing component for dynamic routes and chaining several routers
symfony/class-loader         v2.2.1        Symfony ClassLoader Component
symfony/dependency-injection v2.2.1        Symfony DependencyInjection Component
symfony/event-dispatcher     v2.2.1        Symfony EventDispatcher Component
symfony/http-foundation      v2.2.1        Symfony HttpFoundation Component
symfony/http-kernel          v2.2.1        Symfony HttpKernel Component
symfony/process              v2.2.1        Symfony Process Component
symfony/routing              v2.2.1        Symfony Routing Component
symfony/serializer           v2.2.1        Symfony Serializer Component
symfony/translation          v2.2.1        Symfony Translation Component
symfony/validator            v2.2.1        Symfony Validator Component
symfony/yaml                 v2.2.1        Symfony Yaml Component
twig/twig                    v1.12.3       Twig, the flexible, fast, and secure template language for PHP
RobLoach’s picture

RobLoach
he/him
CreditAttribution: RobLoach commented
Priority: Normal » Major

Pushing to major as ParisLiakos mentioned there are patches being blocked.

ParisLiakos’s picture

ParisLiakos CreditAttribution: ParisLiakos commented

yeah, 1.0.1 is the backwards compatible version it still has the old logger interface. we need dev-master for latest commits:/ would that be a problem?

RobLoach’s picture

RobLoach
he/him
CreditAttribution: RobLoach commented

Which commit do we need that's not in 1.0.1? Could we ask them to issue a 1.0.2 or 1.1.0?

effulgentsia’s picture

effulgentsia CreditAttribution: effulgentsia commented

Use package versions like 2.2.* rather than a specific version.

We used to do that, but changed to specific versions in #1894002: Update vendor libraries and pin them to specific versions in composer.json. I left a comment there asking for reviews here.

Pushing to major as ParisLiakos mentioned there are patches being blocked.

Can we split this into two issues then? One to get us up to date on our needed libraries, and a separate one for changing our approach of how we manage composer.json?

ParisLiakos’s picture

ParisLiakos CreditAttribution: ParisLiakos commented

i need this one
https://github.com/symfony-cmf/Routing/commit/bf35d4877f1c969c363544f51f...

the new 1.0.1 happened just before committing this, cause it breaks BC..from this commit and onwards master branch has switched to 1.1, no releases/tags yet, i guess its too soon, but we could ask

RobLoach’s picture

RobLoach
he/him
CreditAttribution: RobLoach commented

effulgentsia in #9

We used to do that, but changed to specific versions in #1894002: Update vendor libraries and pin them to specific versions in composer.json. I left a comment there asking for reviews here.

That's what composer.lock is for, not composer.json. That issue is a hack. If we want to update just one specific package, you run:

$ drush composer update symfony/yaml
RobLoach’s picture

RobLoach
he/him
CreditAttribution: RobLoach commented
Issue tags: -WSCCI, -Composer

#5: 1977570.patch queued for re-testing.

RobLoach’s picture

RobLoach
he/him
CreditAttribution: RobLoach commented
Issue tags: +WSCCI, +Composer

#5: 1977570.patch queued for re-testing.

ParisLiakos’s picture

ParisLiakos CreditAttribution: ParisLiakos commented

if with this way we cant get in dev versions of packages i would agree with #9 and fix composer.json to rely on versions after code freeze

RobLoach’s picture

RobLoach
he/him
CreditAttribution: RobLoach commented

If we need the dev version, then let's target with @dev until the commit is in a tag.

ParisLiakos’s picture

ParisLiakos CreditAttribution: ParisLiakos commented

ah, cause i will probably need dev version here as well
#1839468: [Followup] Replace aggregator rss parsing with Zend Feed
since the awesome guys in ZF just dropped quite a few dependencies

Soo reroll #5 with dev cmf-routing and move on?

dbu’s picture

dbu CreditAttribution: dbu commented

crell asked me if we could tag the symfony-cmf/routing component. i now just tagged 1.1.0-alpha1 which has the fix about psr0 logger interface.

RobLoach’s picture

RobLoach
he/him
CreditAttribution: RobLoach commented
FileSize
1977570.patch917.06 KB

Thanks a lot dbu, tags are much easier to track than commit hashes. Switched to 1.1.*@alpha, ran a composer update, and it brought in 1.1.0-alpha1.

$ drush composer show --installed
doctrine/common              2.3.0        
easyrdf/easyrdf              0.8.0-beta.1 
guzzle/common                v3.1.2       
guzzle/http                  v3.1.2       
guzzle/parser                v3.1.2       
guzzle/stream                v3.1.2       
kriswallsmith/assetic        v1.1.0-alpha4
phpunit/php-code-coverage    1.2.9        
phpunit/php-file-iterator    1.3.3        
phpunit/php-text-template    1.1.4        
phpunit/php-timer            1.0.4        
phpunit/php-token-stream     1.1.5        
phpunit/phpunit              3.7.19       
phpunit/phpunit-mock-objects 1.2.3        
psr/log                      1.0.0        
symfony-cmf/routing          1.1.0-alpha1 
symfony/class-loader         v2.2.1       
symfony/dependency-injection v2.2.1       
symfony/event-dispatcher     v2.2.1       
symfony/http-foundation      v2.2.1       
symfony/http-kernel          v2.2.1       
symfony/process              v2.2.1       
symfony/routing              v2.2.1       
symfony/serializer           v2.2.1       
symfony/translation          v2.2.1       
symfony/validator            v2.2.1       
symfony/yaml                 v2.2.1       
twig/twig                    v1.12.3

Symfony 2.3 update is here: #1989230: Update to Symfony 2.3

Crell’s picture

Crell CreditAttribution: Crell commented
Status: Needs review » Reviewed & tested by the community
ParisLiakos’s picture

ParisLiakos CreditAttribution: ParisLiakos commented

thanks dbu!
i can confirm the needed changes for #1289536: Switch Watchdog to a PSR-3 logging framework are there

alexpott’s picture

alexpott
he/they
Primary language English
Location 🇪🇺🌍
CreditAttribution: alexpott commented
Issue tags: -WSCCI, -Composer

#18: 1977570.patch queued for re-testing.

Status: Reviewed & tested by the community » Needs work

The last submitted patch, 1977570.patch, failed testing.

ParisLiakos’s picture

ParisLiakos CreditAttribution: ParisLiakos commented
Status: Needs work » Needs review
Issue tags: +WSCCI, +Composer

#18: 1977570.patch queued for re-testing.

ParisLiakos’s picture

ParisLiakos CreditAttribution: ParisLiakos commented
Status: Needs review » Reviewed & tested by the community

random failure indeed.the semaphore table

catch’s picture

catch
he/him
Primary language English
CreditAttribution: catch commented
Status: Reviewed & tested by the community » Fixed

Committed/pushed to 8.x, thanks!

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

cosmicdreams’s picture

cosmicdreams CreditAttribution: cosmicdreams commented

Related #2114829: Use stable versions of imported components

Now that Core and the components that composer brings in have evolved more. We no longer need to refer to alpha or beta builds for all but one of these libraries. We can be a bit more specific in the versions we bring in so that we do not accidentally bring in libraries that include unstable code.

Also, if we ship Drupal 8 with wildcards in composer.json, aren't we allowing a curious dev to update all these libraries at once? I think it's starting to make sense to be very specific about which libraries Drupal 8 is using and then update each library through Drupal's standard patch workflow whenever there is an update. It seems that our current strategy has worked for us thusfar, but could potentially be brittle if a library advances a new version and we're not ready for it.

cosmicdreams’s picture

cosmicdreams CreditAttribution: cosmicdreams commented
Issue summary: View changes

a