doctrine common 2.9 has moved reflection

@devitate instead of requiring the doctrine/common package you can take advantage of composer conflict and add to you composer.json file

  "conflict": {
    "doctrine/common":">2.8"
  }

Adding:

  "conflict": {
    "doctrine/common":">2.8"
  }

to my composer.json gets me this error:

  "conflict": {
    "doctrine/common":">2.8"
  }

,

I think this should be

  "conflict": {
    "doctrine/common":"2.8"
  }

Sorry, the error should have read:

> DrupalProject\composer\ScriptHandler::checkComposerVersion
Loading composer repositories with package information
Installing dependencies (including require-dev) from lock file
Warning: The lock file is not up to date with the latest changes in composer.json. You may be getting outdated dependencies. Run update to update them.
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - drupal-composer/drupal-project dev-event-content-158520905 conflicts with doctrine/common[v2.9.0].
    - doctrine/common v2.9.0 conflicts with drupal-composer/drupal-project[dev-event-content-158520905].
    - Installation request for drupal-composer/drupal-project dev-event-content-158520905 -> satisfiable by drupal-composer/drupal-project[dev-event-content-158520905].
    - Installation request for doctrine/common v2.9.0 -> satisfiable by doctrine/common[v2.9.0].

Using @jmolivas approach worked for me:

  "conflict": {
    "doctrine/common":">2.8"
  }

Given that this bug renders a composer-based install unusable, I would recommend we bump up the priority to critical.

Running a regular composer update can lead to an error like Drupal Fatal error: require(): Failed opening required '/app/vendor/composer/../doctrine/common/lib/Doctrine/Common/Reflection/ClassFinderInterface.php' [...].

Running composer update also got me the error. It updated doctrine/common from v2.8.1 to v2.9.0. @jmolivas approached worked for me.

#3 Solution worked for me.

1. Modified the composer.json file adding the "doctrine/common": "2.8" line in conflict section.
2. Ran composer update after saving the file.

I had to use '2.8' specifically per #5 to get this working

Exact 2.8 did not work for me as the error continued.

"conflict": {
  "doctrine/common": "2.8"
}

The great than (>) syntax did the trick. After running composer update, the output showed - Updating doctrine/common (v2.9.0 => v2.8.0): Loading from cache

"conflict": {
  "doctrine/common": ">2.8"
}

I had this error:

Fatal error: require(): Failed opening required '/var/www/html/vendor/composer/../doctrine/common/lib/Doctrine/Common/Reflection/ClassFinderInterface.php' (include_path='.:/usr/local/lib/php') in /var/www/html/princeton/vendor/symfony/class-loader/ApcClassLoader.php on line 112

After running a composer update and getting that new version of Reflection library.

Since it was an APC error, I just restarted PHP to clear its cache and the error went away. So at least in my case this was just a caching issue.

If this is just an APC classloader issue I don't think it requires any special handling.

">2.8" worked for me.

"conflict": {
  "doctrine/common": ">2.8"
}

As long as the doctrine/reflection is installed and your cache is cleared, this really shouldn't cause any problems. (Works fine on multiple hosts/multiple sites for me)

Since at least this isn't eating anyone's data and would hardly surprise anyone who stages updates, can we go ahead and downgrade this?

Personally, I'd close it as Won't fix or Works as Designed. Needing to clear caches isn't a good reason to hold that version back especially with 8.6 coming out I think.

Yes I'm going to close this as by-design.

Over in #2976407-25: Use drupalci.yml for composer dependency min/max testing - DO NOT COMMIT we see that this incompatibility does, in fact, break Drupal under reasonable circumstances. Here's the CI result where this is visible: https://www.drupal.org/pift-ci-job/1147811 In that CI run, the testbot says composer update, which results in doctrine/common 2.10 being installed and breaking the site.

Reducing this to normal since if it were really critical it would have been re-opened a long time ago.

This patch limits the version constraints for doctrine/common. Core is trying to figure out whether and how to drop Doctrine use altogether, so this is probably a better solution than trying to make core compatible with doctrine/common 2.9.

This seems like a good reflection (see what I did there) of reality. We can open a follow-up for Doctrine 2.9 compatibility.

I think we should update core/lib/Drupal/Component/Annotation/composer.json and core/lib/Drupal/Component/ClassFinder/composer.json too as both of these use Doctrine\Common\Reflection\ClassFinderInterface which is the class that has moved.

Good catch.

Crediting @devitate for opening the issue and myself for a review that affected the final patch.

Committed and pushed 994e7ec786 to 8.7.x and 12173dcf79 to 8.6.x. Thanks!

Backported to 8.6.x because it is a bug there too.

Has the follow-up for Doctrine 2.9 compatibility been opened?

IMHO the assumptions on the basis of which this change has been made are wrong. My suggestion is to revert this change.

1) The patch here proves how, reverting the changes and forcing an update of doctrine/common to 2.10, the entire test suite still runs successfully.

2) In #2976407: Use drupalci.yml for composer dependency min/max testing - DO NOT COMMIT, it looks like after composer install we run a composer update --prefer-lowest, then run the tests, then we run a composer update to load latest dependencies, the run tests again. Some how it looks like Composer autoload map remains set at the lowest dependencies directories... do not know why. But that's the reason for the fail there: ClassFinderInterface will no longer be at /var/www/html/vendor/composer/../doctrine/common/lib/Doctrine/Common/Reflection/ClassFinderInterface.php but rather at
/var/www/html/vendor/composer/../doctrine/reflection/lib/Doctrine/Common/Reflection/ClassFinderInterface.php.

Patch is test-only, not for commit.

Some how it looks like Composer autoload map remains set at the lowest dependencies directories... do not know why.

In order to read the results of #2976407: Use drupalci.yml for composer dependency min/max testing - DO NOT COMMIT you really have to look at the console log, and not the normal results you might be used to from DrupalCI testing.

So let's look at comment #28 over there, since it doesn't change the upper constraint for doctrine/common: #2976407-28: Use drupalci.yml for composer dependency min/max testing - DO NOT COMMIT

If you click through to the full console log at https://dispatcher.drupalci.org/job/drupal_patches/79589/consoleFull

...you'll see that there are a number of times when composer regenerates the autoloader for us.

First we install from core, then DrupalCI re-runs composer because we patched composer.json, then it runs compser run-scripts drupal-update-phpunit. That's all the initial set-up that DrupalCI always does.

We do an install because otherwise we can't perform an update, due to our use of wikimedia/composer-merge-plugin.

So then, for the composer.min task, we do an update with --prefer-lowest, which regenerates the autoloader:

20:41:52   - Downgrading doctrine/common (v2.6.2 => v2.5.0): Downloading (100%)
[...]
20:42:16 Generating autoload files
20:42:16 > Drupal\Core\Composer\Composer::preAutoloadDump
20:42:16 > Drupal\Core\Composer\Composer::ensureHtaccess
20:42:17 ---------------- Finished composer.min in 41.359 seconds ---------------- 

Then we run the tests and some fail, but not because of doctrine/common:

20:42:44 Drupal\Tests\aggregator\Functional\DeleteFeedTest              1 passes 

Then for composer.max we run the composer update for the highest allowed version, and that also generates a new autoloader.

21:22:53   - Updating doctrine/common (v2.5.0 => v2.10.0): Downloading (100%)
[...]
21:23:02 Generating autoload files
21:23:02 > Drupal\Core\Composer\Composer::preAutoloadDump
21:23:03 > Drupal\Core\Composer\Composer::ensureHtaccess
21:23:03 ---------------- Finished composer.max in 31.239 seconds ---------------- 

But then the tests start failing because of doctrine/common:

21:23:51 Drupal\Tests\aggregator\Functional\DeleteFeedTest              0 passes             1 exceptions             
21:23:51 FATAL Drupal\Tests\aggregator\Functional\DeleteFeedTest: test runner returned a non-zero error code (2).

And from the test results page:

1) Drupal\Tests\aggregator\Functional\DeleteFeedTest::testDeleteFeed
Exception: Warning: require(/var/www/html/vendor/composer/../doctrine/common/lib/Doctrine/Common/Reflection/ClassFinderInterface.php): failed to open stream: No such file or directory
require()() (Line: 112)

This shows that the autoloader is always generated, and that some test fails are related to the max install but not the min install.

You can see that this is fixed in the backlog of #29 by limiting the upper constraints of doctrine/common: https://www.drupal.org/pift-ci-job/1147833 DeleteFeedTest is no longer failing there, either for max or min. (The current test run doesn't apply, but some easy sleuthing shows you the past test run.)

I just added #2976407-32: Use drupalci.yml for composer dependency min/max testing - DO NOT COMMIT It's a reroll. I will predict won't have the doctrine/common related fails, which will then show that changing the upper constraint in this issue also fixed those fails.

#29 yes, and if you look at the console log of the job run from the patch @ #28, you will see that doctrine/common 2.10 is loaded as part of the PHPUnit update, along with doctrine/reflection 1.0 and others

12:14:05   - Installing doctrine/reflection (v1.0.0): Downloading (100%)
12:14:05 > Drupal\Core\Composer\Composer::vendorTestCodeCleanup
12:14:06   - Installing doctrine/event-manager (v1.0.0): Downloading (100%)
12:14:06 > Drupal\Core\Composer\Composer::vendorTestCodeCleanup
12:14:06   - Updating doctrine/collections (v1.3.0 => v1.5.0): Loading from cache
12:14:06 > Drupal\Core\Composer\Composer::vendorTestCodeCleanup
12:14:06   - Updating doctrine/cache (v1.6.2 => v1.8.0): Downloading (100%)
12:14:06 > Drupal\Core\Composer\Composer::vendorTestCodeCleanup
12:14:07   - Installing doctrine/persistence (v1.1.0): Downloading (100%)
12:14:07 > Drupal\Core\Composer\Composer::vendorTestCodeCleanup
12:14:07   - Updating doctrine/inflector (v1.1.0 => v1.3.0): Loading from cache
12:14:07 > Drupal\Core\Composer\Composer::vendorTestCodeCleanup
12:14:07   - Updating doctrine/common (v2.6.2 => v2.10.0): Downloading (100%)
12:14:08 > Drupal\Core\Composer\Composer::vendorTestCodeCleanup

and test run successfully. Out of curiosity I also tried to change the PHPUnit update to a general @composer update, that practically executes a build with highest dependencies. It also runs fine with the exception of the page cache test that also fails in #2976407-32: Use drupalci.yml for composer dependency min/max testing - DO NOT COMMIT. See in practice #3015974-14: Ignore, testing issue.

So IMO:
a) doctrine/common 2.10 is compatible with Drupal if install is performed on a code build that already is updated with higest dependencies;
b) Drupal breaks when updating to highest dependencies from an installed Drupal with code built with doctrine/common 2.6, which is the case reported in the IS here and likely what @Mile23 is hitting at #2976407-28: Use drupalci.yml for composer dependency min/max testing - DO NOT COMMIT.

I still think we should revert the change made here.

EDIT: earlier comments indicate caching may affect the thing - can it be the case for #2976407-28: Use drupalci.yml for composer dependency min/max testing - DO NOT COMMIT failures?

if you look at the console log of the job run from the patch @ #28, you will see that doctrine/common 2.10 is loaded as part of the PHPUnit update

Aha, so we get this:

11:14:05   - Installing doctrine/reflection (v1.0.0): Downloading (100%)

Going back to our fails from the full update, hey look, it's a reflection class causing the error:

1) Drupal\Tests\aggregator\Functional\AggregatorAdminTest::testSettingsPage
Exception: Warning: require(/var/www/html/vendor/composer/../doctrine/common/lib/Doctrine/Common/Reflection/ClassFinderInterface.php): failed to open stream: No such file or directory

And here's doctrine/common with release notes for 2.9.0: https://github.com/doctrine/common/releases/tag/v2.9.0

Starting with this release, some of the components were moved to separate packages:

• Event Manager
• Persistence
• Reflection

So for some reason, during the min/max test Composer does indeed fail to account for class loading the new reflection library. Or else our code uses a deprecated namespace, though that would lead to fails in #28 here.

#28 proves that we could revert the changes here, because no one's min/max installing their Drupal site. But if they are, they should really speak up on #2976407: Use drupalci.yml for composer dependency min/max testing - DO NOT COMMIT :-)

So for some reason, during the min/max test Composer does indeed fail to account for class loading the new reflection library.

That's because, even if the namespace remains the same, the files are moved, i.e. from

/var/www/html/vendor/composer/../doctrine/common/lib/Doctrine/Common/Reflection/ClassFinderInterface.php

to

/var/www/html/vendor/composer/../doctrine/reflection/lib/Doctrine/Common/Reflection/ClassFinderInterface.php

If there's an opcode cache, it's likely that PHP still looks for the old location after the composer update. So that would need to be cleared after the composer update and before running tests with highest dependencies. But that's for the other issue.

As per @Mile23 and @mondrake I've reverted this. Thanks for all the investigation. I'm not sure what the status of this issue should be. I think given we are dealing with normal operation - i.e you have to clear APC caches after changing the classloader then I'm going towards the original "closed works as designed" as per #18.

@bkosborne thanks restarting php-fpm did the trick.
service php-fpm restart

@bkosborne thanks restarting php-fpm did the trick for me too.

I'm in local path, so restarting Apache and it's work.

#12 saved my day!!!

I just restarted PHP!

Thanks

#12 worked for me too, I was working on my local machine so I just typed the following command into terminal:

sudo apachectl -k restart

Thanks, @bkosborne. I just restarted the environment as described in #12 and the project works as expected.