Impossible to write unit tests involving Vocabulary, because TAXONOMY_HIERARCHY_(DISABLED|SINGLE|MULTIPLE) are defined in taxonomy.module instead of VocabularyInterface

You need to require_once taxonomy.module in the setUp() method of your unit test instead.

That is what I'm doing.

And @tim.plunkett suggested I do

  public static function setUpBeforeClass() {
    parent::setUpBeforeClass();
    require_once static::getDrupalRoot() . '/core/modules/taxonomy/taxonomy.module';
  }

just like in http://cgit.drupalcode.org/drupal/diff/core/modules/block/tests/src/Kern....

But that also doesn't work; it results in the same errors as above.

I'm starting to suspect somehow the test is resulting in a separate PHP environment. I don't know the internals of BrowserTestBase well enough to know that for sure though, but that's the only explanation I can think of atm.

Continuing to debug this…

I did lots more digging.

This was hinting in the direction of PHPUnit:

Exception: Notice: Use of undefined constant TAXONOMY_HIERARCHY_DISABLED - assumed 'TAXONOMY_HIERARCHY_DISABLED'
Drupal\Core\Cache\DatabaseBackend->prepareItem()() (Line: 138)


/Users/wim.leers/Work/drupal-tres/core/lib/Drupal/Core/Test/HttpClientMiddleware/TestHttpClientMiddleware.php:44
/Users/wim.leers/Work/drupal-tres/vendor/guzzlehttp/promises/src/FulfilledPromise.php:39
/Users/wim.leers/Work/drupal-tres/vendor/guzzlehttp/promises/src/TaskQueue.php:61
/Users/wim.leers/Work/drupal-tres/vendor/guzzlehttp/promises/src/Promise.php:246
/Users/wim.leers/Work/drupal-tres/vendor/guzzlehttp/promises/src/Promise.php:223
/Users/wim.leers/Work/drupal-tres/vendor/guzzlehttp/promises/src/Promise.php:266
/Users/wim.leers/Work/drupal-tres/vendor/guzzlehttp/promises/src/Promise.php:225
/Users/wim.leers/Work/drupal-tres/vendor/guzzlehttp/promises/src/Promise.php:62
/Users/wim.leers/Work/drupal-tres/vendor/guzzlehttp/guzzle/src/Client.php:129
/Users/wim.leers/Work/drupal-tres/vendor/guzzlehttp/guzzle/src/Client.php:87
/Users/wim.leers/Work/drupal-tres/core/modules/rest/tests/src/Functional/EntityResource/EntityResourceTestBase.php:127
/Users/wim.leers/Work/drupal-tres/core/modules/rest/tests/src/Functional/EntityResource/EntityResourceTestBase.php:127

So I looked at TestHttpClientMiddleware.php:44. It's doing this:

                    throw new \Exception($parameters[1] . ': ' . $parameters[0] . "\n" . Error::formatBacktrace([$parameters[2]]));

This is where that output is coming from (Exception: Notice: Use of undefined constant TAXONOMY_HIERARCHY_DISABLED - assumed 'TAXONOMY_HIERARCHY_DISABLED' […])!

So this is dealing with the case of there being a fatal error, or a notice — or in other words: PHP engine output.

By putting different code there, to return early, reveals that it's PHPUnit's Php.php, line 107, that is sending this back in the first place. This lead me to the following PHPUnit issues, which were simply closed without being solved:

1. https://github.com/sebastianbergmann/phpunit/issues/1571
2. https://github.com/sebastianbergmann/phpunit/issues/314

The latter was even commented on in great detail by our very own sun!

Both of these PHPUnit issues indicate that the problem is related to global state being preserved or not. BrowserTestBase does this:

  protected $runTestInSeparateProcess = TRUE;
  protected $preserveGlobalState = FALSE;

This also explains why the recommendations by klausi & tim.plunkett don't have the desired effect.

Still no solution yet, continuing my debugging…

I've found the root cause. You won't believe this. This is both hilarious and sad. But it's definitely epic.

Stay tuned for the full explanation.

Debug process (a successful one, after many attempts)

I personally found it very very tricky to debug this PHP notice at Drupal\Core\Cache\DatabaseBackend->prepareItem()() (Line: 138), because it was happening inside a request made by Guzzle, which made it impossible for me to step through it with a debugger. Worse, it was also very difficult to be able to add meaningful debug output, because it was happening in a place that got many dozens of function calls, and I couldn't figure out which call was the problem.

In the end, I settled on this:

Original code

      $cache->data = unserialize($cache->data);

Modified code (for debugging)

      set_error_handler(function ($errno, $errstr, $errfile, $errline) {
        throw new \ErrorException($errstr, $errno, $errno, $errfile, $errline);
      });
      try {
        $cache->data = unserialize($cache->data);
      }
      catch (\ErrorException $e) {
        throw new \Exception('When unserializing the data for cache item "' . $cache->cid . "\", this problem occurred: \n" . $e->getMessage() . "\n\nStored data:" . $cache->data);
      }
      restore_error_handler();

Pattern

set_error_handler(/* function that converts everything, including notices, to ErrorExceptions */)
PROBLEMATIC CODE
restore_error_handler()

This allowed me to see what exactly was happening, with useful debug output:

Before

1) Drupal\Tests\rest\Functional\EntityResource\Vocabulary\VocabularyJsonAnonTest::testGet
Exception: Notice: Use of undefined constant TAXONOMY_HIERARCHY_DISABLED - assumed 'TAXONOMY_HIERARCHY_DISABLED'
Drupal\Core\Cache\DatabaseBackend->prepareItem()() (Line: 138)


/Users/wim.leers/Work/drupal-tres/core/lib/Drupal/Core/Test/HttpClientMiddleware/TestHttpClientMiddleware.php:44
/Users/wim.leers/Work/drupal-tres/vendor/guzzlehttp/promises/src/FulfilledPromise.php:39
/Users/wim.leers/Work/drupal-tres/vendor/guzzlehttp/promises/src/TaskQueue.php:61
/Users/wim.leers/Work/drupal-tres/vendor/guzzlehttp/promises/src/Promise.php:246
/Users/wim.leers/Work/drupal-tres/vendor/guzzlehttp/promises/src/Promise.php:223
/Users/wim.leers/Work/drupal-tres/vendor/guzzlehttp/promises/src/Promise.php:266
/Users/wim.leers/Work/drupal-tres/vendor/guzzlehttp/promises/src/Promise.php:225
/Users/wim.leers/Work/drupal-tres/vendor/guzzlehttp/promises/src/Promise.php:62
/Users/wim.leers/Work/drupal-tres/vendor/guzzlehttp/guzzle/src/Client.php:129
/Users/wim.leers/Work/drupal-tres/vendor/guzzlehttp/guzzle/src/Client.php:87
/Users/wim.leers/Work/drupal-tres/core/modules/rest/tests/src/Functional/EntityResource/EntityResourceTestBase.php:127
/Users/wim.leers/Work/drupal-tres/core/modules/rest/tests/src/Functional/EntityResource/EntityResourceTestBase.php:127

After

1) Drupal\Tests\rest\Functional\EntityResource\Vocabulary\VocabularyJsonAnonTest::testGet
Exception: Exception: When unserializing the data for cache item "http://localhost/tres/entity/taxonomy_vocabulary/camelids?_format=json:json", this problem occurred: 
Use of undefined constant TAXONOMY_HIERARCHY_DISABLED - assumed 'TAXONOMY_HIERARCHY_DISABLED'

Stored data:O:28:"Drupal\rest\ResourceResponse":8:{s:7:"headers";O:50:"Symfony\Component\HttpFoundation\ResponseHeaderBag":5:{s:23:"*computedCacheControl";a:3:{s:15:"must-revalidate";b:1;s:8:"no-cache";b:1;s:7:"private";b:1;}s:10:"*cookies";a:0:{}s:14:"*headerNames";a:12:{s:13:"cache-control";s:13:"Cache-Control";s:12:"content-type";s:12:"Content-Type";s:22:"x-drupal-dynamic-cache";s:22:"X-Drupal-Dynamic-Cache";s:15:"x-ua-compatible";s:15:"X-UA-Compatible";s:16:"content-language";s:16:"Content-language";s:22:"x-content-type-options";s:22:"X-Content-Type-Options";s:15:"x-frame-options";s:15:"X-Frame-Options";s:19:"x-drupal-cache-tags";s:19:"X-Drupal-Cache-Tags";s:23:"x-drupal-cache-contexts";s:23:"X-Drupal-Cache-Contexts";s:7:"expires";s:7:"Expires";s:4:"vary";s:4:"Vary";s:11:"x-generator";s:11:"X-Generator";}s:10:"*headers";a:12:{s:13:"cache-control";a:1:{i:0;s:34:"must-revalidate, no-cache, private";}s:12:"content-type";a:1:{i:0;s:16:"application/json";}s:22:"x-drupal-dynamic-cache";a:1:{i:0;s:4:"MISS";}s:15:"x-ua-compatible";a:1:{i:0;s:7:"IE=edge";}s:16:"content-language";a:1:{i:0;s:2:"en";}s:22:"x-content-type-options";a:1:{i:0;s:7:"nosniff";}s:15:"x-frame-options";a:1:{i:0;s:10:"SAMEORIGIN";}s:19:"x-drupal-cache-tags";a:1:{i:0;s:110:"config:rest.resource.entity.taxonomy_vocabulary config:taxonomy.vocabulary.camelids config:user.role.anonymous";}s:23:"x-drupal-cache-contexts";a:1:{i:0;s:16:"user.permissions";}s:7:"expires";a:1:{i:0;s:29:"Sun, 19 Nov 1978 05:00:00 GMT";}s:4:"vary";a:0:{}s:11:"x-generator";a:1:{i:0;s:33:"Drupal 8 (https://www.drupal.org)";}}s:15:"*cacheControl";a:2:{s:15:"must-revalidate";b:1;s:8:"no-cache";b:1;}}s:10:"*content";s:174:"{"uuid":"ae28bfe4-d015-47e9-b179-bc4bdae041de","langcode":"en","status":true,"dependencies":[],"name":"Camelids","vid":"camelids","description":null,"hierarchy":0,"weight":0}";s:10:"*version";s:3:"1.0";s:13:"*statusCode";i:200;s:13:"*statusText";s:2:"OK";s:10:"*charset";N;s:23:"*cacheabilityMetadata";O:35:"Drupal\Core\Cache\CacheableMetadata":3:{s:16:"*cacheContexts";a:1:{i:0;s:16:"user.permissions";}s:12:"*cacheTags";a:3:{i:0;s:47:"config:rest.resource.entity.taxonomy_vocabulary";i:1;s:35:"config:taxonomy.vocabulary.camelids";i:2;s:26:"config:user.role.anonymous";}s:14:"*cacheMaxAge";i:-1;}s:15:"*responseData";O:33:"Drupal\taxonomy\Entity\Vocabulary":20:{s:6:"*vid";s:8:"camelids";s:7:"*name";s:8:"Camelids";s:14:"*description";N;s:12:"*hierarchy";i:0;s:9:"*weight";i:0;s:13:"*originalId";s:8:"camelids";s:9:"*status";b:1;s:7:"*uuid";s:36:"ae28bfe4-d015-47e9-b179-bc4bdae041de";s:11:"*langcode";s:2:"en";s:23:"*third_party_settings";a:0:{}s:8:"*_core";a:0:{}s:14:"*trustedData";b:0;s:15:"*entityTypeId";s:19:"taxonomy_vocabulary";s:15:"*enforceIsNew";N;s:12:"*typedData";N;s:16:"*cacheContexts";a:0:{}s:12:"*cacheTags";a:0:{}s:14:"*cacheMaxAge";i:-1;s:14:"*_serviceIds";a:0:{}s:15:"*dependencies";a:0:{}}}
Drupal\Core\Cache\DatabaseBackend->prepareItem()() (Line: 155)


/Users/wim.leers/Work/drupal-tres/core/lib/Drupal/Core/Test/HttpClientMiddleware/TestHttpClientMiddleware.php:51
/Users/wim.leers/Work/drupal-tres/vendor/guzzlehttp/promises/src/FulfilledPromise.php:39
/Users/wim.leers/Work/drupal-tres/vendor/guzzlehttp/promises/src/TaskQueue.php:61
/Users/wim.leers/Work/drupal-tres/vendor/guzzlehttp/promises/src/Promise.php:246
/Users/wim.leers/Work/drupal-tres/vendor/guzzlehttp/promises/src/Promise.php:223
/Users/wim.leers/Work/drupal-tres/vendor/guzzlehttp/promises/src/Promise.php:266
/Users/wim.leers/Work/drupal-tres/vendor/guzzlehttp/promises/src/Promise.php:225
/Users/wim.leers/Work/drupal-tres/vendor/guzzlehttp/promises/src/Promise.php:62
/Users/wim.leers/Work/drupal-tres/vendor/guzzlehttp/guzzle/src/Client.php:129
/Users/wim.leers/Work/drupal-tres/vendor/guzzlehttp/guzzle/src/Client.php:87
/Users/wim.leers/Work/drupal-tres/core/modules/rest/tests/src/Functional/EntityResource/EntityResourceTestBase.php:127
/Users/wim.leers/Work/drupal-tres/core/modules/rest/tests/src/Functional/EntityResource/EntityResourceTestBase.php:127

Analysis

So, unserialization of a Vocabulary entity is occurring and failing not inside the test, but as part of a Page Cache hit! ResourceResponse apparently contains a serialized Vocabulary entity, and it cannot be unserialized, so BAM, everything explodes.

The root cause is simple: the PageCache middleware runs before the KernelPreHandle middleware. The latter is what loads modules. So: taxonomy.module has not yet been loaded by the time that we try to unserialize a serialized, cached Vocabulary entity that uses a constant defined in taxonomy.module.

Solution: still move TAXONOMY_HIERARCHY_* constants to VocabularyInterface, but keep the existing constants, and keep them in sync. This ensures BC is not broken for code that is doing things like if ($vocabulary->getHierarchy() === TAXONOMY_HIERARCHY_*. But then all other code must use VocabularyInterface::HIERARCHY_*.

To convince yourself this works, try this:

<?php

const TAXONOMY_HIERARCHY_DISABLED = 0;

interface VocabularyInterface {
  const HIERARCHY_DISABLED = 0;
}

var_dump(TAXONOMY_HIERARCHY_DISABLED === VocabularyInterface::HIERARCHY_DISABLED);

See https://3v4l.org/K3LRM to see the results of that on all PHP versions. It works.

Performance

Coincidentally, this revealed that ResourceResponse still contains the data it must serialize! And so, whenever (Dynamic) Page Cache retrieves its cache items, the cache system unserializes the response, and so it must also unserialize ResourceResponse::$responseData.

Fixing this leads to a 60% performance boost (for anonymous REST responses). See #2807501: ResourceResponse::$responseData is serialized, but is never used: unserializing it on every Page Cache hit is a huge performance penalty.

Patch

Attached patch includes test coverage that shows the problem.

And here's the fix.

catch independently proposed the same solution yesterday in IRC:

16:10:00 <catch> WimLeers: duplicate the constants to vocabularyinterface and deprecate the old ones?

Moving to 8.2.x, because this is breaking REST and blocking #2737719: EntityResource: Provide comprehensive test coverage: for every entity type, every format, every method, which is necessary for REST to become truly stable/reliable in 8.2.x.

Thanks! Follow-up issue created: #2807785: Move global constants from *.module files into interfaces.

Addressed everything in #14. Thanks for responseHeaderEquals(), I didn't know that one!

CR created: https://www.drupal.org/node/2807795.

#20: Oh, I like that!

#21: that's the same as what #20 says, right?

Much better. Thanks! :)

This must be ported to 8.2.x. It's a hard blocker for #2737719: EntityResource: Provide comprehensive test coverage: for every entity type, every format, every method, which we need to make REST as stable/reliable as possible in 8.2.x. Alex and catch have been marking those issues that still need to be committed to 8.2.x after 8.2.0 as "patch to be ported" and assigned to 8.2.x. So doing the same here.

Please don't yet publish the CR, until we know in which version of 8.2.x this lands, so that we don't need to retroactively update it after publishing.

Yep, they're just deprecated, but they'll continue to work just fine.

Published https://www.drupal.org/node/2807795.

Similar problem is now being seen elsewhere: #2825801: Node type objects (and other objects that depend on *.module files) cannot be part of response objects due to Page Cache: Document or fix!.