Problem/Motivation

There's a regression in D8 compared to the D7 router: if the router table is empty then D8 will silently fail routing instead of automatically running a rebuild). This resulted in a lot of manual rebuild calls in kernel tests because we didn't want to rebuild the router if it was not needed.

Proposed resolution

Add a kernel test only proxy to the router provider which triggers a rebuild automatically when something is called in the provider service. Since kernel tests do not persist anything this is the right thing to do.

Remaining tasks

User interface changes

API changes

Data model changes

Comments

chx created an issue. See original summary.

chx’s picture

Status: Active » Needs review
chx’s picture

Issue summary: View changes
chx’s picture

Title: The router is broken, chapter #462: routing silently fails in kernel tests » Routing silently fails in kernel tests
Issue summary: View changes

Status: Needs review » Needs work

The last submitted patch, route_build_on_demand.patch, failed testing.

chx’s picture

Issue summary: View changes
Status: Needs work » Needs review
StatusFileSize
new26.73 KB

Status: Needs review » Needs work

The last submitted patch, 6: 2605684_6.patch, failed testing.

dawehner’s picture

There's a regression in D8 compared to the D7 router:

It is not a regression, this was an active design decision, see #356399: Optimize the route rebuilding process to rebuild on write

Also, possibly move the router table out of the schema into the router builder.

We had an issue for that, see #2338747: Move {router} out of system.install and create the table lazy

The patch doesn't actually patch the current kernel test base though, see \Drupal\KernelTests\KernelTestBase::register

fgm’s picture

Just got bitten by this writing a test in a module using the KernelTestBase. As chx explained on IRC, such tests can solve the issue for now adding the following fragment, much like the patch does it:

$this->installSchema('system', 'router');
$this->container->get('router.builder')->rebuild();
chx’s picture

Status: Needs work » Needs review
StatusFileSize
new26.2 KB

> see \Drupal\KernelTests\KernelTestBase

I have nothing to do with those.

This should fix a few of them.

Status: Needs review » Needs work

The last submitted patch, 10: 2605684_10.patch, failed testing.

chx’s picture

Issue summary: View changes
StatusFileSize
new27.5 KB

About MenuLinkTreeTest. We hit RouteProvider::lazyLoadItself first when MenuLinkContent::postSave does $menu_link_manager->addDefinition($this->getPluginId(), $this->getPluginDefinition()); and that getPluginDefinition() call has a check on $url_object = $this->getUrlObject() which in turn leads to a routing request from Url::fromInternalUri.

During the router rebuild, MenuRouterRebuildSubscriber::menuLinksRebuild calls MenuLinkManager::rebuild which will call upon MenuTreeStorage::rebuild which saves all top links including our custom link in the tree storage. The link is now fully loadable and the addDefinition call which tries to load the link will promptly throw an exception. Commenting out the exception makes the test pass but it is obviously not the right thing to do.

A tentative fix is attached.

chx’s picture

Status: Needs work » Needs review

In short: checking the $update in postSave is wrong because it says nothing about the newness of the link as demonstrated. Instead, we can just check whether the definition exists and if it does then upgrade it.

dawehner’s picture

Given that my productivity increased a lot with the new test base, here is an issue to also improve that: #2605956: Port #2605684 to the new KernelTest

dawehner’s picture

In short: checking the $update in postSave is wrong because it says nothing about the newness of the link as demonstrated. Instead, we can just check whether the definition exists and if it does then upgrade it.

Sounds like a good fix!

dawehner’s picture

  1. +++ b/core/modules/menu_link_content/src/Entity/MenuLinkContent.php
    @@ -200,15 +200,16 @@ public function postSave(EntityStorageInterface $storage, $update = TRUE) {
         // The menu link can just be updated if there is already an menu link entry
         // on both entity and menu link plugin level.
    

    Mh, this comment now is not really exact anymore

  2. +++ b/core/modules/simpletest/src/TestServiceProvider.php
    @@ -25,4 +27,17 @@ function register(ContainerBuilder $container) {
    +      for ($id = 'router.route_provider'; $container->hasAlias($id); $id = (string) $container->getAlias($id));
    

    This for without a body is valid PHP? Sounds like a while loop for me, which would be IMHO more readable

  3. +++ b/core/modules/simpletest/src/TestServiceProvider.php
    @@ -25,4 +27,17 @@ function register(ContainerBuilder $container) {
    +      $container->setDefinition('router.route_provider', new Definition('Drupal\simpletest\RouteProvider'));
    

    Unneeded suggestion: You could use RouterProvider::class here

chx’s picture

Issue summary: View changes
StatusFileSize
new35.38 KB

This is about all the ones that needed to be removed I hope I have not overshot. Altogether some 49 removed and 51 remains. The followup will be more about DX than anything else, there are not many in kernelTNG tests to be removed.

Edit: next up I will do the table install from the new provider class as well without any schema change.

Status: Needs review » Needs work

The last submitted patch, 17: 2605684_16.patch, failed testing.

chx’s picture

Status: Needs work » Needs review
StatusFileSize
new42.36 KB
new21.25 KB

This version installs the router table on demand. There are more to be removed but I'm getting sleepy :)

Status: Needs review » Needs work

The last submitted patch, 19: 2605684_19.patch, failed testing.

chx’s picture

Status: Needs work » Needs review
StatusFileSize
new46.29 KB
new5.9 KB

Status: Needs review » Needs work

The last submitted patch, 21: 2605684_21.patch, failed testing.

chx’s picture

Status: Needs work » Needs review
StatusFileSize
new46.34 KB
chx’s picture

StatusFileSize
new53.98 KB

There's only one installSchema call left in DbDumpCommandTest but that's followed by an insert call so it can't be avoided. I didn't roll an interdiff as there's no meaningful change.

dawehner’s picture

This is clearly developer experience

+++ b/core/modules/simpletest/src/RouteProvider.php
@@ -0,0 +1,122 @@
+  public static function getSubscribedEvents() {
+    RouteProvider::getSubscribedEvents();
+  }

This would better have a return statement

chx’s picture

StatusFileSize
new53.22 KB
new5.08 KB

that's because it's not an event subscriber :) "simpletest.$original_id" is an event subscriber but this is not. I also folded some of the builder functionality into the semi-proxy class. Destruction is not needed. Let's hope this passes.

dawehner’s picture

Status: Needs review » Reviewed & tested by the community
Issue tags: +rc eligible

All tests pass so we are fine here. I hope we don't get bitten by the additional magic later, because well, this could be tricky for other people to debug.
On the other hand tests aren't part of any supported API.

The only problem I have with this is that this changes things from an explicit system to a every implicit system, which many people like, but I think most of the time implicitness
is problematic.

Given that these are test only patches, this is 'rc eligible'

chx’s picture

Status: Reviewed & tested by the community » Needs review
Issue tags: -rc eligible +rc target triage

No, there's a fix in MenuLinkContent and I pinged pwolanin over it.

dawehner’s picture

Status: Needs review » Reviewed & tested by the community

Let's talk about the change in MenuLinkContent ...

#2605684-12: Routing silently fails in kernel tests describes the problem pretty clear. The solution for this problem is really elegant as it removes the complexity of the code and by that remove the potential
edge cases.

chx’s picture

Thanks. I didn't realize you were a menu maintainer too :)

pwolanin’s picture

Status: Reviewed & tested by the community » Needs work
+++ b/core/modules/menu_link_content/src/Entity/MenuLinkContent.php
@@ -200,15 +200,16 @@ public function postSave(EntityStorageInterface $storage, $update = TRUE) {
-    if ($update && $menu_link_manager->getDefinition($this->getPluginId())) {
+    $definition = $this->getPluginDefinition();
+    if ($menu_link_manager->getDefinition($this->getPluginId(), FALSE)) {

It looks like you are throwing out the check on the $update flag?

chx’s picture

Status: Needs work » Reviewed & tested by the community

Yes. It is meaningless to do so because as described in #12 the link changes from "new" to "existing" by calling the plugin definition. Just check whether there's a definition.

chx’s picture

Status: Reviewed & tested by the community » Needs review

Opsie, sorry! Bad status, wanted NR.

dawehner’s picture

@pwolanin
So yeah $menu_link_manager->getDefinition($this->getPluginId(), FALSE) will return TRUE, if there is already an entry in the menu_tree and FALSE otherwise, which
is IMHO exactly what we want to check.

pwolanin’s picture

Ok, so it sounds like this just needs some code comments explaining that logic and why the flag is (should be) ignored?

chx’s picture

StatusFileSize
new53.62 KB

Added wall of text no other change

pwolanin’s picture

I looked quickly at the call chain but not enough to understand what's happening in detail. Something seems wrong there if we get that mysterious saving behavior, but at least this added comment makes it possible to understand the code.

chx’s picture

Let me attempt again.

What's a plugin definition for a menu link?

  public function getDefinition($plugin_id, $exception_on_invalid = TRUE) {
    $definition = $this->treeStorage->load($plugin_id);

the tree storage entry. Where do we save definitions aside from postSave? MenuTreeStorage::rebuild has this:

  public function rebuild(array $definitions) {
    if ($definitions) {
      foreach ($definitions as $id => $link) {
          $top_links[$id] = $id;
    foreach ($top_links as $id) {
      $this->saveRecursive($id, $children, $links);
    }

Where does $definitions come from? Why, it's the $definitions = $this->getDiscovery()->getDefinitions(); call in MenuLinkManager::rebuild which is , in turn, called from MenuRouterRebuildSubscriber::menuLinksRebuild

So then what definitions are there? Many, but for us the most important is MenuLinkContentDeriver.

So

  1. Router rebuild
  2. Links rebuild
  3. MenuLinkContentDeriver loads from entity storage
  4. Top level links saved into tree storage by MenuTreeStorage::rebuild
  5. postSave tries to insert the definition which which fails because the previous step the definition == tree storage entry already was saved.
chx’s picture

Following up on an IRC discussion: this is a testing artefact but there's nothing stopping a route provider to trigger a rebuildIfNeeded. One could even argue it's a bug not to do so but that's for a different issue.

dawehner’s picture

Status: Needs review » Reviewed & tested by the community

This is still RTBC

xjm’s picture

Status: Reviewed & tested by the community » Needs work
Issue tags: -rc target triage

Can we split the MenuLinkContent fix out into its own issue that blocks this one?

Both parts would potentially be eligible for a patch release, so @effulgentsia and I don't think this needs to be done during RC. If this issue does actually contain something disruptive that wouldn't be eligible for a patch release, let's add that to the issue summary.

chx’s picture

Status: Needs work » Needs review

I thought of it but you can't test it without this issue or not easily.

chx’s picture

StatusFileSize
new6.3 KB

This is the absolute minimal version of this patch: just MenuLinkContent, TestServiceProvider and RouteProvider. We can do the rest in a test only followup. This much is enough to trigger and fix the MLC bug.

chx’s picture

Issue summary: View changes
dawehner’s picture

Status: Needs review » Reviewed & tested by the community

I am still convinced about it, given that it makes things easier for people to write tests.

Status: Reviewed & tested by the community » Needs work

The last submitted patch, 43: 2605684_43.patch, failed testing.

chx’s picture

Status: Needs work » Reviewed & tested by the community

Bot fail.

catch’s picture

Status: Reviewed & tested by the community » Needs review
+++ b/core/modules/menu_link_content/src/Entity/MenuLinkContent.php
@@ -200,15 +200,22 @@ public function postSave(EntityStorageInterface $storage, $update = TRUE) {
+    $definition = $this->getPluginDefinition();
+    // Even when $update is FALSE, for top level links it is possible the link
+    // already is in the storage because of the getPluginDefinition() call
+    // above, see https://www.drupal.org/node/2605684#comment-10515450 for the
+    // call chain. Because of this the $update flag is ignored and only the
+    // existence of the definition (equals to being in the tree storage) is
+    // checked.
+    if ($menu_link_manager->getDefinition($this->getPluginId(), FALSE)) {
       // When the entity is saved via a plugin instance, we should not call
       // the menu tree manager to update the definition a second time.
       if (!$this->insidePlugin) {
-        $menu_link_manager->updateDefinition($this->getPluginId(), $this->getPluginDefinition(), FALSE);
+        $menu_link_manager->updateDefinition($this->getPluginId(), $definition, FALSE);
       }
     }
     else {
-      $menu_link_manager->addDefinition($this->getPluginId(), $this->getPluginDefinition());
+      $menu_link_manager->addDefinition($this->getPluginId(), $definition);

It looks like this change could cause equivalent failures in contrib/custom modules doing the same thing.

So the question for me is:

- is this a bug in MenuLinkContent that will affect production sites, that is exposed by the test coverage
- or, is it something that will only fail in tests

If it's the latter, this starts to look 8.1.x-ish to me - since we're forcing changes to production code in a patch release. Back to CNR to figure out which it is.

chx’s picture

Status: Needs review » Reviewed & tested by the community

It's internal. By the time external modules

    $entity->postSave($this, $update);
    $this->invokeHook($update ? 'update' : 'insert', $entity);

get a chance the behavior is consistent, the definition/tree storage is updated.

You only have a problem if you change the route provider to rebuild on demand. Modules can't get into trouble.

chx’s picture

most recent example of hitting this and needing to manually rebuild was in #2598376: d6_user_settings migration user_register constants don't seem to line up.

alexpott’s picture

Status: Reviewed & tested by the community » Fixed

@catch, @effulgentsia and I discussed this and we feel the benefits outweigh the risks. Committed 817ee92 and pushed to 8.0.x. Thanks!

  • alexpott committed b1c7365 on 8.1.x
    Issue #2605684 by chx, dawehner: Routing silently fails in kernel tests
    

  • alexpott committed 817ee92 on
    Issue #2605684 by chx, dawehner: Routing silently fails in kernel tests...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

dawehner’s picture

Let's remove that in 8.1.x again, see #2605956: Port #2605684 to the new KernelTest