Problem/Motivation
In HookCollectorPass when we apply order operations, we are not checking if that hook is actually implemented.
This scenario is valid because the attribute might target a hook implementation in another module that may or may not be installed, or where that hook implementation only exists in specific versions of that module.
Steps to reproduce
(The examples are for #[ReorderHook] only. Currently #[RemoveHook] does not cause this kind of problems, but we should include it in the tests.)
Problem scenario 1: Trivial example
A #[ReorderHook] targets a non-existing method.
The hook has no implementations.
#[ReorderHook('non_implemented_hook', 'NonExistingClass', 'nonExistingMethod', Order::Last)]
#[ReorderHook('non_implemented_type_alter', 'NonExistingClass', 'nonExistingMethod', Order::Last)]
class MyHooks {...}
Expected: The attributes are ignored.
Actual: The #[ReorderHook] attributes cause problems during rebuild:
TypeError: Drupal\Core\Hook\HookCollectorPass::applyOrderOperations(): Argument #1 ($implementation_list) must be of type array, null given, called in /var/www/html/core/lib/Drupal/Core/Hook/HookCollectorPass.php on line 255
Problem scenario 2: Advanced case
A #[ReorderHook] targets an existing method, which may implement a different hook.
The actual target hook has no implementations.
#[ReorderHook('form_myform_alter', self::class, 'formAlter2', Order::First)]
class MyHooks {
#[Hook('form_alter')]
public function formAlter1(array &$form): void {
$form['#alter_calls'][] = __METHOD__;
}
#[Hook('form_alter')]
public function formAlter2(array &$form): void {
$form['#alter_calls'][] = __METHOD__;
}
}
Expected: The attributes are ignored.
Actual behavior: Error during rebuild, as above.
Working example 1: Reorder a non-existing method
A #[ReorderHook] targets a non-existing method.
The hook has at least one implementation.
// Target non-existing methods for reordering.
#[ReorderHook('regular_hook', 'NonExistingClass', 'nonExistingMethod', Order::Last)]
#[ReorderHook('regular_type_alter', 'NonExistingClass', 'nonExistingMethod', Order::Last)]
class MyHooks {
// Make sure the hooks have at least one implementation.
#[Hook('regular_hook')]
public function regularHook(): void {}
#[Hook('regular_type_alter')]
public function regularTypeAlter(): void {}
}
Expected = Actual: The #[ReorderHook] attributes have no effect.
Working example 2: Reordering across alter hooks
A #[ReorderHook] targets an existing method from a different hook.
The hook has at least one implementation.
#[ReorderHook('form_myform_alter', self::class, 'formAlter2', Order::First)]
class MyHooks {
#[Hook('form_alter')]
public function formAlter1(array &$form): void {}
#[Hook('form_alter')]
public function formAlter2(array &$form): void {}
}
class OtherModuleHooks {
#[Hook('form_myform_alter')]
public function formMyformAlter(array &$form): void {}
}
Expected: The attributes are ignored.
Actual behavior:
- For `->alter('form', ..)`, the #[ReorderHook] attribute has no effect.
The call order is [MyHooks::formAlter1, MyHooks::formAlter2]. - For `->alter(['form', 'form_myform'], ..)`, the #[ReorderHook] attribute causes the formAlter2 method to run first.
The call order is [MyHooks::formAlter2, OtherModuleHooks::formMyformAlter, MyHooks::formAlter1] (also depending on module order).
The reason:
- In HookCollectorPass::applyOrderOperations(), the order operation has no effect, but it is still stored as argument for ModuleHandler.
- In ModuleHandler::getCombinedListeners() for a combination of alter hooks, the order operation does have an effect.
Proposed resolution
https://git.drupalcode.org/project/drupal/-/merge_requests/12858
(changed) Discard ordering directives if the target method is not registered for the target hook specified in the #[ReorderHook] attribute.
(also discard the ordering directive if the target method is missing, or was removed with #[RemoveHook]).
(changed) In combined alter calls, ordering directives can no longer reorder implementations from other hooks.
Effect:
- Problem scenario 1: (changed) The #[ReorderHook] has no effect. No error during rebuild.
- Problem scenario 2: (changed) The #[ReorderHook] has no effect. No error during rebuild.
- Working scenario 1: (unchanged) The #[ReorderHook] has no effect.
- Working scenario 2: (changed) The #[ReorderHook] has no effect.
Remaining tasks
Review
User interface changes
N/A
Introduced terminology
N/A
API changes
[#ReorderHook] must match all aspects for it to take effect and must target an existing implementation. Otherwise it is dropped from the ordering ruleset.
Data model changes
N/A
Release notes snippet
N/A
| Comment | File | Size | Author |
|---|
Issue fork drupal-3536470
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
- 3536470-improve-docs
changes, plain diff MR !14105
- 3536470-deregister-order-nonexistent-v2
changes, plain diff MR !12858
- 11.x
compare
- 3536470-deregister-order-nonexistent
changes, plain diff MR !12814
- 3536470-ReOrderHook-on-non-implemented-hook
changes, plain diff MR !12758
- 3536470-ReOrderHook-on-non-implemented-hook-current-behavior
changes, plain diff MR !12771
Comments
Comment #2
donquixote commentedComment #4
donquixote commentedComment #5
donquixote commentedComment #6
nicxvan commentedProbably major.
Comment #7
xjmThis is at least major. A case could be made for critical.
Comment #8
xjmCan we add some explicit assertions about the existence of the non-existent hook fixture definition in the registry or whatnot? Otherwise the file could be removed without tests failing. Thanks!
Comment #9
xjmI talked myself into erring on the side of critical. This mostly only affects custom code or rather advanced contrib modules that integrate with other contrib modules, which would usually limit the scope to major, but if someone hits this on any one of the many hooks that act on CRUD operations it could theoretically affect data integrity or even something triggered in a post-update hook (and therefore the upgrade path).
Comment #10
xjm@nicxvan pointed out that we should also add test coverage that the hook order is correct with or without the missing hook.
Comment #11
nicxvan commentedHappy to leave this critical.
However,
Post update hooks are not oop.
Also this blows up in the compiler pass, I'm not sure how data integrity will be affected.
The bigger problem might be modules installed hooks not running.
Comment #12
nicxvan commentedRe 10, not quite what I meant.
I meant #[Hook('some_hook', order: OrderAfter(classesAndMethods: MissingClass::non_existent_hook_method'))]
Comment #13
xjmI meant things triggered by them. If they themselves were being ordered by this mechanism I would've definitely marked it critical. (I checked that first.) But yeah, let's leave aside what exactly happens when or whether this is possible. The base issue (and the case for critical) is that there are so many different ways this could break advanced usecases that it's hard to rule out the possibility of the "definitely critical" categories of issue (security, data integrity, upgrade path). So let's err on the side of the higher priority which we do when it's borderline in general.
Comment #14
nicxvan commentedI'm not opposed to this being critical.
I had a chance to investigate why workspace using this on content moderation doesn't blow up.
It turns out that this needs to be an alter hook. It has to try ordering runtime, which means this IS critical since it can blow up during runtime, not collection.
We only pack the operations when it's for an alter. I think when gathering order operations either for packing or applying we should check that the class exists.
In the case of
Reorderif the target doesn't exist we should just outright drop it. WhenOrderBeforeorOrderAftertargets don't exist I think that is safe, but we should confirm we have a test.Regarding my comment on dropping
Reorder@donquixote made this comment in slack:I don't think that is quite right,
ReorderHookcan only target one implementation, if that implementation doesn't exist then it's rules should not apply at all.Specifically:
but if the
form_form_id_alterdoesn't exist then no ordering should happen, if the module want's to also order theform_alterthey should target that separately.Edit:
I think the right place to address this is here.
If the class / method don't exist don't set the operation.
Edit 2:
We may have to consider what to do here too if class and method are used to target an implementation. Maybe we just handle the check in apply? But I really think we shouldn't be packing or storing a target reorder for an implementation that does not exist.
Comment #15
nicxvan commentedComment #16
xjmComment #17
donquixote commentedThis is really about targeting a hook that (currently) has no implementations.
If you target a hook that does have implementations, but the targeted implementation does not (currently) exist, there is no problem.
Also it is not "fatal" per se: A TypeError is catchable, it is up to Drupal whether to catch it.
The "if the form_form_id_alter doesn't exist" does not mean anything.
A hook may never be called or it may have no implementations.
There is no such a thing for a hook as to "not exist".
During discovery, we can only determine whether a hook has implementations or not.
But it may still be called.
Currently this will cause the TypeError for which this issue exists.
But we can easily craft a version of this scenario that does not crash: We simply add an implementation for hook_form_myform_alter().
Comment #18
nicxvan commentedHaven't fully read 17, but this only happens for alter calls, this does not occur as far as I tested for non alter hooks.
Comment #19
donquixote commentedThe "steps to reproduce" show how this applies to non-alter hooks.
Simply add this attribute (and nothing else) on a class in a Hook namespace:
(don't forget the imports, or it won't work)
It also does not matter if the target class and method exist:
- If the hook has (other) implementations, but the target class and method do not exist, there is no error.
- If the hook has no implementations, but the target class and method do exist, just registered for a different hook (or no hook), we do get the TypeError during discovery.
Comment #20
nicxvan commentedSo it had to be a class without an invoke method I suspect.
As part of my testing I enabled workspaces without content moderation and it did not fail.
I then changed the use and class name for content moderation hooks to a non existent class and nothing failed.
I did not dig further, but why would my test not fail?
Comment #21
nicxvan commentedComment #22
nicxvan commentedComment #23
nicxvan commentedOk, I had a long discussion over slack with @donquixote.
I had misunderstood initially that this was about the target implementation for an ordering operation no longer existing. That is not the case. This error is about
ReorderHookonly.This is when there are no remaining implementations for a hook when
ReorderHookis processed. Workspaces doesn't blow up because there are otherentity_presaveimplementations in core.HOWEVER, when it comes to ordering (whether directly or indirectly) the question that now remains is how to appropriately handle this.
Legacy behavior
hook_module_implements_alterThis was a separate hook that executed independent of anything else. The ordering rules applied where applicable especially with extra types.
Modern behavior
Current
order directives (whether on
#[Hook]or#[ReorderHook]after collection are independent of the attribute / implementation they were defined on or before. In this way they are much like HMIA that they were designed to replace.My open question is do we want to preserve that behavior long term. My original thinking was that since order directives are defined in a hook implementation (or in the case of reorder, there is a target implementation) if the hook is removed then the order directives would no longer apply.
I would posit that if a hook implementation that has an order directive (whether directly on the
Hookattribute or indirectly withReorderHook) gets removed somehow the ordering directive would be removed as well.@donquixote disagrees, but I don't see a situation where implementations that don't exist can affect other implementation ordering.
Comment #24
nicxvan commentedTo put it more explicitly after further discussion on slack.
In my opinion, the ordering rules should only be applied or packed for runtime application if the target class and method are known for the defined target hook.
That means in the following cases ordering operations would be dropped:
#[Hook('test_hook', order: OrderBefore(['some_module']))]and there is a#[RemoveHook]elsewhere that targets that implementation#[ReorderHook('test_hook', classAndMethod, order: OrderBefore(['some_module']))] and the class and method do not exist or were removed somehow (like with a
#[RemoveHook]attribute)If the hook implementation defining an order operation does not exist, then it should not affect any ordering whatsoever.
If you need to affect order of something no matter what you can do that with dummy hooks or other workarounds.
This may technically be a change in behavior from 11.2, but I think the fact that ordering operations were sticky is a bug that should be fixed rather than behavior we should preserve. Further, this is incredibly niche and rare. Anyone implementing
#[ReorderHook]should be able to handle a change here.Same with ordering operations in general.
Comment #25
donquixote commentedSo the main difference is this:
This means the scenario in #17 would no longer be supported.
The distinction between "invoked with" and "registered with" is relevant only in multi-type alter calls like ->alter('form', 'form_myform').
An implementation is (typically) registered for only one hook, but the alter call can add more hooks into the mix.
The proposed change would a BC break.
The time to suggest this would have beeen pre 11.2.0 :)
What I would propose is to only fix the error case, and make it align with the non-error case.
EDIT:
There is another interesting case where one implementation is registered for multiple hooks.
There are legitimate reasons to do this, e.g. hook_node_update() and hook_node_insert(), or we want to target two similar form ids with hook_form_FORM_ID_alter().
But if these are not part of the same ->alter() call, there is really nothing special here. And if they are, we get into that notice "The @implementation is registered for more than one of the alter hooks ...".
Comment #26
catchIf I understand correctly, the current behaviour that @nicxvan is arguing to remove and that @donquixote is arguing to retain, is that if you try to affect the order relative to a hook that doesn't exist, this still affects the order of other hooks that do exist. To me this sounds unexpected and a bug, that we can fix in a patch release - since it's a behaviour change it will need a CR, but I disagree it's a 'bc break' this seems like an extreme edge case and undocumented behaviour.
Comment #27
donquixote commentedNot really - unless I misunderstand.
When I see "order relative to", I think of #[ReorderHook(..., new OrderBefore(...))].
This is not what this is about.
The best example is in comment #17.
As you know, ->alter() can be called with more than one "alter type", which then maps to more than one alter hook.
E.g. ->alter(['form', 'form_node_form', 'form_node_article_form'], ..).
The hooks are 'form_alter', 'form_node_form_alter', 'form_node_article_form_alter'.
Each of these hooks can have order operations that were declared either with #[ReorderHook], or by setting an order in the #[Hook] attribute for a specific implementation.
All of these order operations will be applied when the implementations from these multiple hooks are "merged" at runtime.
When this happens, an order operation from e.g. 'form_node_form_alter' is allowed to target and reorder an implementation of 'form_alter'.
How?
Most of the #[ReorderHook] will target an implementation of the same hook that is passed in the first parameter of the attribute.
E.g.
- #[ReorderHook('form_alter', MyClass::class, 'formAlter', Order::First)], or
- #[ReorderHook('form_node_form_alter', MyClass::class, 'formNodeFormAlter', Order::Last)],
assuming that ::formAlter() implements hook_form_alter(), and ::formNodeFormAlter() implements hook_form_node_form_alter().
However, a #[ReorderHook] can target (and thus reorder) an implementation from a different hook.
This operation is only applied when both hooks are used together in an ->alter() call.
E.g. a completely silly example would be completely unrelated hooks #[ReorderHook('mail_alter', SomeClass::class, 'toolbar', Order::First)], assuming that SomeClass::toolbar() implements hook_toolbar().
These are never used together in ->alter(), and the second one ('toolbar') is not even an alter hook.
Therefore this #[ReorderHook] is completely pointless and has no effect.
However, we can #[ReorderHook('form_node_form_alter', MyClass::class, 'formAlter', Order::First)].
Here the target hook is 'form_node_form_alter', but the target method implements 'form_alter'.
What could be the intent behind this?
Currently, this works as described _if_ there is at least one implementation of hook_form_node_form_alter().
This implementation could be from a different module that we don't care about.
If there is no such implementation, we get the TypeError as reported in this issue.
Comment #28
donquixote commentedThe new version is a bit verbose but hopefully more clear.
The test-only branch illustrates the current behavior.
Comment #30
donquixote commentedComment #31
nicxvan commentedI'm going to cut from your test only branch and work on the solution I was suggesting.
I haven't parsed your tests fully so I wouldn't be surprised if some fail since your goal was to preserve current persistent ordering directives.
Comment #33
nicxvan commentedTook a quick pass at my approach using your tests, I also retargeted on 11.x that all MRs in this issue should be on.
Let's see what the tests say, but this filters the order operations by whether the identifies still exists in the filteredImplementations.
We might be able to make this a little more efficient, but this should close what I think is an unexpected consequence of order parameters. (I realize you're arguing that this is expected and should remain, but I still disagree)
Comment #34
nicxvan commentedOk I've pushed up my proposal here https://git.drupalcode.org/project/drupal/-/merge_requests/12814/diffs
All tests that @donquixote pass on my branch too except for one: https://git.drupalcode.org/project/drupal/-/merge_requests/12814/diffs?c...
The conceptual difference is that the class and method targeted by ReorderHook must match the hook assigned to that class and method as well. (in the test reorder is looking for
test_b_subtype_alterand the hook for that class is actuallytest_a_supertype_alter)In donquixote's approach this reorder still has an effect, I think that is also a bug.
My approach has a bit of filtering in HookCollectorPass during build, but it avoids all sorts of ordering side effects and there are no runtime changes.
@donquixote mentioned in slack that he'd like to add a test for:
I think we can do that here or in a follow up.
After seeing both options I think the approach I suggested is the way to go still.
I'm going to hide the test only branch for now.
Comment #36
needs-review-queue-bot commentedThe Needs Review Queue Bot tested this issue. It fails the Drupal core commit checks. Therefore, this issue status is now "Needs work".
This does not mean that the patch necessarily needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.
Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.
Comment #39
donquixote commentedI don't think it should be seen as a bug.
(the cross-hook ordering, not the TypeError)
The desired behavior in this scenario was not discussed in depth at the time we introduced these attributes.
The current behavior seems just as valid as the new behavior proposed by @nicxvan.
Changing it now does somewhat break existing behavior, even though it is a niche case.
This said, I could see one argument to see it as a bug: There is inconsistency with #[RemoveHook], which does not show this cross-hook behavior.
I added the test case for this in 3536470-ReOrderHook-on-non-implemented-hook-current-behavior and in 3536470-ReOrderHook-on-non-implemented-hook.
The goal was to provide a more relevant use case where reordering across hooks might actually be desirable.
In the test it is used to "reinforce" the ordering on a base alter hook implementation so it can overrule ordering for subtype alter implementations.
Overall I feel like our attributes are not well-suited for these competitive situations.
What if you really want to be "last, after X which also wants to be last, but before Y, which also wants to be last"?
And what if X and Y actually come from different hooks, which are all called togeher in ->alter([*, *, *], ..)?
The OrderAfter and OrderBefore were meant for this purpose, but I don't believe in them.
They are going to be very unreliable if the target implementation we order against is also reordered. Now the success of OrderAfter depends on _when_ that rule is applied.
If A.Last is applied first and then B.OrderAfter(A), we get ***, A, B.
If B.OrderAfter(A) is applied first and then A.Last, we get **, B, **, A.
The most reliable solution would be numeric weights/priorities.
We don't need to give weights to implementations, instead we can give weights to ordering rules.
(not in this issue, of course)
If we have a weights system, the use case mentioned before would mostly disappear.
Until then, maybe we can say that the relative order of implementations with competitive Order::First or Order::Last is not fully guaranteed.
Comment #40
nicxvan commentedI'll take a look at the new tests and move them to my branch!
Still think my approach is the more expected behavior and cleaner long term.
Comment #41
nicxvan commentedOk I pulled in your new tests, the crosshookreorder test will fail, I need to read through it to confirm my suspicion that it's just the behavior changes expected with my approach.
Edit that's a bad rebase I'll fix it in the morning.
Comment #44
nicxvan commentedOk I fixed the rebase and updated the order.
I added a comment about the three that changed.
I think this is ready for review again.
Comment #46
nicxvan commentedComment #47
nicxvan commentedComment #48
nicxvan commentedComment #50
donquixote commentedComment #51
donquixote commentedComment #52
catch@donquixote
It's fine to change behaviour in minor releases, we just need some degree of confidence about how many contrib modules are likely to be affected, which given OOP hook re-ordering is very new is going to be low. We can document the behaviour change in a change record so that if someone is affected they'll get a hint what to look for.
Given that I think we should focus here on what the desired behaviour should be - this will then be less likely to require further changes later.
However it's not entirely clear to me how much of the discussion between the two approaches is around what the desired behaviour should be (regardless of bc) vs how much of a behaviour change it would be.
Comment #53
catchDiscussed this briefly with @nicxvan in slack, trying to summarise the arguments.
Let's say:
We have hook implementations A, B and C
A specifies it should run before B
C specifies it should run after B
When B exists in the code base, the order will be A, B, C with both approaches.
When B doesn't exist in the code base:
@donquixote's approach:
A is still guaranteed to run before C, because B's existence is replicated.
@nicxvan's approach:
No extra ordering happens.
However, I think that if A needs to run before C, or if C needs to run after A, then they should be explicitly saying that, and not relying on the existence of B. B could be a hook implementation that is removed by module B, or module B could be uninstalled, or in extreme cases it could be a typo by the author of both A and C. This points towards @nicxvan's approach as being more predictable overall, even if @donquixote's would make things work in some situations where they're relying on implicit dependencies.
Comment #54
nicxvan commentedThank you, I've rebased my branch so it's up to date, I'll update the issue summary to reflect the decision in a bit.
Comment #56
nicxvan commentedComment #57
phenaproximaI like how luxuriously clear and detailed the test coverage is -- those comments make it much more accessible to the casual reader.
Why hold this up any further?
Comment #58
catchYes agreed on the test coverage, will make it much nicer for the next person that looks at it.
Committed/pushed to 11.x and cherry-picked to 11.3.x, thanks!
Comment #64
donquixote commentedSorry that I did not pay attention to this in the recent months.
This looks a bit strange.
It seems as if the "The hook implementation identifier, as "$class::$method", to be changed by." describes the return value.
But even then it is strange, what does "to be changed by" mean?
In the constructor doc, the $identifier parameter is described as "Identifier of the implementation to move to a new position.". It would make sense to replicate this here.
The "for filtering" is also confusing.
Nothing in the class file mentions anything about filtering.
We can either not talk about filtering at all, or we have to explain what it refers to.
The part in the return, "The identifier for the OrderOperation." is also misleading, because the identifier is for the implementation, _not_ for the order operation: There could be multiple order operations that target the same implementation and would have the same return value in ->identify().
One could also consider to rename the method to something like ->getTargetId() or ->getTargetImplementationIdentifier(), but I don't mind if we keep ->identify().
Comment #65
donquixote commentedIn HookCollectorPass we introduced an empty line.
I think this is not intended.
Comment #66
nicxvan commentedNo need to apologize!
Your description is far better, feel free to open a follow up for that and I'll review it.
The extra empty line isn't crucial, we can clean that up as part of the HookCollectorPass ThemeHookCollectorPass consolidation issue: #3547641: Update HookCollectorPass to leverage shared logic from ThemeHookCollectorPass
Comment #68
donquixote commentedI created follow-up MR here.
Maybe a new issue would be the "correct" thing but I don't want to pollute the issue list.
Comment #69
donquixote commentedI opened #3568330: Minor doc tweaks after #3536470 as follow-up for these doc fixes.