Problem/Motivation
Drupal core supports OOP hooks.
Let's convert everything everywhere all at once.
With our testing this can be done in about 10 minutes on a fresh branch.
This allows us to manage this in one commit.
This is unique to most issues since it's a bulk rector pass.
How to review
This can be reviewed even with merge errors.
The review is more about the process and results than an individual branch.
The primary thing to review is post update changes found in patches in the gist here: https://gitlab.com/-/snippets/3763525
You can also investigate individual modules like the system branch.
If you'd like to see a specific module converted alone let me know.
You can review the Rector Rule here: https://github.com/nlighteneddesign/drupal-rector/blob/rectorOOPHooks/sr...
I will periodically regenerate as needed.
I think this should only be put in needs work if there is specific feedback or there are non random test failures.
Current Canonical Branch
3483599-nov13-fullAutomated2
Important do not attempt to rebase this, you must go here and click update fork: https://git.drupalcode.org/issue/drupal-3483599
Then create a new branch and convert.
Important: do not use any patches attached here, use the gist below.
We have a green pipeline!!!!
All remaining tasks have been merged.
There are several follow ups we've identified.
Before
- 69 .inc files
- 241 .module files
After
- 39 .inc files
- 99 .module files
Steps to reproduce
Check core procedural implementations.
Proposed resolution
Automated script: https://gitlab.com/-/snippets/3762254 (See the gist notes for updates that are pending)
It automates:
- A rector rule to do the conversions which move procedural hooks to OOP methods on HookClasses. It creates a hookClass per file it finds with hooks implemented. e.g. a module with a module.module and a module.tokens.inc would get moduleHooks and moduleTokenHooks classes.
- Applying a patch for coder to fix the indentation. #3461148: Drupal.WhiteSpace.ScopeIndent gets confused by attributes on a function. and then runs phpcbf to fix formatting.
- Applying patches to fix tests. This is to revert some conversions that are explicitly to test procedural functionality and some exceptions like system_page_attachments. It also fixes direct calls to hooks in favor of their new locations.
- Various other tasks such as deleting empty .inc and .module files
- Regenerating the baseline.
Most up to date patches are here: https://gitlab.com/-/snippets/3763525 (See the gist notes for updates that are pending)
Remaining tasks
Review process.
The goal is to achieve this in one commit.
Investigate failures as we rebase and re run on HEAD.
The following issues are applied in this mr as they fix regressions or things preventing accurate conversion.
These are all applied as part of this conversion, but have been reviewed separately with context in their respective issues.
- #3482618: ModuleHandler->alterDeprecated does not properly handle deprecated OOP hooks Fix deprecations
- #3486462: Support #Hook for several hooks called by ModuleInstaller Support hooks invoked by ModuleInstaller::invokeAll()
- #3484754: Ordering of hooks needs some attention hook module implements alter fix
- #3485690: Update Implements hook comments to be accurate or add them if missing Correct documentation
- #3485470: views_form_views_exposed_form_alter is documented as hook_form_alter instead of hook_form_FORM_ID_alter Correct documentation
#3484105: Automatically included .inc files are no longer included hook_hook_info and include. Inc files with no hook implementations after conversion.
Follow ups
These tasks are not required for this issue to merge, but are worth follow up.
- #3483899: hook_module_implements_alter hook_cache_flush can only be procedural
hook_module_implements_alteris called in a procedural manner during build time so it must remain procedural for the rest of its life may that be short - #3482464: Backport Hook and LegacyHook Attribute - Backport Attributes
- #3472165: [Meta] Track support for Install and Theme hook implementations to OOP - Convert remaining hooks
- #3485658: Add an advisory AltersImplementations attribute Add an advisory
ImplementsAlterattribute which can be used onhook_module_implements_alterimplementations to indicate which hook this alter is using. - #3485659: Remove HookOrder Remove HookOrder temporarily
Other Exceptions:
- system_theme
- system_page_attachments #3428339: Remove calls to system_page_attachments()]
These are both directly called during Drupal install so are a special case and cannot be converted.
We undo or prevent conversions for:
- core/tests/fixtures/empty_file.php.module it is testing empty .module so we cannot remove it.
- core/modules/system/tests/modules/deprecation_test it is testing deprecated procedural hooks so we must keep them until there are no more.
- core/modules/system/tests/modules/module_test this is testing loading order of .module and .inc files so we must keep it.
- core/tests/Drupal/Tests/Core/Extension/modules/module_handler_test_all1/module_handler_test_all1.module this is used to test hook ordering in mixed scenarios
- core/tests/Drupal/Tests/Core/Extension/modules/module_handler_test_all2/module_handler_test_all2.module this is used to test hook ordering in mixed scenarios
User interface changes
N/A
Introduced terminology
N/A
API changes
...
Data model changes
N/A
Release notes snippet
| Comment | File | Size | Author |
|---|---|---|---|
| #167 | drush_gen_hook-3483599-167.png | 12.67 KB | joseph.olstad |
Issue fork drupal-3483599
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
- 3483599-ignore-convert-everything
changes, plain diff MR !9963
- 3483599-conversion3
changes, plain diff MR !9970
- 3483599-conversion4
changes, plain diff MR !9977
- 3483599-conversion5
changes, plain diff MR !9979
- 3483599-conversion6
changes, plain diff MR !9982
- 3483599-conversion7
changes, plain diff MR !9983
- 3483599-conversion8
changes, plain diff MR !9986
- 3483599-conversion9
changes, plain diff MR !9997
- 3483599-conversion10
changes, plain diff MR !10012
- 3483599-patchReroll1
changes, plain diff MR !10014
- 3483599-nov1-fullAutomated1
changes, plain diff MR !10033
- 3483599-nov4-fullAutomated1NewHMIA
changes, plain diff MR !10056
- 3483599-nov4-fullAutomated2
changes, plain diff MR !10058
- 3483599-nov4-fullAutomated3
changes, plain diff MR !10059
- 3483599-nov5-fullAutomated1
changes, plain diff MR !10067
- 3483599-nov7-fullAutomated1
changes, plain diff MR !10086
- 3483599-nov7-fullAutomated2
changes, plain diff MR !10091
- 3483599-nov7-fullAutomated3
changes, plain diff MR !10099
- 3483599-nov10fullAutomated1
changes, plain diff MR !10126
- 3483599-nov11-fullAutomated1Installer
changes, plain diff MR !10139
- 3483599-nov11-fullAutomated2
changes, plain diff MR !10142
- 3483599-nov11-fullAutomated3
changes, plain diff MR !10146
- 3483599-nov11-fullAutomated4
changes, plain diff MR !10149
- 3483599-nov1-systemOnlyNoPatches1
changes, plain diff MR !10031
- 3483599-nov1-nodeOnlyNoPatches
changes, plain diff MR !10030
- 3483599-nov1-viewsOnlyNoPatches
changes, plain diff MR !10032
- 3483599-nov13-fullAutomated2
changes, plain diff MR !10178
- 11.x
compare
- 3483599-nov13-fullAutomated1
compare
- 3483599-nov1-fullAutomated1Installer
compare
- 3483599-nov1-systemOnlyNoPatches
compare
- 3483599-conversionPatchReroll
compare
- 3483599-NOCONVERSIONS
compare
- 3483599-conversion2
compare
Comments
Comment #3
nicxvan commentedI think we have a path forward here:
apply patch to convert some that need manual conversion e.g module implemented alter and the alter deprecation
Then run conversion
Then apply test fix patches in test conversion issue
Then apply patches that revert conversions that can't be converted yet not created yet but this includes tests like the one for .inc loading
Wrangle coding standards, initial testing this is probably a few minor things.
Check for any failures that arise from all things being converted and integrate into above step.
Run process during code freeze if possible.
All tests have resolutions except the functional htaccess test.
Comment #5
nicxvan commentedDo not use these patches!
Most up to date patches are here: https://gitlab.com/-/snippets/3763525 (See the gist notes for updates that are pending)
Comment #7
nicxvan commentedOk after the automated conversion, patch application, and new baseline it looks like there are three functions in the baseline that shouldn't be there, not sure why, but that's pretty minor, let's see if there are any new test failures.
Comment #8
nicxvan commentedMany, many failures, mostly due to function calls missing, it seems that tests call hooks directly more than expected. I'll see if we can address this with rector or identify another pattern.
The test conversion is working though so we can always fall back to one bulk update of the tests, then group the modules as needed.
Comment #10
nicxvan commentedGood news I unintentionally deleted some proxy files.
We have some patches and a follow up.
Comment #11
nicxvan commentedDo not use these patches!
Most up to date patches are here: https://gitlab.com/-/snippets/3763525 (See the gist notes for updates that are pending)
Comment #12
nicxvan commentedDo not use these patches!
Most up to date patches are here: https://gitlab.com/-/snippets/3763525 (See the gist notes for updates that are pending)
Comment #14
nicxvan commentedok I have converted everything.
Patches are out of date
Comment #17
nicxvan commentedGreat news, only minor codestyle issues.
There are another 25 functions that were converted, but those fixes are minor I can add patches for them.
Comment #20
nicxvan commentedTests are failing now mostly due to how I incorrectly fixed the missing hook functions now that they are methods on classes.
I need to call the actual hooks directly still. I'll update my local patch, that will take care of the majority of the issues.
Comment #23
nicxvan commentedOk missing functions should be all fixed now, I also have a bigpipe patch so that is no longer manual.
Comment #25
nicxvan commented0003 has an update coming, but in order to not lose anything here are the patches used in conversion7Also 8 doesn't apply, but it's super easy to just fix the bigpipehooks arrays manually.
Do not use these patches!
Most up to date patches are here: https://gitlab.com/-/snippets/3763525 (See the gist notes for updates that are pending)
Comment #29
nicxvan commentedPatches are out of date.
Comment #33
nicxvan commentedComment #34
nicxvan commentedComment #35
nicxvan commentedComment #36
nicxvan commentedComment #37
ghost of drupal pastComment #38
ghost of drupal pastComment #39
ghost of drupal pastComment #40
nicxvan commentedComment #46
nicxvan commentedComment #47
nicxvan commentedComment #48
nicxvan commentedComment #49
nicxvan commentedComment #57
nicxvan commentedComment #58
nicxvan commentedComment #59
nicxvan commentedComment #60
nicxvan commentedComment #67
nicxvan commentedComment #68
nicxvan commentedComment #69
nicxvan commentedComment #70
nicxvan commentedComment #71
nicxvan commentedComment #74
nicxvan commentedComment #75
longwaveThis looks amazing, well done.
A couple of thoughts/questions:
1. With things like
This shouldn't be considered API, but I would bet that someone somewhere is calling a procedural hook like this directly. Should we leave the existing code in place and apply the
#[LegacyHook]attribute to it?2. In turn this made me consider BC. If you were hook_module_implements_alter'ing to remove or reorder core hooks before, now we have broken this I think? But if we have the
#[LegacyHook]attribute could we add metadata to point to the new OO hook name - and then we can provide a BC layer based on this information? This does sound like it could be quite difficult and perhaps not performant, however.Comment #76
nicxvan commentedThank you for taking a look!
1. I really don't think it's necessary to keep LegacyHook for most of core, I also think it would have performance implications.
However, if you just meant a few select ones that probably makes sense to keep some like `block_rebuild`. Would it be acceptable to move that to a follow up so we can highlight them?
2. We had to circle back to handle BC for hook_module_implements_alter, that ticket is ready for review with tests here #3484754: Ordering of hooks needs some attention (that is merged into this branch, it's required to make things like ckeditor5 work).
Comment #77
longwaveI think the problem with #75.1 is that we don't really know which procedural hooks someone might be calling directly; we could search contrib but we have no idea about custom code. The BC policy says that hook implementations are not part of the BC promise so perhaps we just hard break by removing all these, unless there are a small number that are widely used - but again I wouldn't know how to find that out.
The "gigantic mess" has put me off #3484754: Ordering of hooks needs some attention :D but I will try to look soon!
Comment #78
catchI think it is OK for us to remove the procedural hook implementations - we do that for individual hook implementations every so often, and it's very clearly documented that you shouldn't call them directly. Also it will be an easy thing for a module to adapt to. My concern with leaving all the procedural hook implementations in is that we'll be doing hook discovery on those files until Drupal 12 then, which anecdotally has put our memory requirements through the roof. Core is big enough that fully converting core might mitigate some of this, got another issue open to hopefully mitigate it further but it's just an idea with no code still.
Comment #79
nicxvan commentedI'd love to just do a clean break, we can add examples in the CR that this will inevitably need.
I've been meaning to update that title, it has now been changed, but it is fully ready for review.
There are some optional follow ups there as well, one of which is RTBC as well.
Comment #80
nicxvan commentedComment #81
nicxvan commentedComment #82
nicxvan commentedComment #83
catchI think this will actually be OK after #3484754: Ordering of hooks needs some attention.
hook_module_implements_alter() works on an array keyed by the module name, so as long as a module only implements one hook, the alter will continue to work the same way.
Once modules start having two implementations of the same hook, then it won't be possible to use h_m_i_a() to put an implementation between the two implementation, but that will be a result of refactoring of hook implementations (in 11.2 or later, or contrib) rather than the conversion itself. At that point we'll need a different ordering mechanism which will eventually replace h_m_i_a().
Comment #84
nicxvan commentedI should also note, that while #3484754: Ordering of hooks needs some attention has it's own tests, it is applied in this issue so we can see it works with the alters ckeditor needs to work with editor and media.
Comment #85
nicxvan commentedComment #86
nicxvan commentedAll remaining tasks are now rtbc.
Comment #87
nicxvan commentedComment #88
nicxvan commentedComment #91
nicxvan commentedComment #96
nicxvan commentedComment #97
nicxvan commentedComment #98
nicxvan commented#3486462: Support #Hook for several hooks called by ModuleInstaller is ready for review and can be rolled into this issue if we want. ( tested here #3486495: [ignore] Test extra conversions )
This allows for converting:
This also includes: #3484747: Fix invocation of hook_cache_flush in ModuleInstaller uninstall to allow OOP implementations so that is now a duplicate.
Comment #99
nicxvan commentedComment #100
nicxvan commentedComment #101
nicxvan commentedComment #102
nicxvan commentedComment #103
nicxvan commentedComment #104
nicxvan commentedComment #105
nicxvan commentedComment #108
nicxvan commentedComment #109
nicxvan commentedComment #110
nicxvan commentedComment #113
nicxvan commentedComment #117
nicxvan commentedComment #118
nicxvan commentedComment #119
catchI spot checked the first handful of converted modules and the conversion looks really great - all comments and formatting preserved where it needs to be. Overall I agree committing this in one go will be less disruptive than module-by-module, even if a few more MRs have to be re-rolled at least it's one hit instead of over and over again. Tagging with beta target (in the sense of good to do during the beta).
Comment #120
nicxvan commentedI'm regenerating since #3484754: Ordering of hooks needs some attention got merged!
Comment #123
nicxvan commentedOk is been updated.
Comment #124
nicxvan commentedThe two remaining tasks were merged, I'm gonna update again.
Comment #127
nicxvan commentedIt has been updated again since the last two remaining issues have been merged.
Comment #128
nicxvan commentedComment #129
fabianx commentedRTBC - I did not look at every single line of code, but performed isolated spot checks and the conversion seems fully automatic to me.
I am also +1 to do this in one full sweep, so that we can get off .module files ASAP.
This MR also clearly shows that hooks are as easy to implement as before the conversion, so that’s a huge plus.
Comment #130
nicxvan commentedThis will need to be reconverted since some upstream issues updated baseline.
Currently working through getting some of the return type issues in that are converted so I will wait for those to reconvert.
Comment #131
nicxvan commentedGonna regenerate now since the two hook return type issues got in that we convert:
#3483065: Add void return type to all hook_form_alter, hook_form_FORM_ID_alter and hook_form_BASE_FORM_ID_alter implementations
#3484249: Add void return type to all hook_entity_type_alter implementations
#3483050: Add int return type to all hook_update_last_removed implementations Might get in soon too, but we don't convert those hooks here, so while it requires a reroll due to baseline won't affect this branch.
Last one just landed, rerolling now.
Comment #135
nicxvan commentedI've rerolled, all tests are still green.
Comment #136
nicxvan commentedComment #137
nicxvan commentedComment #138
nicxvan commentedComment #139
fabianx commentedI also reviewed the snippets of patches applied after the automatic conversion manually.
They all also look good to me.
The change record makes sense and even in D7 it was the right way to use module handler to invoke a single hook in another module.
Still RTBC
Comment #140
nicxvan commentedComment #141
nicxvan commentedJust leaving a note here that @longwave confirmed in slack that his questions have been answered as well.
Comment #144
longwaveThanks again for keeping on top of this.
Discussed this with @catch and @nod_ and we agree it would be good to get this into beta1. As it's such a big change and removes a lot of procedural code it would be good to shake out any issues as early as possible.
I have some nits, but they can wait for followups:
I also realised that we can infer the hook name from the method name to avoid repetition:
could just be
Even for e.g.
we can swap camelCase for snake_case automatically. But again this easily can wait for a followup.
Committed and pushed 8aeb2ca5992 to 11.x and 059e256c285 to 11.1.x. Thanks!
Comment #145
longwaveTagging as a release highlight and also for the release notes as it's such a big change.
Comment #146
nicxvan commentedThank you so much!
I can't thank @ghost of drupal past enough for all of his help working through some of the tests and regressions we discovered working on this. I learned so much working through the myriad of issues we uncovered.
I'd also like to thank @catch for his advice and attention on several follow up issues and process for an issue like this.
@godotislate, @lendude, @berdir, @smustgrave and @bbrala all helped me uncover some test solutions, reviewed related issues in a way that was pertinent to this issue or helped with rector.
And of course thank you @longwave and @fabianx for your reviews here.
Comment #147
mondrake👏 great advancement!
Comment #153
longwaveAdding credits from #146.
Comment #154
nicxvan commentedI'm happy to work on some follow ups, however:
I don't think we want to do that for a few reasons:
Unless you mean infer only if we're missing the hook parameter in the hook attribute, but I think point 3 covers that, it means if it's missing you have to magically name you're method.
We also probably are going to use that for documentation similar to implements so it helps there too.
Comment #155
ghost of drupal pastExplicit is better than implicit. But we have a core committer expressing his opinion and if that's the direction core wants to take, I wouldn't say no. nicxvan quotes the IS but it has the serious problem of being written by me and as such it's poorly worded -- I speak much better PHP than English. In this case I wanted to make sure we are not ending up with another solution where the only way to write a hook is a magic string. But I think an optional empty hook attribute with the method name converted to snake case as the default hook is okay. It just shouldn't be mandatory. And yeah, it's not usable for either angle of multiples but that's also ok -- it's a sensible default after all, nothing more.. It'll cover a lot of cases and when it doesn't it's very easy to do more. Change $hook to optional in Hook, put a comment on it "defaults to snake case converted method name" and in HookCollectorPass add said conversion:
So simple ;) but really, this is just 1 line change besides factoring out $hook->hook into $hookValue. Your test string should be something like FooENtityInsert to check against a) first capital letter doesn't cause an underscore -- this is why the regex has a dot b) two capital letters next to each other do cause an underscore -- this is why the regex uses a positive lookbehind instead of a plain dot, this way the previous character doesn't get consumed.
This could be a second rector rule -- one that is much, much simpler than the first and way, way more aligned with what rector does.
Edit: on further thinking there's significant challenges here with numbers. Like, do we put an underscore before a block of numbers, between them...? Camelcasing is a lossy operation for numbers. Careful here. Perhaps throw an UnexpectedValueException when this conversion is being done but there are numbers to force the developer to be explicit?
Comment #156
nicxvan commentedI could create a follow up for that, are you suggesting we do a second pass to update core to use this?
Does this complicate hooks with extra types more?
Comment #157
ghost of drupal pastThat's what I am suggesting, it should be a whole lot easier since the rule would only remove superfluous strings from Hook attributes. It will not change anything behavior wise, it's free, no test failures, no nothing. It's just less strings. Not to mention how easy it is to write the rule. It doesn't matter whether it's extra types or not. It could look at
fieldWidgetCompleteTestFieldWidgetMultipleSingleValueFormAlter, convert it tofield_widget_complete_test_field_widget_multiple_single_value_form_altercheck the first arg to Hook and see it's the same, so it deletes it. If it is not the same, the rule does nothing.It would also skip ckeditor5PluginInfoAlter for reasons outlined above, it's the only eligible Hook implementation which is followed by a number.
Comment #158
berdirLets discuss that in a new issue? That said, -1 from me on that. It will make it harder to search/grep for implementations and understand for a human what on earth
fieldWidgetCompleteTestFieldWidgetMultipleSingleValueFormAlteris exactly. I'd rather do the opposite and have the full Hook attribute and possibly use a shorter method name.Comment #159
ghost of drupal pastCertainly. I said "if core wants to go that way" and outlined how it could look, what would it take in core and in rector. I am not calling any shots.
Comment #160
catchYes as someone who constantly greps looking for code and who doesn't use an IDE, it would be nice to retain this ability.
Comment #166
joseph.olstaddrush gen hook appears to have been forgotten for this refactor. Still no OOP option for drush gen hook !
Comment #167
joseph.olstadscreenshot of
drush gen hookwhen using Drupal 10.3.6 and also occurs with Drupal 11 and Drush Commandline Tool 13.3.3.0Comment #168
joseph.olstadhttps://github.com/drush-ops/drush/issues/6175
Comment #169
joseph.olstadIt's actually higher than drush, it's the Chi-teck drupal-code-generator
https://github.com/Chi-teck/drupal-code-generator/issues/194