Problem/Motivation
Scenario:
- A new node type
'article'
creates a new'edit any article content'
permission. - A new text format
'safe_html'
creates a new'use text format safe_html'
permission.
When those permissions are granted to a role, that role config entity will contain the 2 permissions.
If an object that is the foundation of a permission is removed the role doesn't revoke that permission. This is proved in the attached test.
(Steps to reproduce are in this duplicate issue -- it's for a D8 site, but I think they'll work regardless -- in case it helps anyone with testing.)
This is a bug because when a module is uninstalled roles are not cleaned up. This dangerous because if the module is re-enabled a role could have an unintended permission already configured.
Proposed resolution
- Make role config entity dependent on the provider of each granted permission.
- Make role config entity dependent on each entity (config or content) that was used to build that permission.
- The dynamic permission callbacks should optionally, return the dependencies used to build a specific permission (except the module which is already added as 'provider').
- The user_role config entity should implement
onDependencyRemoval()
. - Add proper dependencies to all dynamic permissions
- Trigger a deprecation if a permission does not exist when saving a role. Skip this for migrations - this will be resolved in #2953111: Only migrate role permissions that exist on the destination
The changes in this issue require updates to several existing tests.
- Only use permissions defined by modules. Fix typos. Do not use random strings.
- If a test uninstalls a module and then installs it again, related permissions need to be restored to test users.
- Some tests install
config_translation
but notlocale
. If any user role is saved, then all permission callbacks are exercised, and a service provided by thelocale
module is called.
Remaining tasks
Un-postpone #2953111: Only migrate role permissions that exist on the destination when this issue is fixed.Un-postpone or close #3211113: Add update path to remove permissions that do not exist when this issue is fixed, depending on whether we decide to include the update function with this issue.Add a follow-up issue to the Upgrade Status module. See #169.2.Comment on and/or un-postpone related issues that need to be updated. At least #2809291: Add "edit block $type" permissions, #3213736: Add "create block $type" permissions.
User interface changes
None.
API changes
Dynamic permission callbacks to optionally return the permission dependencies in the array.
Data model changes
Roles will have dependency information according to what permissions are assigned.
Release notes snippet
Each user role now depends on the modules that provide the role's permissions. This means that permissions will be automatically cleaned up when a module is uninstalled. Additionally, if a custom or contributed module uses a permission callback, see how to add additional dependency to the callback's return value. Existing roles will be updated to remove permissions that do not exist and dependencies will be added.
Comment | File | Size | Author |
---|---|---|---|
#225 | 2571235-9.3.x-225.patch | 83.92 KB | alexpott |
#225 | 223-225-interdiff.txt | 745 bytes | alexpott |
#223 | 2571235-9.3.x-223.patch | 84.07 KB | alexpott |
#223 | 217-223-interdiff.txt | 711 bytes | alexpott |
#217 | 2571235-9.3.x-217.patch | 83.38 KB | alexpott |
Comments
Comment #2
claudiu.cristeaComment #5
claudiu.cristeaQ.E.D.
Comment #6
claudiu.cristeaUpdate issue summary.
Comment #7
claudiu.cristeaFull patch.
I'm using
filter_format
as use case. For the others, I will fill followups.Comment #10
claudiu.cristeaComment #13
claudiu.cristeaComment #16
claudiu.cristeaFixed the missed 'dependencies' key in permissions array.
Comment #17
claudiu.cristeaOn a medium/long term I think that classes providing dynamic permissions must implement an interface and dependencies should be provided in a separate method of the class. I opened an 8.1.x followup #2574251: Dynamic permissions callbacks classes must implement an interface for this.
Added beta evaluation.
Comment #20
claudiu.cristeaReroll.
Comment #23
claudiu.cristeatest fixing.
Comment #25
xjmSee https://groups.drupal.org/node/484788 for more information on the rc target tag.
Comment #26
claudiu.cristeaRerolled.
@xjm, I added that tag because there's reverse issue #2569741: [PP-1] FilterFormat should add the roles as config dependencies tagged by @catch in the same way. They are both dealing with config dependencies.
Comment #27
claudiu.cristeaA better plan, IMO, would be:
permission_callbacks
array from*.permissions.yml
. But keep it till 9.0.x to allow the existing custom modules to work.DynamicPermissionsInterface
:Here, calculatePermissionsDependencies(), would return dependencies in the same format as usual, but nested under a top-level permission key:
DynamicPermissionsInterface
are placed in the "Permissions" namespace. Example:Drupal\node\Permissions
. Can beDrupal\node\DynamicPermissions
?.
\Drupal\user\PermissionHandler::buildPermissionsYaml()
to collect also permissions from all the classes implementingDynamicPermissionsInterface
and stored in the namespaceDrupal\{module}\Permissions
and store all permissions in config entities of type 'permission'.DynamicPermissionsInterface
. This will be magically, appended to the list of dependencies inPermissionHandler
.In 9.0.x just:
*.permissions.yml
file. Provide static permissions as default config .yml files, undercore/module/{module}/config/install
as{module}.permission.{permission}.yml
. Discuss how the {permission} name will be built (we can rely on a processed ->id() that only replaces spaces with dash '-'.PermissionHandler
to only grab dynamic permissions from theDrupal\{module}\Permissions
namespace.This approach has the benefit of providing the BC but still prepare for a cleaner 9.0.0 approach and assuring permission caching through standard config entity cache.
And this is doable right now, in 8.0.x because it breaks no API and assures the BC!
EDIT: Added 9.0.x tasks.
Comment #28
alexpottSo I don't think we're going to get round to #27 now. We do have a problem with cleaning up roles when things are deleted that provide dynamic permissions. We have exactly the same problems for the node type based permissions.
Comment #29
claudiu.cristea@alexpott, off-topic but related: permissions must be config entities in 9.x. Then we simply do config dependencies format <=> permission <=> role. That would fix a lot of other issues (like caching the permissions)
Comment #30
alexpottOh and we still don't cleanup permissions on uninstall of a module either.
Comment #33
dpiThis issue is pretty much the same as #607238: Permissions are assumed to be unique among modules, but uniqueness is not enforced
Comment #36
alexpottRe-rolled the patch after a couple of years. I think we should do this instead of #2026983: [regression] Grants for permissions defined by a module are not revoked / removed from Role config entities when that module is uninstalled.
No interdiff because it's been 2 years and by the looks of it #26 and quite a few unrelated changes in it so an interdiff is just going to confuse matters and be unnecessarily big.
Comment #37
alexpottWe need to implement and test for:
We need to add test coverage for:
Comment #39
alexpottcore/modules/taxonomy/src/TaxonomyPermissions.php has changed in 8.5.x so we're going to need to do this there. And I think it is likely this will be 8.5.x only because it will need an upgrade path.
Comment #40
alexpottWhoops.
Comment #42
dawehnerThis does not yet resolve the test failures.
Comment #43
alexpottAh the fun of module install :) So the fails are caused by calling $format->url() whilst creating configuration during a module install and the route provider can't kind the new routes of previous installed modules. Fortunately we already have a fix for this - #2913912: URL generator may have a stale route provider during module installation.
Comment #49
alexpottI have closed #2026983: [regression] Grants for permissions defined by a module are not revoked / removed from Role config entities when that module is uninstalled in favour of this issue because the approach to fixing the issue is more correct. However that means that this issue should be critical for the reasons that #2026983: [regression] Grants for permissions defined by a module are not revoked / removed from Role config entities when that module is uninstalled was. This is also a regression from Drupal 7. I'm crediting the people who worked on that issue.
Comment #50
alexpottAdded blockers to issue summary.
Comment #51
alexpottHere's an interdiff for #42
Comment #52
alexpottHere's the patch with the fix from #2913912: URL generator may have a stale route provider during module installation to see if anything else is broken.
Comment #54
claudiu.cristeaSome thoughts:
Nits:
Let's drop the @todo. Seems that #2569741: [PP-1] FilterFormat should add the roles as config dependencies is a "won't fix"
We use to use the standard "Check that ..." in comments explaining the assertions.
I don't think assertions are returning something in PHPUnit. Probably this if() is left here from the WebTestBase version of the test?
Comment #55
alexpottThanks for the review @claudiu.cristea
I outlined the need for an upgrade path and tests in #37 and #39.
The static permission thing - I'm not sure we should even support that - what would the use case be? These only really can depend on their provider.
Comment #57
dawehnerHere are some test failure fixes, not all yet.
Nice catch!
I don't think we have anything like a standard about that :)
Comment #59
alexpottSome more fixes. The implementation of
\Drupal\user\Entity\Role::onDependencyRemoval
has been rewritten.Comment #61
alexpottOh wow - so a lot of the API first tests added are depending on permissions that don't actually exist!
Comment #63
Wim Leers#61: Oh, hah! So evidently it's possible to grant non-existent permissions and have them be respected by the permission checking logic … otherwise those API-First tests could never have passed!
Comment #64
Wim LeersThis helps API-First Drupal become more reliable (in core, but also in the contrib JSON API and GraphQL modules), so tagging
.Comment #65
dawehnerIts indeed possible. I'm wondering whether we should add validation of some kind on the
grantPermissio
level. This would still lead to a fast runtime, but we have a form of validation at least.
Comment #66
alexpott@dawehner I think we should explore whether or not we should do that in a follow-up. I guess there is a question about whether or not this change in behaviour is allowed even in a minor. The problem being that granting non-existent perms is a critical as well because I think it is a data integrity issue.
Here's a patch that might be green. We have test coverage for:
Missing test coverage for
There is some implied test coverage for node and filter in UserRoleDeleteTest::testDependenciesRemoval() but I think we should have explicit test coverage that the dependencies are added correctly in each module that provides dynamic permissions that depend on things other than the module.
In the issue summary it used to say
I don't think this is a good way to go because once we add dependencies to roles than we need to try to keep them correct all the time. Doing this in steps just means we have lots and lots of upgrade headaches. Rather than one. We still need to write the upgrade path and tests for it.
The interdiff attached is pseudo interdiff because #2913912: URL generator may have a stale route provider during module installation landed so I had to rebase.
Comment #67
Wim LeersThe only other blocker, #2892821: Core modules check node module's "access content" permission for displaying things that have nothing to do with nodes, landed!
Comment #68
Wim LeersExpanded test coverage to explicitly ensure that the
access content
case is handled correctly. I'm not sure if it really is that very useful though…@alexpott: feel free to remove this small bit of extra test coverage for checking the #2892821: Core modules check node module's "access content" permission for displaying things that have nothing to do with nodes special case explicitly, if you feel it doesn't add much (and feel free to uncredit me).
Comment #69
alexpottThe interdiff on #68 is not correct. Here's the correct one. I think having explicit testing about not removing the
access content
permission is fine. It proves that other non-related permissions are maintained through module uninstall.Comment #70
Wim Leers:O how the hell did that #68 interdiff end up looking like THAT!? I haven't seen any changes like that in as long as I can remember. I'd swear d.o mixed up files here… :O
The #69 interdiff is the one I remember seeing an hour ago, thanks for uploding that.
And: great!
Comment #72
alexpottPatch adds test coverage for
And reverts the change to ModuleUninstallTest because
access content
is now provided by System there are no config updates when uninstalling Node.Comment #73
alexpott@Berdir talked to me about this issue on IRC and he brought up a great point. During Drupal install there's the ability for install profiles to provide alternate configuration to be used in place of the configuration provided by the module. People are using this to provide alternate user.role.anonymous and user.role.authenticated config entities. These entities have permissions that do not exist when user is installed. Therefore we have a big problem because the current implementation of this patch is likely to remove those permissions. Ideas of how to fix this are welcome!
Comment #74
BerdirThanks for writing that down here.
I mostly just wanted to make sure that we are aware of that problem, install profiles config/overrides are excluded from BC, we've broken that before e.g. for views and so on. But if we do, we should at least make sure we are aware of the impact and to document in the change record what should be done.
The only viable alternative that currently exists is to move that to hook_install() of the install profile and grant the permissions there once all modules are enabled.
That said, this is a general problem that doesn't just affect this, it's especially common in form/view displays, views and other complex config with a lot of dependencies.. e.g when you want to add another custom field that some module in your install profile adds to admin/content. A possible generic solution would be a config/post_override folder, which is config that would only be replaced once the installation is finished.. Or config_installer based install profiles, although I could imagine problems there as well, e.g. when the auth/anon user roles are installed only very late due to dependencies and other modules somehow rely on them in code.
Comment #75
alexpottHere's a patch which shows the nature of the problem @Berdir is talking about... it should fail.
Comment #77
alexpottSo one idea would be to not override configuration entities that are provided by profiles. And just update the configuration when the profile is installed. This seems to work. Perhaps we could just do all the profile updates when the profile is being installed and not do the early swap out.
Hmmm... actually that won't work because profiles can provide overrides for modules that are optional... ie. be installed after Drupal has be installed.
Patch attached tries to get the best of both worlds - also ConfigInstallProfileUnmetDependenciesTest is still failing because in this instance the dependencies can now be met!
Comment #79
alexpottSo let's split off the installer stuff into a new blocker to make it easier to review / think about in isolation... created #2922417: Profile provided configuration entities can have unmeetable dependencies
Comment #80
alexpottPatch includes an upgrade path and a test and the current patch from #2922417: Profile provided configuration entities can have unmeetable dependencies
So I think once the blocker lands we are good to go here.
Comment #81
effulgentsia CreditAttribution: effulgentsia at Acquia commentedI don't understand the nature of the problem in #75.
Why isn't the following enough to fix that?
Comment #82
effulgentsia CreditAttribution: effulgentsia at Acquia commentedOh, never mind. I'll comment in #2922417: Profile provided configuration entities can have unmeetable dependencies.
Comment #83
Berdir@effulgentsia because you can't install any module before installing user.module, that won't end well :) Also, config dependency order doesnot override module order, it just makes it fail if the dependencies aren't already installed. But maybe your "never mind" meant you got that already :)
Comment #84
Wim LeersOn second thought, I don't think this is a significant impact for API-First. I'm disagreeing with myself in #64 :) Untagging.
Comment #85
larowlanCan we get a change record here too - because obviously contrib etc will need to add their support for it.
I really like the approach here - nice work
@effulgentsia, @xjm, @plach, @catch and I discussed this on a triage call and agree this is critical, primarily because the presence of the permission on the role grants access, even if the permission doesn't exist.
I assume this is because we're using permissions with dependencies in tests - where previously those modules weren't enabled?
unrelated?
Should we have a follow up for this?
can we use strict checking here because they're strings
nit 'as a dependency'
Comment #89
jibranNW for #85.
Comment #90
dawehnerAdded one
Yes, in the exactly you've given, it's the "administer node form display" permission.
Seems so, I reverted it
See #3055557: Role migration results in migration permissions that do not exist on the target site
Comment #92
dawehnerI hope these are the remaining failures.
Comment #94
alexpottThe plumbing has made it in to the ConfigInstaller. Now we can focus on user role dependencies and not fixing the config system. Rerolled.
Comment #95
alexpottThis should use the ConfigEntityUpdater class.
These changes look odd - can't recall why they are necessary need to have a look at the issue history.
Comment #97
alexpottSome test fixes.
Comment #99
alexpottLooks like Views UI is not defining some link relation types. And this exposes this because Views entities use an admin permission that's provided by Views UI so the tests are not currently exposing this.
Comment #100
claudiu.cristeaI wonder why the patch decreases from 44 (in #97) to 6KB (in #99).
Comment #102
gabesulliceI think the test failures in #97 are exposing some fundamentally odd things about entity link templates: They use link relation types as their keys and REST is automatically trying to expose them.
What does an
edit-display-form
link relation type even mean to a REST client? In 99% of cases, that serialization format is going to be JSON/XML, not HTML, and neither of these media types define form semantics like HTML does.IMHO, we shouldn't be adding
Link
headers to REST responses for every link template. Especially if a matching link relation type is not defined for it. All that does is create a noisy and poor DX for anyone using REST. Worse, our . link relation types are not even validThe attached removes the assumption that every link template has an associated link relation type. I think we should do something to this effect and then define some follow-ups to disassociate them everywhere else.
n.b. the interdiff is to #97, not #99
Comment #104
alexpottThanks for looking at this @gabesullice - so the whole link relation thing happens because we have to install Views UI for the rest tests. We have to install Views UI because it provides the permission 'administer views' which these tests use. This permission is defined on the View entity provided by the Views module - so that's a weird circular dependency. Once we install the Views UI module that exposes a load of undefined link relationships. It's a completely separate bug that is being surfaced here and not really part of this issue. On HEAD ff you install views ui and rest and try and get views via rest you'll have exactly the same problem. I'll open a separate issue about this.
Comment #105
alexpottOpened #3088282: Missing Views link relationships when Views UI installed to address this link relation stuff.
Comment #107
gabesullice@alexpott, did you mean to add the link relation types in this issue? It sounded like you were instead going to address the permission placement instead. The interdiff just shows 3 new link relations.
Comment #108
alexpott@gabesullice well the permission placement is one solution. I'm not sure it's right though. If we put it in Views then people will have an "Administer views" permission with no UI which is odd.
Comment #109
gabesulliceIf users have JSON:API + JSON:API Extras in contrib, it's possible, although extremely unlikely, that users could be editing Views display config without a UI.
Comment #110
alexpottThe link relation type stuff has been put into #3088282: Missing Views link relationships when Views UI installed which is now RTBC and kinda blocks this one. Here are the remaining fixes to make the patch green. The patch includes #3088282: Missing Views link relationships when Views UI installed.
Comment #111
alexpottThis change shows how important this change is. In HEAD atm if you uninstall the field UI module you still get messages implying you can edit field settings whereas in reality there is no way to edit them.
Comment #112
alexpottWe can fallback to \Drupal\Tests\rest\Functional\EntityResource\EntityResourceTestBase::getExpectedCacheContexts() now. No need to override.
Comment #113
alexpottNow that #3088282: Missing Views link relationships when Views UI installed has landed rerolled the patch.
Comment #115
alexpottNeeded to re-apply #111 to the correct test.
Comment #116
alexpottNeed to convert to using the 8.8 file.
Need to convert to using the ConfigEntityUpdater
Comment #117
alexpottAddressing #116
Comment #119
alison(Caveat: I'm on 8.8.5, so I'm not going to change to RTBC, but...)
The functionality worked PERFECTLY for me when I tested it with a new vocabulary (full steps to reproduce/test in my duplicate issue from a couple months ago, in case it helps anyone else with testing).
Two questions:
Thank you @alexpott and everybody who's worked on this issue!!
Comment #120
alisonAlso, didn't notice this before: I get a slew of "notice" log messages when performing certain actions that I think trigger the check for permission configs -- here's an example message, when I was deleting my test vocabulary:
Notice: Undefined index: use text format full_html in Drupal\user\Entity\Role->onDependencyRemoval() (line 243 of /path/to/core/modules/user/src/Entity/Role.php)
>> Text format "full_html" no longer exists on this site, so "use text format full_html" is one of the many orphaned permissions we have. This example log message happened when I was deleting my test vocab (to test the patch) -- there were 6 such log messages at the time, idk how it decided which orphaned permissions to complain about. (I say that because ⤵️ )
>> I happened to uninstall a module while trying out this patch, and when I did that, the logs were ***FLOODED*** with a "notice" log message like the example above, for every orphaned permission -- we're talking several pages of log messages.
Anyway, FYI!
Comment #121
IJsbrandy CreditAttribution: IJsbrandy at Betawerk commentedRerolled patch from #117 for latest 8.9.x
Comment #122
alexpottHere's a patch for 9.1.x since this will only get commit to 9.x at this point and probably only 9.1.x since it has a post update.
Comment #123
alexpottRe #119 / @alisonjo315 this patch does clean up roles with permissions that do not exist. You need to run update.php / updatedb
This function will remove non-existing permissions.
This tests this functionality.
Comment #124
alexpottWe need to address this - one thought is that we could support adding non existent permissions in D9 but deprecate it and then remove support in D10. This might help contrib / custom tests deal with this change in a graceful way.
Comment #126
alisonAdd link to steps to reproduce to issue summary -- feel free to remove, esp since it's on a D8 site, just adding in case they're helpful, but I won't be offended if it's determined they don't belong here.
Comment #127
alisonFWIW, #121 works great for me on 8.9.12!
@alexpott Thank you for the comment - sorry I disappeared - the update hook totally worked, as you said, idk what happened to me back in May, but it all definitely works, so, hooray! I 💜 my permission config entities now....
Comment #128
alexpottRe-rolled for 9.2.x. It'd be great to get this to rtbc and committed.
Comment #129
alexpottAdded release note.
Comment #131
alexpottFixing the broken tests.
Comment #132
alexpottRemoving the @todo and seeing what is broken....
Comment #133
alexpottResolving some of the fails in #132 which shows the importance of resolving most of #3055557: Role migration results in migration permissions that do not exist on the target site in this issue. I think in a future patch will need to trigger a deprecation when saving with a permission that does not exist as that will help contrib / custom deal with this and prevent core from regressing again.
Comment #137
alexpottDiscussed the migrate fails with @heddn. Given the complexity of the migrate part of this I think it is better to punt that to #3055557: Role migration results in migration permissions that do not exist on the target site a re-purpose that to working out what to do about migrations.
Discussed where to put the role clean-up on filter disable with @catch. After considering putting in the Filter entity, on the filter module we plumped for the user module because this means tests like EditorFilterIntegrationTest don't have to change and it is Roles changing when a filter is disabled and Roles are owned by the user module and that's what the hook is for.
Comment #139
alexpottFixing the last fail.
Comment #140
alexpottImproved the issue summary.
Comment #141
alexpottThese tests depend on the 'administer node fields' permission and that is provided by the field ui module. Perhaps we should consider moving that to the field module? There's a similar thing going on with Actions and Views UI. They both provide permissions which are used by entities provided by other modules.
The other permissions are:
I think given JSON:API in core we should consider moving these permissions to the module that provides the entity that depends on these permissions. So...
I'm not sure where 'administer display modes' should be. I think looking at the usage field_ui.permissions.yml is fine - it's where it current is. EntityViewDisplay and EntityFormDisplay both use permissions created by \Drupal\field_ui\FieldUiPermissions which I think should be moved.
Comment #142
daffie CreditAttribution: daffie commented@alexpott I think that you are right. Those permissions should be moved. I have created #3193983: Move entity permissions to the module that is providing the entity for it. Also postponing this issue for it.
Comment #143
alexpottRerolled the patch on HEAD. Conflict in core/modules/filter/tests/src/Kernel/FilterCrudTest.php due to #3193955: Swap assertEqual arguments in preparation to replace with assertEquals.
I'm not sure that #3193983: Move entity permissions to the module that is providing the entity needs to be blocking - it raises a number of tricky issues about whether permissions that using a used by a UI should be moved to modules that don't provide the UI.
Comment #144
alexpottComment #145
alexpottTurns out [#3055557 ] is a dupe of #2953111: Only migrate role permissions that exist on the destination
Comment #146
benjifisherIt is not clear to me what the plan is for migrations. Currently, the
d6_user_role
andd7_user_role
migrations happily copy over whatever permissions are listed on the source site, with no validation.The patch in #143 adds
to those two migrations. Then, when calculating dependencies for the Role config object,
Minor point: I do not like the name
migrating
for this flag. It describes where it is set, not what it does. I prefervalidate_permissions
as in #2953111: Only migrate role permissions that exist on the destination.Use the core migration directly
Suppose someone uses the core migration directly, either with Drush or with the UI provided by the
migrate_drupal_ui
module. Then themigrating
flag is set, so we never get to the@trigger_error()
line. No errors, no warnings, no messages of any kind.In another year or two, the same developer starts working on a migration from Drupal 7 to Drupal 10. Is the plan to keep the
migrating
flag and replace the deprecation with an exception? Or remove the flag and turn the deprecation into an exception? In the first case, we keep migrating non-permissions. In the second case, the developer is surprised by the exception messages, reads the release notes and/or change record, and gets back on track.Use a copy of the core migration
A developer may have a custom migration,
my_module/migrations/my_user_role.yml
, a modified version of one of the core migrations. Probably more common, using themigrate_plus
module, the developer has a config entity exported as a YAML file.The user-role migrations are done early in the project, with Drupal 9.1. Once the modified migration is created, it will not be affected by changes to the core migration. Then Drupal 9.2 is released, shortly before the deadline for the migration project. (I love the swooshing sound ...) As the team starts the final tests before running the migrations on the prod database, they start hitting the
@trigger_error()
line. Since the error level is onlyE_USER_DEPRECATION
, does that even generate a message? Where does it show up: in the logs, in the migrate message table, somewhere else?Recurring migrations
Not all migrations are site upgrades. The Migrate API supports continuing migrations such as feeds, user imports from other systems, or incremental upgrades.
If I am importing from an external system, am I likely to have extra permissions floating around that do not exist on the target site? Call it an edge case. Maybe a module was disabled on the target site, and neither the source system nor the migration got updated.
The previous two headings are relevant: are we using the core migrations directly, or are we using custom/copied migrations? If we are migrating from an external system, probably the latter. If access to the source system is tightly controlled, and we never set up the test/staging site to do that, then the deprecation messages will only show up in the logs for our cron jobs, if at all.
If a site has a recurring migration and does not use the core migration directly, then bad things will happen when Drupal 10 is released and the deprecation is replaced with an exception. I want to understand where the deprecation notices will appear before that happens.
Comment #147
alexpottThe flag added here never ends up in storage. It's only used while the object is being saved by migrations. Once the migration is over and you manage your permissions on the site then the deprecation would be triggered every time a role with a missing permission is saved. The D10 upgrade module can easy be made to check this for an existing site.
FWIW if you have a recurring migration that changes permissions - I think you are living on the edge and need to have a big think about how you are doing things - and if it is adding a permission that does not exist and cannot be managed by the site - well ideally your migration would fail but with this patch it'll work and the same statement about how and when deprecations happen would apply.
For in play migrations that don't have the migrating flag - you're right the deprecation will only be caught if they have deprecation testing but as above this is fine and to the point we will only drop support for permissions that don't exist in D10 and that'll have an upgrade path. The reason the migrating flag exists is so we can pass core's tests and not have to deal with migrations in this issue. It'll be perfectly okay for #2953111: Only migrate role permissions that exist on the destination to fix all the core migrations and remove the flag.
Comment #148
benjifisherNow that #3055557 is closed as a duplicate of #2953111: Only migrate role permissions that exist on the destination, we should update the comment. I still do not like calling the option
migrating
.Same comment about the
@todo
comment.When deprecation messages are generated, we will get a separate message for each invalid permission. That is a little verbose, but it is good that we see all the invalid permissions. But when we change the deprecation to an exception, we will get only one error message, for the first invalid permission.
I suggest calculating
array_diff()
andarray_intersect()
. If the first is not empty, then generate a single message containing all the invalid permissions. If it is empty, or if themigrating
flag is set, then loop through the valid permissions.I think this change will also simplify the control flow (cyclomatic complexity).
Comment #149
benjifisherThe changes to
core/modules/filter/src/Entity/FilterFormat.php
look like unrelated coding standards cleanup.Why this?
I do not have time right now, but I would like to compare with the list of core modules that have permission callbacks: #2953111-40: Only migrate role permissions that exist on the destination. Are all of them updated in this issue?
Comment #150
alexpottI've changed the flag to
skip_missing_permission_deprecation
- doesn't hurt be be super descriptive.Nice spot of the FilterFormat - wonder how that crept in /shrug.
I disagree that the control flow would get better if we have to build up a list of missing permissions. In D10 we do want to throw an exception on the first missing permission. If we build up this list now then we'll be tempted to keep more complexity later when it offers no benefit. Also this is not a common case - normally permissions exist and roles are valid. So any optimisations (if they are actually optimisations) would be for the uncommon and exceptional case.
Re changes to PathAliasResourceTestBase thats because these tests have an unbelievably complex class hierarchy and the code is in \Drupal\Tests\rest\Functional\EntityResource\EntityResourceTestBase::getExpectedCacheContexts() which now returns (as it should)
All entity types that have an admin permission with have a cache context of user.permissions. The url.site is because in order to have the admin permission we have to enable the path module... there is further discussion of the entity permission issue in #3193983: Move entity permissions to the module that is providing the entity
All the ones that have callbacks that create permissions determined by config, content or a module being installed are changed here - to add in the dependency information.
Comment #151
benjifisher@alexpott, thanks for the answers in #150. I will follow up later on PathAliasResourceTestBase.
I am starting a comprehensive review of the patch in #150. I will start with something pretty simple, the
user.role.*.yml
files in the core profiles.I checked the
user.role.*.yml
files in the core profiles that are not touched by the patch in #150. Indemo_umami
andstandard
, that means theadministrator
role. Those haveis_admin: true
andpermissions: { }
, so that is fine. Curiously, thetesting_config_overrides
profile already declares a dependency on thetour
module.I also installed the
demo_umami
andstandard
profiles, exported configuration, saved each role in the admin UI, and exported configuration again. There were no changes.I thought that
access content
was defined by thenode
module, but there is a comment insystem.permissions.yml
that explains my confusion.The
system
anduser
modules are both required, so I guess we do not need to declare dependencies on them. Most of the roles in the core profiles declare a dependency onsystem
, but none declare a dependency onuser
. Theauthor
andeditor
roles in thedemo_umami
profile include thecancel account
permission from theuser
module.I reverted
core/profiles/*
to match 9.2.x, then installeddemo_umami
andstandard
and exported configuration. During the profile installation, the roles were saved, and that is when dependencies were added (I think). I then restored the profiles to the version provided by the patch and compared to the exported configuration.In the
demo_umami
profile, neither theauthor
noreditor
roles depended on theworkflows.workflow.editorial
configuration, even though this dependency is added in the patch and both roles include permissions likeuse editorial transition archive
. This seems like a bug.I checked the list of core modules that provide permission callbacks: #2953111-40: Only migrate role permissions that exist on the destination. Most of these callbacks are updated in this patch, but not the ones from
layout_builder
normedia
. That deserves a closer look.It is getting close to my bed time, so that is all for today.
Comment #152
alexpott@benjifisher thank you for your review.
Re #151.2 configuration does not need a dependency on the module that provides that configuration. views.view.blah does not have an explicit dependency on views same for user.role.blah - the dependency is implicit and ensured by the config system itself. This is how config dependencies work for simple configuration such as system.site, user.mail, node.settings etc...
Re #151.3 The workflow here is exactly how people get bugs with the configuration system. Without the dependency information correct during install it doesn't know that it has to import the role after creating the workflows.workflow.editorial config. This is how config dependencies work during install. FWIW if you go and resave the role on
admin/people/roles/manage/author
the dependencies will be re-calculated and the role's dependencies would be fixed. Also \Drupal\Tests\demo_umami\Functional\DemoUmamiProfileTest::testConfig() exists - this ensures that the configuration in the db matches the config in the profile after install.Re #151.4 That's definitely a bug. These were added well after work on this patch started - lol.
Comment #153
alexpottAdding the missing dependencies to layout_builder and media modules. Need to add test coverage of this - let's see if the existing test coverage can help us find where best to add this.
Comment #154
alexpottHo hum... thought I'd fixed that before creating the patch.
Comment #155
benjifisher@alexpott, thanks for the explanations.
In #151.3, I wrote,
All the other dependencies were added, but not
workflows.workflow.editorial
configuration. I repeated the test, saving the roles before exporting, and the dependency was added.Here are some comments on the comments (sorry) in the permission handler and its interface:
A few nits with this comment:
Are we building a "permission name"? Maybe just "permission" or "An array of entities on which this permission depends, structured …".
Add a comma after "For example".
Keep sentences short. Instead of "because", start a new sentence: "It is automatically parsed …".
Same snippet: Can we remove "(optional)" from the
provider
key, or is that considered an API change? The implementation always provides a value, and the new code inDrupal\user\Entity\Role
does not check whether it is there.I do not like adding this example here, since the code snippet is introduced with "Here is an example from the core filter module (comments have been added):". In fact, there are no examples in core
.permissions.yml
files of dependencies being declared.Can we correct the line "restrict access: false", or is that out of scope?
There are no examples in core, but either a YAML file or a callback could explicitly set the
provider
key. Should we document this? What about thewarning
key?Maybe we should shorten the comment on
permission_callbacks
in the code snippet, then add something like this after the snippet:The second "…" is because I am confused by "the return config entities dependencies".
Comment #156
benjifisherFollow-up to #148, #150: this looks simpler to me.
The error message needs some work. I am not sure why you want the error message (when we turn it into an exception) to mention only the first invalid permission, but it would be easy to use
reset()
instead ofimplode()
.Comment #157
benjifisherI have already made a few comments on the changes to
Drupal\user\Entity\Role
, but this is the first time I have looked atonDependencyRemoval()
.I think this makes the intent clearer:
Same snippet: I am confused by the condition
in_array($type, ['config', 'content'], TRUE)
. I read the API docs foronDependencyRemoval()
and followed the link to the class documentation forConfigDependencyManager
. That explicitly lists the outer keys of the$dependencies
array:config
,content
,module
, andtheme
. (Hereenforced
does not apply.) I guess that$dependencies['config']
and$dependencies['content']
are arrays of entities, and$dependencies['module']
,$dependencies['theme']
are arrays of strings.If I have that right, then I will submit a little documentation patch to say so explicitly.
Consider changing
in_array(...)
to$dependency instanceof EntityInterface
. If you do that, then update the comment.If you prefer to test
type
, then why not do it in the outer loop? Or make the outer loopforeach (['config', 'content'] as $type)
and the inner loopforeach ($dependencies[$type] ?? [] as $key => $dependency)
.Nit: add a comma at the end of the first comment line.
Same snippet: I suggest moving the definition of
$permission_definitions
closer to this loop.Same snippet: after
$changed = TRUE
in theelseif
clause, we couldbreak
out of the inner loop orcontinue 2
the outer loop.Comment #158
alexpott@benjifisher re demo_umami installation without the presence of the dependency in user.role.author - there's no way the configuration installation can work out that workflows.workflow.editorial needs to be created before the user role. This is how config dependencies work during install.
re #155.1 Fixed.
#155.2 The provider should stay optional - it is possible to use this to provide permissions on behalf of another module. Not a great idea but it is the way it is. This is the same as other provider based things in config and plugins.
#155.3 I agree - while this is possible it is pretty nonsensical so let's not document it here. This was added before I started working on this issue :) Oh I see why this was added - because you need to know about dependencies for the permission callback section. We definitely shouldn't be add anything about the warning key here - that's out-of-scope. Feels worth a separate issue to add documentation about
warning
to\Drupal\user\PermissionHandler
. However I do think we should point people more to\Drupal\user\PermissionHandlerInterface::getPermissions()
because that has the definite list of keys.#155.4 as per the answer to .2 - i don't think we should document this any more than the documentation in \Drupal\user\PermissionHandlerInterface::getPermissions() - it's not really an intentional use of the API.
I like #156 - with a few tweaks to make it work and clearer variable names.
#157.1 I've refactored this to make it more explicit that we're only dealing with content and config
#157.2 Te refactor has dealt with this. +1 to a separate issue to improve the docs on \Drupal\Core\Config\Entity\ConfigEntityInterface::onDependencyRemoval()
#157.3 Fixed
#157.4 I disagree - getting the list of perms is less important than the loop that flattens the dependencies being removed.
#157.5 Good spot - implemented.
Comment #159
benjifisher@alexpott:
Sorry, I was not clear: the comment at the start of #155 was meant as confirmation that it worked the way you said in #152.
This comment completes my review of the non-test code for this issue.
Speaking of tests, are you planning any additions related to the
layout_builder
andmedia
permissions callbacks (#153, #154)?Follow-up to #155.2:
If the
provider
key really is optional, then we should add a!empty()
check for it.I think the same point applies to
onDependencyRemoval()
.Follow-up to #155.3: Can we correct the line "restrict access: false", in
PermissionHandler.php
or is that out of scope?Thanks, I think this is a lot clearer:
I only wish that PHP made it easier to apply a method. I thought I saw something about simpler syntax for anonymous functions in PHP 8, but I cannot find it now. :-(
This looks fine, although I might do
$dependencies = ['config' => [$filter_format], 'module' => []];
outside the loop:The real question is whether there are any similar hooks we should implement. Where else do we have the option to disable some configuration without removing it? The only thing that comes to mind is views, and I do not think there are any per-view permissions.
I did not know about the
ConfigEntityUpdater
class. Good to know!This passes a callback to the
update()
method, so the default callback will not be used. But the callback returnsTRUE
, meaning that the entity will be saved, and that will trigger calculating dependencies, right?I guess this calls for the "T" part of RTBC. I repeated the earlier test, installing the
demo_umami
profile with configuration from 9.2.x. This time, I also reverteduser.post_update.php
before the site install. After the initial config export, neither theauthor
noreditor
role had theworkflows.workflow.editorial
dependency. I ran the database update and re-exported configuration. The dependency was added. Test passed!This gets the job done, but it is not easy to read:
This is inside the case where
$permission_granularity = $entity_type->getPermissionGranularity()
evaluates to'bundle'
. Do we still need that complicatedif(...)
?Same snippet: it would be simpler if we could first calculate
$dependencies
and thenIf the answer to the previous question is that we need to keep part or all of the test, is there any harm in adding
'dependencies' => []
?Nit: remove "should".
This follows the pattern that I suggested in the previous point.
For consistency, can we use the same pattern here?
It is simpler to add the dependencies in
buildPermissions()
:If I count correctly, it even saves 2 lines, even though you have to add the
'dependencies'
key 5 times.Same here. It is simpler, but it might add a line or two.
Here I agree that we should add the permissions here instead of in the plugins, but I have two nits:
Nit 1: add an
array
type declaration for$permission_info
.Nit 2: use the
+=
pattern one more time:I would use a
foreach()
loop instead ofarray_map()
, but I think that is an individual preference.Again, it is simpler to add the dependencies in
buildPermissions()
:Comment #160
benjifisherFollow-up to #157.2: I added #3209296: Fix API docs for ConfigDependencyManager and #3209309: Improve documentation for ConfigEntityInterface::onDependencyRemoval(). I am not sure what the correct fix is for the second.
Follow-up to #159.13: Maybe it is simpler to use
NestedArray::mergeDeep()
.Comment #161
alexpottRe #159.2 It's optional from the point of view of permission.yml or the a callback implementation - in fact it should never be set. It's always present though because that is guaranteed.
#159.3 it's out of scope imo
#159.5 I don't think we do - also this is only about things that can determine a permission. Plus config entity disabling is not really well supported at all.
#159.6 Yes that is how it works.
#159.7 Yes - it is possible that bundles might not be configuration - there are some test entity types that use state for example.
#159.8 I don't see the benefit here of doing that.
#159.9 Fixed
#159.10 Sure I've changed this.
#159.11 Doing things the same way everywhere is necessarily good. In think here there are two benefits to the array_map approach. One, \Drupal\media\MediaPermissions::buildPermissions is really simple to read. Two, if a new permission is added to \Drupal\media\MediaPermissions::buildPermissions it'll automatically get the dependency.
#159.12 As above
#159.13 NestedArray::merge is a good idea
#159.14 as per .11/.12 I disagree.
Added more explicit test coverage of RestPermissions - still need to add coverage for Media and Layout Builder.
Comment #162
benjifisher@alexpott:
The patch in #161 removes "(optional)" in
PermissionHandlerInterface
, so I guess we agree on #155.2 after all.+1 for eliminating
$perms_to_add
in the permissions callbacks and just using$permissions += array_map(...)
. "Get rid of the middleman."I will not press on most of the changes to the permissions callbacks, but I would like to see some simplification in
ContentTranslationPermissions
(#159.7, #159.8). The benefit of #159.8 is that long lines like this are hard to read, hard to understand, and therefore harder to maintain and less secure.We will get an exception from
$this->entityTypeManager->getStorage($bundle_entity_type)
if$bundle_entity_type
is NULL or an empty string, but also if it is'not_valid_bundle_type'
. We can protect against that and keep the lines short with something like this (untested):I probably will not have time to look at the tests until next weekend.
Comment #163
benjifisherCan we add some feedback and logging to the update function?
Suppose that my site has the Easter module. I enable it one day a year, then uninstall it, but keep the related permissions. I know this is not best practice, but it seems to work. I should be notified.
I suggest one log message for each role that has permissions removed, and one message from the update function. Draft text:
It might even be a good idea to postpone the update function until Drupal 10.
Comment #164
benjifisherI am starting to review the changes to the tests.
Reviewing the tests seems like a big task, with 43 files changed. Luckily, about 10 of them are like this:
@larowlan asked about this in #85.1. In #141, @alexpott explained it is because these tests use the ‘administer node fields’ permission, and suggested moving that permission to the
field
module. I was going to suggest a follow-up, but I see we already have #3193983: Move entity permissions to the module that is providing the entity.Why do we need to rebuild the router? I think a code comment would help.
Nit: I do not like the pattern of defining a variable inside an explicit array like this:
Can we use
$role = Role::load($role->id())
instead? Alternatively, put$rid = ...
on a separate line so it is easier to scan.The machine name and title do not match.
Nit: can we expand the first line of the doc block?
I do not see where this permission is used:
Is it related to this change?
I am taking a break for lunch. I will assign the issue to myself for the rest of the review.
Comment #165
benjifisherI finished reviewing the changes to the tests. I do not have any more changes to request: just a couple of comments and a bunch of questions.
I see that I already asked about
getExpectedCacheContexts()
in #149, and I just re-read the reply in #150. I am still trying to understand this change and how it relates to this issue.We remove the override in four places, falling back to the method defined in
PathAliasResourceTestBase
. In one case (views
module) this is a no-op, so the only question is why the change is in scope for this issue. In the other three cases, removing the override means we are adding theurl.site
context. In each of those cases, we are also enabling an additional module. In the test for thefield
module, we enablefield_ui
; forpath_alias
, we enablepath
; insystem
, we enableaction
.In all cases, we need to enable the extra module because it defines permissions that we want to test. These changes would not be needed if we fixed #3193983: Move entity permissions to the module that is providing the entity before this issue.
I am afraid I still do not see how enabling the extra module adds the
url.site
cache context.The other question is whether we are making these changes in the right place. We are enabling an extra module in four blase classes, and adding an expected cache context in three of these. Why do it there and not in the child test classes?
@quietone and I already discussed why we need to add
locale
to the migration tests: see #2953111-45: Only migrate role permissions that exist on the destination. For example:I think this is correcting a typo:
If I am right, then why is it in scope? If I am wrong, then we are intentionally adding a bogus permission, and I would expect further changes to the test.
Where are these test permissions referenced?
It makes sense to enable these modules instead of the
user
module, but I wonder why it worked before. How is it in scope for this issue?I am surprised we do not have more changes like this one. From now on, our tests have to assign real permissions when they want to change a role.
Why this change?
I see that the test fails without this change, but I do not see why.
Do we test somewhere that the permissions change?
I see that the test fails without this change, but I do not understand why. Changing permissions means that the permissions hash changes, so we do not get the cached copy of the page. But I am still missing something: why does this test work before this issue?
Comment #166
alexpottRe #163
I think moving the update to follow-up and something we discuss separately. Including timing is a great idea. This is possible because we're deprecating support for permissions that don't exist. There's a slight oddness because permissions will still be removed on uninstall if we can work out that the permission depends on a module that is being uninstalled but I think that that is okay.
Re #164
2. We no longer need the route rebuild - we fixed route rebuilding during module install a while back.
3. Fixed - this uncovered that we really should be forcing a proper reload of the role.
4. Fixed
5. Fixed
6. \Drupal\Tests\page_cache\Functional\PageCacheTest uses the
pet llamas
permission. This can be found by searching the code base. We don't want to start a new standard of documenting where permissions are used.Re #165
1. This is because we're now generating a url because the test now includes the module that defines the admin permission. I do not think that fixing #3193983: Move entity permissions to the module that is providing the entity is simple and maybe doing so is not even desirable. However in order to have full API coverage of these entities and have their admin permissions actually exist we need to enable these modules. This is a good thing because it reflects the current reality. On a real Drupal site you'd struggle to use the JSON API to create views without the Views UI module being enabled (for example). So this issue is stopping us from pretending things make sense in the tests - when actually they don't.
3. That permission does not exist so we need to fix that in this issue - this is another massive benefit of making this change. When you use a permission in tests it'll be a real permission. We could file a follow-up because I'm pretty sure that we don't need to assign these permissions anyway. The test is making these transitions to the entity under test but it is doing do on the API level and therefore permissions are not checked (as far as I know). I prefer to fix the permission to use the real and intended permission here.
4. The codebase can be grepped to reveal this. I don't think that adding comments benefits us here. The module names tell you that these permissions are being used for access testing.
5. This all because the tests in HEAD are assigning the admin permission without it existing.
6. Well it really doesn't make sense to assign a permission that does not exist. Also it is dangerous because often then you'll not be testing what you think you are testing.
7. Because \Drupal\media_library\Plugin\Field\FieldWidget\MediaLibraryWidget::getNoMediaTypesAvailableMessage() changes the message depending on whether field_ui is installed or not and whether the user has a permission. This permission is now correctly removed when the field UI module is uninstalled.
8.
$this->assertDifferentHash();
does that. This hash is built from permission. If it is different then permissions have changed. This is documented in the test.Added test coverage of Media permission dependencies - still need to cover Layout Builder.
Comment #167
alexpottCreated #3211113: Add update path to remove permissions that do not exist - whilst doing that I realised that there is a massive downside of not including the update here. That is, we're not fixing the gotcha for existing sites. They'll still be left if the position that enabling a module might result in roles having active permissions that they'd normally not have. I'm not sure removing the update is such a great idea anymore. Hmmm.
Comment #168
benjifisher@alexpott:
I will look at the new patch, and the comments in #166, after work today.
As for the update function, we are also not fixing the problem when saving a role. We just issue deprecation messages. We have lived with the status quo for a long time. I think we should put up with it until a major version upgrade.
Maybe expand #3211113: Add update path to remove permissions that do not exist to include changing the deprecation messages to exceptions.
So far, I have done a lot of R, but only a little T. When I save a role with bogus permissions, will I see the deprecation messages in the admin UI? In the log?
Have you made any changes in response to #162?
Comment #169
benjifisherI did some testing.
I inspected and modified permissions for the
authenticated
role using the config import/export pages under/admin/config/development/configuration/single
.I saved the role by saving its permission form,
/admin/people/permissions/authenticated
. I did not test, but I think I would get the same results using/admin/people/roles/manage/authenticated
.demo_umami
profile.foo
andbar
text formats, and give theauthenticated
role permission to use them.foo
andbar
content types, and give theauthenticated
role permission to create them.authenticated
role. (Part of the previous step.)user.role.authenticated
configuration: it has the expected permissions, no dependencies.foo
text format.foo
content type.authenticated
role.user.role.authenticated
configuration: no changes.authenticated
role.user.role.authenticated
configuration: it now has dependencies on thebar
text format and thebar
content type, and all the same permissions.bar
text format.bar
content type. Theauthenticated
role is listed as configuration that will be updated.authenticated
role.user.role.authenticated
configuration: thebar
dependency and permissions have been removed.So far, so good: that is what we expected to happen.
Checking the logs, I see a lot of PHP Notices ("Undefined index") from this line:
We need to add a check that
$permission_definitions[$permission]
is defined. This will come up if we defer the update function to a follow-up issue, if we choose not to but some site owners neglect to run it, or if permissions are migrated without the defining module. NW for this.From #147, replying to one of my questions:
Does "D10 upgrade module" mean the version of Upgrade Status that will run on Drupal 9 and prepare for upgrades to Drupal 10? If so, then I think we should add an issue now for that module.
Unless I am missing something, there is no more natural way to see the deprecation messages. There is a PHP setting that would make them show up in the log, right? But almost no one would enable that on a live site, and almost no one would think to test saving user roles during deprecation testing. (That is why we have to add an issue now to do that.)
We should expand the Permissions must exist change record. In #163, I suggested the hypothetical Easter module, enabled just one day a year. What about the Monday module? If I enable it once a week, and then disable it the next day, I do not want to set up permissions every time. Currently, I do not have to. After this issue goes in, I will. This impacts site owners, not just module developers. Maybe we should add a third CR instead of adding this to an existing one.
The test was passing before, but I will take your word that it is a good idea to reset the cache here. It is a little odd to use
Role::load()
instead of$role_storage->load()
. Why not use the class from the container?Re #164.6: I see. The test already assigned the
pet llamas
permission to a role, even though the permission did not exist. After this issue, that test would fail. #164.7 is similar.I am still trying to understand #164.8. Maybe it will be clearer in the morning.
Comment #170
benjifisherI did not even need to sleep on it. I just needed to step away from the computer for a bit.
After disabling the
taxonomy
module, the admin menu does change. The problem is after enabling it again: unless we restore the "administer taxonomy" permission to our test user, the admin menu will not change.I think it is a good idea to give the admin user an extra role, with just the "administer taxonomy" permission, after re-installing the
taxonomy
module, instead of giving the admin user a true admin role (all permissions) at the start of the test method. Among other things, it makes the test more like a realistic use case, where you always assign permissions after installing a module.Something like this, before the second call to
assertDifferentHash()
:Sorry, I am rambling. I really do need to get some sleep.
Comment #172
benjifisherI am adding some notes to the issue description to help with the final review.
Looking back on earlier comments, I think these still need to be addressed:
Comment #173
alexpottPatch attached fixes:
Review comments:
$bundle_entity_type = $entity_type->getBundleEntityType();
out of the loop and the if - which makes the line shorter.I've thought about #167 a bit more and I think we need the updates in this patch because we can't issue the deprecations unless we've taken steps to clean them up and this patch also adds removal on uninstall so not cleaning up will introduce some very very odd behaviour.
Comment #174
alexpottLet's update the deprecations to 9.3.0 - this won't land in 9.2.x now.
Comment #175
benjifisher@alexpott, thanks for the updates.
Notice that I have added references to this issue from three other open issues recently:
If we are restoring the update function to this issue, then can we add some feedback and logging to the update function? I think it should log a message for each role that has permissions removed, listing all of them. Then it should return a message listing those roles, with the usual "check the logs for details". It should be pretty much the same logic we added to generate one deprecation message per role. I suggested text for the return message in #163.
I think the update function should be mentioned in a CR and in the release-notes snippet.
Setting
$bundle_entity_type = $entity_type->getBundleEntityType();
on its own line helps, but I still find this really hard to read:I think this is a lot easier:
Missing type declaration:
If you do not like the suggestion in #170, then please change this comment (see #165.8 for context):
I found this comment very confusing. The point is not that permissions change. The point is that the links in the admin menu (and therefore the tree hash) change when the
taxonomy
module is uninstalled and when it is enabled again.General point: I often quip that writing documentation is really very easy … provided that the audience is the same as the author. The hard part is writing documentation that will be helpful to others, and that is really quite hard.
Along the same lines, you are not the best judge of whether the code you have written might be confusing to others, just as I am not the best judge of the code that I write. Please take seriously the fact that I have trouble with the long lines of code in
ContentTranslationPermissions
and the comment inToolbarAdminMenuTest
.Comment #176
benjifisherComment #177
alexpott1. This is great idea. I will do that in a follow-up patch.
2. I'll do this in the follow-up.
3. I disagree with the local variable approach because rather than removing complexity it adds complexity for the reader. There are more variables to keep in your head and more possibilities for side effects. This code suffers from the complexity of the situation - grafting translation permissions on top of the entity system is complex and our existing implementation makes this hard because bundles can be anything - in core tests we have a bundle implementation that relies on state!
However given the disagreement let's go a whole step further and encapsulate the bundle permission array building in a private function where it's easier to document the bundle stuff and also prevents side effects. For example in HEAD atm the $t_args array is mutated as we loop which can result in t() being called with incorrect arguments that happen to be unused in core's translations. Using a private function here is good because this method has no meaning outside of the call from \Drupal\content_translation\ContentTranslationPermissions::contentPermissions().
4. Fixed
5. I've improved the comment - you're right that this should be clearer. Hopefully the new version will help it explain this us when we're looking at this again in 6 months time!
Comment #178
benjifisherI think the patch in #177 needs a reroll after #3131281: Replace assertEqual() with assertEquals().
Comment #179
benjifisherI confirmed the fixes for #175.4, #175.5.
The new helper function is a big help for #175.3. Even little things help reduce cognitive load when reading code, like not having to read
"translate $bundle $entity_type_id"
twice (and check that it is the same both times).I just have a few tiny suggestions for the new function:
The
@param
comment for$t_args
could usestring[]
instead ofarray
. Can we add something likeRequired key: @entity_label.
to the comment?Similarly, can we add to the
@return
comment? Something likeThe permission details, keyed by 'title' and, if available, 'dependencies'.
Suggestion: call
$entity_type->getBundleEntityType()
in the calling function, so the first argument becomesstring|null $bundle_entity_type
. Then you do not need to add theuse
statement at the top of the file.Simplify the first line of the function:
Or delete that line and use
$t_args + ['%bundle_label' => $bundle_info['label'] ?? $bundle]
inside$this->t()
.Comment #180
alexpottThanks for the review @benjifisher
#179.1 I've removed the $t_args thing - it is a premature optimisation that doesn't optimise. \Drupal\Core\Entity\EntityTypeInterface::getSingularLabel is statically cached anyways.
#179.2 I wish we had Symfony's policy on private method documentation. But sure added this.
#179.3 If you make the first argument optional then all the other args have to be optional - we could make entity type the last arg but it is the superset - ie all bundles belong to an entity type so that doesn't make sense to me. Anyway now I've remove $t_args this point is moot.
#179.4 $t_args is removed - see point 1.
Still need to add more logging to the update function.
Comment #182
benjifisherThanks, I like the changes in #180.
Is there an honest way to generate an interdiff after a reroll?
Is it my imagination or is the testbot generating more random fails than usual in the last couple of weeks? It looks as though the patch in #180 failed, but you (or someone) triggered a retest.
Still NW for #175.1, #175.2.
Comment #183
benjifisherThe test failure looks like a case of #3041318: Various random fails due to mis-triggered Mink deprecation error.
Comment #184
alexpottPatch addresses #175.1 and tests it. Interestingly this reveals that our update db has both the anonymous and authenticated role with a non-existent permission.
Will update the CR re #175.2
@benjifisher my interdiff is honest. It is the diff of my changes to the patch after resolving the patch conflicts. There is no way to generate an interdiff that can apply to the previous patch - that just impossible because it's not the patch that has changed it's the code it is applying to.
Comment #185
alexpottHere be files.
Comment #186
alexpottUpdated https://www.drupal.org/node/3193348 for #175.2
Comment #187
anish.a CreditAttribution: anish.a at QBurst commented@benjifisher,
The patch uses assertEquals() everywhere, as I am seeing.
Also, #3131281: Replace assertEqual() with assertEquals() is fixed. I am unclear on what needs to be done in the context of Needs reroll tag here. Can you explain?
Comment #188
benjifisherSorry, the patch was rerolled in #180. @alexpott and I forgot to remove the tag. I will do so now.
Yes, that is why the patch in #177 needed a reroll.
Comment #189
alison("sidenote" about reroll interdiffs -- just to help set some minds at ease! https://www.drupal.org/docs/develop/git/using-git-to-contribute-to-drupa... )
-------
I'm still running #121 on my 8.9.x sites, and I don't have a D9 site to test on :( I think/hope this will get backported, but chiming in to express a desire for it, just in case. Thank you all!
Comment #190
alisonOne more thing -- and if this is 8.9.x specific, kinda never mind, I realize things are targeting 9.x -- or if there's another place I should bring this up, just let me know! -- for now, I'm mentioning here juuuuust in case it's an issue with the patch syntax --
Is anyone else getting extra directories when you apply the patch? -- this does happen to me from time to time with core patches, idk if it's an issue with drupal + cweagans/composer-patches, or what, but I get the following "extra directories" when I apply this patch (#121 on 8.9.x) -- again, it's not the first time I've run into this with core patches, but it doesn't happen "every time," and I haven't figured out a pattern yet...
Comment #191
benjifisher@alexpott:
Thanks for the logging in the update function, and the updated tests. We are almost there!
I guess that the
coder
module has a new sniff, because now the testbot is complaining about this@var
comment:Nit: add ending punctuation to the log message. (Adjust the test.)
Same snippet, suggestion: change "permissions" to "permission(s)". I agree that PluralTranslatableMarkup is overkill.
Same function: I notice that you chose an explicit
return NULL
instead of "falling off the end". I guess that is because of the explicitreturn
inside theif
block.I see you added a note about the update function to [#3193348]. (There are two CRs for this issue.) I think the update function is relevant to "Site builders, administrators, editors", so I checked off that audience for the CR. I will add to the text soon.
Comment #192
benjifisherNW for #191.1, optionally #191.2, 3.
Interdiffs: Yes, my understanding is that there is no way to create an interdiff for a reroll. I was wondering if @alexpott knew something I did not. We never posted a straight reroll of the patch in #177. If we had, then the interdiff in #180 would be an ordinary comparison of that to the patch in #180.
@alisonjo315, what problem does this issue cause for your sites? Is it just clutter or something worse? My understanding of the problems here is mostly theoretical/potential.
BTW, looking at your earlier comments on this issue, I see you reported the PHP notices in #120 that were still there when I tested in #169. (I think those are the same messages.) They should be gone now.
I think it is unlikely that this issue will be back-ported to anything before 9.3.x. For one thing, it is disruptive, as shown by the many changes to the existing tests. I still worry about sites that may rely on the current behavior, although that also falls into the "theoretical/potential" category. Drupal 8.9 is now in security maintenance, although that does not rule out Major bug fixes (and this issue is classified as a Critical bug fix).
@alisonjo315, the extra directories you see could be a result of the composer patches plugin. See https://github.com/cweagans/composer-patches/tree/1.x#allowing-to-force-... (the patchLevel option).
Comment #193
alexpottAddressing #192
Comment #194
benjifisher@alexpott, thanks for fixing those last nits. I am satisfied: RTBC.
I updated the change record Permissions must exist, so I am removing that item from the list of remaining tasks in the issue summary. I think the remaining tasks should be done when this issue is actually fixed.
Comment #195
AaronMcHaleJust had a quick skim over the patch and change record.
I don't want to be a wet blanket here and disrupt a great RTBC issue, but a couple of things came to mind:
As a final note, once this is done it would be great if some effort could then be directed towards #2809177: Introduce entity permission providers. As I said, I've really enjoyed using the contrib Entity API module Permission Providers, takes a nice bit of manual effort out of custom entity type development; I feel this issue provides an even stronger case for that issue to be implemented.
Thanks,
-Aaron
Comment #196
alexpott@AaronMcHale config cannot depend on an entity type - since an entity type is neither a module, theme, config entity or content entity. These dependencies map to config dependencies. RE Entity API Permission Providers - it's going to be up to what ever hooks that into the permissions system to provide the correct dependencies - probably the entity API module. If something is collecting / creating permissions on an entity type's behalf then it'll need to add a dependency to the permission on the module providing the entity type - exactly the same way field_ui is doing in the patch.
Comment #197
benjifisher@AaronMcHale:
There are two CRs for this issue. Did you look at both? I think that Permissions can define dependencies has the sort of examples you suggested in #195.1.
Be careful: the permissions are not config entities, so they do not have dependencies. The user roles have dependencies.
I think the suggestion in #27 to turn permissions into config entities made a lot of sense. I did not read the discussion starting there to see why that suggestion was dropped.
Comment #198
AaronMcHaleThanks both for the helpful replies :)
@alexpott: Ah yes that all makes sense, thanks for clarifying that.
@benjifisher: Thank you for linking the other CR, I wasn't expecting there to be more than one and that one does indeed cover what I mentioned.
Yeah that is an interesting idea, I suppose it could have some advantages. I wonder if there would be any performance impacts (positive or negative) for sites that have a large number of permissions.
Comment #199
alison@benjifisher
Clutter! Drives me bananas. And the clutter occasionally causes legit confusion, like when I was uninstalling a contrib module I didn't end up using (printable), and wanted to make sure the associated permissions were gone, and there was a leftover permission for accessing printer-friendly pages provided by "book" module (the site I'm working on right now was built from a clone of a site that used the book module -- "book" is uninstalled on the site I'm working on now, but the permissions were left behind). I guess even that example of confusion is based in me trying to eliminate clutter, but still. Sometimes it's just, so much clutter!!!!!!!!!!!!
Comment #200
alexpottRe #27 and the config entity idea - that scares me a lot. And there are a number of things in that idea that don't map to configuration that well but the biggest and most important point is that permissions are not configurable and idea of making them possibly configurable is part of what scares me. Then there are config overrides and also sorts of other config functionality that just doesn't apply. The ability to override the 'restrict access' value of a permission in settings.php?!?!?
Comment #201
larowlanDoesn't apply anymore, haven't reviewed yet, but this is on my list.
I note that this will have implications for install profiles that try to ship permissions for optional modules.
Comment #202
larowlanComment #203
alexpottRebased on top of 9.3.x - #2977495: Content Moderation missing permission descriptions caused the conflicts cause it adds new descriptions to the content moderation permissions. No interdiff because its a rebase - attaching a diff of the two patches so people can see not much has changed...
Comment #204
larowlanThis looks great, awesome work everyone.
If there is no bundle entity type, should we instead add a dependency on the entity-type provider like we do for the entity-path?
note to self: we don't need to add the bundle and entity-type provider here because the entity-view-display already depends on them
we usually favour full words for variable names.
Given we're touching this here and introducing a new variable (perm) should we use the full spelling of the words?
with media types, node types, and now vocabularies, this is the third time we've got similar code, by the rule of 3 is this worth adding a trait for?
There's at least a few contrib modules that will need the same change
Looking at the code in
\Drupal\Core\Config\ConfigManager::callOnDependencyRemoval
it looks like we already do this in the passed values, there's an array_combine in there that sets the keys to the config dependency names. So we should be able to use array_keys instead?we have a mix of elseif and continue here, should we standardise on continue and avoid elseif?
I think this should also add a
@see
to\Drupal\Core\Config\Entity\ConfigDependencyManager
where the structure of dependencies is documented in fullAs above, the docs at
\Drupal\Core\Config\Entity\ConfigDependencyManager
are more complete and would be a better reference point in my opinionComment #205
alexpott@larowlan thanks the review.
Re #204
EDIT: fixed numbering
Comment #207
benjifisher@alexpott:
If you can fix the failing tests, then I will review after work today.
Can you edit #205 and fix the numbering?
Related to #204.7, 8: I opened #3209296: Fix API docs for ConfigDependencyManager while reviewing this issue, but #12 there suggests that issue will be closed as "won't fix".
Comment #208
alexpottFixed up the failing tests. The direct call to $role->onDependencyRemoval() is interesting. I think it is fine - we don't want to go through \Drupal\Core\Config\ConfigManagerInterface::getConfigEntitiesToChangeOnDependencyRemoval() as that will find other config entities. And we don't want to duplicate the code in \Drupal\user\Entity\Role::calculateDependencies().
Comment #210
alexpottComment #211
benjifisherSorry, I did not get to this yesterday as I said I would.
Why is this in the Core namespace? It deals with permissions, which are defined in the
user
module, and it is odd to have@see
references from Core to a module.(same snippet) Nit: "… to simplify generating bundle level permissions." Also, I would say "bundle-level", but reasonable people can disagree on hyphens.
Is there a reason to make this a trait instead of making
generatePermissions()
a public, static method?We would not have to pass
$permission_builder
if the trait declaredabstract protected function buildPermissions()
. Then I would withdraw the suggestion of making this a public, static method.Why not use
+=
?A function should not be aware of where it is used. If that is really what is going on here, then the problem is in the other implementations of
onDependencyRemoval()
. But maybe it is not really a problem, and all we need is the first sentence of the comment.Can we split this long line?
Perhaps
Comparing to the previous patch, I guess we are adding a key to the innermost array (used to be
[$filter_format]
) to be consistent with the change suggested in #204.5.The more I think about it, the more I wonder about #204.5. In
Role::onDependencyRemoval()
, we implement a function declared inConfigEntityInterface
. The doc block there says something about the keys of$dependencies
, but nothing to suggest that the values are again keyed arrays. Maybe we should update that doc block, but that is an API change. Is any of this marked@internal
?I will defer to @alexpott and @larowlan if they agree that I am making a mountain out of a mole hill.
Comment #212
alexpottThanks for the review @benjifisher
Re #211
\Drupal\Core
for example:\Drupal\Core\Access\AccessResult::allowedIfHasPermission
oradmin_permission = "administer site configuration",
in\Drupal\Core\Datetime\Entity\DateFormat
. We already lots of @see to \Drupal\system - this would be the first \Drupal\user but I think this is okay because system and user are bothI've added a test for the new trait to ensure it works regardless of core implementations and also to test an edge case not covered by core usages.
Comment #213
alexpottHo hum test groups... now we don't have a SImpletest UI in core these are largely pointless...
Comment #214
longwaveThis sentence doesn't make sense - which -> for?
Comment #215
alexpottFixed #214
Comment #216
benjifisher@alexpott:
At first, I assumed that the testbot's OOM error was an infrastructure problem. It turns out that this is the error you get if your patch causes so many failures that the testbot cannot record them all. :-(
Regarding #211.3 (and reply in #212 ... thanks for fixing the numbering) I read Traits in the PHP manual carefully before making that suggestion. Unfortunately, that document is not correct for PHP 8.0. According to PHP 8.0: Fatal errors on incompatible method signatures we no longer have the freedom to mismatch function signatures in this case. (I do not know how authoritative that reference is. It turned up in a search. First I installed PHP 8.0 and got the same failures that the testbot reported.)
Well, it seemed like a good idea. I really like having a proper docblock for
buildPermissions()
instead of an@var
comment for$permission_builder
. But I guess we either go back to that or we change the function signatures when implementingbuildPermissions()
. I will go along with either approach.Other than feeling like scope creep on an issue that is already complicated, I like the idea of making the implementations compatible with the declaration in the trait. They are
protected
, so it is not an API change. We could use{@inheritdoc}
. The type declarationEntityInterface
is more generic than the existing declarations, so that should not cause problems if contrib/custom code extends these classes. Maybe we would have to remove the return type to allow for that. And the only methods used on the arguments areid()
andlabel()
, soEntityInterface
is a good choice.Back to #211.2, which I already described as a nit: you replied to the part after "Also", but the comment still says "... to simplify generate bundle level permissions" instead of my suggested "… to simplify generating bundle level permissions."
At first glance, the refactoring in response to #211.4-6 looks like an improvement, but I have not looked at it closely yet.
Comment #217
alexpottBoo to PHP 8 becoming more sensible! :)
Changing the typehints to EntityInterface will break contrib modules - see http://grep.xnddx.ru/node/30981393 for example. So let's go with the anonymous function approach again...
Comment #218
alexpottSo #3059984: Add new “Content Editor” role to Standard Profile has landed. This issue was not implemented correctly because it is adding permission that don't exist to a role. We'll have to adjust that role on this issue and fix it to use hook_modules_installed in the standard profile.
Comment #220
benjifisher@alexpott:
I mentioned #3059984: Add new “Content Editor” role to Standard Profile in #175. I was thinking that if that issue was fixed first (as it was) then we could remove the undefined permissions as an update to this issue. I see you have already opened #3221258: Fix content editor role to only have permissions that exist, so I will review that instead.
I also mentioned #2809291: Add "edit block $type" permissions, which is currently RTBC. I was going to suggest that you mark it postponed on this issue, but I will do that myself.
P.S. The good news is that the tests worked as designed. It is lucky that I was aware of all these issues and linked them to each other, but we do not have to rely on that luck.
Comment #221
alexpottThe non-existent permissions have been removed as #3221258: Fix content editor role to only have permissions that exist has landed - let's see if we green here again...
Comment #223
alexpottAdding the missing dependencies to the new role...
Comment #224
benjifisherNit: this comment is left over from an earlier patch:
To be on the safe side, can we restore the line from the patch in #210 that ensures the keys
content
andconfig
are present? Otherwise, this snippet might generate PHP notices. See my comment in #211.4.In fact, I restored the test added by that patch to
UserRoleEntityTest
, and it failed.Maybe we should restore the test, too.This is certainly simpler than the earlier version. I do not have time to think this through completely: I am a little worried about the order of operations, and I have not looked up when this hook is invoked. When the role is saved, that will recalculate the dependencies. If the filter format has already been removed, then that should work. We have test coverage for this, right?
Since saving the role will trigger
onDependencyRemoval()
, do we even need to explicitly revoke the permission?Why create two bundles in this test?
Were you thinking of changing these lines to peek at the bundle name?
Comment #225
alexpottThanks for the review @benjifisher
\Drupal\Core\Config\ConfigManager::callOnDependencyRemoval()
- as per earlier comments\Drupal\Core\Config\Entity\ConfigEntityBase::onDependencyRemoval()
is not designed to be called by anything else. The old test UserRoleEntityTest added a test for callingonDependencyRemoval()
because that's what we were doing fromuser_filter_format_disable()
but we're now no longer doing that so that test is unnecessary. FWIW all other implementation of ::onDependencyRemoval() assume the structure of $dependencies too.core/modules/filter/tests/src/Functional/FilterAdminTest.php
.The bundle ID is part of the config dependency name and this is tested.
Been doing some more thinking about #3211113: Add update path to remove permissions that do not exist and whether or not we should delay the update till D10. Still not 100% convinced either way - going to try to chat to @catch about this one as @catch tends to have good thoughts about this type of problem.
Comment #226
benjifisher@alexpott:
onDependencyRemoval()
mentioned the expected keys in its doc block, but that is out of scope for this issue.generatePermissions()
handles both cases (add to existing dependencies or no existing dependencies). But your explanation in #225 is good enough for me.I think this issue is ready. Back to RTBC.
We should update the CR "Permissions can define dependencies", using the new trait. I will do that after work today if you do not do it first. For now, I am adding it to the issue summary.
Comment #227
alexpott@benjifisher re #226.5 yeah I was considering moving the update out again. I talked to @catch who said that he preferred to leave the update in this issue. One thing that is interesting once we fix the migrate part of this we're going to need to have an update that does exacting the same things.
@benjifisher thanks for all the reviews and effort on this one. It's not a simple one at all.
Comment #228
benjifisherYou can say that again!
Comment #229
benjifisherI updated the change record Permissions can define dependencies, so I am removing the "Remaining task" I added in #226.
Since #3059984: Add new “Content Editor” role to Standard Profile has landed, then modified in #3221258: Fix content editor role to only have permissions that exist and #223 of this issue, I am removing it from the list of issues to be updated when this issue is fixed.
Comment #230
larowlanadding review credits
Comment #232
larowlanI applied this branch locally and went through all the changes again in my IDE
Other than some personal-preference nitpicks, I couldn't fault it.
And it doesn't make sense to hold up a critical on those.
I think the longer this is in 9.3, the more chance there is of finding any issues.
Committed cffb02a and pushed to 9.3.x. Thanks!
Published both change records.
Comment #233
larowlanThis is worth highlighting in release notes
Comment #234
AaronMcHaleGreat work here everyone 🎉
So, just to be super clear, when a module is uninstalled, because the permissions it has created will depend on the module, they will automatically be removed from relevant roles? In other words, a module developer doesn't need to implement a uninstall hook to remove any permissions?
If that is the case, might be worth adding a line to one of the CRs explaining that, or at least explain what action (if any) module developers needs to take, as the CR isn't clear if any action is required as a result of this change.
Thanks,
-Aaron
Comment #235
alexpott@AaronMcHale you are correct a module developer doesn't need to do anything for a module's permissions to be removed from a role on module uninstall.
I added the word automatically to
This means that permissions will be automatically cleaned up when a module is uninstalled.
Comment #236
AaronMcHale@alexpott Great sounds good, thanks for the addition :)
Comment #237
benjifisherI took care of the remaining tasks listed in the issue summary. Part of that is a new issue for the Upgrade Status module: #3223068: Check for invalid permissions, will throw runtime exceptions in Drupal 10.
Do we need a follow-up issue to convert the deprecations to exceptions, or do we manage that by searching for deprecation messages when getting closer to Drupal 10?
Comment #238
alexpott@benjifisher thanks for doing that. We can create follow-up already for the work to convert the deprecations to exceptions.
Comment #240
bkline@claudiu.cristea
What is the name used for this? I'm looking in the database for a Drupal 9.x site, and
SELECT name FROM config WHERE name LIKE '%perm%'
gets an empty results set.Comment #241
benjifisher@bkline:
If you read the responses to that comment, you will see that the final decision was not to make permissions into config objects. As before, user roles are config objects.
Comment #242
bkline@benjifisher:
You make it sound so easy, given that this is not a threaded interface. :-) I did find references to that topic after wading through 178 more comments, though I haven't yet found the definitive final decision to which you referred.
Perhaps it would have been more prudent if he had said "it has been proposed that ..." instead of "permissions must be config entities in 9.x," which implied that the change had already made its way into the 9.x code base.
Comment #243
neclimdulOuch, this hurt. This was a "feature" for me that bundled with config split allowed me to have a role that had permissions for modules that where only available in certain environments. Namely a developer role that it would get access to devel.module in development environments but also have access to certain power user developer tools in QA and Production where devel isn't installed.
Note: not actually to upset, this issue has some cool benefits I'm looking forward to. Just identifying a possible pain point or regression we might see.
Comment #244
bircherRE #243
Yea you have to use Config Split 2.x in that scenario, this issue here was one of the reasons I worked on 2.x over the summer.
Comment #245
John Pitcairn CreditAttribution: John Pitcairn commentedOuch indeed.
If a module provides permissions that are in use by any user role, and that module is config-excluded in
$settings['config_exclude_modules']
, then any user roles with those permissions checked will also be excluded from config export/import.This change tripped me up while upgrading a site from 9.2.x to 9.3.x.
Comment #246
bircherRE #245
Yes this is one of the consequences of this. The solution would be #3230825: Change config as if the modules defined in config_exclude_modules had been uninstalled.
But there are quite a few difficulties. Config Split 2.x is experimenting with solutions for this.