Problem/Motivation

Scenario:

  • A new node type 'article' creates a new 'edit any article content' permission.
  • A new text format 'safe_html' creates a new 'use text format safe_html' permission.

When those permissions are granted to a role, that role config entity will contain the 2 permissions.

If an object that is the foundation of a permission is removed the role doesn't revoke that permission. This is proved in the attached test.

(Steps to reproduce are in this duplicate issue -- it's for a D8 site, but I think they'll work regardless -- in case it helps anyone with testing.)

This is a bug because when a module is uninstalled roles are not cleaned up. This dangerous because if the module is re-enabled a role could have an unintended permission already configured.

Proposed resolution

  1. Make role config entity dependent on the provider of each granted permission.
  2. Make role config entity dependent on each entity (config or content) that was used to build that permission.
  3. The dynamic permission callbacks should optionally, return the dependencies used to build a specific permission (except the module which is already added as 'provider').
  4. The user_role config entity should implement onDependencyRemoval().
  5. Add proper dependencies to all dynamic permissions
  6. Trigger a deprecation if a permission does not exist when saving a role. Skip this for migrations - this will be resolved in #2953111: Only migrate role permissions that exist on the destination

The changes in this issue require updates to several existing tests.

  • Only use permissions defined by modules. Fix typos. Do not use random strings.
  • If a test uninstalls a module and then installs it again, related permissions need to be restored to test users.
  • Some tests install config_translation but not locale. If any user role is saved, then all permission callbacks are exercised, and a service provided by the locale module is called.

Remaining tasks

User interface changes

None.

API changes

Dynamic permission callbacks to optionally return the permission dependencies in the array.

Data model changes

Roles will have dependency information according to what permissions are assigned.

Release notes snippet

Each user role now depends on the modules that provide the role's permissions. This means that permissions will be automatically cleaned up when a module is uninstalled. Additionally, if a custom or contributed module uses a permission callback, see how to add additional dependency to the callback's return value. Existing roles will be updated to remove permissions that do not exist and dependencies will be added.

CommentFileSizeAuthor
#225 2571235-9.3.x-225.patch83.92 KBalexpott
#225 223-225-interdiff.txt745 bytesalexpott
#223 2571235-9.3.x-223.patch84.07 KBalexpott
#223 217-223-interdiff.txt711 bytesalexpott
#217 2571235-9.3.x-217.patch83.38 KBalexpott
#217 215-217-interdiff.txt4.79 KBalexpott
#215 2571235-9.3.x-215.patch83.46 KBalexpott
#215 213-215-interdiff.txt1.26 KBalexpott
#213 2571235-9.3.x-213.patch83.33 KBalexpott
#213 212-213-interdiff.txt604 bytesalexpott
#212 2571235-9.3.x-212.patch83.31 KBalexpott
#212 210-212-interdiff.txt9.16 KBalexpott
#210 2571235-9.3.x-210.patch81.38 KBalexpott
#210 208-210-interdiff.txt2.72 KBalexpott
#208 2571235-9.3.x-208.patch80.74 KBalexpott
#208 205-208-interdiff.txt662 bytesalexpott
#205 2571235-9.3.x-205.patch80.7 KBalexpott
#205 203-205-interdiff.txt13.09 KBalexpott
#203 2571235-9.3.x-203.patch77.34 KBalexpott
#203 193-203-real-diff.txt7.11 KBalexpott
#203 2571235-9.3.x-203.patch77.34 KBalexpott
#193 2571235-9.3.x-193.patch76.72 KBalexpott
#193 185-193-interdiff.txt2.88 KBalexpott
#185 2571235-9.3.x-184.patch76.71 KBalexpott
#185 180-184-interdiff.txt3.65 KBalexpott
#180 2571235-9.3.x-180.patch74.99 KBalexpott
#180 177-180-interdiff.txt2.89 KBalexpott
#177 2571235-9.3.x-177.patch74.73 KBalexpott
#177 174-177-interdiff.txt5.88 KBalexpott
#174 2571235-9.3.x-174.patch72.79 KBalexpott
#174 173-174-interdiff.txt3.41 KBalexpott
#173 2571235-9.2.x-173.patch72.79 KBalexpott
#173 166-178-interdiff.txt7.32 KBalexpott
#166 2571235-9.2.x-166.patch64.71 KBalexpott
#166 161-166-interdiff.txt6.59 KBalexpott
#161 2571235-9.2.x-161.patch66.61 KBalexpott
#161 158-161-interdiff.txt10.07 KBalexpott
#158 2571235-9.2.x-158.patch64.85 KBalexpott
#158 154-158-interdiff.txt8.77 KBalexpott
#154 2571235-9.2.x-154.patch64.02 KBalexpott
#154 153-154-interdiff.txt686 bytesalexpott
#153 2571235-9.2.x-153.patch64.02 KBalexpott
#153 150-153-interdiff.txt5.57 KBalexpott
#150 2571235-9.2.x-150.patch60.86 KBalexpott
#150 144-150-interdiff.txt3.92 KBalexpott
#144 2571235-9.2.x-143.patch62.17 KBalexpott
#139 2571235-9.2.x-139.patch62.05 KBalexpott
#139 137-139-interdiff.txt1.18 KBalexpott
#137 2571235-9.2.x-135.patch62.05 KBalexpott
#137 133-135-interdiff.txt12.69 KBalexpott
#133 2571235-9.2.x-133.patch52.32 KBalexpott
#133 132-133-interdiff.txt8.57 KBalexpott
#132 2571235-9.2.x-132.patch45.35 KBalexpott
#132 131-132-interdiff.txt1.53 KBalexpott
#131 2571235-9.2.x-130.patch45.67 KBalexpott
#131 128-130-interdiff.txt1.91 KBalexpott
#128 2571235-9.2.x-128.patch45.01 KBalexpott
#122 2571235-9.1.x-122.patch45.01 KBalexpott
#121 reroll_diff_17-18.txt2.57 KBIJsbrandy
#121 2571235-3-118.patch46.66 KBIJsbrandy
#117 2571235-3-117.patch46.87 KBalexpott
#117 115-117-interdiff.txt2.58 KBalexpott
#115 2571235-3-115.patch46.7 KBalexpott
#115 113-115-interdiff.txt998 bytesalexpott
#113 2571235-3-113.patch45.73 KBalexpott
#112 2571235-2-112.patch48.75 KBalexpott
#112 110-112-interdiff.txt563 bytesalexpott
#110 2571235-2-110.patch48.77 KBalexpott
#110 104-110-interdiff.txt2.79 KBalexpott
#104 2571235-2-104.patch45.18 KBalexpott
#104 97-104-interdiff.txt765 bytesalexpott
#102 2571235-102.patch46.19 KBgabesullice
#102 interdiff-97-102.txt1.83 KBgabesullice
#99 3086824-99.patch6.19 KBalexpott
#99 97-99-interdiff.txt500 bytesalexpott
#97 2571235-2-97.patch44.43 KBalexpott
#97 94-97-interdiff.txt4.89 KBalexpott
#94 2571235-2-94.patch40.09 KBalexpott
#92 2571235-92.patch44.55 KBdawehner
#92 interdiff-2571235.txt4.29 KBdawehner
#90 interdiff-2571235.txt3.16 KBdawehner
#90 2571235-90.patch43.56 KBdawehner
#80 2571235-80.patch48.7 KBalexpott
#80 77-80-interdiff.txt6.84 KBalexpott
#77 2571235-77.patch42.5 KBalexpott
#77 74-77-interdiff.txt2.08 KBalexpott
#75 2571235-74.patch40.42 KBalexpott
#75 72-74-interdiff.txt523 bytesalexpott
#72 2571235-72.patch39.91 KBalexpott
#72 68-72-interdiff.txt6.12 KBalexpott
#69 66-68-interdiff.txt1.12 KBalexpott
#68 2571235-68.patch35.72 KBWim Leers
#68 interdiff.txt1.36 KBWim Leers
#66 2571235-66.patch35.6 KBalexpott
#66 61-66-interdiff.txt2.93 KBalexpott
#61 2571235-61.patch34.69 KBalexpott
#61 59-61-interdiff.txt5.17 KBalexpott
#59 2571235-59.patch29.5 KBalexpott
#59 55-59-interdiff.txt17.1 KBalexpott
#57 interdiff-2571235.txt4.36 KBdawehner
#57 2571235-55.patch19.85 KBdawehner
#52 2571235-52.patch19.73 KBalexpott
#52 42-52-interdiff.txt2.03 KBalexpott
#51 40-42-interdiff.txt3.87 KBalexpott
#42 2571235-42.patch17.71 KBdawehner
#40 36-40-interdiff.txt454 bytesalexpott
#40 2571235-40.patch16.21 KBalexpott
#36 2571235-36.patch16.21 KBalexpott
#26 2571235-26.patch14.29 KBclaudiu.cristea
#23 2571235-23.patch14.22 KBclaudiu.cristea
#20 2571235-20.patch12.96 KBclaudiu.cristea
#16 interdiff.txt806 bytesclaudiu.cristea
#16 2571235-16.patch13.6 KBclaudiu.cristea
#13 interdiff.txt6.46 KBclaudiu.cristea
#13 2571235-13.patch13.59 KBclaudiu.cristea
#10 interdiff.txt2.79 KBclaudiu.cristea
#10 2571235-10.patch11.63 KBclaudiu.cristea
#7 2571235-7.patch10.29 KBclaudiu.cristea
failing_test.patch2.76 KBclaudiu.cristea
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

claudiu.cristea created an issue. See original summary.

claudiu.cristea’s picture

Title: Roles should depend on objects that are responsible to build permissions » Roles should depend on objects that are building the granted permissions

Status: Needs review » Needs work

The last submitted patch, failing_test.patch, failed testing.

The last submitted patch, failing_test.patch, failed testing.

claudiu.cristea’s picture

Status: Needs work » Active

Q.E.D.

claudiu.cristea’s picture

Issue summary: View changes

Update issue summary.

claudiu.cristea’s picture

Issue summary: View changes
Status: Active » Needs review
FileSize
10.29 KB

Full patch.

I'm using filter_format as use case. For the others, I will fill followups.

Status: Needs review » Needs work

The last submitted patch, 7: 2571235-7.patch, failed testing.

The last submitted patch, 7: 2571235-7.patch, failed testing.

claudiu.cristea’s picture

Status: Needs work » Needs review
FileSize
11.63 KB
2.79 KB

Status: Needs review » Needs work

The last submitted patch, 10: 2571235-10.patch, failed testing.

The last submitted patch, 10: 2571235-10.patch, failed testing.

claudiu.cristea’s picture

Status: Needs work » Needs review
Issue tags: +rc target
FileSize
13.59 KB
6.46 KB

Status: Needs review » Needs work

The last submitted patch, 13: 2571235-13.patch, failed testing.

The last submitted patch, 13: 2571235-13.patch, failed testing.

claudiu.cristea’s picture

Status: Needs work » Needs review
FileSize
13.6 KB
806 bytes

Fixed the missed 'dependencies' key in permissions array.

claudiu.cristea’s picture

Issue summary: View changes

On a medium/long term I think that classes providing dynamic permissions must implement an interface and dependencies should be provided in a separate method of the class. I opened an 8.1.x followup #2574251: Dynamic permissions callbacks classes must implement an interface for this.

Added beta evaluation.

claudiu.cristea queued 16: 2571235-16.patch for re-testing.

Status: Needs review » Needs work

The last submitted patch, 16: 2571235-16.patch, failed testing.

claudiu.cristea’s picture

Status: Needs work » Needs review
FileSize
12.96 KB

Reroll.

Status: Needs review » Needs work

The last submitted patch, 20: 2571235-20.patch, failed testing.

The last submitted patch, 20: 2571235-20.patch, failed testing.

claudiu.cristea’s picture

Status: Needs work » Needs review
FileSize
14.22 KB

test fixing.

Status: Needs review » Needs work

The last submitted patch, 23: 2571235-23.patch, failed testing.

xjm’s picture

Issue tags: -rc target

See https://groups.drupal.org/node/484788 for more information on the rc target tag.

claudiu.cristea’s picture

Status: Needs work » Needs review
FileSize
14.29 KB

Rerolled.

@xjm, I added that tag because there's reverse issue #2569741: [PP-1] FilterFormat should add the roles as config dependencies tagged by @catch in the same way. They are both dealing with config dependencies.

claudiu.cristea’s picture

A better plan, IMO, would be:

  1. Deprecate permission_callbacks array from *.permissions.yml. But keep it till 9.0.x to allow the existing custom modules to work.
  2. Create a new config entity type 'permission', holding everything that belongs to a permission: id (the actual permission key), label, description, etc.
  3. Add a new interface DynamicPermissionsInterface:
    interface DynamicPermissionsInterface {
      public function getPermissions();
      public function calculatePermissionsDependencies();
    }
      

    Here, calculatePermissionsDependencies(), would return dependencies in the same format as usual, but nested under a top-level permission key:

    return [
      'use text format full_html' => [
        'config' => [...],
        'content' => [...],
        ...
      ],
      ...
    ];
    
  4. Classes implementing DynamicPermissionsInterface are placed in the "Permissions" namespace. Example: Drupal\node\Permissions. Can be Drupal\node\DynamicPermissions?
  5. .

  6. Rework \Drupal\user\PermissionHandler::buildPermissionsYaml() to collect also permissions from all the classes implementing DynamicPermissionsInterface and stored in the namespace Drupal\{module}\Permissions and store all permissions in config entities of type 'permission'.
  7. The permission provider (the module) doesn't have to be added via DynamicPermissionsInterface. This will be magically, appended to the list of dependencies in PermissionHandler.
  8. Implements all needed dependency procedures in permissions and roles config entities. Example: (1) permission will depend on provider, (2) when the provider is uninstalled, the permission gets deleted but this means (3) all roles depending on that permission will be updated because we are implementing onDependencyRemoval on Role configurable.
  9. Convert the 12 occurrences from core modules to use the new API.

In 9.0.x just:

  1. Drop *.permissions.yml file. Provide static permissions as default config .yml files, under core/module/{module}/config/install as {module}.permission.{permission}.yml. Discuss how the {permission} name will be built (we can rely on a processed ->id() that only replaces spaces with dash '-'.
  2. Adapt PermissionHandler to only grab dynamic permissions from the Drupal\{module}\Permissions namespace.

This approach has the benefit of providing the BC but still prepare for a cleaner 9.0.0 approach and assuring permission caching through standard config entity cache.

And this is doable right now, in 8.0.x because it breaks no API and assures the BC!

EDIT: Added 9.0.x tasks.

alexpott’s picture

So I don't think we're going to get round to #27 now. We do have a problem with cleaning up roles when things are deleted that provide dynamic permissions. We have exactly the same problems for the node type based permissions.

claudiu.cristea’s picture

@alexpott, off-topic but related: permissions must be config entities in 9.x. Then we simply do config dependencies format <=> permission <=> role. That would fix a lot of other issues (like caching the permissions)

alexpott’s picture

Oh and we still don't cleanup permissions on uninstall of a module either.

Version: 8.0.x-dev » 8.1.x-dev

Drupal 8.0.6 was released on April 6 and is the final bugfix release for the Drupal 8.0.x series. Drupal 8.0.x will not receive any further development aside from security fixes. Drupal 8.1.0-rc1 is now available and sites should prepare to update to 8.1.0.

Bug reports should be targeted against the 8.1.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.9 was released on September 7 and is the final bugfix release for the Drupal 8.1.x series. Drupal 8.1.x will not receive any further development aside from security fixes. Drupal 8.2.0-rc1 is now available and sites should prepare to upgrade to 8.2.0.

Bug reports should be targeted against the 8.2.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

dpi’s picture

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.6 was released on February 1, 2017 and is the final full bugfix release for the Drupal 8.2.x series. Drupal 8.2.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.3.0 on April 5, 2017. (Drupal 8.3.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.3.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.6 was released on August 2, 2017 and is the final full bugfix release for the Drupal 8.3.x series. Drupal 8.3.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.4.0 on October 4, 2017. (Drupal 8.4.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.4.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

alexpott’s picture

Re-rolled the patch after a couple of years. I think we should do this instead of #2026983: [regression] Grants for permissions defined by a module are not revoked / removed from Role config entities when that module is uninstalled.

No interdiff because it's been 2 years and by the looks of it #26 and quite a few unrelated changes in it so an interdiff is just going to confuse matters and be unnecessarily big.

alexpott’s picture

We need to implement and test for:

  • content_translation
  • rest

We need to add test coverage for:

  • content_moderation
  • field_ui
  • vocabulary

Status: Needs review » Needs work

The last submitted patch, 36: 2571235-36.patch, failed testing. View results

alexpott’s picture

Version: 8.4.x-dev » 8.5.x-dev
Status: Needs work » Needs review
Issue tags: +Needs tests, +Needs upgrade path

core/modules/taxonomy/src/TaxonomyPermissions.php has changed in 8.5.x so we're going to need to do this there. And I think it is likely this will be 8.5.x only because it will need an upgrade path.

alexpott’s picture

Status: Needs review » Needs work

The last submitted patch, 40: 2571235-40.patch, failed testing. View results

dawehner’s picture

This does not yet resolve the test failures.

alexpott’s picture

Ah the fun of module install :) So the fails are caused by calling $format->url() whilst creating configuration during a module install and the route provider can't kind the new routes of previous installed modules. Fortunately we already have a fix for this - #2913912: URL generator may have a stale route provider during module installation.

alexpott credited Berdir.

alexpott credited dmouse.

alexpott credited herom.

alexpott credited longwave.

alexpott’s picture

Title: Roles should depend on objects that are building the granted permissions » [regression] Roles should depend on objects that are building the granted permissions
Priority: Major » Critical

I have closed #2026983: [regression] Grants for permissions defined by a module are not revoked / removed from Role config entities when that module is uninstalled in favour of this issue because the approach to fixing the issue is more correct. However that means that this issue should be critical for the reasons that #2026983: [regression] Grants for permissions defined by a module are not revoked / removed from Role config entities when that module is uninstalled was. This is also a regression from Drupal 7. I'm crediting the people who worked on that issue.

alexpott’s picture

Issue summary: View changes

Added blockers to issue summary.

alexpott’s picture

FileSize
3.87 KB

Here's an interdiff for #42

alexpott’s picture

Status: Needs work » Needs review
FileSize
2.03 KB
19.73 KB

Here's the patch with the fix from #2913912: URL generator may have a stale route provider during module installation to see if anything else is broken.

Status: Needs review » Needs work

The last submitted patch, 52: 2571235-52.patch, failed testing. View results

claudiu.cristea’s picture

Some thoughts:

  • Shouldn't we test also the case with dependencies declared on a static permission?
  • It still needs upgrade path and tests. We have to test if the stale permissions are removed from roles.

Nits:

  1. +++ b/core/modules/user/src/Entity/Role.php
    @@ -186,4 +186,67 @@ public function preSave(EntityStorageInterface $storage) {
    +      // @todo Remove this if() when https://www.drupal.org/node/2569741 is in.
    

    Let's drop the @todo. Seems that #2569741: [PP-1] FilterFormat should add the roles as config dependencies is a "won't fix"

  2. +++ b/core/modules/user/tests/src/Kernel/UserRoleDeleteTest.php
    @@ -70,5 +74,73 @@ public function testRoleDeleteUserRoleReferenceDelete() {
    +    // The role $role has the permissions to use $format and edit $node_type.
    ...
    +    // The $role config entity exists before removing dependencies.
    ...
    +    // The $role config entity exists after removing the config dependency.
    ...
    +      // The $format permission should have been revoked.
    ...
    +    // The $role config entity exists after removing the module dependency.
    ...
    +      // The $node_type permission should have been revoked too.
    

    We use to use the standard "Check that ..." in comments explaining the assertions.

  3. +++ b/core/modules/user/tests/src/Kernel/UserRoleDeleteTest.php
    @@ -70,5 +74,73 @@ public function testRoleDeleteUserRoleReferenceDelete() {
    +    if ($this->assertNotNull($role = Role::load($rid))) {
    ...
    +    if ($this->assertNotNull($role = Role::load($rid))) {
    

    I don't think assertions are returning something in PHPUnit. Probably this if() is left here from the WebTestBase version of the test?

alexpott’s picture

Status: Needs work » Needs review

Thanks for the review @claudiu.cristea

  1. I'm not sure why this if actually exists - if a permission does not exist as this point then shouldn't it be removed?
  2. I've never come across this standard and don't feel particularly strongly about that. These are comments you wrote btw :) this test is from your patch in #26
  3. Good point. fixed.

I outlined the need for an upgrade path and tests in #37 and #39.
The static permission thing - I'm not sure we should even support that - what would the use case be? These only really can depend on their provider.

The last submitted patch, 42: 2571235-42.patch, failed testing. View results

dawehner’s picture

Here are some test failure fixes, not all yet.

I don't think assertions are returning something in PHPUnit. Probably this if() is left here from the WebTestBase version of the test?

Nice catch!

We use to use the standard "Check that ..." in comments explaining the assertions.

I don't think we have anything like a standard about that :)

Status: Needs review » Needs work

The last submitted patch, 57: 2571235-55.patch, failed testing. View results

alexpott’s picture

Status: Needs work » Needs review
FileSize
17.1 KB
29.5 KB

Some more fixes. The implementation of \Drupal\user\Entity\Role::onDependencyRemoval has been rewritten.

Status: Needs review » Needs work

The last submitted patch, 59: 2571235-59.patch, failed testing. View results

alexpott’s picture

Status: Needs work » Needs review
FileSize
5.17 KB
34.69 KB

Oh wow - so a lot of the API first tests added are depending on permissions that don't actually exist!

Status: Needs review » Needs work

The last submitted patch, 61: 2571235-61.patch, failed testing. View results

Wim Leers’s picture

#61: Oh, hah! So evidently it's possible to grant non-existent permissions and have them be respected by the permission checking logic … otherwise those API-First tests could never have passed!

Wim Leers’s picture

Issue tags: +API-First Initiative

This helps API-First Drupal become more reliable (in core, but also in the contrib JSON API and GraphQL modules), so tagging API-First Initiative.

dawehner’s picture

#61: Oh, hah! So evidently it's possible to grant non-existent permissions and have them be respected by the permission checking logic … otherwise those API-First tests could never have passed!

Its indeed possible. I'm wondering whether we should add validation of some kind on the grantPermissio
level. This would still lead to a fast runtime, but we have a form of validation at least.

alexpott’s picture

Issue summary: View changes
Status: Needs work » Needs review
Issue tags: +Needs followup
FileSize
2.93 KB
35.6 KB

@dawehner I think we should explore whether or not we should do that in a follow-up. I guess there is a question about whether or not this change in behaviour is allowed even in a minor. The problem being that granting non-existent perms is a critical as well because I think it is a data integrity issue.

Here's a patch that might be green. We have test coverage for:

  • content_moderation
  • content_translation
  • field_ui

Missing test coverage for

  • rest
  • node
  • filter
  • taxonomy

There is some implied test coverage for node and filter in UserRoleDeleteTest::testDependenciesRemoval() but I think we should have explicit test coverage that the dependencies are added correctly in each module that provides dynamic permissions that depend on things other than the module.

In the issue summary it used to say

Fix at least one use case. Let's fix filter_format in this issue.

I don't think this is a good way to go because once we add dependencies to roles than we need to try to keep them correct all the time. Doing this in steps just means we have lots and lots of upgrade headaches. Rather than one. We still need to write the upgrade path and tests for it.

The interdiff attached is pseudo interdiff because #2913912: URL generator may have a stale route provider during module installation landed so I had to rebase.

Wim Leers’s picture

Expanded test coverage to explicitly ensure that the access content case is handled correctly. I'm not sure if it really is that very useful though…

@alexpott: feel free to remove this small bit of extra test coverage for checking the #2892821: Core modules check node module's "access content" permission for displaying things that have nothing to do with nodes special case explicitly, if you feel it doesn't add much (and feel free to uncredit me).

alexpott’s picture

FileSize
1.12 KB

The interdiff on #68 is not correct. Here's the correct one. I think having explicit testing about not removing the access content permission is fine. It proves that other non-related permissions are maintained through module uninstall.

Wim Leers’s picture

:O how the hell did that #68 interdiff end up looking like THAT!? I haven't seen any changes like that in as long as I can remember. I'd swear d.o mixed up files here… :O

The #69 interdiff is the one I remember seeing an hour ago, thanks for uploding that.

And: great!

Status: Needs review » Needs work

The last submitted patch, 68: 2571235-68.patch, failed testing. View results

alexpott’s picture

Issue summary: View changes
Status: Needs work » Needs review
FileSize
6.12 KB
39.91 KB

Patch adds test coverage for

  • rest
  • node
  • filter
  • taxonomy

And reverts the change to ModuleUninstallTest because access content is now provided by System there are no config updates when uninstalling Node.

alexpott’s picture

Issue summary: View changes

@Berdir talked to me about this issue on IRC and he brought up a great point. During Drupal install there's the ability for install profiles to provide alternate configuration to be used in place of the configuration provided by the module. People are using this to provide alternate user.role.anonymous and user.role.authenticated config entities. These entities have permissions that do not exist when user is installed. Therefore we have a big problem because the current implementation of this patch is likely to remove those permissions. Ideas of how to fix this are welcome!

Berdir’s picture

Thanks for writing that down here.

I mostly just wanted to make sure that we are aware of that problem, install profiles config/overrides are excluded from BC, we've broken that before e.g. for views and so on. But if we do, we should at least make sure we are aware of the impact and to document in the change record what should be done.

The only viable alternative that currently exists is to move that to hook_install() of the install profile and grant the permissions there once all modules are enabled.

That said, this is a general problem that doesn't just affect this, it's especially common in form/view displays, views and other complex config with a lot of dependencies.. e.g when you want to add another custom field that some module in your install profile adds to admin/content. A possible generic solution would be a config/post_override folder, which is config that would only be replaced once the installation is finished.. Or config_installer based install profiles, although I could imagine problems there as well, e.g. when the auth/anon user roles are installed only very late due to dependencies and other modules somehow rely on them in code.

alexpott’s picture

Here's a patch which shows the nature of the problem @Berdir is talking about... it should fail.

Status: Needs review » Needs work

The last submitted patch, 75: 2571235-74.patch, failed testing. View results

alexpott’s picture

Status: Needs work » Needs review
FileSize
2.08 KB
42.5 KB

So one idea would be to not override configuration entities that are provided by profiles. And just update the configuration when the profile is installed. This seems to work. Perhaps we could just do all the profile updates when the profile is being installed and not do the early swap out.

Hmmm... actually that won't work because profiles can provide overrides for modules that are optional... ie. be installed after Drupal has be installed.

Patch attached tries to get the best of both worlds - also ConfigInstallProfileUnmetDependenciesTest is still failing because in this instance the dependencies can now be met!

Status: Needs review » Needs work

The last submitted patch, 77: 2571235-77.patch, failed testing. View results

alexpott’s picture

So let's split off the installer stuff into a new blocker to make it easier to review / think about in isolation... created #2922417: Profile provided configuration entities can have unmeetable dependencies

alexpott’s picture

Issue summary: View changes
Status: Needs work » Needs review
Issue tags: -Needs tests, -Needs upgrade path
FileSize
6.84 KB
48.7 KB

Patch includes an upgrade path and a test and the current patch from #2922417: Profile provided configuration entities can have unmeetable dependencies

So I think once the blocker lands we are good to go here.

effulgentsia’s picture

I don't understand the nature of the problem in #75.

These entities have permissions that do not exist when user is installed.

Why isn't the following enough to fix that?

+++ b/core/profiles/testing_config_overrides/config/install/user.role.authenticated.yml
@@ -0,0 +1,11 @@
+dependencies:
+  module:
+    - tour
effulgentsia’s picture

Berdir’s picture

@effulgentsia because you can't install any module before installing user.module, that won't end well :) Also, config dependency order doesnot override module order, it just makes it fail if the dependencies aren't already installed. But maybe your "never mind" meant you got that already :)

Wim Leers’s picture

Issue tags: -API-First Initiative

On second thought, I don't think this is a significant impact for API-First. I'm disagreeing with myself in #64 :) Untagging.

larowlan’s picture

Can we get a change record here too - because obviously contrib etc will need to add their support for it.

I really like the approach here - nice work

@effulgentsia, @xjm, @plach, @catch and I discussed this on a triage call and agree this is critical, primarily because the presence of the permission on the role grants access, even if the permission doesn't exist.

  1. +++ b/core/modules/rest/tests/src/Functional/EntityResource/EntityFormDisplay/EntityFormDisplayResourceTestBase.php
    @@ -11,7 +11,7 @@
    -  public static $modules = ['node'];
    +  public static $modules = ['node', 'field_ui'];
    

    I assume this is because we're using permissions with dependencies in tests - where previously those modules weren't enabled?

  2. +++ b/core/modules/rest/tests/src/Functional/EntityResource/EntityResourceTestBase.php
    @@ -450,12 +450,17 @@ public function testGet() {
    +      // Filter out definitions that do not have a plugin.
    +      // @see \Drupal\rest\Plugin\rest\resource\EntityResource::addLinkHeaders()
    +      $link_relations = array_values(array_filter(array_keys($this->entity->getEntityType()->getLinkTemplates()), function ($relation_name) use ($link_relation_type_manager) {
    +        return $link_relation_type_manager->hasDefinition($relation_name);
    +      }));
           $expected_link_relation_headers = array_map(function ($relation_name) use ($link_relation_type_manager) {
             $link_relation_type = $link_relation_type_manager->createInstance($relation_name);
             return $link_relation_type->isRegistered()
               ? $link_relation_type->getRegisteredName()
               : $link_relation_type->getExtensionUri();
    -      }, array_keys($this->entity->getEntityType()->getLinkTemplates()));
    +      }, $link_relations);
    

    unrelated?

  3. +++ b/core/modules/user/src/Entity/Role.php
    @@ -186,4 +186,68 @@ public function preSave(EntityStorageInterface $storage) {
    +      // @todo Why is this necessary? Surely for a permission to be on role it
    +      // has to exist.
    

    Should we have a follow up for this?

  4. +++ b/core/modules/user/src/Entity/Role.php
    @@ -186,4 +186,68 @@ public function preSave(EntityStorageInterface $storage) {
    +        $name = in_array($type, ['config', 'content']) ? $dependency->getConfigDependencyName() : $dependency;
    ...
    +      if (in_array($permission_definitions[$permission]['provider'], $dependencies['module'], TRUE)) {
    

    can we use strict checking here because they're strings

  5. +++ b/core/modules/user/src/PermissionHandlerInterface.php
    @@ -34,7 +34,20 @@
    +   *     The module providing this permission doesn't have to be added as
    +   *     dependency, because is automatically parsed, stored and retrieved from
    

    nit 'as a dependency'

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.0-alpha1 will be released the week of January 17, 2018, which means new developments and disruptive changes should now be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.6.x-dev » 8.7.x-dev

Drupal 8.6.0-alpha1 will be released the week of July 16, 2018, which means new developments and disruptive changes should now be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

jibran’s picture

dawehner’s picture

Status: Needs work » Needs review
Issue tags: -Needs followup, -Needs change record
FileSize
43.56 KB
3.16 KB

Can we get a change record here too - because obviously contrib etc will need to add their support for it.

Added one

I assume this is because we're using permissions with dependencies in tests - where previously those modules weren't enabled?

Yes, in the exactly you've given, it's the "administer node form display" permission.

unrelated?

Seems so, I reverted it

Should we have a follow up for this?

See #3055557: Role migration results in migration permissions that do not exist on the target site

Status: Needs review » Needs work

The last submitted patch, 90: 2571235-90.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

dawehner’s picture

Status: Needs work » Needs review
FileSize
4.29 KB
44.55 KB

I hope these are the remaining failures.

Status: Needs review » Needs work

The last submitted patch, 92: 2571235-92.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

alexpott’s picture

Status: Needs work » Needs review
FileSize
40.09 KB

The plumbing has made it in to the ConfigInstaller. Now we can focus on user role dependencies and not fixing the config system. Rerolled.

alexpott’s picture

+++ b/core/modules/user/user.post_update.php
@@ -20,3 +20,17 @@ function user_post_update_enforce_order_of_permissions() {
+  $existing_permissions = array_keys(\Drupal::service('user.permissions')->getPermissions());
+  $entity_save = function (Role $role) use ($existing_permissions) {
+    $permissions = array_intersect($role->getPermissions(), $existing_permissions);
+    $role
+      ->set('permissions', $permissions)
+      ->save();
+  };
+  array_map($entity_save, Role::loadMultiple());

This should use the ConfigEntityUpdater class.

+++ b/core/tests/Drupal/FunctionalTests/Rest/EntityViewDisplayResourceTestBase.php
@@ -11,7 +11,7 @@ abstract class EntityViewDisplayResourceTestBase extends EntityResourceTestBase
-  public static $modules = ['node'];
+  public static $modules = ['node', 'field_ui'];

These changes look odd - can't recall why they are necessary need to have a look at the issue history.

Status: Needs review » Needs work

The last submitted patch, 94: 2571235-2-94.patch, failed testing. View results

alexpott’s picture

Status: Needs work » Needs review
FileSize
4.89 KB
44.43 KB

Some test fixes.

Status: Needs review » Needs work

The last submitted patch, 97: 2571235-2-97.patch, failed testing. View results

alexpott’s picture

Status: Needs work » Needs review
FileSize
500 bytes
6.19 KB

Looks like Views UI is not defining some link relation types. And this exposes this because Views entities use an admin permission that's provided by Views UI so the tests are not currently exposing this.

claudiu.cristea’s picture

I wonder why the patch decreases from 44 (in #97) to 6KB (in #99).

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

gabesullice’s picture

I think the test failures in #97 are exposing some fundamentally odd things about entity link templates: They use link relation types as their keys and REST is automatically trying to expose them.

What does an edit-display-form link relation type even mean to a REST client? In 99% of cases, that serialization format is going to be JSON/XML, not HTML, and neither of these media types define form semantics like HTML does.

IMHO, we shouldn't be adding Link headers to REST responses for every link template. Especially if a matching link relation type is not defined for it. All that does is create a noisy and poor DX for anyone using REST. Worse, our extension link relation types are not even valid.

The attached removes the assumption that every link template has an associated link relation type. I think we should do something to this effect and then define some follow-ups to disassociate them everywhere else.

n.b. the interdiff is to #97, not #99

Status: Needs review » Needs work

The last submitted patch, 102: 2571235-102.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

alexpott’s picture

Status: Needs work » Needs review
FileSize
765 bytes
45.18 KB

Thanks for looking at this @gabesullice - so the whole link relation thing happens because we have to install Views UI for the rest tests. We have to install Views UI because it provides the permission 'administer views' which these tests use. This permission is defined on the View entity provided by the Views module - so that's a weird circular dependency. Once we install the Views UI module that exposes a load of undefined link relationships. It's a completely separate bug that is being surfaced here and not really part of this issue. On HEAD ff you install views ui and rest and try and get views via rest you'll have exactly the same problem. I'll open a separate issue about this.

alexpott’s picture

Opened #3088282: Missing Views link relationships when Views UI installed to address this link relation stuff.

Status: Needs review » Needs work

The last submitted patch, 104: 2571235-2-104.patch, failed testing. View results

gabesullice’s picture

@alexpott, did you mean to add the link relation types in this issue? It sounded like you were instead going to address the permission placement instead. The interdiff just shows 3 new link relations.

alexpott’s picture

@gabesullice well the permission placement is one solution. I'm not sure it's right though. If we put it in Views then people will have an "Administer views" permission with no UI which is odd.

gabesullice’s picture

If users have JSON:API + JSON:API Extras in contrib, it's possible, although extremely unlikely, that users could be editing Views display config without a UI.

alexpott’s picture

Status: Needs work » Needs review
FileSize
2.79 KB
48.77 KB

The link relation type stuff has been put into #3088282: Missing Views link relationships when Views UI installed which is now RTBC and kinda blocks this one. Here are the remaining fixes to make the patch green. The patch includes #3088282: Missing Views link relationships when Views UI installed.

alexpott’s picture

+++ b/core/modules/media_library/tests/src/FunctionalJavascript/MediaLibraryTest.php
@@ -282,7 +282,7 @@ public function testWidgetWithoutMediaTypes() {
-    $field_ui_uninstalled_message = 'There are no allowed media types configured for this field. Edit the field settings to select the allowed media types.';
+    $field_ui_uninstalled_message = 'There are no allowed media types configured for this field. Please contact the site administrator.';

This change shows how important this change is. In HEAD atm if you uninstall the field UI module you still get messages implying you can edit field settings whereas in reality there is no way to edit them.

alexpott’s picture

We can fallback to \Drupal\Tests\rest\Functional\EntityResource\EntityResourceTestBase::getExpectedCacheContexts() now. No need to override.

alexpott’s picture

Status: Needs review » Needs work

The last submitted patch, 113: 2571235-3-113.patch, failed testing. View results

alexpott’s picture

Status: Needs work » Needs review
FileSize
998 bytes
46.7 KB

Needed to re-apply #111 to the correct test.

alexpott’s picture

  1. +++ b/core/modules/user/tests/src/Functional/Update/UserUpdateRoleDependenciesTest.php
    @@ -0,0 +1,47 @@
    +      __DIR__ . '/../../../../../system/tests/fixtures/update/drupal-8-rc1.bare.standard.php.gz',
    

    Need to convert to using the 8.8 file.

  2. +++ b/core/modules/user/user.post_update.php
    @@ -20,3 +20,17 @@ function user_post_update_enforce_order_of_permissions() {
    +  $existing_permissions = array_keys(\Drupal::service('user.permissions')->getPermissions());
    +  $entity_save = function (Role $role) use ($existing_permissions) {
    +    $permissions = array_intersect($role->getPermissions(), $existing_permissions);
    +    $role
    +      ->set('permissions', $permissions)
    +      ->save();
    +  };
    +  array_map($entity_save, Role::loadMultiple());
    

    Need to convert to using the ConfigEntityUpdater

alexpott’s picture

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

alison’s picture

(Caveat: I'm on 8.8.5, so I'm not going to change to RTBC, but...)

The functionality worked PERFECTLY for me when I tested it with a new vocabulary (full steps to reproduce/test in my duplicate issue from a couple months ago, in case it helps anyone else with testing).

Two questions:

  1. The patch didn't clean up existing orphaned permissions (i.e. permissions for creating/editing/deleting terms in vocabularies that were removed from our site ages ago). Have y'all already discussed whether this (really wonderful!!!!!!) change should be cleaning up orphaned permissions that existed on a site before applying the patch? If not, I urge you to do so -- though I'd certainly be ok with it being addressed in a follow-up issue, instead of holding up this one.
  2. Would a backport to 8.8.x be permitted? I'm happy to help, but wanted to see if it would be accepted, or if this issue doesn't qualify.

Thank you @alexpott and everybody who's worked on this issue!!

alison’s picture

Also, didn't notice this before: I get a slew of "notice" log messages when performing certain actions that I think trigger the check for permission configs -- here's an example message, when I was deleting my test vocabulary:
Notice: Undefined index: use text format full_html in Drupal\user\Entity\Role->onDependencyRemoval() (line 243 of /path/to/core/modules/user/src/Entity/Role.php)

>> Text format "full_html" no longer exists on this site, so "use text format full_html" is one of the many orphaned permissions we have. This example log message happened when I was deleting my test vocab (to test the patch) -- there were 6 such log messages at the time, idk how it decided which orphaned permissions to complain about. (I say that because ⤵️ )
>> I happened to uninstall a module while trying out this patch, and when I did that, the logs were ***FLOODED*** with a "notice" log message like the example above, for every orphaned permission -- we're talking several pages of log messages.

Anyway, FYI!

IJsbrandy’s picture

Rerolled patch from #117 for latest 8.9.x

alexpott’s picture

Here's a patch for 9.1.x since this will only get commit to 9.x at this point and probably only 9.1.x since it has a post update.

alexpott’s picture

Re #119 / @alisonjo315 this patch does clean up roles with permissions that do not exist. You need to run update.php / updatedb

  1. +++ b/core/modules/user/user.post_update.php
    @@ -13,3 +16,15 @@ function user_removed_post_updates() {
    +/**
    + * Calculate role dependencies and remove non-existent permissions.
    + */
    +function user_post_update_update_roles(&$sandbox = NULL) {
    +  $existing_permissions = array_keys(\Drupal::service('user.permissions')->getPermissions());
    +  \Drupal::classResolver(ConfigEntityUpdater::class)->update($sandbox, 'user_role', function (Role $role) use ($existing_permissions) {
    +    $permissions = array_intersect($role->getPermissions(), $existing_permissions);
    +    $role->set('permissions', $permissions);
    +    return TRUE;
    +  });
    +}
    

    This function will remove non-existing permissions.

  2. +++ b/core/modules/user/tests/src/Functional/Update/UserUpdateRoleDependenciesTest.php
    @@ -0,0 +1,47 @@
    +  public function testRolePermissions() {
    +    // Edit the role to have a non-existent permission.
    +    $raw_config = $this->config('user.role.authenticated');
    +    $permissions = $raw_config->get('permissions');
    +    $permissions[] = 'does_not_exist';
    +    $raw_config
    +      ->set('permissions', $permissions)
    +      ->save();
    +
    +    $authenticated = Role::load('authenticated');
    +    $this->assertTrue($authenticated->hasPermission('does_not_exist'), 'Authenticated role has a permission that does not exist');
    +    $this->assertEquals([], $authenticated->getDependencies());
    +
    +    $this->runUpdates();
    +    $authenticated = Role::load('authenticated');
    +    $this->assertFalse($authenticated->hasPermission('does_not_exist'), 'Authenticated role does not have a permission that does not exist');
    +    $this->assertEquals(['config' => ['filter.format.basic_html'], 'module' => ['comment', 'contact', 'filter', 'shortcut', 'system']], $authenticated->getDependencies());
    +  }
    

    This tests this functionality.

alexpott’s picture

+++ b/core/modules/user/src/Entity/Role.php
@@ -193,4 +193,68 @@ public function preSave(EntityStorageInterface $storage) {
+      // @todo Why is this necessary? Surely for a permission to be on role it
+      // has to exist.
+      if (isset($permissions[$permission])) {

We need to address this - one thought is that we could support adding non existent permissions in D9 but deprecate it and then remove support in D10. This might help contrib / custom tests deal with this change in a graceful way.

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

alison’s picture

Issue summary: View changes

Add link to steps to reproduce to issue summary -- feel free to remove, esp since it's on a D8 site, just adding in case they're helpful, but I won't be offended if it's determined they don't belong here.

alison’s picture

FWIW, #121 works great for me on 8.9.12!

@alexpott Thank you for the comment - sorry I disappeared - the update hook totally worked, as you said, idk what happened to me back in May, but it all definitely works, so, hooray! I 💜 my permission config entities now....

alexpott’s picture

Re-rolled for 9.2.x. It'd be great to get this to rtbc and committed.

alexpott’s picture

Issue summary: View changes

Added release note.

Status: Needs review » Needs work

The last submitted patch, 128: 2571235-9.2.x-128.patch, failed testing. View results

alexpott’s picture

Status: Needs work » Needs review
FileSize
1.91 KB
45.67 KB

Fixing the broken tests.

alexpott’s picture

Removing the @todo and seeing what is broken....

alexpott’s picture

Resolving some of the fails in #132 which shows the importance of resolving most of #3055557: Role migration results in migration permissions that do not exist on the target site in this issue. I think in a future patch will need to trigger a deprecation when saving with a permission that does not exist as that will help contrib / custom deal with this and prevent core from regressing again.

Status: Needs review » Needs work

The last submitted patch, 133: 2571235-9.2.x-133.patch, failed testing. View results

alexpott credited catch.

alexpott credited heddn.

alexpott’s picture

Status: Needs work » Needs review
FileSize
12.69 KB
62.05 KB

Discussed the migrate fails with @heddn. Given the complexity of the migrate part of this I think it is better to punt that to #3055557: Role migration results in migration permissions that do not exist on the target site a re-purpose that to working out what to do about migrations.

Discussed where to put the role clean-up on filter disable with @catch. After considering putting in the Filter entity, on the filter module we plumped for the user module because this means tests like EditorFilterIntegrationTest don't have to change and it is Roles changing when a filter is disabled and Roles are owned by the user module and that's what the hook is for.

Status: Needs review » Needs work

The last submitted patch, 137: 2571235-9.2.x-135.patch, failed testing. View results

alexpott’s picture

Status: Needs work » Needs review
FileSize
1.18 KB
62.05 KB

Fixing the last fail.

alexpott’s picture

Issue summary: View changes

Improved the issue summary.

alexpott’s picture

+++ b/core/modules/field/tests/src/Functional/Rest/FieldConfigResourceTestBase.php
@@ -12,7 +12,7 @@ abstract class FieldConfigResourceTestBase extends EntityResourceTestBase {
-  protected static $modules = ['field', 'node'];
+  protected static $modules = ['field', 'field_ui', 'node'];

+++ b/core/modules/field/tests/src/Functional/Rest/FieldStorageConfigResourceTestBase.php
@@ -10,7 +10,7 @@ abstract class FieldStorageConfigResourceTestBase extends EntityResourceTestBase
-  protected static $modules = ['node'];
+  protected static $modules = ['field_ui', 'node'];

+++ b/core/modules/jsonapi/tests/src/Functional/ActionTest.php
@@ -16,7 +16,7 @@ class ActionTest extends ResourceTestBase {
-  protected static $modules = ['user'];
+  protected static $modules = ['action'];

+++ b/core/modules/jsonapi/tests/src/Functional/BaseFieldOverrideTest.php
@@ -16,7 +16,7 @@ class BaseFieldOverrideTest extends ResourceTestBase {
-  protected static $modules = ['field', 'node'];
+  protected static $modules = ['field', 'node', 'field_ui'];

+++ b/core/modules/jsonapi/tests/src/Functional/EntityFormDisplayTest.php
@@ -16,7 +16,7 @@ class EntityFormDisplayTest extends ResourceTestBase {
-  protected static $modules = ['node'];
+  protected static $modules = ['node', 'field_ui'];

+++ b/core/modules/jsonapi/tests/src/Functional/EntityViewDisplayTest.php
@@ -16,7 +16,7 @@ class EntityViewDisplayTest extends ResourceTestBase {
-  protected static $modules = ['node'];
+  protected static $modules = ['node', 'field_ui'];

+++ b/core/modules/jsonapi/tests/src/Functional/FieldConfigTest.php
@@ -19,7 +19,7 @@ class FieldConfigTest extends ResourceTestBase {
-  protected static $modules = ['field', 'node'];
+  protected static $modules = ['field', 'node', 'field_ui'];

+++ b/core/modules/jsonapi/tests/src/Functional/FieldStorageConfigTest.php
@@ -15,7 +15,7 @@ class FieldStorageConfigTest extends ResourceTestBase {
-  protected static $modules = ['node'];
+  protected static $modules = ['node', 'field_ui'];

+++ b/core/modules/jsonapi/tests/src/Functional/PathAliasTest.php
@@ -16,7 +16,7 @@ class PathAliasTest extends ResourceTestBase {
-  protected static $modules = ['user'];
+  protected static $modules = ['path'];

+++ b/core/modules/jsonapi/tests/src/Functional/ViewTest.php
@@ -15,7 +15,7 @@ class ViewTest extends ResourceTestBase {
-  protected static $modules = ['views'];
+  protected static $modules = ['views', 'views_ui'];

+++ b/core/modules/path_alias/tests/src/Functional/Rest/PathAliasResourceTestBase.php
@@ -14,7 +14,7 @@ abstract class PathAliasResourceTestBase extends EntityResourceTestBase {
-  protected static $modules = ['path_alias'];
+  protected static $modules = ['path', 'path_alias'];

+++ b/core/modules/system/tests/src/Functional/Rest/ActionResourceTestBase.php
@@ -11,7 +11,7 @@ abstract class ActionResourceTestBase extends EntityResourceTestBase {
-  protected static $modules = ['user'];
+  protected static $modules = ['action', 'user'];

+++ b/core/tests/Drupal/FunctionalTests/Rest/BaseFieldOverrideResourceTestBase.php
@@ -11,7 +11,7 @@ abstract class BaseFieldOverrideResourceTestBase extends EntityResourceTestBase
-  protected static $modules = ['field', 'node'];
+  protected static $modules = ['field', 'field_ui', 'node'];

+++ b/core/tests/Drupal/FunctionalTests/Rest/EntityFormDisplayResourceTestBase.php
@@ -11,7 +11,7 @@ abstract class EntityFormDisplayResourceTestBase extends EntityResourceTestBase
-  protected static $modules = ['node'];
+  protected static $modules = ['node', 'field_ui'];

+++ b/core/tests/Drupal/FunctionalTests/Rest/EntityViewDisplayResourceTestBase.php
@@ -11,7 +11,7 @@ abstract class EntityViewDisplayResourceTestBase extends EntityResourceTestBase
-  protected static $modules = ['node'];
+  protected static $modules = ['node', 'field_ui'];

These tests depend on the 'administer node fields' permission and that is provided by the field ui module. Perhaps we should consider moving that to the field module? There's a similar thing going on with Actions and Views UI. They both provide permissions which are used by entities provided by other modules.

The other permissions are:

  • 'administer actions'
  • 'administer views'

I think given JSON:API in core we should consider moving these permissions to the module that provides the entity that depends on these permissions. So...

  • 'administer views' => views.permissions.yml
  • 'administer actions' => system.permissions.yml
  • 'administer url aliases' => path_alias.permissions.yml
  • 'create url aliases' => path_alias.permissions.yml
  • \Drupal\field_ui\FieldUiPermissions => \Drupal\field\FieldPermissions ???

I'm not sure where 'administer display modes' should be. I think looking at the usage field_ui.permissions.yml is fine - it's where it current is. EntityViewDisplay and EntityFormDisplay both use permissions created by \Drupal\field_ui\FieldUiPermissions which I think should be moved.

daffie’s picture

Status: Needs review » Postponed
Related issues: +#3193983: Move entity permissions to the module that is providing the entity

@alexpott I think that you are right. Those permissions should be moved. I have created #3193983: Move entity permissions to the module that is providing the entity for it. Also postponing this issue for it.

alexpott’s picture

Status: Postponed » Needs review

Rerolled the patch on HEAD. Conflict in core/modules/filter/tests/src/Kernel/FilterCrudTest.php due to #3193955: Swap assertEqual arguments in preparation to replace with assertEquals.

I'm not sure that #3193983: Move entity permissions to the module that is providing the entity needs to be blocking - it raises a number of tricky issues about whether permissions that using a used by a UI should be moved to modules that don't provide the UI.

alexpott’s picture

alexpott’s picture

benjifisher’s picture

It is not clear to me what the plan is for migrations. Currently, the d6_user_role and d7_user_role migrations happily copy over whatever permissions are listed on the source site, with no validation.

The patch in #143 adds

  migrating:
    plugin: default_value
    default_value: true

to those two migrations. Then, when calculating dependencies for the Role config object,

+  public function calculateDependencies() {
+    parent::calculateDependencies();
+    // Load all permissions.
+    $permissions = \Drupal::service('user.permissions')->getPermissions();
+    foreach ($this->permissions as $permission) {
+      if (!isset($permissions[$permission])) {
+        // User role migrations set a special flag so that permissions that do
+        // not exist yet can be migrated.
+        // @todo Remove in https://www.drupal.org/project/drupal/issues/3055557.
+        if ($this->get('migrating')) {
+          continue;
+        }
+        @trigger_error('Adding the non-existent permission "' . $permission . '" to a role is deprecated in drupal:9.2.0 and triggers a runtime exception before drupal:10.0.0. The permission is either incorrect or should be defined in a permissions.yml file or a permission callback. See https://www.drupal.org/node/3193348', E_USER_DEPRECATED);
+        continue;
+      }

Minor point: I do not like the name migrating for this flag. It describes where it is set, not what it does. I prefer validate_permissions as in #2953111: Only migrate role permissions that exist on the destination.

Use the core migration directly

Suppose someone uses the core migration directly, either with Drush or with the UI provided by the migrate_drupal_ui module. Then the migrating flag is set, so we never get to the @trigger_error() line. No errors, no warnings, no messages of any kind.

In another year or two, the same developer starts working on a migration from Drupal 7 to Drupal 10. Is the plan to keep the migrating flag and replace the deprecation with an exception? Or remove the flag and turn the deprecation into an exception? In the first case, we keep migrating non-permissions. In the second case, the developer is surprised by the exception messages, reads the release notes and/or change record, and gets back on track.

Use a copy of the core migration

A developer may have a custom migration, my_module/migrations/my_user_role.yml, a modified version of one of the core migrations. Probably more common, using the migrate_plus module, the developer has a config entity exported as a YAML file.

The user-role migrations are done early in the project, with Drupal 9.1. Once the modified migration is created, it will not be affected by changes to the core migration. Then Drupal 9.2 is released, shortly before the deadline for the migration project. (I love the swooshing sound ...) As the team starts the final tests before running the migrations on the prod database, they start hitting the @trigger_error() line. Since the error level is only E_USER_DEPRECATION, does that even generate a message? Where does it show up: in the logs, in the migrate message table, somewhere else?

Recurring migrations

Not all migrations are site upgrades. The Migrate API supports continuing migrations such as feeds, user imports from other systems, or incremental upgrades.

If I am importing from an external system, am I likely to have extra permissions floating around that do not exist on the target site? Call it an edge case. Maybe a module was disabled on the target site, and neither the source system nor the migration got updated.

The previous two headings are relevant: are we using the core migrations directly, or are we using custom/copied migrations? If we are migrating from an external system, probably the latter. If access to the source system is tightly controlled, and we never set up the test/staging site to do that, then the deprecation messages will only show up in the logs for our cron jobs, if at all.

If a site has a recurring migration and does not use the core migration directly, then bad things will happen when Drupal 10 is released and the deprecation is replaced with an exception. I want to understand where the deprecation notices will appear before that happens.

alexpott’s picture

The flag added here never ends up in storage. It's only used while the object is being saved by migrations. Once the migration is over and you manage your permissions on the site then the deprecation would be triggered every time a role with a missing permission is saved. The D10 upgrade module can easy be made to check this for an existing site.

FWIW if you have a recurring migration that changes permissions - I think you are living on the edge and need to have a big think about how you are doing things - and if it is adding a permission that does not exist and cannot be managed by the site - well ideally your migration would fail but with this patch it'll work and the same statement about how and when deprecations happen would apply.

For in play migrations that don't have the migrating flag - you're right the deprecation will only be caught if they have deprecation testing but as above this is fine and to the point we will only drop support for permissions that don't exist in D10 and that'll have an upgrade path. The reason the migrating flag exists is so we can pass core's tests and not have to deal with migrations in this issue. It'll be perfectly okay for #2953111: Only migrate role permissions that exist on the destination to fix all the core migrations and remove the flag.

benjifisher’s picture

Issue summary: View changes
Status: Needs review » Needs work
+++ b/core/modules/user/migrations/d6_user_role.yml
@@ -35,6 +35,11 @@ process:
...
+  # A special flag so we can migrate permissions that do not exist yet.
+  # @todo Remove in https://www.drupal.org/project/drupal/issues/3055557.
+  migrating:
+    plugin: default_value
+    default_value: true
...
+++ b/core/modules/user/migrations/d7_user_role.yml
@@ -33,6 +33,11 @@ process:
...
+  # A special flag so we can migrate permissions that do not exist yet.
+  # @todo Remove in https://www.drupal.org/project/drupal/issues/3055557.
+  migrating:
+    plugin: default_value
+    default_value: true

Now that #3055557 is closed as a duplicate of #2953111: Only migrate role permissions that exist on the destination, we should update the comment. I still do not like calling the option migrating.

+++ b/core/modules/user/src/Entity/Role.php
@@ -193,4 +193,71 @@ public function preSave(EntityStorageInterface $storage) {
...
+  public function calculateDependencies() {
+    parent::calculateDependencies();
+    // Load all permissions.
+    $permissions = \Drupal::service('user.permissions')->getPermissions();
+    foreach ($this->permissions as $permission) {
+      if (!isset($permissions[$permission])) {
+        // User role migrations set a special flag so that permissions that do
+        // not exist yet can be migrated.
+        // @todo Remove in https://www.drupal.org/project/drupal/issues/3055557.
+        if ($this->get('migrating')) {
+          continue;
+        }
+        @trigger_error('Adding the non-existent permission "' . $permission . '" to a role is deprecated in drupal:9.2.0 and triggers a runtime exception before drupal:10.0.0. The permission is either incorrect or should be defined in a permissions.yml file or a permission callback. See https://www.drupal.org/node/3193348', E_USER_DEPRECATED);
+        continue;
+      }

Same comment about the @todo comment.

When deprecation messages are generated, we will get a separate message for each invalid permission. That is a little verbose, but it is good that we see all the invalid permissions. But when we change the deprecation to an exception, we will get only one error message, for the first invalid permission.

I suggest calculating array_diff() and array_intersect(). If the first is not empty, then generate a single message containing all the invalid permissions. If it is empty, or if the migrating flag is set, then loop through the valid permissions.

I think this change will also simplify the control flow (cyclomatic complexity).

benjifisher’s picture

The changes to core/modules/filter/src/Entity/FilterFormat.php look like unrelated coding standards cleanup.

Why this?

+++ b/core/modules/path_alias/tests/src/Functional/Rest/PathAliasResourceTestBase.php
@@ -116,11 +116,4 @@ protected function getNormalizedPostEntity() {
...
-  /**
-   * {@inheritdoc}
-   */
-  protected function getExpectedCacheContexts() {
-    return ['user.permissions'];
-  }

I do not have time right now, but I would like to compare with the list of core modules that have permission callbacks: #2953111-40: Only migrate role permissions that exist on the destination. Are all of them updated in this issue?

alexpott’s picture

Status: Needs work » Needs review
FileSize
3.92 KB
60.86 KB

I've changed the flag to skip_missing_permission_deprecation - doesn't hurt be be super descriptive.

Nice spot of the FilterFormat - wonder how that crept in /shrug.

I disagree that the control flow would get better if we have to build up a list of missing permissions. In D10 we do want to throw an exception on the first missing permission. If we build up this list now then we'll be tempted to keep more complexity later when it offers no benefit. Also this is not a common case - normally permissions exist and roles are valid. So any optimisations (if they are actually optimisations) would be for the uncommon and exceptional case.

Re changes to PathAliasResourceTestBase thats because these tests have an unbelievably complex class hierarchy and the code is in \Drupal\Tests\rest\Functional\EntityResource\EntityResourceTestBase::getExpectedCacheContexts() which now returns (as it should)

      'url.site',
      'user.permissions',

All entity types that have an admin permission with have a cache context of user.permissions. The url.site is because in order to have the admin permission we have to enable the path module... there is further discussion of the entity permission issue in #3193983: Move entity permissions to the module that is providing the entity

I do not have time right now, but I would like to compare with the list of core modules that have permission callbacks. Are all of them updated in this issue?

All the ones that have callbacks that create permissions determined by config, content or a module being installed are changed here - to add in the dependency information.

benjifisher’s picture

@alexpott, thanks for the answers in #150. I will follow up later on PathAliasResourceTestBase.

I am starting a comprehensive review of the patch in #150. I will start with something pretty simple, the user.role.*.yml files in the core profiles.

  1. I checked the user.role.*.yml files in the core profiles that are not touched by the patch in #150. In demo_umami and standard, that means the administrator role. Those have is_admin: true and permissions: { }, so that is fine. Curiously, the testing_config_overrides profile already declares a dependency on the tour module.

    I also installed the demo_umami and standard profiles, exported configuration, saved each role in the admin UI, and exported configuration again. There were no changes.

    I thought that access content was defined by the node module, but there is a comment in system.permissions.yml that explains my confusion.

  2. The system and user modules are both required, so I guess we do not need to declare dependencies on them. Most of the roles in the core profiles declare a dependency on system, but none declare a dependency on user. The author and editor roles in the demo_umami profile include the cancel account permission from the user module.

  3. I reverted core/profiles/* to match 9.2.x, then installed demo_umami and standard and exported configuration. During the profile installation, the roles were saved, and that is when dependencies were added (I think). I then restored the profiles to the version provided by the patch and compared to the exported configuration.

    In the demo_umami profile, neither the author nor editor roles depended on the workflows.workflow.editorial configuration, even though this dependency is added in the patch and both roles include permissions like use editorial transition archive. This seems like a bug.

  4. I checked the list of core modules that provide permission callbacks: #2953111-40: Only migrate role permissions that exist on the destination. Most of these callbacks are updated in this patch, but not the ones from layout_builder nor media. That deserves a closer look.

It is getting close to my bed time, so that is all for today.

alexpott’s picture

@benjifisher thank you for your review.
Re #151.2 configuration does not need a dependency on the module that provides that configuration. views.view.blah does not have an explicit dependency on views same for user.role.blah - the dependency is implicit and ensured by the config system itself. This is how config dependencies work for simple configuration such as system.site, user.mail, node.settings etc...

Re #151.3 The workflow here is exactly how people get bugs with the configuration system. Without the dependency information correct during install it doesn't know that it has to import the role after creating the workflows.workflow.editorial config. This is how config dependencies work during install. FWIW if you go and resave the role on admin/people/roles/manage/author the dependencies will be re-calculated and the role's dependencies would be fixed. Also \Drupal\Tests\demo_umami\Functional\DemoUmamiProfileTest::testConfig() exists - this ensures that the configuration in the db matches the config in the profile after install.

Re #151.4 That's definitely a bug. These were added well after work on this patch started - lol.

alexpott’s picture

Adding the missing dependencies to layout_builder and media modules. Need to add test coverage of this - let's see if the existing test coverage can help us find where best to add this.

alexpott’s picture

Ho hum... thought I'd fixed that before creating the patch.

benjifisher’s picture

@alexpott, thanks for the explanations.

In #151.3, I wrote,

During the profile installation, the roles were saved, and that is when dependencies were added (I think).

All the other dependencies were added, but not workflows.workflow.editorial configuration. I repeated the test, saving the roles before exporting, and the dependency was added.

Here are some comments on the comments (sorry) in the permission handler and its interface:

  1. A few nits with this comment:

     +++ b/core/modules/user/src/PermissionHandlerInterface.php
     @@ -34,7 +34,20 @@ interface PermissionHandlerInterface {
     ...
     +   *   - dependencies: (optional) An array of dependency entities used when
     +   *     building this permission name, structured in the same way as the return
     +   *     of ConfigEntityInterface::calculateDependencies(). For example if this
     +   *     permission was generated as effect of the existence of node type
     +   *     'article', then value of the dependency key is:
     +   *     @code
     +   *     'dependencies' => ['config' => ['node.type.article']]
     +   *     @endcode
     +   *     The module providing this permission doesn't have to be added as a
     +   *     dependency, because is automatically parsed, stored and retrieved from
     +   *     the 'provider' key.
         *   - provider: (optional) The provider name of the permission.
     +   *
     +   * @see \Drupal\Core\Config\Entity\ConfigEntityInterface::calculateDependencies()
         */
        public function getPermissions();

    Are we building a "permission name"? Maybe just "permission" or "An array of entities on which this permission depends, structured …".

    Add a comma after "For example".

    Keep sentences short. Instead of "because", start a new sentence: "It is automatically parsed …".

  2. Same snippet: Can we remove "(optional)" from the provider key, or is that considered an API change? The implementation always provides a value, and the new code in Drupal\user\Entity\Role does not check whether it is there.

  3. I do not like adding this example here, since the code snippet is introduced with "Here is an example from the core filter module (comments have been added):". In fact, there are no examples in core .permissions.yml files of dependencies being declared.

     +++ b/core/modules/user/src/PermissionHandler.php
     @@ -33,6 +33,11 @@
       *   # (optional) Boolean, when set to true a warning about site security will
       *   # be displayed on the Permissions page. Defaults to false.
       *   restrict access: false
     + *   # (optional) Dependency array following the same structure as the return
     + *   # config entities dependencies.
     + *   dependencies:
     + *     config:
     + *       - node.type.article
       *
       * # An array of callables used to generate dynamic permissions.
       * permission_callbacks:
       *   # Each item in the array should return an associative array with one or
       *   # more permissions following the same keys as the permission defined above.
       *   - Drupal\filter\FilterPermissions::permissions

    Can we correct the line "restrict access: false", or is that out of scope?

  4. There are no examples in core, but either a YAML file or a callback could explicitly set the provider key. Should we document this? What about the warning key?

    Maybe we should shorten the comment on permission_callbacks in the code snippet, then add something like this after the snippet:

    • The callback should return an associative array, keyed by permission name. In
    • addition to the title, description, and ‘restrict access’ keys described
    • above, a permission can have the following keys, whether it is defined in a
    • yml file or a callback.
      • warning (optional): …
      • dependencies (optional): A dependency array following the same structure as
      • provider (optional): …

    The second "…" is because I am confused by "the return config entities dependencies".

benjifisher’s picture

Follow-up to #148, #150: this looks simpler to me.

  public function calculateDependencies() {
    parent::calculateDependencies();
    // Load all permissions.
    $permissions = \Drupal::service('user.permissions')->getPermissions();
    $valid_permissions = array_intersect($this->permissions, $permissions);
    $invalid_permissions = array_diff($this->permissions, $permissions);
    if (!empty($invalid_permissions) && !$this->get('skip_missing_permission_deprecation')) {
      @trigger_error('Adding the non-existent permission "' . implode(', ', $invalid_permissions) . '" to a role is deprecated in drupal:9.2.0 and triggers a runtime exception before drupal:10.0.0. The permission is either incorrect or should be defined in a permissions.yml file or a permission callback. See https://www.drupal.org/node/3193348', E_USER_DEPRECATED);
    }
    foreach ($valid_permissions as $permission) {
      // Depend on the module that is providing this permissions.
      $this->addDependency('module', $permissions[$permission]['provider']);
      // Depend on any other dependencies defined by permissions granted to
      // this role.
      if (!empty($permissions[$permission]['dependencies'])) {
        $this->addDependencies($permissions[$permission]['dependencies']);
      }
    }
    return $this;
  }

The error message needs some work. I am not sure why you want the error message (when we turn it into an exception) to mention only the first invalid permission, but it would be easy to use reset() instead of implode().

benjifisher’s picture

I have already made a few comments on the changes to Drupal\user\Entity\Role, but this is the first time I have looked at onDependencyRemoval().

  1.   +++ b/core/modules/user/src/Entity/Role.php
      @@ -193,4 +193,71 @@ public function preSave(EntityStorageInterface $storage) {
      ...
      +    foreach ($dependencies as $type => $items) {
      +      foreach ($items as $key => $dependency) {
      +        // Get the dependency name, based on dependency type.
      +        $name = in_array($type, ['config', 'content'], TRUE) ? $dependency->getConfigDependencyName() : $dependency;
      +        $dependencies[$type][$key] = $name;
      +      }
      +    }

    I think this makes the intent clearer:

          foreach ($dependencies as $type => $items) {
            foreach ($items as $key => $dependency) {
              // Get the dependency name, based on dependency type.
              if (in_array($type, ['config', 'content'], TRUE)) {
                $dependencies[$type][$key] = $dependency->getConfigDependencyName();
              }
            }
          }
  2. Same snippet: I am confused by the condition in_array($type, ['config', 'content'], TRUE). I read the API docs for onDependencyRemoval() and followed the link to the class documentation for ConfigDependencyManager. That explicitly lists the outer keys of the $dependencies array: config, content, module, and theme. (Here enforced does not apply.) I guess that $dependencies['config'] and $dependencies['content'] are arrays of entities, and $dependencies['module'], $dependencies['theme'] are arrays of strings.

    If I have that right, then I will submit a little documentation patch to say so explicitly.

    Consider changing in_array(...) to $dependency instanceof EntityInterface. If you do that, then update the comment.

    If you prefer to test type, then why not do it in the outer loop? Or make the outer loop foreach (['config', 'content'] as $type) and the inner loop foreach ($dependencies[$type] ?? [] as $key => $dependency).

  3. Nit: add a comma at the end of the first comment line.

     +    // If the permission is dependent on anything being deleted or uninstalled
     +    // then remove the permission from the role.
     +    foreach ($this->permissions as $key => $permission) {
     +      if (in_array($permission_definitions[$permission]['provider'], $dependencies['module'], TRUE)) {
     +        unset($this->permissions[$key]);
     +        $changed = TRUE;
     +      }
     +      elseif (isset($permission_definitions[$permission]['dependencies'])) {
     +        foreach ($permission_definitions[$permission]['dependencies'] as $type => $list) {
     +          if (array_intersect($list, $dependencies[$type])) {
     +            unset($this->permissions[$key]);
     +            $changed = TRUE;
     +          }
     +        }
     +      }
     +    }
  4. Same snippet: I suggest moving the definition of $permission_definitions closer to this loop.

  5. Same snippet: after $changed = TRUE in the elseif clause, we could break out of the inner loop or continue 2 the outer loop.

alexpott’s picture

@benjifisher re demo_umami installation without the presence of the dependency in user.role.author - there's no way the configuration installation can work out that workflows.workflow.editorial needs to be created before the user role. This is how config dependencies work during install.

re #155.1 Fixed.

#155.2 The provider should stay optional - it is possible to use this to provide permissions on behalf of another module. Not a great idea but it is the way it is. This is the same as other provider based things in config and plugins.

#155.3 I agree - while this is possible it is pretty nonsensical so let's not document it here. This was added before I started working on this issue :) Oh I see why this was added - because you need to know about dependencies for the permission callback section. We definitely shouldn't be add anything about the warning key here - that's out-of-scope. Feels worth a separate issue to add documentation about warning to \Drupal\user\PermissionHandler. However I do think we should point people more to \Drupal\user\PermissionHandlerInterface::getPermissions() because that has the definite list of keys.

#155.4 as per the answer to .2 - i don't think we should document this any more than the documentation in \Drupal\user\PermissionHandlerInterface::getPermissions() - it's not really an intentional use of the API.

I like #156 - with a few tweaks to make it work and clearer variable names.

#157.1 I've refactored this to make it more explicit that we're only dealing with content and config
#157.2 Te refactor has dealt with this. +1 to a separate issue to improve the docs on \Drupal\Core\Config\Entity\ConfigEntityInterface::onDependencyRemoval()
#157.3 Fixed
#157.4 I disagree - getting the list of perms is less important than the loop that flattens the dependencies being removed.
#157.5 Good spot - implemented.

benjifisher’s picture

Status: Needs review » Needs work

@alexpott:

Sorry, I was not clear: the comment at the start of #155 was meant as confirmation that it worked the way you said in #152.

This comment completes my review of the non-test code for this issue.

  1. Speaking of tests, are you planning any additions related to the layout_builder and media permissions callbacks (#153, #154)?

  2. Follow-up to #155.2:

     +++ b/core/modules/user/src/Entity/Role.php
     @@ -193,4 +194,67 @@ public function preSave(EntityStorageInterface $storage) {
     ...
     +    foreach ($valid_permissions as $permission) {
     +      // Depend on the module that is providing this permissions.
     +      $this->addDependency('module', $permission_definitions[$permission]['provider']);
     +      // Depend on any other dependencies defined by permissions granted to
     +      // this role.
     +      if (!empty($permission_definitions[$permission]['dependencies'])) {
     +        $this->addDependencies($permission_definitions[$permission]['dependencies']);
     +      }
     +    }

    If the provider key really is optional, then we should add a !empty() check for it.

    I think the same point applies to onDependencyRemoval().

  3. Follow-up to #155.3: Can we correct the line "restrict access: false", in PermissionHandler.php or is that out of scope?

      * Here is an example from the core filter module (comments have been added):
      * @code
     ...
      *   restrict access: false
  4. Thanks, I think this is a lot clearer:

     +    foreach (['content', 'config'] as $type) {
     +      if (isset($dependencies[$type])) {
     +        $dependencies[$type] = array_map(function (EntityInterface $entity) {
     +          return $entity->getConfigDependencyName();
     +        }, $dependencies[$type]);
     +      }
     +    }

    I only wish that PHP made it easier to apply a method. I thought I saw something about simpler syntax for anonymous functions in PHP 8, but I cannot find it now. :-(

  5. This looks fine, although I might do $dependencies = ['config' => [$filter_format], 'module' => []]; outside the loop:

     +++ b/core/modules/user/user.module
     @@ -1305,3 +1306,16 @@ function user_form_system_regional_settings_submit($form, FormStateInterface $fo
     ...
     +function user_filter_format_disable(FilterFormatInterface $filter_format) {
     +  // Remove permissions from any roles.
     +  /** @var \Drupal\user\Entity\Role $role */
     +  foreach (Role::loadMultiple() as $role) {
     +    if ($role->onDependencyRemoval(['config' => [$filter_format], 'module' => []])) {
     +      $role->save();
     +    }
     +  }
     +}

    The real question is whether there are any similar hooks we should implement. Where else do we have the option to disable some configuration without removing it? The only thing that comes to mind is views, and I do not think there are any per-view permissions.

  6. I did not know about the ConfigEntityUpdater class. Good to know!

     +++ b/core/modules/user/user.post_update.php
     @@ -13,3 +16,15 @@ function user_removed_post_updates() {
     ...
     +/**
     + * Calculate role dependencies and remove non-existent permissions.
     + */
     +function user_post_update_update_roles(&$sandbox = NULL) {
     +  $existing_permissions = array_keys(\Drupal::service('user.permissions')->getPermissions());
     +  \Drupal::classResolver(ConfigEntityUpdater::class)->update($sandbox, 'user_role', function (Role $role) use ($existing_permissions) {
     +    $permissions = array_intersect($role->getPermissions(), $existing_permissions);
     +    $role->set('permissions', $permissions);
     +    return TRUE;
     +  });
     +}

    This passes a callback to the update() method, so the default callback will not be used. But the callback returns TRUE, meaning that the entity will be saved, and that will trigger calculating dependencies, right?

    I guess this calls for the "T" part of RTBC. I repeated the earlier test, installing the demo_umami profile with configuration from 9.2.x. This time, I also reverted user.post_update.php before the site install. After the initial config export, neither the author nor editor role had the workflows.workflow.editorial dependency. I ran the database update and re-exported configuration. The dependency was added. Test passed!

  7. This gets the job done, but it is not easy to read:

     +++ b/core/modules/content_translation/src/ContentTranslationPermissions.php
     @@ -84,6 +84,9 @@ public function contentPermissions() {
                      $permission["translate $bundle $entity_type_id"] = [
                        'title' => $this->t('Translate %bundle_label @entity_label', $t_args),
                      ];
     +                if (($bundle_entity_type = $entity_type->getBundleEntityType()) && $bundle_entity = $this->entityTypeManager->getStorage($bundle_entity_type)->load($bundle)) {
     +                  $permission["translate $bundle $entity_type_id"]['dependencies'][$bundle_entity->getConfigDependencyKey()][] = $bundle_entity->getConfigDependencyName();
     +                }

    This is inside the case where $permission_granularity = $entity_type->getPermissionGranularity() evaluates to 'bundle'. Do we still need that complicated if(...)?

  8. Same snippet: it would be simpler if we could first calculate $dependencies and then

     $permission["translate $bundle $entity_type_id"] = [
       'title' => ...,
       'dependencies' => $dependencies,
     ]

    If the answer to the previous question is that we need to keep part or all of the test, is there any harm in adding 'dependencies' => []?

  9. Nit: remove "should".

     +++ b/core/modules/field_ui/src/FieldUiPermissions.php
     @@ -48,17 +48,22 @@ public function fieldPermissions() {
    
          foreach ($this->entityTypeManager->getDefinitions() as $entity_type_id => $entity_type) {
            if ($entity_type->get('field_ui_base_route')) {
     +        // The permissions should depend on the module that provides the entity.
     +        $dependencies = ['module' => [$entity_type->getProvider()]];

    This follows the pattern that I suggested in the previous point.

  10. For consistency, can we use the same pattern here?

     +++ b/core/modules/layout_builder/src/LayoutBuilderOverridesPermissions.php
     @@ -79,23 +79,32 @@ public function permissions() {
     ...
     +      // These permissions are generated on behalf of $entity_display entity
     +      // display, therefore add this entity display as a config dependency.
     +      $base_permission = [
     +        'dependencies' => [
     +          $entity_display->getConfigDependencyKey() => [
     +            $entity_display->getConfigDependencyName(),
     +          ],
     +        ],
     +      ];
            if ($entity_type->hasKey('bundle')) {
              $permissions["configure all $bundle $entity_type_id layout overrides"] = [
                'title' => $this->t('%entity_type - %bundle: Configure all layout overrides', $args),
                'warning' => $this->t('Warning: Allows configuring the layout even if the user cannot edit the @entity_type_singular itself.', $args),
     -        ];
     +        ] + $base_permission;
  11. It is simpler to add the dependencies in buildPermissions():

     +++ b/core/modules/media/src/MediaPermissions.php
     @@ -53,6 +53,18 @@ public function mediaTypePermissions() {
            ->getStorage('media_type')->loadMultiple();
          foreach ($media_types as $type) {
            $perms += $this->buildPermissions($type);
     +      $perms_to_add = array_map(
     +        function ($perm) use ($type) {
     +          // This permission is generated on behalf of a media type, therefore
     +          // add the media type as a config dependency.
     +          $perm['dependencies'] = [
     +            $type->getConfigDependencyKey() => [$type->getConfigDependencyName()],
     +          ];
     +          return $perm;
     +        },
     +        $this->buildPermissions($type));
     +      $perms += $perms_to_add;

    If I count correctly, it even saves 2 lines, even though you have to add the 'dependencies' key 5 times.

  12. Same here. It is simpler, but it might add a line or two.

     +++ b/core/modules/node/src/NodePermissions.php
     @@ -21,9 +21,20 @@ class NodePermissions {
     ...
          foreach (NodeType::loadMultiple() as $type) {
     -      $perms += $this->buildPermissions($type);
     +      $perms_to_add = array_map(
     +        function ($perm) use ($type) {
     +          // This permission is generated on behalf of a node type, therefore
     +          // add the node type as a config dependency.
     +          $perm['dependencies'] = [
     +            $type->getConfigDependencyKey() => [$type->getConfigDependencyName()],
     +          ];
     +          return $perm;
     +        },
     +        $this->buildPermissions($type));
     +      $perms += $perms_to_add;
  13. Here I agree that we should add the permissions here instead of in the plugins, but I have two nits:

     +++ b/core/modules/rest/src/RestPermissions.php
     @@ -57,7 +57,17 @@ public function permissions() {
          $resource_configs = $this->resourceConfigStorage->loadMultiple();
          foreach ($resource_configs as $resource_config) {
            $plugin = $resource_config->getResourcePlugin();
     -      $permissions = array_merge($permissions, $plugin->permissions());
     +
     +      $permissions_to_add = array_map(function ($permission_info) use ($resource_config) {
     +        $permission_info += ['dependencies' => []];
     +        $permission_info['dependencies'] += [
     +          $resource_config->getConfigDependencyKey() => [
     +            $resource_config->getConfigDependencyName(),
     +          ],
     +        ];
     +        return $permission_info;
     +      }, $plugin->permissions());
     +      $permissions = array_merge($permissions, $permissions_to_add);
          }

    Nit 1: add an array type declaration for $permission_info.

    Nit 2: use the += pattern one more time:

     $permission_info['dependencies'] += [
       $resource_config->getConfigDependencyKey() => [],
     ];
     $permission_info['dependencies'][$resource_config->getConfigDependencyKey()] += [
       $resource_config->getConfigDependencyName(),
     ];

    I would use a foreach() loop instead of array_map(), but I think that is an individual preference.

  14. Again, it is simpler to add the dependencies in buildPermissions():

     +++ b/core/modules/taxonomy/src/TaxonomyPermissions.php
     @@ -50,7 +50,17 @@ public static function create(ContainerInterface $container) {
        public function permissions() {
          $permissions = [];
          foreach (Vocabulary::loadMultiple() as $vocabulary) {
     -      $permissions += $this->buildPermissions($vocabulary);
     +      $perms_to_add = array_map(
     +        function ($perm) use ($vocabulary) {
     +          // This permission is generated on behalf of a vocabulary, therefore
     +          // add the vocabulary as a config dependency.
     +          $perm['dependencies'] = [
     +            $vocabulary->getConfigDependencyKey() => [$vocabulary->getConfigDependencyName()],
     +          ];
     +          return $perm;
     +        },
     +        $this->buildPermissions($vocabulary));
     +      $permissions += $perms_to_add;
          }
benjifisher’s picture

Follow-up to #157.2: I added #3209296: Fix API docs for ConfigDependencyManager and #3209309: Improve documentation for ConfigEntityInterface::onDependencyRemoval(). I am not sure what the correct fix is for the second.

Follow-up to #159.13: Maybe it is simpler to use NestedArray::mergeDeep().

alexpott’s picture

Status: Needs work » Needs review
FileSize
10.07 KB
66.61 KB

Re #159.2 It's optional from the point of view of permission.yml or the a callback implementation - in fact it should never be set. It's always present though because that is guaranteed.
#159.3 it's out of scope imo
#159.5 I don't think we do - also this is only about things that can determine a permission. Plus config entity disabling is not really well supported at all.
#159.6 Yes that is how it works.
#159.7 Yes - it is possible that bundles might not be configuration - there are some test entity types that use state for example.
#159.8 I don't see the benefit here of doing that.
#159.9 Fixed
#159.10 Sure I've changed this.
#159.11 Doing things the same way everywhere is necessarily good. In think here there are two benefits to the array_map approach. One, \Drupal\media\MediaPermissions::buildPermissions is really simple to read. Two, if a new permission is added to \Drupal\media\MediaPermissions::buildPermissions it'll automatically get the dependency.
#159.12 As above
#159.13 NestedArray::merge is a good idea
#159.14 as per .11/.12 I disagree.

Added more explicit test coverage of RestPermissions - still need to add coverage for Media and Layout Builder.

benjifisher’s picture

@alexpott:

The patch in #161 removes "(optional)" in PermissionHandlerInterface, so I guess we agree on #155.2 after all.

+1 for eliminating $perms_to_add in the permissions callbacks and just using $permissions += array_map(...). "Get rid of the middleman."

I will not press on most of the changes to the permissions callbacks, but I would like to see some simplification in ContentTranslationPermissions (#159.7, #159.8). The benefit of #159.8 is that long lines like this are hard to read, hard to understand, and therefore harder to maintain and less secure.

We will get an exception from $this->entityTypeManager->getStorage($bundle_entity_type) if $bundle_entity_type is NULL or an empty string, but also if it is 'not_valid_bundle_type'. We can protect against that and keep the lines short with something like this (untested):

                $dependencies = [];
                $bundle_entity_type = $entity_type->getBundleEntityType();
		try {
                  $bundle_entity = $this->entityTypeManager
                    ->getStorage($bundle_entity_type)
                    ->load($bundle);
                  if ($bundle_entity) {
                    $dependencies[$bundle_entity->getConfigDependencyKey()] = [
                      $bundle_entity->getConfigDependencyName(),
                    ];
                  }
		}
		catch (PluginNotFoundException $e) {
                  // Do not add dependencies because $bundle_entity_type is not
                  // valid.
		}
                $permission["translate $bundle $entity_type_id"] = [
                  'title' => $this->t('Translate %bundle_label @entity_label', $t_args),
                  'dependencies' => $dependencies,
                ];

I probably will not have time to look at the tests until next weekend.

benjifisher’s picture

Can we add some feedback and logging to the update function?

Suppose that my site has the Easter module. I enable it one day a year, then uninstall it, but keep the related permissions. I know this is not best practice, but it seems to work. I should be notified.

I suggest one log message for each role that has permissions removed, and one message from the update function. Draft text:

Invalid permissions were removed from the following roles: .... These permissions are listed in the logs. See [link to CR] for more information.

It might even be a good idea to postpone the update function until Drupal 10.

benjifisher’s picture

Assigned: Unassigned » benjifisher

I am starting to review the changes to the tests.

  1. Reviewing the tests seems like a big task, with 43 files changed. Luckily, about 10 of them are like this:

     +++ b/core/tests/Drupal/FunctionalTests/Rest/EntityViewDisplayResourceTestBase.php
     @@ -11,7 +11,7 @@ abstract class EntityViewDisplayResourceTestBase extends EntityResourceTestBase
        /**
         * {@inheritdoc}
         */
     -  protected static $modules = ['node'];
     +  protected static $modules = ['node', 'field_ui'];
    
        /**
         * {@inheritdoc}

    @larowlan asked about this in #85.1. In #141, @alexpott explained it is because these tests use the ‘administer node fields’ permission, and suggested moving that permission to the field module. I was going to suggest a follow-up, but I see we already have #3193983: Move entity permissions to the module that is providing the entity.

  2. Why do we need to rebuild the router? I think a code comment would help.

     +++ b/core/modules/user/tests/src/Kernel/UserRoleDeleteTest.php
     @@ -71,4 +74,79 @@ public function testRoleDeleteUserRoleReferenceDelete() {
     ...
     +  public function testDependenciesRemoval() {
     +    $this->enableModules(['node', 'filter']);
     +    $this->container->get('router.builder')->rebuild();
  3. Nit: I do not like the pattern of defining a variable inside an explicit array like this:

     +++ b/core/modules/user/tests/src/Kernel/UserRoleDeleteTest.php
     @@ -71,4 +74,79 @@ public function testRoleDeleteUserRoleReferenceDelete() {
     ...
     +    $role = Role::create([
     +      'id' => $rid = mb_strtolower($this->randomMachineName()),
     +      'label' => $this->randomString(),
     +    ]);
     +    $role->save();
     ...
     +    $role = Role::load($rid);

    Can we use $role = Role::load($role->id()) instead? Alternatively, put $rid = ... on a separate line so it is easier to scan.

  4. The machine name and title do not match.

     +++ b/core/modules/system/tests/modules/entity_test/entity_test.permissions.yml
     @@ -18,6 +18,8 @@ view all entity_test_query_access entities:
     ...
     +create entity_test entity_test_with_bundle entities:
     +  title: 'Create entity_test content'
  5. Nit: can we expand the first line of the doc block?

     +++ b/core/modules/system/tests/modules/entity_test/src/Entity/EntityTestMulWithBundle.php
     @@ -0,0 +1,49 @@
     +<?php
     +
     +namespace Drupal\entity_test\Entity;
     +
     +/**
     + * Defines the test entity class.
  6. I do not see where this permission is used:

     +++ b/core/modules/system/tests/modules/system_test/system_test.permissions.yml
     @@ -1,2 +1,5 @@
     ...
     +pet llamas:
     +  title: 'Permission for page cache testing'

    Is it related to this change?

     @@ -70,15 +70,6 @@ protected function getExpectedNormalizedEntity() {
     ...
     -  /**
     -   * {@inheritdoc}
     -   */
     -  protected function getExpectedCacheContexts() {
     -    return [
     -      'user.permissions',
     -    ];

I am taking a break for lunch. I will assign the issue to myself for the rest of the review.

benjifisher’s picture

Assigned: benjifisher » Unassigned
Status: Needs review » Needs work

I finished reviewing the changes to the tests. I do not have any more changes to request: just a couple of comments and a bunch of questions.

  1. I see that I already asked about getExpectedCacheContexts() in #149, and I just re-read the reply in #150. I am still trying to understand this change and how it relates to this issue.

    We remove the override in four places, falling back to the method defined in PathAliasResourceTestBase. In one case (views module) this is a no-op, so the only question is why the change is in scope for this issue. In the other three cases, removing the override means we are adding the url.site context. In each of those cases, we are also enabling an additional module. In the test for the field module, we enable field_ui; for path_alias, we enable path; in system, we enable action.

    In all cases, we need to enable the extra module because it defines permissions that we want to test. These changes would not be needed if we fixed #3193983: Move entity permissions to the module that is providing the entity before this issue.

    I am afraid I still do not see how enabling the extra module adds the url.site cache context.

    The other question is whether we are making these changes in the right place. We are enabling an extra module in four blase classes, and adding an expected cache context in three of these. Why do it there and not in the child test classes?

  2. @quietone and I already discussed why we need to add locale to the migration tests: see #2953111-45: Only migrate role permissions that exist on the destination. For example:

     +++ b/core/modules/block/tests/src/Kernel/Migrate/d6/MigrateBlockContentTranslationTest.php
     @@ -24,6 +24,7 @@ class MigrateBlockContentTranslationTest extends MigrateDrupal6TestBase {
          'block_content',
          'config_translation',
          'language',
     +    'locale',
  3. I think this is correcting a typo:

     +++ b/core/modules/jsonapi/tests/src/Functional/ResourceTestBase.php
     @@ -2373,7 +2373,7 @@ public function testPatchIndividual() {
          $this->grantPermissionsToTestedRole([
            'use editorial transition create_new_draft',
            'use editorial transition archived_published',
     -      'use editorial transition published',
     +      'use editorial transition publish',

    If I am right, then why is it in scope? If I am wrong, then we are intentionally adding a bogus permission, and I would expect further changes to the test.

  4. Where are these test permissions referenced?

     +++ b/core/modules/jsonapi/tests/modules/jsonapi_test_field_access/jsonapi_test_field_access.permissions.yml
     @@ -0,0 +1,6 @@
     +'field_jsonapi_test_entity_ref edit access':
     +  title: 'Tests JSON:API field edit access'
     +'field_jsonapi_test_entity_ref update access':
     +  title: 'Tests JSON:API field update access'
     +'field_jsonapi_test_entity_ref view access':
     +  title: 'Tests JSON:API field view access'
    
     +++ b/core/modules/jsonapi/tests/modules/jsonapi_test_field_filter_access/jsonapi_test_field_filter_access.permissions.yml
     @@ -0,0 +1,2 @@
     +'filter by spotlight field':
     +  title: 'Tests JSON:API filter access'
  5. It makes sense to enable these modules instead of the user module, but I wonder why it worked before. How is it in scope for this issue?

     +++ b/core/modules/jsonapi/tests/src/Functional/ActionTest.php
     @@ -16,7 +16,7 @@ class ActionTest extends ResourceTestBase {
     ...
     -  protected static $modules = ['user'];
     +  protected static $modules = ['action'];
    
     +++ b/core/modules/jsonapi/tests/src/Functional/PathAliasTest.php
     @@ -16,7 +16,7 @@ class PathAliasTest extends ResourceTestBase {
     ...
     -  protected static $modules = ['user'];
     +  protected static $modules = ['path'];
  6. I am surprised we do not have more changes like this one. From now on, our tests have to assign real permissions when they want to change a role.

     +++ b/core/modules/basic_auth/tests/src/Functional/BasicAuthTest.php
     @@ -234,7 +234,7 @@ public function testCacheabilityOf401Response() {
     ...
     -    $this->grantPermissions(Role::load(Role::ANONYMOUS_ID), [$this->randomMachineName()]);
     +    $this->grantPermissions(Role::load(Role::ANONYMOUS_ID), ['access content']);
  7. Why this change?

     +++ b/core/modules/media_library/tests/src/FunctionalJavascript/WidgetWithoutTypesTest.php
     @@ -129,7 +129,7 @@ public function testWidgetWithoutMediaTypes() {
     ...
     -    $field_ui_uninstalled_message = 'There are no allowed media types configured for this field. Edit the field settings to select the allowed media types.';
     +    $field_ui_uninstalled_message = 'There are no allowed media types configured for this field. Please contact the site administrator.';

    I see that the test fails without this change, but I do not see why.

  8. Do we test somewhere that the permissions change?

     +++ b/core/modules/toolbar/tests/src/Functional/ToolbarAdminMenuTest.php
     @@ -114,6 +115,14 @@ protected function setUp(): void {
     ...
        public function testModuleStatusChangeSubtreesHashCacheClear() {
     +    // Give the user an admin role so permissions change when modules are
     +    // installed and uninstalled.
     +    $role = Role::load($this->createRole([]));
     +    $role->setIsAdmin(TRUE);
     +    $role->save();
     +    $this->adminUser->addRole($role->id());
     +    $this->adminUser->save();

    I see that the test fails without this change, but I do not understand why. Changing permissions means that the permissions hash changes, so we do not get the cached copy of the page. But I am still missing something: why does this test work before this issue?

alexpott’s picture

Status: Needs work » Needs review
FileSize
6.59 KB
64.71 KB

Re #163
I think moving the update to follow-up and something we discuss separately. Including timing is a great idea. This is possible because we're deprecating support for permissions that don't exist. There's a slight oddness because permissions will still be removed on uninstall if we can work out that the permission depends on a module that is being uninstalled but I think that that is okay.


Re #164
2. We no longer need the route rebuild - we fixed route rebuilding during module install a while back.

3. Fixed - this uncovered that we really should be forcing a proper reload of the role.

4. Fixed

5. Fixed

6. \Drupal\Tests\page_cache\Functional\PageCacheTest uses the pet llamas permission. This can be found by searching the code base. We don't want to start a new standard of documenting where permissions are used.


Re #165
1. This is because we're now generating a url because the test now includes the module that defines the admin permission. I do not think that fixing #3193983: Move entity permissions to the module that is providing the entity is simple and maybe doing so is not even desirable. However in order to have full API coverage of these entities and have their admin permissions actually exist we need to enable these modules. This is a good thing because it reflects the current reality. On a real Drupal site you'd struggle to use the JSON API to create views without the Views UI module being enabled (for example). So this issue is stopping us from pretending things make sense in the tests - when actually they don't.

3. That permission does not exist so we need to fix that in this issue - this is another massive benefit of making this change. When you use a permission in tests it'll be a real permission. We could file a follow-up because I'm pretty sure that we don't need to assign these permissions anyway. The test is making these transitions to the entity under test but it is doing do on the API level and therefore permissions are not checked (as far as I know). I prefer to fix the permission to use the real and intended permission here.

4. The codebase can be grepped to reveal this. I don't think that adding comments benefits us here. The module names tell you that these permissions are being used for access testing.

5. This all because the tests in HEAD are assigning the admin permission without it existing.

6. Well it really doesn't make sense to assign a permission that does not exist. Also it is dangerous because often then you'll not be testing what you think you are testing.

7. Because \Drupal\media_library\Plugin\Field\FieldWidget\MediaLibraryWidget::getNoMediaTypesAvailableMessage() changes the message depending on whether field_ui is installed or not and whether the user has a permission. This permission is now correctly removed when the field UI module is uninstalled.

8. $this->assertDifferentHash(); does that. This hash is built from permission. If it is different then permissions have changed. This is documented in the test.


Added test coverage of Media permission dependencies - still need to cover Layout Builder.
alexpott’s picture

Created #3211113: Add update path to remove permissions that do not exist - whilst doing that I realised that there is a massive downside of not including the update here. That is, we're not fixing the gotcha for existing sites. They'll still be left if the position that enabling a module might result in roles having active permissions that they'd normally not have. I'm not sure removing the update is such a great idea anymore. Hmmm.

benjifisher’s picture

@alexpott:

I will look at the new patch, and the comments in #166, after work today.

As for the update function, we are also not fixing the problem when saving a role. We just issue deprecation messages. We have lived with the status quo for a long time. I think we should put up with it until a major version upgrade.

Maybe expand #3211113: Add update path to remove permissions that do not exist to include changing the deprecation messages to exceptions.

So far, I have done a lot of R, but only a little T. When I save a role with bogus permissions, will I see the deprecation messages in the admin UI? In the log?

Have you made any changes in response to #162?

benjifisher’s picture

Status: Needs review » Needs work

I did some testing.

I inspected and modified permissions for the authenticated role using the config import/export pages under /admin/config/development/configuration/single.

I saved the role by saving its permission form, /admin/people/permissions/authenticated. I did not test, but I think I would get the same results using /admin/people/roles/manage/authenticated.

  1. Check out 9.2.x.
  2. Install Drupal with the demo_umami profile.
  3. Add two bogus permissions.
  4. Add foo and bar text formats, and give the authenticated role permission to use them.
  5. Add foo and bar content types, and give the authenticated role permission to create them.
  6. Save the authenticated role. (Part of the previous step.)
  7. Check the user.role.authenticated configuration: it has the expected permissions, no dependencies.
  8. Disable the foo text format.
  9. Delete the foo content type.
  10. Save the authenticated role.
  11. Check the user.role.authenticated configuration: no changes.
  12. Check out the feature branch (or apply the patch from #166).
  13. Save the authenticated role.
  14. Check the user.role.authenticated configuration: it now has dependencies on the bar text format and the bar content type, and all the same permissions.
  15. Disable the bar text format.
  16. Delete the bar content type. The authenticated role is listed as configuration that will be updated.
  17. Save the authenticated role.
  18. Check the user.role.authenticated configuration: the bar dependency and permissions have been removed.

So far, so good: that is what we expected to happen.

  1. Checking the logs, I see a lot of PHP Notices ("Undefined index") from this line:

     +++ b/core/modules/user/src/Entity/Role.php
     @@ -193,4 +194,67 @@ public function preSave(EntityStorageInterface $storage) {
     ...
     +      if (in_array($permission_definitions[$permission]['provider'], $dependencies['module'], TRUE)) {

    We need to add a check that $permission_definitions[$permission] is defined. This will come up if we defer the update function to a follow-up issue, if we choose not to but some site owners neglect to run it, or if permissions are migrated without the defining module. NW for this.

  2. From #147, replying to one of my questions:

    … the deprecation would be triggered every time a role with a missing permission is saved. The D10 upgrade module can easy be made to check this for an existing site.

    Does "D10 upgrade module" mean the version of Upgrade Status that will run on Drupal 9 and prepare for upgrades to Drupal 10? If so, then I think we should add an issue now for that module.

    Unless I am missing something, there is no more natural way to see the deprecation messages. There is a PHP setting that would make them show up in the log, right? But almost no one would enable that on a live site, and almost no one would think to test saving user roles during deprecation testing. (That is why we have to add an issue now to do that.)

  3. We should expand the Permissions must exist change record. In #163, I suggested the hypothetical Easter module, enabled just one day a year. What about the Monday module? If I enable it once a week, and then disable it the next day, I do not want to set up permissions every time. Currently, I do not have to. After this issue goes in, I will. This impacts site owners, not just module developers. Maybe we should add a third CR instead of adding this to an existing one.

  4. The test was passing before, but I will take your word that it is a good idea to reset the cache here. It is a little odd to use Role::load() instead of $role_storage->load(). Why not use the class from the container?

     +++ b/core/modules/user/tests/src/Kernel/UserRoleDeleteTest.php
     @@ -71,4 +74,83 @@ public function testRoleDeleteUserRoleReferenceDelete() {
     ...
     +  public function testDependenciesRemoval() {
     +    $this->enableModules(['node', 'filter']);
     +    /** @var \Drupal\user\RoleStorage $role_storage */
     +    $role_storage = $this->container->get('entity_type.manager')->getStorage('user_role');
     ...
     +    $role_storage->resetCache();
     +    $role = Role::load($role->id());
     ...
     +    $role_storage->resetCache();
     +    $role = Role::load($role->id());
     ...
  5. Re #164.6: I see. The test already assigned the pet llamas permission to a role, even though the permission did not exist. After this issue, that test would fail. #164.7 is similar.

  6. I am still trying to understand #164.8. Maybe it will be clearer in the morning.

benjifisher’s picture

I did not even need to sleep on it. I just needed to step away from the computer for a bit.

After disabling the taxonomy module, the admin menu does change. The problem is after enabling it again: unless we restore the "administer taxonomy" permission to our test user, the admin menu will not change.

I think it is a good idea to give the admin user an extra role, with just the "administer taxonomy" permission, after re-installing the taxonomy module, instead of giving the admin user a true admin role (all permissions) at the start of the test method. Among other things, it makes the test more like a realistic use case, where you always assign permissions after installing a module.

Something like this, before the second call to assertDifferentHash():

    // Give the user a role with the permission that was removed when the
    // taxonomy module was uninstalled.
    $rid = $this->drupalCreateRole(['administer taxonomy']);
    $this->adminUser->addRole($rid);
    $this->adminUser->save();

Sorry, I am rambling. I really do need to get some sleep.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

benjifisher’s picture

Issue summary: View changes

I am adding some notes to the issue description to help with the final review.

Looking back on earlier comments, I think these still need to be addressed:

  • #169.1
  • #159.7, #159.8, #162 (thread)
  • #164.8, #170 (thread)
  • (minor) reply to #169.4.
alexpott’s picture

Status: Needs work » Needs review
FileSize
7.32 KB
72.79 KB

Patch attached fixes:

  • #169.1
  • Add test coverage of layout builder permission dependencies.

Review comments:

  • #159.7 Yes we do. Bundles do not have to be entities.
  • #159.8/#162 I don't see the benefit here of doing that. I still disagree that the suggestions are better and represent more secure code. We can however move $bundle_entity_type = $entity_type->getBundleEntityType(); out of the loop and the if - which makes the line shorter.
  • Re #170 / #168.4 FWIW I think using admin roles is actually more realistic and representative a real experience of someone installing the module as user 1 or a user with an admin role.

I've thought about #167 a bit more and I think we need the updates in this patch because we can't issue the deprecations unless we've taken steps to clean them up and this patch also adds removal on uninstall so not cleaning up will introduce some very very odd behaviour.

alexpott’s picture

Let's update the deprecations to 9.3.0 - this won't land in 9.2.x now.

benjifisher’s picture

Status: Needs review » Needs work
Issue tags: +Needs change record updates

@alexpott, thanks for the updates.

Notice that I have added references to this issue from three other open issues recently:

  1. If we are restoring the update function to this issue, then can we add some feedback and logging to the update function? I think it should log a message for each role that has permissions removed, listing all of them. Then it should return a message listing those roles, with the usual "check the logs for details". It should be pretty much the same logic we added to generate one deprecation message per role. I suggested text for the return message in #163.

  2. I think the update function should be mentioned in a CR and in the release-notes snippet.

  3. Setting $bundle_entity_type = $entity_type->getBundleEntityType(); on its own line helps, but I still find this really hard to read:

           if ($this->contentTranslationManager->isEnabled($entity_type_id, $bundle)) {
             $t_args['%bundle_label'] = isset($bundle_info['label']) ? $bundle_info['label'] : $bundle;
             $permission["translate $bundle $entity_type_id"] = [
               'title' => $this->t('Translate %bundle_label @entity_label', $t_args),
             ];
             if ($bundle_entity_type && $bundle_entity = $this->entityTypeManager->getStorage($bundle_entity_type)->load($bundle)) {
               $permission["translate $bundle $entity_type_id"]['dependencies'][$bundle_entity->getConfigDependencyKey()][] = $bundle_entity->getConfigDependencyName();
             }
           }

    I think this is a lot easier:

           if ($this->contentTranslationManager->isEnabled($entity_type_id, $bundle)) {
             $dependencies = [];
             if ($bundle_entity_type && $bundle_entity = $this->entityTypeManager->getStorage($bundle_entity_type)->load($bundle)) {
               $dependencies[$bundle_entity->getConfigDependencyKey()] = [
                 $bundle_entity->getConfigDependencyName(),
               ];
             }
             $t_args['%bundle_label'] = isset($bundle_info['label']) ? $bundle_info['label'] : $bundle;
             $permission["translate $bundle $entity_type_id"] = [
               'title' => $this->t('Translate %bundle_label @entity_label', $t_args),
               'dependencies' => $dependencies,
             ];
           }
  4. Missing type declaration:

     +++ b/core/modules/layout_builder/tests/src/Functional/LayoutBuilderAccessTest.php
     @@ -66,8 +66,10 @@ protected function setUp(): void {
     ...
     +   * @param array $permission_dependencies
     +   *   An array of expected permission dependencies.
         */
     -  public function testAccessWithBundles(array $permissions, $default_access, $non_editable_access, $editable_access) {
     +  public function testAccessWithBundles(array $permissions, $default_access, $non_editable_access, $editable_access, $permission_dependencies) {
  5. If you do not like the suggestion in #170, then please change this comment (see #165.8 for context):

      +    // Give the user an admin role so permissions change when modules are
      +    // installed and uninstalled.

    I found this comment very confusing. The point is not that permissions change. The point is that the links in the admin menu (and therefore the tree hash) change when the taxonomy module is uninstalled and when it is enabled again.

  6. General point: I often quip that writing documentation is really very easy … provided that the audience is the same as the author. The hard part is writing documentation that will be helpful to others, and that is really quite hard.

    Along the same lines, you are not the best judge of whether the code you have written might be confusing to others, just as I am not the best judge of the code that I write. Please take seriously the fact that I have trouble with the long lines of code in ContentTranslationPermissions and the comment in ToolbarAdminMenuTest.

benjifisher’s picture

Issue summary: View changes
alexpott’s picture

Status: Needs work » Needs review
FileSize
5.88 KB
74.73 KB

1. This is great idea. I will do that in a follow-up patch.
2. I'll do this in the follow-up.
3. I disagree with the local variable approach because rather than removing complexity it adds complexity for the reader. There are more variables to keep in your head and more possibilities for side effects. This code suffers from the complexity of the situation - grafting translation permissions on top of the entity system is complex and our existing implementation makes this hard because bundles can be anything - in core tests we have a bundle implementation that relies on state!

However given the disagreement let's go a whole step further and encapsulate the bundle permission array building in a private function where it's easier to document the bundle stuff and also prevents side effects. For example in HEAD atm the $t_args array is mutated as we loop which can result in t() being called with incorrect arguments that happen to be unused in core's translations. Using a private function here is good because this method has no meaning outside of the call from \Drupal\content_translation\ContentTranslationPermissions::contentPermissions().

4. Fixed
5. I've improved the comment - you're right that this should be clearer. Hopefully the new version will help it explain this us when we're looking at this again in 6 months time!

benjifisher’s picture

Status: Needs review » Needs work
Issue tags: +Needs reroll

I think the patch in #177 needs a reroll after #3131281: Replace assertEqual() with assertEquals().

benjifisher’s picture

I confirmed the fixes for #175.4, #175.5.

The new helper function is a big help for #175.3. Even little things help reduce cognitive load when reading code, like not having to read "translate $bundle $entity_type_id" twice (and check that it is the same both times).

I just have a few tiny suggestions for the new function:

  1. The @param comment for $t_args could use string[] instead of array. Can we add something like Required key: @entity_label. to the comment?

     +++ b/core/modules/content_translation/src/ContentTranslationPermissions.php
     ...
     +  /**
     +   * Builds a content translation permission array for a bundle.
     +   *
     +   * @param \Drupal\Core\Entity\EntityTypeInterface $entity_type
     +   *   The entity type.
     +   * @param string $bundle
     +   *   The bundle to build the translation permission for.
     +   * @param array $bundle_info
     +   *   The bundle info.
     +   * @param array $t_args
     +   *   The arguments for translation.
     +   *
     +   * @return array
     +   *   The permission details.
     +   */
     +  private function buildBundlePermission(EntityTypeInterface $entity_type, string $bundle, array $bundle_info, array $t_args) {
     +    $t_args['%bundle_label'] = isset($bundle_info['label']) ? $bundle_info['label'] : $bundle;
  2. Similarly, can we add to the @return comment? Something like The permission details, keyed by 'title' and, if available, 'dependencies'.

  3. Suggestion: call $entity_type->getBundleEntityType() in the calling function, so the first argument becomes string|null $bundle_entity_type. Then you do not need to add the use statement at the top of the file.

  4. Simplify the first line of the function:

         $t_args['%bundle_label'] = $bundle_info['label'] ?? $bundle;

    Or delete that line and use $t_args + ['%bundle_label' => $bundle_info['label'] ?? $bundle] inside $this->t().

alexpott’s picture

Status: Needs work » Needs review
FileSize
2.89 KB
74.99 KB

Thanks for the review @benjifisher

#179.1 I've removed the $t_args thing - it is a premature optimisation that doesn't optimise. \Drupal\Core\Entity\EntityTypeInterface::getSingularLabel is statically cached anyways.
#179.2 I wish we had Symfony's policy on private method documentation. But sure added this.
#179.3 If you make the first argument optional then all the other args have to be optional - we could make entity type the last arg but it is the superset - ie all bundles belong to an entity type so that doesn't make sense to me. Anyway now I've remove $t_args this point is moot.
#179.4 $t_args is removed - see point 1.

Still need to add more logging to the update function.

Status: Needs review » Needs work

The last submitted patch, 180: 2571235-9.3.x-180.patch, failed testing. View results

benjifisher’s picture

Thanks, I like the changes in #180.

Is there an honest way to generate an interdiff after a reroll?

Is it my imagination or is the testbot generating more random fails than usual in the last couple of weeks? It looks as though the patch in #180 failed, but you (or someone) triggered a retest.

Still NW for #175.1, #175.2.

benjifisher’s picture

alexpott’s picture

Issue summary: View changes
Status: Needs work » Needs review

Patch addresses #175.1 and tests it. Interestingly this reveals that our update db has both the anonymous and authenticated role with a non-existent permission.

Will update the CR re #175.2

@benjifisher my interdiff is honest. It is the diff of my changes to the patch after resolving the patch conflicts. There is no way to generate an interdiff that can apply to the previous patch - that just impossible because it's not the patch that has changed it's the code it is applying to.

alexpott’s picture

alexpott’s picture

anish.a’s picture

@benjifisher,

The patch uses assertEquals() everywhere, as I am seeing.

Also, #3131281: Replace assertEqual() with assertEquals() is fixed. I am unclear on what needs to be done in the context of Needs reroll tag here. Can you explain?

benjifisher’s picture

Issue tags: -Needs reroll

Sorry, the patch was rerolled in #180. @alexpott and I forgot to remove the tag. I will do so now.

Also, [3131281] is fixed.

Yes, that is why the patch in #177 needed a reroll.

alison’s picture

("sidenote" about reroll interdiffs -- just to help set some minds at ease! https://www.drupal.org/docs/develop/git/using-git-to-contribute-to-drupa... )

-------
I'm still running #121 on my 8.9.x sites, and I don't have a D9 site to test on :( I think/hope this will get backported, but chiming in to express a desire for it, just in case. Thank you all!

alison’s picture

One more thing -- and if this is 8.9.x specific, kinda never mind, I realize things are targeting 9.x -- or if there's another place I should bring this up, just let me know! -- for now, I'm mentioning here juuuuust in case it's an issue with the patch syntax --

Is anyone else getting extra directories when you apply the patch? -- this does happen to me from time to time with core patches, idk if it's an issue with drupal + cweagans/composer-patches, or what, but I get the following "extra directories" when I apply this patch (#121 on 8.9.x) -- again, it's not the first time I've run into this with core patches, but it doesn't happen "every time," and I haven't figured out a pattern yet...

	web/core/b/
	web/core/core/
benjifisher’s picture

@alexpott:

Thanks for the logging in the update function, and the updated tests. We are almost there!

  1. I guess that the coder module has a new sniff, because now the testbot is complaining about this @var comment:

     +++ b/core/modules/node/src/NodePermissions.php
     @@ -21,9 +21,20 @@ class NodePermissions {
         */
        public function nodeTypePermissions() {
          $perms = [];
     +    /* @var \Drupal\node\Entity\NodeType $type */
  2. Nit: add ending punctuation to the log message. (Adjust the test.)

     +++ b/core/modules/user/user.post_update.php
     @@ -13,3 +17,34 @@ function user_removed_post_updates() {
     ...
     +      \Drupal::logger('update')->notice(
     +        'The role %role has had the following non-existent permissions removed: %permissions',
     +        ['%role' => $role->label(), '%permissions' => implode(', ', $removed_permissions)]
     +      );
  3. Same snippet, suggestion: change "permissions" to "permission(s)". I agree that PluralTranslatableMarkup is overkill.

  4. Same function: I notice that you chose an explicit return NULL instead of "falling off the end". I guess that is because of the explicit return inside the if block.

  5. I see you added a note about the update function to [#3193348]. (There are two CRs for this issue.) I think the update function is relevant to "Site builders, administrators, editors", so I checked off that audience for the CR. I will add to the text soon.

benjifisher’s picture

Status: Needs review » Needs work

NW for #191.1, optionally #191.2, 3.

Interdiffs: Yes, my understanding is that there is no way to create an interdiff for a reroll. I was wondering if @alexpott knew something I did not. We never posted a straight reroll of the patch in #177. If we had, then the interdiff in #180 would be an ordinary comparison of that to the patch in #180.

@alisonjo315, what problem does this issue cause for your sites? Is it just clutter or something worse? My understanding of the problems here is mostly theoretical/potential.

BTW, looking at your earlier comments on this issue, I see you reported the PHP notices in #120 that were still there when I tested in #169. (I think those are the same messages.) They should be gone now.

I think it is unlikely that this issue will be back-ported to anything before 9.3.x. For one thing, it is disruptive, as shown by the many changes to the existing tests. I still worry about sites that may rely on the current behavior, although that also falls into the "theoretical/potential" category. Drupal 8.9 is now in security maintenance, although that does not rule out Major bug fixes (and this issue is classified as a Critical bug fix).

@alisonjo315, the extra directories you see could be a result of the composer patches plugin. See https://github.com/cweagans/composer-patches/tree/1.x#allowing-to-force-... (the patchLevel option).

alexpott’s picture

Status: Needs work » Needs review
FileSize
2.88 KB
76.72 KB

Addressing #192

benjifisher’s picture

Issue summary: View changes
Status: Needs review » Reviewed & tested by the community
Issue tags: -Needs change record updates

@alexpott, thanks for fixing those last nits. I am satisfied: RTBC.

I updated the change record Permissions must exist, so I am removing that item from the list of remaining tasks in the issue summary. I think the remaining tasks should be done when this issue is actually fixed.

AaronMcHale’s picture

Just had a quick skim over the patch and change record.

I don't want to be a wet blanket here and disrupt a great RTBC issue, but a couple of things came to mind:

  1. Should the CR provide instructions/examples of what a module developer should do to ensure their permissions are properly depending on the right thing, for example I might want entity-specific permissions to depend on the entity type. I'm just thinking that I use the contrib Entity API Permission Providers in some of my custom Entity Types, the CR doesn't tell me if there's anything I need to do here.
  2. Similarly, I noticed in the patch that field_ui permissions would depend on the relevant module, however wouldn't we want them to depend on the entity type, as that's what they directly relate to - so like: field_ui permissions, depend on entity_type, which depends on module - you get to the same place in the end, but feel it might be better to be specific.

As a final note, once this is done it would be great if some effort could then be directed towards #2809177: Introduce entity permission providers. As I said, I've really enjoyed using the contrib Entity API module Permission Providers, takes a nice bit of manual effort out of custom entity type development; I feel this issue provides an even stronger case for that issue to be implemented.

Thanks,
-Aaron

alexpott’s picture

@AaronMcHale config cannot depend on an entity type - since an entity type is neither a module, theme, config entity or content entity. These dependencies map to config dependencies. RE Entity API Permission Providers - it's going to be up to what ever hooks that into the permissions system to provide the correct dependencies - probably the entity API module. If something is collecting / creating permissions on an entity type's behalf then it'll need to add a dependency to the permission on the module providing the entity type - exactly the same way field_ui is doing in the patch.

benjifisher’s picture

@AaronMcHale:

There are two CRs for this issue. Did you look at both? I think that Permissions can define dependencies has the sort of examples you suggested in #195.1.

field_ui permissions would depend on the relevant module

Be careful: the permissions are not config entities, so they do not have dependencies. The user roles have dependencies.

I think the suggestion in #27 to turn permissions into config entities made a lot of sense. I did not read the discussion starting there to see why that suggestion was dropped.

AaronMcHale’s picture

Thanks both for the helpful replies :)

@alexpott: Ah yes that all makes sense, thanks for clarifying that.

@benjifisher: Thank you for linking the other CR, I wasn't expecting there to be more than one and that one does indeed cover what I mentioned.

I think the suggestion in #27 to turn permissions into config entities made a lot of sense. I did not read the discussion starting there to see why that suggestion was dropped.

Yeah that is an interesting idea, I suppose it could have some advantages. I wonder if there would be any performance impacts (positive or negative) for sites that have a large number of permissions.

alison’s picture

@benjifisher

@alisonjo315, what problem does this issue cause for your sites? Is it just clutter or something worse? My understanding of the problems here is mostly theoretical/potential.

Clutter! Drives me bananas. And the clutter occasionally causes legit confusion, like when I was uninstalling a contrib module I didn't end up using (printable), and wanted to make sure the associated permissions were gone, and there was a leftover permission for accessing printer-friendly pages provided by "book" module (the site I'm working on right now was built from a clone of a site that used the book module -- "book" is uninstalled on the site I'm working on now, but the permissions were left behind). I guess even that example of confusion is based in me trying to eliminate clutter, but still. Sometimes it's just, so much clutter!!!!!!!!!!!!

alexpott’s picture

Re #27 and the config entity idea - that scares me a lot. And there are a number of things in that idea that don't map to configuration that well but the biggest and most important point is that permissions are not configurable and idea of making them possibly configurable is part of what scares me. Then there are config overrides and also sorts of other config functionality that just doesn't apply. The ability to override the 'restrict access' value of a permission in settings.php?!?!?

larowlan’s picture

Issue tags: +Needs reroll
error: patch failed: core/modules/content_moderation/src/Permissions.php:30
error: core/modules/content_moderation/src/Permissions.php: patch does not apply
error: patch failed: core/modules/content_moderation/tests/src/Kernel/ContentModerationPermissionsTest.php:59
error: core/modules/content_moderation/tests/src/Kernel/ContentModerationPermissionsTest.php: patch does not apply

Doesn't apply anymore, haven't reviewed yet, but this is on my list.

I note that this will have implications for install profiles that try to ship permissions for optional modules.

larowlan’s picture

Status: Reviewed & tested by the community » Needs work
alexpott’s picture

Status: Needs work » Reviewed & tested by the community
FileSize
77.34 KB
7.11 KB
77.34 KB

Rebased on top of 9.3.x - #2977495: Content Moderation missing permission descriptions caused the conflicts cause it adds new descriptions to the content moderation permissions. No interdiff because its a rebase - attaching a diff of the two patches so people can see not much has changed...

larowlan’s picture

Status: Reviewed & tested by the community » Needs review
Issue tags: -Needs reroll +Bug Smash Initiative

This looks great, awesome work everyone.

  1. +++ b/core/modules/content_translation/src/ContentTranslationPermissions.php
    @@ -99,6 +96,36 @@ public function contentPermissions() {
    +    // If the entity type uses bundle entities, add a dependency on the bundle.
    +    $bundle_entity_type = $entity_type->getBundleEntityType();
    +    if ($bundle_entity_type && $bundle_entity = $this->entityTypeManager->getStorage($bundle_entity_type)->load($bundle)) {
    +      $permission['dependencies'][$bundle_entity->getConfigDependencyKey()][] = $bundle_entity->getConfigDependencyName();
    +    }
    

    If there is no bundle entity type, should we instead add a dependency on the entity-type provider like we do for the entity-path?

  2. +++ b/core/modules/layout_builder/src/LayoutBuilderOverridesPermissions.php
    @@ -79,22 +79,33 @@ public function permissions() {
    +          'dependencies' => $dependencies,
    

    note to self: we don't need to add the bundle and entity-type provider here because the entity-view-display already depends on them

  3. +++ b/core/modules/media/src/MediaPermissions.php
    @@ -52,7 +52,17 @@ public function mediaTypePermissions() {
    -      $perms += $this->buildPermissions($type);
    +      $perms += array_map(
    +        function (array $perm) use ($type) {
    
    +++ b/core/modules/node/src/NodePermissions.php
    @@ -21,9 +21,20 @@ class NodePermissions {
    -      $perms += $this->buildPermissions($type);
    +      $perms += array_map(
    

    we usually favour full words for variable names.
    Given we're touching this here and introducing a new variable (perm) should we use the full spelling of the words?

  4. +++ b/core/modules/taxonomy/src/TaxonomyPermissions.php
    @@ -50,7 +50,17 @@ public static function create(ContainerInterface $container) {
    +      $permissions += array_map(
    +        function (array $perm) use ($vocabulary) {
    +          // This permission is generated on behalf of a vocabulary, therefore
    

    with media types, node types, and now vocabularies, this is the third time we've got similar code, by the rule of 3 is this worth adding a trait for?

    There's at least a few contrib modules that will need the same change

  5. +++ b/core/modules/user/src/Entity/Role.php
    @@ -193,4 +194,71 @@ public function preSave(EntityStorageInterface $storage) {
    +        $dependencies[$type] = array_map(function (EntityInterface $entity) {
    +          return $entity->getConfigDependencyName();
    +        }, $dependencies[$type]);
    

    Looking at the code in \Drupal\Core\Config\ConfigManager::callOnDependencyRemoval it looks like we already do this in the passed values, there's an array_combine in there that sets the keys to the config dependency names. So we should be able to use array_keys instead?

  6. +++ b/core/modules/user/src/Entity/Role.php
    @@ -193,4 +194,71 @@ public function preSave(EntityStorageInterface $storage) {
    +        continue;
    ...
    +      elseif (isset($permission_definitions[$permission]['dependencies'])) {
    

    we have a mix of elseif and continue here, should we standardise on continue and avoid elseif?

  7. +++ b/core/modules/user/src/PermissionHandler.php
    @@ -36,11 +36,15 @@
    + *   # The callable should return an associative array with one or more
    + *   # permissions. Each permission array can use the same keys as the example
    + *   # permission defined above. Additionally, a dependencies key is supported.
    + *   # For more information about permission dependencies see
    + *   # PermissionHandlerInterface::getPermissions().
    ...
    + * @see \Drupal\user\PermissionHandlerInterface::getPermissions()
    

    I think this should also add a @see to \Drupal\Core\Config\Entity\ConfigDependencyManager where the structure of dependencies is documented in full

  8. +++ b/core/modules/user/src/PermissionHandlerInterface.php
    @@ -34,7 +34,21 @@ interface PermissionHandlerInterface {
    +   * @see \Drupal\Core\Config\Entity\ConfigEntityInterface::calculateDependencies()
    

    As above, the docs at \Drupal\Core\Config\Entity\ConfigDependencyManager are more complete and would be a better reference point in my opinion

alexpott’s picture

@larowlan thanks the review.

Re #204

  1. Excellent point - fixed and added test coverage.
  2. I thought about adding a note to the code but I'm not sure it really helps
  3. Sure why not - fixed for NodePermissions too
  4. This nixes the changes in 3... but yeah why not have a trait. I considered an abstract class to extend but I like the trait better because passing a callable in for the permission array per bundle allows you to keep the type hint on the actual bundle entity interface.
  5. Good point - fixed.
  6. Yep +1
  7. Hmmm I think the change in point 8 is the one to make - you're dealing permissions - go to \Drupal\user\PermissionHandlerInterface::getPermissions - and then you want more info on config dependencies go to \Drupal\Core\Config\Entity\ConfigDependencyManager
  8. Yep +1

EDIT: fixed numbering

Status: Needs review » Needs work

The last submitted patch, 205: 2571235-9.3.x-205.patch, failed testing. View results

benjifisher’s picture

@alexpott:

If you can fix the failing tests, then I will review after work today.

Can you edit #205 and fix the numbering?

Related to #204.7, 8: I opened #3209296: Fix API docs for ConfigDependencyManager while reviewing this issue, but #12 there suggests that issue will be closed as "won't fix".

alexpott’s picture

Status: Needs work » Needs review
FileSize
662 bytes
80.74 KB

Fixed up the failing tests. The direct call to $role->onDependencyRemoval() is interesting. I think it is fine - we don't want to go through \Drupal\Core\Config\ConfigManagerInterface::getConfigEntitiesToChangeOnDependencyRemoval() as that will find other config entities. And we don't want to duplicate the code in \Drupal\user\Entity\Role::calculateDependencies().

Status: Needs review » Needs work

The last submitted patch, 208: 2571235-9.3.x-208.patch, failed testing. View results

alexpott’s picture

Status: Needs work » Needs review
FileSize
2.72 KB
81.38 KB
benjifisher’s picture

Status: Needs review » Needs work

Sorry, I did not get to this yesterday as I said I would.

  1. Why is this in the Core namespace? It deals with permissions, which are defined in the user module, and it is odd to have @see references from Core to a module.

     +++ b/core/lib/Drupal/Core/Entity/BundlePermissionHandlerTrait.php
     @@ -0,0 +1,44 @@
     +<?php
     +
     +namespace Drupal\Core\Entity;
     +
     +/**
     + * Provides a method to simplify generate bundle level permissions.
     + */
     +trait BundlePermissionHandlerTrait {
  2. (same snippet) Nit: "… to simplify generating bundle level permissions." Also, I would say "bundle-level", but reasonable people can disagree on hyphens.

  3. Is there a reason to make this a trait instead of making generatePermissions() a public, static method?

     +   * @param callable $permission_builder
     +   *   A callable to generate the permissions for a particular bundle. Returns
     +   *   an array of permissions. See PermissionHandlerInterface::getPermissions()
     +   *   for the array structure.
     +   *
     +   * @return array
     +   *   Permissions array. See PermissionHandlerInterface::getPermissions() for
     +   *   the array structure.
     +   *
     +   * @see \Drupal\user\PermissionHandlerInterface::getPermissions()
     +   */
     +  protected function generatePermissions(array $bundles, callable $permission_builder) {

    We would not have to pass $permission_builder if the trait declared abstract protected function buildPermissions(). Then I would withdraw the suggestion of making this a public, static method.

  4. Why not use +=?

     +++ b/core/modules/user/src/Entity/Role.php
     @@ -193,4 +193,77 @@ public function preSave(EntityStorageInterface $storage) {
     ...
     +  public function onDependencyRemoval(array $dependencies) {
     +    // Ensure $dependencies has the expected keys. Unlike other implementations
     +    // of ConfigEntityInterface::onDependencyRemoval() this is called from code
     +    // other than \Drupal\Core\Config\ConfigManager::callOnDependencyRemoval()
     +    $dependencies = $dependencies + ['config' => [], 'content' => [], 'module' => []];

    A function should not be aware of where it is used. If that is really what is going on here, then the problem is in the other implementations of onDependencyRemoval(). But maybe it is not really a problem, and all we need is the first sentence of the comment.

  5. Can we split this long line?

     +++ b/core/modules/user/user.module
     @@ -1305,3 +1306,16 @@ function user_form_system_regional_settings_submit($form, FormStateInterface $fo
     ...
     +function user_filter_format_disable(FilterFormatInterface $filter_format) {
     +  // Remove permissions from any roles.
     +  /** @var \Drupal\user\Entity\Role $role */
     +  foreach (Role::loadMultiple() as $role) {
     +    if ($role->onDependencyRemoval(['config' => [$filter_format->getConfigDependencyName() => $filter_format]])) {

    Perhaps

       $dependencies['config'][$filter_format->getConfigDependencyName()] = $filter_format;
       /** @var \Drupal\user\Entity\Role $role */
       foreach (Role::loadMultiple() as $role) {
         if ($role->onDependencyRemoval($dependencies)) {

    Comparing to the previous patch, I guess we are adding a key to the innermost array (used to be [$filter_format]) to be consistent with the change suggested in #204.5.

  6. The more I think about it, the more I wonder about #204.5. In Role::onDependencyRemoval(), we implement a function declared in ConfigEntityInterface. The doc block there says something about the keys of $dependencies, but nothing to suggest that the values are again keyed arrays. Maybe we should update that doc block, but that is an API change. Is any of this marked @internal?

    I will defer to @alexpott and @larowlan if they agree that I am making a mountain out of a mole hill.

alexpott’s picture

Status: Needs work » Needs review
FileSize
9.16 KB
83.31 KB

Thanks for the review @benjifisher

Re #211

  1. There's plenty of permission stuff in the \Drupal\Core for example: \Drupal\Core\Access\AccessResult::allowedIfHasPermission or admin_permission = "administer site configuration", in \Drupal\Core\Datetime\Entity\DateFormat. We already lots of @see to \Drupal\system - this would be the first \Drupal\user but I think this is okay because system and user are both
  2. There's 4 bundle-level (in only 1 file) and 5 bundle level in core already.. so I guess I'll stick with bundle level as that's the most popular atm
  3. Let's use an abstract method. I thought we couldn't because of types but it works because abstract methods in traits d not enforce the signature (which is interesting).
  4. I've refactored all of this to use hasPermission() and revokePermission() which means we don't have to think about all the onDependencyRemoval and how public it really is.
  5. See 4.
  6. See 4.

I've added a test for the new trait to ensure it works regardless of core implementations and also to test an edge case not covered by core usages.

alexpott’s picture

Ho hum test groups... now we don't have a SImpletest UI in core these are largely pointless...

longwave’s picture

+++ b/core/lib/Drupal/Core/Entity/BundlePermissionHandlerTrait.php
@@ -0,0 +1,52 @@
+   * This method will call the ::buildPermissions() method which each bundle and

This sentence doesn't make sense - which -> for?

alexpott’s picture

benjifisher’s picture

Status: Needs review » Needs work

@alexpott:

At first, I assumed that the testbot's OOM error was an infrastructure problem. It turns out that this is the error you get if your patch causes so many failures that the testbot cannot record them all. :-(

Regarding #211.3 (and reply in #212 ... thanks for fixing the numbering) I read Traits in the PHP manual carefully before making that suggestion. Unfortunately, that document is not correct for PHP 8.0. According to PHP 8.0: Fatal errors on incompatible method signatures we no longer have the freedom to mismatch function signatures in this case. (I do not know how authoritative that reference is. It turned up in a search. First I installed PHP 8.0 and got the same failures that the testbot reported.)

Well, it seemed like a good idea. I really like having a proper docblock for buildPermissions() instead of an @var comment for $permission_builder. But I guess we either go back to that or we change the function signatures when implementing buildPermissions(). I will go along with either approach.

Other than feeling like scope creep on an issue that is already complicated, I like the idea of making the implementations compatible with the declaration in the trait. They are protected, so it is not an API change. We could use {@inheritdoc}. The type declaration EntityInterface is more generic than the existing declarations, so that should not cause problems if contrib/custom code extends these classes. Maybe we would have to remove the return type to allow for that. And the only methods used on the arguments are id() and label(), so EntityInterface is a good choice.


Back to #211.2, which I already described as a nit: you replied to the part after "Also", but the comment still says "... to simplify generate bundle level permissions" instead of my suggested "… to simplify generating bundle level permissions."

At first glance, the refactoring in response to #211.4-6 looks like an improvement, but I have not looked at it closely yet.

alexpott’s picture

Status: Needs work » Needs review
FileSize
4.79 KB
83.38 KB

Boo to PHP 8 becoming more sensible! :)

Changing the typehints to EntityInterface will break contrib modules - see http://grep.xnddx.ru/node/30981393 for example. So let's go with the anonymous function approach again...

alexpott’s picture

So #3059984: Add new “Content Editor” role to Standard Profile has landed. This issue was not implemented correctly because it is adding permission that don't exist to a role. We'll have to adjust that role on this issue and fix it to use hook_modules_installed in the standard profile.

Status: Needs review » Needs work

The last submitted patch, 217: 2571235-9.3.x-217.patch, failed testing. View results

benjifisher’s picture

@alexpott:

I mentioned #3059984: Add new “Content Editor” role to Standard Profile in #175. I was thinking that if that issue was fixed first (as it was) then we could remove the undefined permissions as an update to this issue. I see you have already opened #3221258: Fix content editor role to only have permissions that exist, so I will review that instead.

I also mentioned #2809291: Add "edit block $type" permissions, which is currently RTBC. I was going to suggest that you mark it postponed on this issue, but I will do that myself.

P.S. The good news is that the tests worked as designed. It is lucky that I was aware of all these issues and linked them to each other, but we do not have to rely on that luck.

alexpott’s picture

Status: Needs work » Needs review

The non-existent permissions have been removed as #3221258: Fix content editor role to only have permissions that exist has landed - let's see if we green here again...

Status: Needs review » Needs work

The last submitted patch, 217: 2571235-9.3.x-217.patch, failed testing. View results

alexpott’s picture

Status: Needs work » Needs review
FileSize
711 bytes
84.07 KB

Adding the missing dependencies to the new role...

benjifisher’s picture

Status: Needs review » Needs work
  1. Nit: this comment is left over from an earlier patch:

     +++ b/core/lib/Drupal/Core/Entity/BundlePermissionHandlerTrait.php
     @@ -0,0 +1,45 @@
     ...
     +   * This method will call the ::buildPermissions() method for each bundle and
     +   * add bundle dependencies to the resulting permissions array.
  2. To be on the safe side, can we restore the line from the patch in #210 that ensures the keys content and config are present? Otherwise, this snippet might generate PHP notices. See my comment in #211.4.

    In fact, I restored the test added by that patch to UserRoleEntityTest, and it failed.Maybe we should restore the test, too.

     +++ b/core/modules/user/src/Entity/Role.php
     @@ -193,4 +193,72 @@ public function preSave(EntityStorageInterface $storage) {
     ...
     +  public function onDependencyRemoval(array $dependencies) {
     ...
     +    foreach (['content', 'config'] as $type) {
     +      $dependencies[$type] = array_keys($dependencies[$type]);
     +    }
  3. This is certainly simpler than the earlier version. I do not have time to think this through completely: I am a little worried about the order of operations, and I have not looked up when this hook is invoked. When the role is saved, that will recalculate the dependencies. If the filter format has already been removed, then that should work. We have test coverage for this, right?

    Since saving the role will trigger onDependencyRemoval(), do we even need to explicitly revoke the permission?

     +++ b/core/modules/user/user.module
     @@ -1305,3 +1306,17 @@ function user_form_system_regional_settings_submit($form, FormStateInterface $fo
     ...
     +function user_filter_format_disable(FilterFormatInterface $filter_format) {
     +  // Remove the permission from any roles.
     +  $permission = $filter_format->getPermissionName();
     +  /** @var \Drupal\user\Entity\Role $role */
     +  foreach (Role::loadMultiple() as $role) {
     +    if ($role->hasPermission($permission)) {
     +      $role->revokePermission($permission)->save();
     +    }
     +  }
     +}
  4. Why create two bundles in this test?

     +++ b/core/tests/Drupal/KernelTests/Core/Entity/BundlePermissionHandlerTraitTest.php
     @@ -0,0 +1,81 @@
     ...
     +  public function testGeneratePermissions() {
     +    EntityTestBundle::create([
     +      'id' => 'test1',
     +    ])->save();
     +    EntityTestBundle::create([
     +      'id' => 'test2',
     +    ])->save();

    Were you thinking of changing these lines to peek at the bundle name?

     +        // Ensure it is possible for buildPermissions to add additional
     +        // dependencies.
     +        'dependencies' => ['config' => ["test_module.entity.{$bundle->id()}"], 'module' => ['test_module']],
alexpott’s picture

Status: Needs work » Needs review
FileSize
745 bytes
83.92 KB

Thanks for the review @benjifisher

  1. Good spot - fixed.
  2. I don't think we should do this. The array structure is guaranteed by \Drupal\Core\Config\ConfigManager::callOnDependencyRemoval() - as per earlier comments \Drupal\Core\Config\Entity\ConfigEntityBase::onDependencyRemoval() is not designed to be called by anything else. The old test UserRoleEntityTest added a test for calling onDependencyRemoval() because that's what we were doing from user_filter_format_disable() but we're now no longer doing that so that test is unnecessary. FWIW all other implementation of ::onDependencyRemoval() assume the structure of $dependencies too.
  3. Saving a role does not trigger onDependencyRemoval and yes we have test coverage - see the changes to core/modules/filter/tests/src/Functional/FilterAdminTest.php.
  4. Two bundles to test that it is looping and not just a singleton. I don't understand
    Were you thinking of changing these lines to peek at the bundle name?"

    The bundle ID is part of the config dependency name and this is tested.

Been doing some more thinking about #3211113: Add update path to remove permissions that do not exist and whether or not we should delay the update till D10. Still not 100% convinced either way - going to try to chat to @catch about this one as @catch tends to have good thoughts about this type of problem.

benjifisher’s picture

Issue summary: View changes
Status: Needs review » Reviewed & tested by the community

@alexpott:

  1. Thanks.
  2. OK. I would be happier if onDependencyRemoval() mentioned the expected keys in its doc block, but that is out of scope for this issue.
  3. Thanks for the reminders. I had another look at the test, and it reassured me.
  4. I was just trying to guess the reason for two bundles. It would not hurt to test that generatePermissions() handles both cases (add to existing dependencies or no existing dependencies). But your explanation in #225 is good enough for me.
  5. Just to clarify: you are reconsidering moving the database update to #3211113: Add update path to remove permissions that do not exist and postponing that to Drupal 10. You are not considering delaying this issue, right?

I think this issue is ready. Back to RTBC.

We should update the CR "Permissions can define dependencies", using the new trait. I will do that after work today if you do not do it first. For now, I am adding it to the issue summary.

alexpott’s picture

@benjifisher re #226.5 yeah I was considering moving the update out again. I talked to @catch who said that he preferred to leave the update in this issue. One thing that is interesting once we fix the migrate part of this we're going to need to have an update that does exacting the same things.

@benjifisher thanks for all the reviews and effort on this one. It's not a simple one at all.

benjifisher’s picture

It's not a simple one at all.

You can say that again!

benjifisher’s picture

Issue summary: View changes

I updated the change record Permissions can define dependencies, so I am removing the "Remaining task" I added in #226.

Since #3059984: Add new “Content Editor” role to Standard Profile has landed, then modified in #3221258: Fix content editor role to only have permissions that exist and #223 of this issue, I am removing it from the list of issues to be updated when this issue is fixed.

larowlan’s picture

adding review credits

  • larowlan committed cffb02a on 9.3.x
    Issue #2571235 by alexpott, claudiu.cristea, dawehner, Wim Leers,...
larowlan’s picture

Status: Reviewed & tested by the community » Fixed

I applied this branch locally and went through all the changes again in my IDE

Other than some personal-preference nitpicks, I couldn't fault it.

And it doesn't make sense to hold up a critical on those.

I think the longer this is in 9.3, the more chance there is of finding any issues.

Committed cffb02a and pushed to 9.3.x. Thanks!

Published both change records.

larowlan’s picture

Issue tags: +9.3.0 release notes

This is worth highlighting in release notes

AaronMcHale’s picture

Great work here everyone 🎉

So, just to be super clear, when a module is uninstalled, because the permissions it has created will depend on the module, they will automatically be removed from relevant roles? In other words, a module developer doesn't need to implement a uninstall hook to remove any permissions?

If that is the case, might be worth adding a line to one of the CRs explaining that, or at least explain what action (if any) module developers needs to take, as the CR isn't clear if any action is required as a result of this change.

Thanks,
-Aaron

alexpott’s picture

Issue summary: View changes

@AaronMcHale you are correct a module developer doesn't need to do anything for a module's permissions to be removed from a role on module uninstall.

I added the word automatically to This means that permissions will be automatically cleaned up when a module is uninstalled.

AaronMcHale’s picture

@alexpott Great sounds good, thanks for the addition :)

benjifisher’s picture

Issue summary: View changes

I took care of the remaining tasks listed in the issue summary. Part of that is a new issue for the Upgrade Status module: #3223068: Check for invalid permissions, will throw runtime exceptions in Drupal 10.

Do we need a follow-up issue to convert the deprecations to exceptions, or do we manage that by searching for deprecation messages when getting closer to Drupal 10?

alexpott’s picture

@benjifisher thanks for doing that. We can create follow-up already for the work to convert the deprecations to exceptions.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

bkline’s picture

@claudiu.cristea

permissions must be config entities in 9.x

What is the name used for this? I'm looking in the database for a Drupal 9.x site, and SELECT name FROM config WHERE name LIKE '%perm%' gets an empty results set.

benjifisher’s picture

@bkline:

If you read the responses to that comment, you will see that the final decision was not to make permissions into config objects. As before, user roles are config objects.

bkline’s picture

@benjifisher:

If you read the responses to that comment, ...

You make it sound so easy, given that this is not a threaded interface. :-) I did find references to that topic after wading through 178 more comments, though I haven't yet found the definitive final decision to which you referred.

Perhaps it would have been more prudent if he had said "it has been proposed that ..." instead of "permissions must be config entities in 9.x," which implied that the change had already made its way into the 9.x code base.

neclimdul’s picture

Ouch, this hurt. This was a "feature" for me that bundled with config split allowed me to have a role that had permissions for modules that where only available in certain environments. Namely a developer role that it would get access to devel.module in development environments but also have access to certain power user developer tools in QA and Production where devel isn't installed.

Note: not actually to upset, this issue has some cool benefits I'm looking forward to. Just identifying a possible pain point or regression we might see.

bircher’s picture

RE #243
Yea you have to use Config Split 2.x in that scenario, this issue here was one of the reasons I worked on 2.x over the summer.

John Pitcairn’s picture

Ouch indeed.

If a module provides permissions that are in use by any user role, and that module is config-excluded in $settings['config_exclude_modules'], then any user roles with those permissions checked will also be excluded from config export/import.

This change tripped me up while upgrading a site from 9.2.x to 9.3.x.

bircher’s picture

RE #245
Yes this is one of the consequences of this. The solution would be #3230825: Change config as if the modules defined in config_exclude_modules had been uninstalled.
But there are quite a few difficulties. Config Split 2.x is experimenting with solutions for this.