Part of #3371246: [Meta, Plan] Pitch-Burgh: Policy based access in core

Summary

We want to convert the Flexible Permissions module into core code. This will allow for any access policy to be translated into a set of permissions so that access checks can run. These permissions are calculated once during a build phase and then pulled through an alter phase. After both phases are complete, the end result is poured into an immutable value object and cached.

This value object can then be used by a permission checker (such as the one introduced here #3347873: Centralize permission checks in a service) to verify if an account has permission to a given access check.

A key aspect that may not seem useful to core is the use of scopes. By default, all access policies will return permissions for the "drupal" scope, but to make it so access modules in contrib don't have to copy all of this logic, the use of scopes will allow them to define permissions in other scopes, such as Group, Domain, Commerce Stores, etc. where the calculated permissions will only have effect within the scope of, say, one domain or group.

Overview of new API

New concept: Build and alter phase

The goal is to loop over all access policies in a build phase and come up with a set of permissions that are from then on immutable. These permissions respect cache contexts (and are thus added to the variation cache), so you could be getting a different set depending on the time of day, your user roles, etc. The reason the permissions have to be immutable is because we still cache everything by user.permissions and if your permissions could change during runtime, that would quickly become a security nightmare.

Right before we turn the built permissions into this immutable object, however, we allow all policies to have a final alter pass of the fully built permissions. This adds some extra flexibility for people who want to alter other modules' (or core's) access policy behavior from the outside. After the alter pass, the immutable object is built and cached.

New concept: Scopes and identifiers

Furthermore, these permissions are built for a given scope and identifier within said scope. For Drupal core, both of these simply default to AccessPolicyInterface::SCOPE_DRUPAL and seemingly do nothing, but it's contrib where the addition of this concept will truly shine. You see, for Group, Domain, or similar we don't necessarily care about what permissions you have across the entire website, but rather what you can do within a subset, e.g. a single domain.

The main question then becomes "To what do these permissions apply?" Because Group is a bit complex to explain here (it has 3 scopes), let's use Domain as an example: If you want to discern who can make changes to content on an individual domain (whether active or not), then you would hand out those permissions in SCOPE_DOMAIN, where the identifier is the machine name of the domain. Doing this would for example allow the "Belgian team" to change the content of the ".be" website, but not the ".nl".

AccessPolicyProcessor and VariationCache

This is where the AccessPolicyProcessor (APP) and VariationCache (part of core) come into play. The APP is a service collector that looks for services which are tagged as access_policy. It then asks all of these policies what they initially vary by (i.e. cache contexts) and evaluates what they end up varying by at the end of the permission processing. This concept of "initial cache contexts" and "final cache contexts" is what powers cache redirects and is required to store something in the variation cache.

Before any of this runs, however, it first asks each individual access policy whether it applies to the scope and does not do anything with those that don't. So any Group or Domain specific access policies will not interfere with your sitewide Drupal permissions or vice versa.

(Refinable)CalculatedPermissions and CalculatedPermissionsItem

These value objects simply represent all of what was described above. The CalculatedPermissions object holds entries for an infinite amount of identifiers across an infinite amount of scopes. Keep in mind: In case of Drupal core it would be one scope and one identifier.

At each scope-identifier address sits a CalculatedPermissionsItem that indicates what permissions you should get for a given identifier within a scope and whether or not you have admin rights (i.e. all permissions) there. If you try to add multiple CPI to the same address, they get merged or overwritten depending on a parameter in the RefinableCalculatedPermissions::addItem() method.

Which leads to the next point: The difference between RefinableCalculatedPermissions and CalculatedPermissions. Simply put, the former allows you to make changes and is passed around during the build phase, the latter is immutable and passed around to those services who request what your fully built permissions are.

Permission checkers

This is not part of the new API but I'll explain it here rather than the implementation issue. How do you use these built permissions? Well, you use a central permission checker such as the one introduced in #3347873: Centralize permission checks in a service and call the AccessPolicyProcessor. Then you ask for the permission item at the scope-identifier (SCOPE_DRUPAL/SCOPE_DRUPAL for core) address you seek and simply run hasPermission() on it.

Here's what that would look like (taken from the implementation issue):

  /**
   * {@inheritdoc}
   */
  public function hasPermission(string $permission, AccountInterface $account): bool {
    $item = $this->processor->processAccessPolicies($account)->getItem();
    return $item && $item->hasPermission($permission);
  }

Sane defaults

See anything missing in the example above? We didn't have to specify AccessPolicyInterface::SCOPE_DRUPAL anywhere because the system defaults to the drupal scope and identifier. Contrib scopes would have to specify these parameters in the processAccessPolicies() and getItem() call.

To make life easier on people wanting to add an access policy to core from within a contrib module (such as office hours), CalculatedPermissionsItem also defaults to adding permissions to the SCOPE_DRUPAL/SCOPE_DRUPAL address. This should make access policies aimed at core easier on the eyes.

Here's what that would look like (taken from the implementation issue):

    foreach ($user_roles as $user_role) {
      $calculated_permissions
        ->addItem(new CalculatedPermissionsItem($user_role->getPermissions(), $user_role->isAdmin()))
        ->addCacheableDependency($user_role);
    }

Issue fork drupal-3376843

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

kristiaanvandeneynde created an issue. See original summary.

kristiaanvandeneynde’s picture

Issue summary: View changes

kristiaanvandeneynde’s picture

Status: Active » Needs review

Some things worth mentioning:

  1. I needed a fast memory cache and the one freely available continuously serializes/unserializes its data whenever its used. As permissions are asked for often during a request, I decided to use the MemoryCache class also used by jsonapi. The only downside is that that class has no factory like all the other ones and so I added it here. I'll ping @catch and let him decide what to do with this (or guide me on a good alternative)
  2. I've made the "drupal" scope the default in the applies() check, but in the CalculatedPermissions* classes you still need to pass "drupal" as the scope and identifier. Should we also somehow make these default to "drupal" to make it easier for people to use in combination with core? If so, suggestions welcome.
  3. I'll convert the unit tests I had in Flexible Permissions ASAP and add them here, but I've run out of time for the day.
  4. I think it makes more sense to have applies() also be taken into account for the alter phase, will add that ASAP too
kristiaanvandeneynde’s picture

Last test is choking on $cache_static->set(Argument::cetera())->willReturn(NULL); as the set method has a void return type. There is a workaround like this: $cache_static->set(Argument::cetera())->will(function(){}); as per https://github.com/phpspec/prophecy/issues/280.

However, newer versions of prophecy shouldn't have this issue as it got fixed here: https://github.com/phpspec/prophecy/pull/398

Added a commit that removes any mention of willReturn and can add in the will workaround if they go red. Hopefully, we are on a recent enough prophecy version and do not need the workaround.

partdigital’s picture

@kristiaanvandeneynde

Any thoughts on an API like this being applicable with access policy module? https://www.drupal.org/project/access_policy

This provides a powerful UI for allowing site builders to define their own access policies. It currently works on Drupal 9 and 10 but I'd be open to releasing version 2.0 to use this API if you feel it's a good fit.

kristiaanvandeneynde’s picture

Two more things I want to try:

  • Change AccessPolicyChain into AccessPolicyProcessor and remove the notion of it being chainable. Also do not implement AccessPolicyInterface anymore
  • Change CalculatedPermissionsItem to default to Drupal scope and identifier by turning order of parameters into:
    1. permissions
    2. isAdmin (optional)
    3. scope (optional)
    4. identifier (optional)
kristiaanvandeneynde’s picture

Re #6 I've looked into your module and the whole idea of building access policies using a UI is incompatible with this API-first approach. The goal here is to build an API that both core and contrib can use and then allow contrib to build any UI on top of that.

Your module comes with plugins that make assumptions about the types of access policies, limiting what can be done with it unless you introduce more plugins. This work comes with tagged services that translate an access policy into a set of permissions that can then be handed out on a particular level: core, Group, Domain, Commerce Stores, ...

Secondly, evaluating access policies during runtime is unsafe because we'd still be caching things with user.permissions even though your permissions may vary based on what access policies apply during your current and next request. The API presented here prevents that by introducing a one-time build phase and then caching your processed permissions using cache contexts. This way, it's still safe to use the user.permissions cache context because we know that your access does not change between requests unless you no longer match your policies' criteria (e.g. you lose a user role).

partdigital’s picture

re #8

Just to clarify, I’m not suggesting moving the access policy module into core. I’m wondering if I could use your new API as part of the contrib module?

In regards to your other comments, did you get a chance to try out the access policy module? It can be difficult to interpret how it works just by looking at the code. For example, you can make as many access policies as you like. The Access policy type plugin merely defines which operations you want to control. The access policies themselves are config entities.

Secondly, evaluating access policies during runtime is unsafe because we'd still be caching things with user.permissions even though your permissions may vary based on what access policies apply during your current and next request.

The access policy module uses a combination of permissions and access rules (plugins that return TRUE/FALSE) to control access. So any user.permission cache context still applies. If the user has permission then it will evaluate the access rules via hook_entity_access() which uses the user cache context. It sounds like, for this reason, it might not be viable to use your API in access policy?

However, I don’t want to hijack this issue, so I’d happy to open another issue or even hop on a brief call.

Thanks, and nice work! Looking forward to seeing this get in!

kristiaanvandeneynde’s picture

I will add documentation and an example on how to use this, so that should help in evaluating whether you could leverage it in your module.

lauriii’s picture

Tagging for a framework manager review. I think it would be helpful for the review to have the API documented on a high level in the issue summary / draft CR.

catch’s picture

Yeah I'm missing an issue summary here or in the meta issue. Left a note about memory cache.

kristiaanvandeneynde’s picture

Issue summary: View changes

Thanks for the reviews so far!

Updated the IS to document the new API a bit.

kristiaanvandeneynde’s picture

Issue summary: View changes
kristiaanvandeneynde’s picture

As the commit says: I made the scope and identifier 'drupal' the default across the board now. So core code should look cleaner. Should still go green and I consider this the final version, aside from obvious discussions regarding naming etc.

But functionally this is feature complete.

kristiaanvandeneynde’s picture

Actually working on one minor change: Add the account as a parameter to the alter function. Will post an update with that functionality soon.

smustgrave’s picture

Status: Needs review » Reviewed & tested by the community

Tried my best with this one. Read the issue summary a few times, kudos btw for all the detail, everything looks okay.

Seems it's been through some review as all the threads appear to be resolved.

Going to move this one to the committer bucket?

Not sure if the change record will be needed for this ticket or one of the related tickets?

lauriii’s picture

Issue summary has been updated since #12. We do need a change record eventually for the new API so adding that tag. We could use the issue summary as a starting point for that.

kristiaanvandeneynde’s picture

Issue summary: View changes

Thanks so much for the review! Updated the IS a bit and will write a change record when I have more time.

kristiaanvandeneynde’s picture

Added a CR. It mainly copies the IS from this issue, but I've changed the language and intro a bit so that we can also have a CR for the implementation issue that merely focuses on the changes to the permission checker and the 2 core access policies.

https://www.drupal.org/node/3385551

quietone’s picture

Issue summary: View changes

I'm triaging RTBC issues. I read the IS and the comments. I didn't find any unanswered questions.

The change record has been added but has not been reviewed. That still needs to be done. I added a remaining tasks section to the IS that include this. I did skim the change record and I think that as new API there should be a guide at Drupal APIs that can contain pages as needed.

I did not review the code.

Leaving at RTBC because this still needs framework manager review.

kristiaanvandeneynde’s picture

Suggested documentation is linked from the meta issue: #3381453: [policy/docs, no patch] Document the new access policy API

kristiaanvandeneynde’s picture

Thanks for the review @mmxr576!

My main takeaway is that we could improve the DX surrounding the exception and cache , but I would like to wait for core maintainers to sign off before I go through the trouble (as I'm on the DA's budget here).

Your other suggestions I would mark as personal taste and I tried to stick as close to what core already does as possible (hard-coding cache tag, not mentioning Drupal or Core in a base class, ...)

mxr576’s picture

The AccessPolicyCacheWrapper also raised another question, do we want to mark certain code as @internal? JSONAPI was and still is very strict about its programming APIs... Could other components do that, like this one?

So to reduce complexity in AccessPolicyProcessor, AccessPolicyCacheWrapper could be a viable answer, but we could be uncertain about marking that as a public API at day 1.

kristiaanvandeneynde’s picture

I'll try to go over your comments on Gitlab and see which ones I can reply to. I again want to iterate that I can't currently spend time on your suggestions regarding the exceptions and additional custom cache layer as I'm on a budget and I don't want to use it up by getting sidetracked without core committers voicing support for these topics.

Having said that, marker interfaces are kind of an anti-pattern as they use empty contracts (interfaces) to project meaning. I'd rather not go down that route as we're better off writing dedicated exceptions for each scenario then; something I also wanted to avoid by simply having a more generic exception.

kristiaanvandeneynde’s picture

Issue tags: -Needs change record

I ended up reviewing all but one comment as, while I was looking at them, it became clear that we should not use a custom cache nor prefix our base classes with Drupal. The only thing that I left up for discussion is what we should do with the exceptions, but as I stated above I am not a fan of marker interfaces so I'll leave that one up for core maintainers to decide.

Also removing needs change record tag as we have a draft ready.

mxr576’s picture

Having said that, marker interfaces are kind of an anti-pattern as they use empty contracts (interfaces) to project meaning.

I had the exact same mindset years ago and tbh I am still bit on a fence because sometimes they prove to be useful... there are design patterns that makes time necessary for example, e.g. Command Bus.

Thanks for reviewing my comments. I definitely do not want to be the cause any over budget type of issues. The quality of this MR is already very high so IMO this can be also merged as is.

kristiaanvandeneynde’s picture

Yeah and thanks for the reviews, I really appreciate it.

larowlan’s picture

Left a review, great work @kristiaanvandeneynde

kristiaanvandeneynde’s picture

Re #29, thanks @larowlan. I've tried to solve as many things as possible and added some explanations left and right. I've not marked things as resolved when I felt you might still have feedback on my answers.

kristiaanvandeneynde’s picture

@larowlan I had to change your code from:
return array_reduce($this->items, fn($carry, $scope_items) => [...$carry, ...$scope_items], []);

to:
return array_reduce($this->items, fn($carry, $scope_items) => [...$carry, ...array_values($scope_items)], []);

As the original would keep keys and keys can potentially be strings here, leading to overriding of elements rather than adding them all to one big array. The latter takes care of that problem by calling array_values() first.

Also, not entirely sold on the readability of the above versus:

    $item_sets = [];
    foreach ($this->items as $scope_items) {
      $item_sets[] = array_values($scope_items);
    }
    return array_merge(...$item_sets);
kingdutch’s picture

I have no knowledge of this issue but stumbled upon the code snippet in #31 and have previously played with similar code. It's not just readability. The closure and overhead of multiple copy/spread is also an order of magnitude slower and has different memory characteristics for large arrays.

Using a sufficiently large items array this can be shown with a small sandbox. I tried upping the iterations to 10.000 but the 3v4l sandbox rand out of memory for the array_reduce example.

$iterations = 1000;

$start = microtime(true);

for ($i=0 ; $i<$iterations ; $i++) {
    $reduced = array_reduce($items, fn($carry, $scope_items) => [...$carry, ...array_values($scope_items)], []);
}

$split = microtime(true);

for ($i=0 ; $i<$iterations ; $i++) {
  $item_sets = [];
  foreach ($items as $scope_items) {
    $item_sets[] = array_values($scope_items);
  }
  $looped = array_merge(...$item_sets);
}

$end = microtime(TRUE);

echo "Reduce: " . ($split-$start)/$iterations . " sec\n";
echo "Merge: " . ($end-$split)/$iterations . " sec\n";
Reduce: 0.0002404670715332 sec # That's 2E-4
Merge: 1.0043859481812E-5 sec   # vs 1E-5
kristiaanvandeneynde’s picture

Thanks for chiming in! I'll change it to the foreach then.

kristiaanvandeneynde’s picture

Important topic brought up in the sibling issue: #3376846-21: Implement the new access policy API

Summarized quote:

However, this means we allow unsaved user's permissions to be calculated and the question is whether that is something we should allow. ... Should we update AccessPolicyProcessor to throw an exception or something if $account->id() is NULL?

kristiaanvandeneynde’s picture

One more discussion I'd like to have before the names are set in stone. We currently have:

  • Drupal\Core\Session\(Refinable)CalculatedPermissions(Interface) -> Stores all permissions from calculatePermissions and alterPermissions
  • Drupal\Core\Session\CalculatedPermissionsItem(Interface) -> A single value object within the above, matching a specific scope-identifier pair
  • calculatePermissions and alterPermissions methods on access policies

If you don't mind having short names, we could also rename these to:

  • Drupal\Core\Session\(Refinable)Permissions(Interface)
  • Drupal\Core\Session\PermissionsItem(Interface)
  • buildPermissions and alterPermissions methods, as the docs talk about build phase and alter phase

The shorter class names are a bit less verbose, but at the same time less explicit than the original ones. So this comes down to preference.

larowlan’s picture

Personally I prefer the more verbose names but I don't feel that strongly either way

kristiaanvandeneynde’s picture

Yeah me too, it's just something I wanted to bring up proactively rather than have people ask about it after the fact :)

mxr576’s picture

Let me second what @Kingdutch wrote in #32

The closure and overhead of multiple copy/spread is also an order of magnitude slower and has different memory characteristics for large arrays.

I also bumped into this problem and it is actually a known and "won't fix" PHP bug tracked under https://github.com/php/php-src/issues/8283
but there is also a potential workaround mentioned in the thread: https://github.com/php/php-src/issues/8283#issuecomment-1084143580

catch’s picture

Couple of minor comments on the MR, I'm happy with the account switching logic now, very little code in the wild actually checks permissions for not-the-current-user and we've minimised any effects to only when that happens.

kristiaanvandeneynde’s picture

Thanks for the review! I dropped the constructor docs and moved the user ID check outside of the loop (good suggestion). I've also updated the inline docs regarding the account switcher to mention that we only switch when the accounts don't match.

In the implementation issue I was able to drop a few more constructor docs.

catch’s picture

There is one unresolved thread (that I can actually see, MR says there's 4) that is still open, but for me it's not blocking. I am pretty happy at this point but leaving the FM tag on a bit longer in case @larowlan wants to take a final look.

kristiaanvandeneynde’s picture

The only unresolved thread I can still see is @larowlan's proposed refactoring of early returning cache hits, which I implemented. I merely kept that one unresolved for Lee to flag as resolved as I left a comment there:

I adjusted the code, but we now have some minor code duplication: When we retrieve from the DB cache, we run the same code to set it to the static cache as we do at the very end. Namely: wrapping it in an immutable object and adding the same comment to explain why we do so.

Aside from that all threads show up as resolved on my end.

larowlan’s picture

Removing the tag as @catch and I have gone over this a few times

kristiaanvandeneynde’s picture

Assigned: kristiaanvandeneynde » Unassigned

Awesome! So is it safe to assume that this is now considered "ready"? Asking so I can give Alex Moreno an update regarding the sponsorship. Also unassigning myself then as there seems to be nothing left to do.

larowlan’s picture

I think it should be fine to update Alex

ndf’s picture

Hi kristiaanvandeneynde, RTBC it's finally happening!

What a major accomplishment for you personally and a great gift to the wider Drupal community.
I know you have been working on this for years (2016? #540008: Add a container parameter that can remove the special behavior of UID#1, #777578: Add an entity query access API and deprecate hook_query_ENTITY_TYPE_access_alter()) and you kept on going and find your way.

Thanks a lot!

larowlan’s picture

Issue summary: View changes
Status: Reviewed & tested by the community » Needs work

Gave this one final pass review. Left some minor questions/nits. Fine to self RTBC after addressing/replying
Thanks @kristiaanvandeneynde!

Adding issue credits.

kristiaanvandeneynde’s picture

Status: Needs work » Reviewed & tested by the community

Answered all comments. Hopefully the one about VariationCache is satisfactory. I felt it would be a bit overkill to explain how API A works in the context of API B's implementation of A.

kristiaanvandeneynde’s picture

Also thanks @ndf for your kind words!

kristiaanvandeneynde’s picture

Assigned: Unassigned » kristiaanvandeneynde
Status: Reviewed & tested by the community » Needs work

Just discussed the newly introduced factory with @catch and until we can come up with a better solution I'm going to go with a manually instantiated VariationCache object instead. This due to the fact that VariationCacheFactory only supports bins that can be instantiated by CacheFactory and MemoryCache should have no factory because it's not a true cache bin.

So we're not adding a factory for MemoryCache and I'll adjust the MR. Probably on Monday as I'm off tomorrow.

Setting to NW, will update, ping @catch for approval and then set back to RTBC.

kristiaanvandeneynde’s picture

Assigned: kristiaanvandeneynde » Unassigned
Status: Needs work » Needs review

Didn't want to keep people waiting 3 days. If @catch approves of the new approach then we can remove the CR I wrote here: https://www.drupal.org/node/3402104

Will also merge in the Implement issue to see if things break there or not. They shouldn't, but theory and practice don't always agree.

catch’s picture

Yes using the memory cache directly and tagging the service as a cache tag invalidator seems better to me.

My concern with tagging it as a cache bin is that cache bins serialize and unserialize objects, whereas MemoryCache specifically gives you back what you put in (as documented in MemoryCacheInterface). If we start mixing and matching the patterns, it could get (and is) very confusing. We have existing issues open like #3016690: Audit usages of cache.static and #2973286: Clean up MemoryCache and MemoryBackend naming issues to try to untangle this.

kristiaanvandeneynde’s picture

Tests in pipelines are now failing on seemingly random frontend stuff. I fixed the obvious services.yml file mistakes (that's what you get for trying to get something in before your long weekend), but I think everything is fine now. I pressed the "rerun" button on the failing FE tests.

Also, it's really hard to find what failed in the GitLab UI compared to the old test results we'd see on DO itself. Any tips?

kristiaanvandeneynde’s picture

Ah, all green again. Cya on Monday :D

kristiaanvandeneynde’s picture

Hmm, with the cache factory approach all tests in the follow-up "implement" issue go green. With this new approach without a factory, UserPermissionsTest::testUserPermissionChanges() goes red. So maybe cache tags aren't being cleared the way they should be with this approach?

kristiaanvandeneynde’s picture

Yeah just confirmed that tests go green with:

  cache.access_policy_memory:
    class: Drupal\Core\Cache\MemoryCache\MemoryCache
    tags:
      - { name: cache.bin }

But red with:

  cache.access_policy_memory:
    class: Drupal\Core\Cache\MemoryCache\MemoryCache
    tags:
      - { name: cache_tags_invalidator }

So there is a difference somewhere. Will have to figure that one out.

kristiaanvandeneynde’s picture

Found it!

UiHelperTrait::submitForm() and UiHelperTrait::drupalGet() both call $this->refreshVariables();, but if we look into that:

protected function refreshVariables() {
    // Clear the tag cache.
    \Drupal::service('cache_tags.invalidator')->resetChecksums();
    foreach (Cache::getBins() as $backend) {
      if (is_callable([$backend, 'reset'])) {
        $backend->reset();
      }
    }

    \Drupal::service('config.factory')->reset();
    \Drupal::service('state')->resetCache();
  }

It's clearly only resetting cache.bin tagged services, which is why we are seeing this testfail. If I manually add in this line:

\Drupal::service('cache.access_policy_memory')->reset();

Then the tests go green again.

So it's purely a test fail because refreshVariables() doesn't pick up on cache tag invalidators that also support a reset() method.

kristiaanvandeneynde’s picture

Status: Needs review » Reviewed & tested by the community

With the above confirmed and @catch's earlier approval we can set this issue back to RTBC and we'll have to work around what I found here in the Implement issue.

  • larowlan committed 77638be3 on 11.x
    Issue #3376843 by kristiaanvandeneynde, larowlan, catch, mxr576,...
larowlan’s picture

Committed and pushed to 11.x!

Published change record.

Congratulations @kristiaanvandeneynde on a huge effort to get a major new API into core (in record time no less)

larowlan’s picture

To avoid confusion in the future, I deleted the redundant CR for the memory cache factory

smustgrave’s picture

Congrats!

larowlan’s picture

Status: Reviewed & tested by the community » Fixed

Added the docs page to the handbook

larowlan’s picture

kristiaanvandeneynde’s picture

Awesome!

Thanks to everyone who helped review and move this forward.

nevergone’s picture

Maybe 10.2.0?

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.