Part of #3371246: [Meta, Plan] Pitch-Burgh: Policy based access in core
Summary
We want to convert the Flexible Permissions module into core code. This will allow for any access policy to be translated into a set of permissions so that access checks can run. These permissions are calculated once during a build phase and then pulled through an alter phase. After both phases are complete, the end result is poured into an immutable value object and cached.
This value object can then be used by a permission checker (such as the one introduced here #3347873: Centralize permission checks in a service) to verify if an account has permission to a given access check.
A key aspect that may not seem useful to core is the use of scopes. By default, all access policies will return permissions for the "drupal" scope, but to make it so access modules in contrib don't have to copy all of this logic, the use of scopes will allow them to define permissions in other scopes, such as Group, Domain, Commerce Stores, etc. where the calculated permissions will only have effect within the scope of, say, one domain or group.
Overview of new API
New concept: Build and alter phase
The goal is to loop over all access policies in a build phase and come up with a set of permissions that are from then on immutable. These permissions respect cache contexts (and are thus added to the variation cache), so you could be getting a different set depending on the time of day, your user roles, etc. The reason the permissions have to be immutable is because we still cache everything by user.permissions and if your permissions could change during runtime, that would quickly become a security nightmare.
Right before we turn the built permissions into this immutable object, however, we allow all policies to have a final alter pass of the fully built permissions. This adds some extra flexibility for people who want to alter other modules' (or core's) access policy behavior from the outside. After the alter pass, the immutable object is built and cached.
New concept: Scopes and identifiers
Furthermore, these permissions are built for a given scope and identifier within said scope. For Drupal core, both of these simply default to AccessPolicyInterface::SCOPE_DRUPAL and seemingly do nothing, but it's contrib where the addition of this concept will truly shine. You see, for Group, Domain, or similar we don't necessarily care about what permissions you have across the entire website, but rather what you can do within a subset, e.g. a single domain.
The main question then becomes "To what do these permissions apply?" Because Group is a bit complex to explain here (it has 3 scopes), let's use Domain as an example: If you want to discern who can make changes to content on an individual domain (whether active or not), then you would hand out those permissions in SCOPE_DOMAIN, where the identifier is the machine name of the domain. Doing this would for example allow the "Belgian team" to change the content of the ".be" website, but not the ".nl".
AccessPolicyProcessor and VariationCache
This is where the AccessPolicyProcessor (APP) and VariationCache (part of core) come into play. The APP is a service collector that looks for services which are tagged as access_policy. It then asks all of these policies what they initially vary by (i.e. cache contexts) and evaluates what they end up varying by at the end of the permission processing. This concept of "initial cache contexts" and "final cache contexts" is what powers cache redirects and is required to store something in the variation cache.
Before any of this runs, however, it first asks each individual access policy whether it applies to the scope and does not do anything with those that don't. So any Group or Domain specific access policies will not interfere with your sitewide Drupal permissions or vice versa.
(Refinable)CalculatedPermissions and CalculatedPermissionsItem
These value objects simply represent all of what was described above. The CalculatedPermissions object holds entries for an infinite amount of identifiers across an infinite amount of scopes. Keep in mind: In case of Drupal core it would be one scope and one identifier.
At each scope-identifier address sits a CalculatedPermissionsItem that indicates what permissions you should get for a given identifier within a scope and whether or not you have admin rights (i.e. all permissions) there. If you try to add multiple CPI to the same address, they get merged or overwritten depending on a parameter in the RefinableCalculatedPermissions::addItem() method.
Which leads to the next point: The difference between RefinableCalculatedPermissions and CalculatedPermissions. Simply put, the former allows you to make changes and is passed around during the build phase, the latter is immutable and passed around to those services who request what your fully built permissions are.
Permission checkers
This is not part of the new API but I'll explain it here rather than the implementation issue. How do you use these built permissions? Well, you use a central permission checker such as the one introduced in #3347873: Centralize permission checks in a service and call the AccessPolicyProcessor. Then you ask for the permission item at the scope-identifier (SCOPE_DRUPAL/SCOPE_DRUPAL for core) address you seek and simply run hasPermission() on it.
Here's what that would look like (taken from the implementation issue):
/**
* {@inheritdoc}
*/
public function hasPermission(string $permission, AccountInterface $account): bool {
$item = $this->processor->processAccessPolicies($account)->getItem();
return $item && $item->hasPermission($permission);
}
Sane defaults
See anything missing in the example above? We didn't have to specify AccessPolicyInterface::SCOPE_DRUPAL anywhere because the system defaults to the drupal scope and identifier. Contrib scopes would have to specify these parameters in the processAccessPolicies() and getItem() call.
To make life easier on people wanting to add an access policy to core from within a contrib module (such as office hours), CalculatedPermissionsItem also defaults to adding permissions to the SCOPE_DRUPAL/SCOPE_DRUPAL address. This should make access policies aimed at core easier on the eyes.
Here's what that would look like (taken from the implementation issue):
foreach ($user_roles as $user_role) {
$calculated_permissions
->addItem(new CalculatedPermissionsItem($user_role->getPermissions(), $user_role->isAdmin()))
->addCacheableDependency($user_role);
}
Issue fork drupal-3376843
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #2
kristiaanvandeneyndeComment #4
kristiaanvandeneyndeSome things worth mentioning:
Comment #5
kristiaanvandeneyndeLast test is choking on
$cache_static->set(Argument::cetera())->willReturn(NULL);as the set method has a void return type. There is a workaround like this:$cache_static->set(Argument::cetera())->will(function(){});as per https://github.com/phpspec/prophecy/issues/280.However, newer versions of prophecy shouldn't have this issue as it got fixed here: https://github.com/phpspec/prophecy/pull/398
Added a commit that removes any mention of willReturn and can add in the will workaround if they go red. Hopefully, we are on a recent enough prophecy version and do not need the workaround.
Comment #6
partdigital commented@kristiaanvandeneynde
Any thoughts on an API like this being applicable with access policy module? https://www.drupal.org/project/access_policy
This provides a powerful UI for allowing site builders to define their own access policies. It currently works on Drupal 9 and 10 but I'd be open to releasing version 2.0 to use this API if you feel it's a good fit.
Comment #7
kristiaanvandeneyndeTwo more things I want to try:
Comment #8
kristiaanvandeneyndeRe #6 I've looked into your module and the whole idea of building access policies using a UI is incompatible with this API-first approach. The goal here is to build an API that both core and contrib can use and then allow contrib to build any UI on top of that.
Your module comes with plugins that make assumptions about the types of access policies, limiting what can be done with it unless you introduce more plugins. This work comes with tagged services that translate an access policy into a set of permissions that can then be handed out on a particular level: core, Group, Domain, Commerce Stores, ...
Secondly, evaluating access policies during runtime is unsafe because we'd still be caching things with user.permissions even though your permissions may vary based on what access policies apply during your current and next request. The API presented here prevents that by introducing a one-time build phase and then caching your processed permissions using cache contexts. This way, it's still safe to use the user.permissions cache context because we know that your access does not change between requests unless you no longer match your policies' criteria (e.g. you lose a user role).
Comment #9
partdigital commentedre #8
Just to clarify, I’m not suggesting moving the access policy module into core. I’m wondering if I could use your new API as part of the contrib module?
In regards to your other comments, did you get a chance to try out the access policy module? It can be difficult to interpret how it works just by looking at the code. For example, you can make as many access policies as you like. The Access policy type plugin merely defines which operations you want to control. The access policies themselves are config entities.
The access policy module uses a combination of permissions and access rules (plugins that return TRUE/FALSE) to control access. So any user.permission cache context still applies. If the user has permission then it will evaluate the access rules via hook_entity_access() which uses the user cache context. It sounds like, for this reason, it might not be viable to use your API in access policy?
However, I don’t want to hijack this issue, so I’d happy to open another issue or even hop on a brief call.
Thanks, and nice work! Looking forward to seeing this get in!
Comment #10
kristiaanvandeneyndeI will add documentation and an example on how to use this, so that should help in evaluating whether you could leverage it in your module.
Comment #11
lauriiiTagging for a framework manager review. I think it would be helpful for the review to have the API documented on a high level in the issue summary / draft CR.
Comment #12
catchYeah I'm missing an issue summary here or in the meta issue. Left a note about memory cache.
Comment #13
kristiaanvandeneyndeThanks for the reviews so far!
Updated the IS to document the new API a bit.
Comment #14
kristiaanvandeneyndeComment #15
kristiaanvandeneyndeAs the commit says: I made the scope and identifier 'drupal' the default across the board now. So core code should look cleaner. Should still go green and I consider this the final version, aside from obvious discussions regarding naming etc.
But functionally this is feature complete.
Comment #16
kristiaanvandeneyndeActually working on one minor change: Add the account as a parameter to the alter function. Will post an update with that functionality soon.
Comment #17
smustgrave commentedTried my best with this one. Read the issue summary a few times, kudos btw for all the detail, everything looks okay.
Seems it's been through some review as all the threads appear to be resolved.
Going to move this one to the committer bucket?
Not sure if the change record will be needed for this ticket or one of the related tickets?
Comment #18
lauriiiIssue summary has been updated since #12. We do need a change record eventually for the new API so adding that tag. We could use the issue summary as a starting point for that.
Comment #19
kristiaanvandeneyndeThanks so much for the review! Updated the IS a bit and will write a change record when I have more time.
Comment #20
kristiaanvandeneyndeAdded a CR. It mainly copies the IS from this issue, but I've changed the language and intro a bit so that we can also have a CR for the implementation issue that merely focuses on the changes to the permission checker and the 2 core access policies.
https://www.drupal.org/node/3385551
Comment #21
quietone commentedI'm triaging RTBC issues. I read the IS and the comments. I didn't find any unanswered questions.
The change record has been added but has not been reviewed. That still needs to be done. I added a remaining tasks section to the IS that include this. I did skim the change record and I think that as new API there should be a guide at Drupal APIs that can contain pages as needed.
I did not review the code.
Leaving at RTBC because this still needs framework manager review.
Comment #22
kristiaanvandeneyndeSuggested documentation is linked from the meta issue: #3381453: [policy/docs, no patch] Document the new access policy API
Comment #23
kristiaanvandeneyndeThanks for the review @mmxr576!
My main takeaway is that we could improve the DX surrounding the exception and cache , but I would like to wait for core maintainers to sign off before I go through the trouble (as I'm on the DA's budget here).
Your other suggestions I would mark as personal taste and I tried to stick as close to what core already does as possible (hard-coding cache tag, not mentioning Drupal or Core in a base class, ...)
Comment #24
mxr576The AccessPolicyCacheWrapper also raised another question, do we want to mark certain code as @internal? JSONAPI was and still is very strict about its programming APIs... Could other components do that, like this one?
So to reduce complexity in AccessPolicyProcessor, AccessPolicyCacheWrapper could be a viable answer, but we could be uncertain about marking that as a public API at day 1.
Comment #25
kristiaanvandeneyndeI'll try to go over your comments on Gitlab and see which ones I can reply to. I again want to iterate that I can't currently spend time on your suggestions regarding the exceptions and additional custom cache layer as I'm on a budget and I don't want to use it up by getting sidetracked without core committers voicing support for these topics.
Having said that, marker interfaces are kind of an anti-pattern as they use empty contracts (interfaces) to project meaning. I'd rather not go down that route as we're better off writing dedicated exceptions for each scenario then; something I also wanted to avoid by simply having a more generic exception.
Comment #26
kristiaanvandeneyndeI ended up reviewing all but one comment as, while I was looking at them, it became clear that we should not use a custom cache nor prefix our base classes with Drupal. The only thing that I left up for discussion is what we should do with the exceptions, but as I stated above I am not a fan of marker interfaces so I'll leave that one up for core maintainers to decide.
Also removing needs change record tag as we have a draft ready.
Comment #27
mxr576I had the exact same mindset years ago and tbh I am still bit on a fence because sometimes they prove to be useful... there are design patterns that makes time necessary for example, e.g. Command Bus.
Thanks for reviewing my comments. I definitely do not want to be the cause any over budget type of issues. The quality of this MR is already very high so IMO this can be also merged as is.
Comment #28
kristiaanvandeneyndeYeah and thanks for the reviews, I really appreciate it.
Comment #29
larowlanLeft a review, great work @kristiaanvandeneynde
Comment #30
kristiaanvandeneyndeRe #29, thanks @larowlan. I've tried to solve as many things as possible and added some explanations left and right. I've not marked things as resolved when I felt you might still have feedback on my answers.
Comment #31
kristiaanvandeneynde@larowlan I had to change your code from:
return array_reduce($this->items, fn($carry, $scope_items) => [...$carry, ...$scope_items], []);to:
return array_reduce($this->items, fn($carry, $scope_items) => [...$carry, ...array_values($scope_items)], []);As the original would keep keys and keys can potentially be strings here, leading to overriding of elements rather than adding them all to one big array. The latter takes care of that problem by calling array_values() first.
Also, not entirely sold on the readability of the above versus:
Comment #32
kingdutchI have no knowledge of this issue but stumbled upon the code snippet in #31 and have previously played with similar code. It's not just readability. The closure and overhead of multiple copy/spread is also an order of magnitude slower and has different memory characteristics for large arrays.
Using a sufficiently large items array this can be shown with a small sandbox. I tried upping the iterations to 10.000 but the 3v4l sandbox rand out of memory for the array_reduce example.
Comment #33
kristiaanvandeneyndeThanks for chiming in! I'll change it to the foreach then.
Comment #34
kristiaanvandeneyndeImportant topic brought up in the sibling issue: #3376846-21: Implement the new access policy API
Summarized quote:
Comment #35
kristiaanvandeneyndeOne more discussion I'd like to have before the names are set in stone. We currently have:
If you don't mind having short names, we could also rename these to:
The shorter class names are a bit less verbose, but at the same time less explicit than the original ones. So this comes down to preference.
Comment #36
larowlanPersonally I prefer the more verbose names but I don't feel that strongly either way
Comment #37
kristiaanvandeneyndeYeah me too, it's just something I wanted to bring up proactively rather than have people ask about it after the fact :)
Comment #38
mxr576Let me second what @Kingdutch wrote in #32
I also bumped into this problem and it is actually a known and "won't fix" PHP bug tracked under https://github.com/php/php-src/issues/8283
but there is also a potential workaround mentioned in the thread: https://github.com/php/php-src/issues/8283#issuecomment-1084143580
Comment #39
catchCouple of minor comments on the MR, I'm happy with the account switching logic now, very little code in the wild actually checks permissions for not-the-current-user and we've minimised any effects to only when that happens.
Comment #40
kristiaanvandeneyndeThanks for the review! I dropped the constructor docs and moved the user ID check outside of the loop (good suggestion). I've also updated the inline docs regarding the account switcher to mention that we only switch when the accounts don't match.
In the implementation issue I was able to drop a few more constructor docs.
Comment #41
catchThere is one unresolved thread (that I can actually see, MR says there's 4) that is still open, but for me it's not blocking. I am pretty happy at this point but leaving the FM tag on a bit longer in case @larowlan wants to take a final look.
Comment #42
kristiaanvandeneyndeThe only unresolved thread I can still see is @larowlan's proposed refactoring of early returning cache hits, which I implemented. I merely kept that one unresolved for Lee to flag as resolved as I left a comment there:
Aside from that all threads show up as resolved on my end.
Comment #43
larowlanRemoving the tag as @catch and I have gone over this a few times
Comment #44
kristiaanvandeneyndeAwesome! So is it safe to assume that this is now considered "ready"? Asking so I can give Alex Moreno an update regarding the sponsorship. Also unassigning myself then as there seems to be nothing left to do.
Comment #45
larowlanI think it should be fine to update Alex
Comment #46
ndf commentedHi kristiaanvandeneynde, RTBC it's finally happening!
What a major accomplishment for you personally and a great gift to the wider Drupal community.
I know you have been working on this for years (2016? #540008: Add a container parameter that can remove the special behavior of UID#1, #777578: Add an entity query access API and deprecate hook_query_ENTITY_TYPE_access_alter()) and you kept on going and find your way.
Thanks a lot!
Comment #47
larowlanGave this one final pass review. Left some minor questions/nits. Fine to self RTBC after addressing/replying
Thanks @kristiaanvandeneynde!
Adding issue credits.
Comment #48
kristiaanvandeneyndeAnswered all comments. Hopefully the one about VariationCache is satisfactory. I felt it would be a bit overkill to explain how API A works in the context of API B's implementation of A.
Comment #49
kristiaanvandeneyndeAlso thanks @ndf for your kind words!
Comment #50
kristiaanvandeneyndeJust discussed the newly introduced factory with @catch and until we can come up with a better solution I'm going to go with a manually instantiated VariationCache object instead. This due to the fact that VariationCacheFactory only supports bins that can be instantiated by CacheFactory and MemoryCache should have no factory because it's not a true cache bin.
So we're not adding a factory for MemoryCache and I'll adjust the MR. Probably on Monday as I'm off tomorrow.
Setting to NW, will update, ping @catch for approval and then set back to RTBC.
Comment #51
kristiaanvandeneyndeDidn't want to keep people waiting 3 days. If @catch approves of the new approach then we can remove the CR I wrote here: https://www.drupal.org/node/3402104
Will also merge in the Implement issue to see if things break there or not. They shouldn't, but theory and practice don't always agree.
Comment #52
catchYes using the memory cache directly and tagging the service as a cache tag invalidator seems better to me.
My concern with tagging it as a cache bin is that cache bins serialize and unserialize objects, whereas MemoryCache specifically gives you back what you put in (as documented in MemoryCacheInterface). If we start mixing and matching the patterns, it could get (and is) very confusing. We have existing issues open like #3016690: Audit usages of cache.static and #2973286: Clean up MemoryCache and MemoryBackend naming issues to try to untangle this.
Comment #53
kristiaanvandeneyndeTests in pipelines are now failing on seemingly random frontend stuff. I fixed the obvious services.yml file mistakes (that's what you get for trying to get something in before your long weekend), but I think everything is fine now. I pressed the "rerun" button on the failing FE tests.
Also, it's really hard to find what failed in the GitLab UI compared to the old test results we'd see on DO itself. Any tips?
Comment #54
kristiaanvandeneyndeAh, all green again. Cya on Monday :D
Comment #55
kristiaanvandeneyndeHmm, with the cache factory approach all tests in the follow-up "implement" issue go green. With this new approach without a factory, UserPermissionsTest::testUserPermissionChanges() goes red. So maybe cache tags aren't being cleared the way they should be with this approach?
Comment #56
kristiaanvandeneyndeYeah just confirmed that tests go green with:
But red with:
So there is a difference somewhere. Will have to figure that one out.
Comment #57
kristiaanvandeneyndeFound it!
UiHelperTrait::submitForm() and UiHelperTrait::drupalGet() both call
$this->refreshVariables();, but if we look into that:It's clearly only resetting cache.bin tagged services, which is why we are seeing this testfail. If I manually add in this line:
Then the tests go green again.
So it's purely a test fail because refreshVariables() doesn't pick up on cache tag invalidators that also support a reset() method.
Comment #58
kristiaanvandeneyndeWith the above confirmed and @catch's earlier approval we can set this issue back to RTBC and we'll have to work around what I found here in the Implement issue.
Comment #60
larowlanCommitted and pushed to 11.x!
Published change record.
Congratulations @kristiaanvandeneynde on a huge effort to get a major new API into core (in record time no less)
Comment #61
larowlanTo avoid confusion in the future, I deleted the redundant CR for the memory cache factory
Comment #62
smustgrave commentedCongrats!
Comment #63
larowlanAdded the docs page to the handbook
Comment #64
larowlanComment #65
kristiaanvandeneyndeAwesome!
Thanks to everyone who helped review and move this forward.
Comment #66
nevergoneMaybe 10.2.0?