Problem/Motivation
The Cookie authentication provider currently executes SQL queries in order to construct an authenticated UserSession.
This is different from the BasicAuth authentication provider which simply loads the user from entity store.
It is quite common to actually have to load the user entity anyway, this could simplify some code (keep AccountInterface, but if authenticated user, then you can assume that you have a user entity).
Steps to reproduce
Proposed resolution
Simplify Cookie:authenticate() and load the user entity directly.
In order to prevent triggering the field translation subsystem prematurely, introduce ContentEntityBase::getFieldValue() which returns an untranslated value and use that for simple user entity properties (User::getLastAccessedTime(), User::isActive(), User::getTimezone(), User::getAccountName()).
Remaining tasks
- Review
- Commit
User interface changes
None.
API changes
API addition: ContentEntityBase::getFieldValue()
Data model changes
None.
Release notes snippet
| Comment | File | Size | Author |
|---|---|---|---|
| #170 | Screenshot_20260326_224708.png | 139.35 KB | berdir |
| #152 | 2345611-152.patch | 11.49 KB | _utsavsharma |
| #152 | interdiff_d11.txt | 11.49 KB | _utsavsharma |
| #145 | 2345611-145.patch | 11.47 KB | eugene.brit |
| #143 | interdiff_141_143.txt | 581 bytes | ameymudras |
Issue fork drupal-2345611
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #1
berdirFirst patch.
Comment #3
andypostDoes entity api requires user to be usable?
Seems only access checking "prepareUser()" cares about user in entity manager
Anyway EM dependency should be declared
should be injected properly
Comment #4
berdirSecond and better patch.
I have a feeling that all the session related methods on user/account are completely bogus and unused...
@andypost: Don't get the first part, but sure, it should be injected, will care about that when I know that it is actually working..
Comment #6
andypostI mean that chicken-egg about session and session account that uses entity api.
and yes, currentUser() is fragile at least with global $user
Comment #7
berdirSee #2248297: Ensure routes are rebuilt when install modules.
That and lots of translation issues.
Comment #11
berdirSupport TranslationWrapper in config casting (this is a bit weird, as we put translated values there, but we did that before to.
Also merge in the fixes from #2248297: Ensure routes are rebuilt when install modules
Comment #13
berdirFixing those last tests. TranslationWrapper *is* tricky...
Comment #14
catchIs this hunk intentional?
Overall this looks good. The entity system dependency in sessions isn't great, but it's just making an existing dependency explicit.
Comment #15
znerol commentedI'm very in favor of this. In the long run we should load and set the current user in the
AuthenticationManagerand that will move the entity manager dependency out of the session handler again.Still it looks like this reverses the decoupling which was introduced in #1825332: Introduce an AccountInterface to represent the current user.
Comment #16
znerol commentedFiled #2347799: Remove bugged session-related methods from AccountInterface
Comment #17
znerol commentedWhat about introducing a core
AccountProviderInterfaceand implement it in the user module?Comment #18
berdirI had a similar plan yes, firing an event or something after having loaded the session and then user can chime in and provide the user.
I have no idea yet how that would work with basic auth yet, though.
Also, I've planned to remove the methods here, because they are alraedy non-functional now and are obviously unused.
Comment #19
znerol commentedI think this is overkill (and probably out of scope here). I do not think that we should support more than one user-provider (at least not in core) and therefore firing an event is not appropriate here.
Instead the responsibility of identifying the user-id (or perhaps even the user account) associated with a given session should be moved to the cookie authentication provider. However this requires us to pass that information (i.e. the uid) form the
SessionHandler::read()to the authentication provider somehow. I planned to simply use the symfony session for that (#2229145: Register symfony session components in the DIC and inject the session service into the request object). This would have the effect that theuidwould be duplicated though, once it would be recorded in the uid-column and then also in the serialized session record. I do not consider that a problem. The uid-column on the session table simply is an additional index used to kill all sessions of a given user for example.Given the above, cookie provider would then work like basic auth instead of the other way around.
Great, in addition you should kill
getHostnameand movegetLastAccessedTimeto theUserentity. The former is unused and the latter is really only useful to order the list of users in the administrative interface. Also the code inSessionHandler::write()which updates the access-timestamp should be moved to a TERMINATE event handler in the user module.Comment #20
berdirSure, that works too for me.
We could just put the uid column value inside the session data so that it is available to the cookie provider?
I will pick this up when #2313135: setting page_cache_without_database in settings.php prevents the container from being dumped is in.
Comment #21
znerol commentedThat would be wrong, because at this point in time the
$_SESSIONis not yet present (and also the symfony session bags are not yet associated with it). It will only be populated later from the return value ofSessionHandler::read().Comment #22
berdirOk, that got in, so here is a re-roll + I also included #2347799: Remove bugged session-related methods from AccountInterface.
Comment #23
znerol commentedThis can be removed after #2313135: setting page_cache_without_database in settings.php prevents the container from being dumped
Not sure about that, but maybe related to #2248297: Ensure routes are rebuilt when install modules or #2230091: Route rebuilding is not guaranteed to finish in time for the next request
Removing
getLastAccessedTime()from theAccountInterfacewould make it possible to kill that entirely. In additiongetHostname()is unused and can be removed too.Comment #24
berdir1. 2. Yes, those changes were added here because I needed this. This issue is the reason I started working on all those routing issues :)
Comment #25
berdirRerolled and removed those methods.
Comment #26
berdir#2363099: Using translated field definition descriptions in entity schema results in different schema definitions, resulting in update.php changes would avoid all those translation wrapper thingies.
Comment #28
znerol commentedLooks like a leftover?
Dito.
Comment #29
berdirYes. Removed that. Also the module handler change seems to fix InstallUninstallTest, some weird caching thing going on. Again.
Comment #30
znerol commentedComment #34
berdirRe-roll.
Comment #36
berdirSuch fear. much scared.
Comment #37
berdirReroll, let's see if this change fixes those tests, content_translation tried accessing new services after being installed but we didn't replace the global container yet.
Comment #39
znerol commented#2363099: Using translated field definition descriptions in entity schema results in different schema definitions, resulting in update.php changes is in.
Comment #40
berdirCorrect, but that wasn't fixed in a way that helps us here. That part was postponed to #2418521: Translating field definition descriptions problematic due to early t(), hard to test.
The problem with moving forward without that is that adding a field to users that does not use TranslationWrapper() could suddenly completely break your site. I'm not sure that's a risk we can take.
Comment #41
berdirReroll.
Comment #43
znerol commentedShould be much easier now after #2228393: Decouple session from cookie based user authentication went in.
Comment #44
berdirYeah, the TranslationWrapper problem stil exists, though, but if someone wants to re-roll this, now would be the time :)
Comment #45
almaudoh commented@Berdir: could you explain the TranslationWrapper problem, it is not reflected in the IS and not very clear from the comments.
I also re-opened #2347799: Remove bugged session-related methods from AccountInterface so we can finish that piece quickly, even though it had been incorporated into this patch.
Comment #46
znerol commentedLet's see whether I'm getting that right: The
AuthenticationSubscriberneeds to run beforeLanguageRequestSubscriberfor some reason. At the time theauthenticate()method is executed, the language subsystem is not yet completely initialized.In the
BasicAuthprovider we currently load the whole user entity. But in theCookieprovider we execute the queries manually. The question now is whether the former is broken in multilingual sites or whether it would be possible to convert the latter to entities now that we do not authenticated at random points in the request response cycle?Comment #47
berdirFresh start, with the discussed getRoles() implementation that avoids fields definitions. Manual testing worked well.
Comment #51
berdirFixed the unit tests, will need more.
Test took forever but I really can't explain why that would be related to the patch. Let's see if it's still that slow.
Comment #53
berdirAgain the test got terminated after 3h. I guess one of them loops forever, but how... will need to diff the test list I guess...
Comment #54
znerol commentedDrupal\user\Tests\UserCancelTestseems to be missing.Comment #55
berdirRight, that was looping in a weird way because I forgot to check the user status. I added that now and used the entity key to ensure that we can load and check that without having to load the field definitions. And also explicitly unset the uid in the session during the user cancel process.
Comment #56
andypostGreat! Only user entity changes needs clarification - why status key value used to return field value?
result of
\Drupal\Core\Entity\ContentEntityBase::getEntityKey()described aselse superflowous
this looks strange, getEntityKey() should return a value of key (string "status")
Comment #57
berdir1. the else is to remove the session key in case the user is no longer active, I think that's useful cleanup so that we don't have to try and load a blocked user on every request.
2. It posts the *value* of the key, which is 0/1. getEntityKey() is an optimization for entity keys that allows to access their values without having to load field definitions and field values without having to go through field objects.
Comment #58
andypostI mean that
Comment #59
znerol commentedI'm trying to come up with a sensible performance test. Trying the following alternative front controller
test.php:ab -c1 -CSESSxxxxx=yyyyyy -n 1000 -k http://localhost:3000/test.php > /tmp/ab-head-test.txtab -c1 -CSESSxxxxx=yyyyyy -n 1000 -k http://localhost:3000/test.php > /tmp/ab-patch-test.txtI do not think that test is fair, in
headthe whole entity system is likely not instantiated.Comment #60
znerol commentedComment #61
znerol commentedAnother test with the path
admin/get-node-onewired up to the following controller (see attached module):HEAD:
patch:
Comment #62
berdirYeah...
My results, your test.php (1000 requests) and frontpage with one node (
Did some profiling to figure out what makes it slower. Turns out, we do have a few calls to methods that do have to load fields. I noticed getTimezone(), getLastAccessedTime() and getUsername().
Comment #63
znerol commentedHacked my way around authentication in order to get xhprof-kit working for authenticated users (see attached diff). There is something weird going on: Only 50% of the whooping 4ms increase is contributed by
Cookie::authenticate(), the rest comes fromAccountProxy::setAccount(). The heavy path there isAccountProxy::setAccount() - drupal_get_user_timezone() - AccountProxy::getTimeZone() - User::getTimeZone() - ContentEntityBase::get() - ContentEntityBase::getTranslatedField() - ...See also attached logs and screenshots. Not really sure whether there is room for optimization down that rabbit hole.
Comment #64
catchFor performance test there's two things to look at:
1. Where the current user is not loaded at all on the request with HEAD.
2. Where the current user is loaded in the current request already.
A controller that returns a minimal response, then one that loads a user and does the same would work for testing.
I'd expect #1 to be slower - since we have a user load where there was previously none (although with entity caching this might not be bad either). But on the other hand #2 should be faster, because we only load the user information once rather than doing the direct query also.
Comment #65
znerol commentedRegrettably entity caching does not help here. The profiling revealed that most of the regression is due to accessing data through fields on the user entity (username, last accessed time, timezone). @Berdir correct me if I'm wrong.
Comment #66
znerol commentedNeeds a reroll after #2347799: Remove bugged session-related methods from AccountInterface.
I've applied the idea of #47 (sidestep field definitions) to a couple of other methods in the user entity (
getPassword(), getCreatedTime(), getLastAccessedTime(), getLastLoginTime(), getTimeZone(), getUsername()), and profiled again. The regression due to the expensive field definition stuff is gone. I.e. the expensiveAccountProxy::setAccount()visible in #63 is gone (see theon-request-authenticatescreenshots.From the xhprof results it may look like
EntityManager::getStorage()is still a regression. However, that is not the case at all. The method is used a couple of times throughout the request/response cycle and it just happens to be called inCookie::authenticate()for the first time. The profiler output forEntityManager::getStorage()shows the same inclusive wall-time even though the method is called 28 times (patch) instead of 27 (head).There is still a minor regression of about 0.5ms(?) due to
EntityStorageBase::loadwhich seems to be a little bit slower than manual queries. In my opinion that is acceptable.Given those results, is there anything in the entity subsystem which we can leverage to access selected properties directly if necessary? I think there is a category of data where translation / typed data should be optional (e.g. timestamps, timezones, boolean flags, password hash) and direct access should be possible.
Comment #67
znerol commentedComment #68
andypostRe-roll and removal of
protected getUserFromSessionto inline code+1
Comment #69
dawehnerIt is though a little bit sad that, as it is in HEAD, we basically have a dependency to user module here, but yeah this is not a new thing.
Should we point to the code using it, I guess its hasPermission?
Comment #70
andypostJust re-roll
#69,2 makes sense, looks we can revert that part because it's no more a case since we use AnonymousUser for that, and we should never get a User object half-initialized
Comment #72
znerol commented#70 is not a good idea. @Berdir: Could you please explain the reason for #47. I only recall that the hunk in
getRoles()is necessary because the field definitions somehow depend on the language subsystem being initialized (and that one depends on authentication already being performed). It would be fantastic if you also could give some direction on what to do about the slowness ofgetTimeZone()and related methods.Comment #73
andypostFailed test because
UserTestinherited fromUserSessionTestProbably we need decouple them
Comment #74
andypostlooks we need to deprecate here
UserSessionor make core depends on user entity as #69.1 saidand how to update #2477213: Move AccountInterface out of Session namespace now
Comment #75
mgiffordUnassigning stale issue. Hopefully someone else will pursue this.
Comment #77
berdirStill applies with git apply -3.
The reason for #47 is the same as the answer to make getTimezone() faster. Skipping the slow field item list + field item creation.
#2580551: Optimize getCommentedEntity() is an attempt at making that more generic, but I'm not sure if we can or want to support multiple fields.
Comment #79
berdirLets try this instead.
Comment #80
berdirThat doesn't work, here's a combined patch with getCommentedEntity() and using getFieldValue().
Comment #83
berdirFixes and cleanup.
Tested this, getFieldDefinitions() for user is no longer called with this on normal pages. Only place it is called is in the comment form. And when viewing the user profile, but only the first time. And both cases are already the case right now.
Comment #85
dawehnerShould we document atht this is optimized for performance reasons?
Just curious, could you specify a third
$deltaparameter with it?Comment #86
berdir1. Was thinking about that, not sure.
2. I thought about that too, actually had it at one point. it's a bit strange with the non-delta specific values, but I think we can simply only check those for delta 0.
Comment #88
berdirThis should fix that test fail.
Comment #90
jibranI think we should add delta as an optional argument.
Comment #92
richgerdesI came across this issue while working working on a project. The method used to setup the user session make it harder to work with the current user since its not a real user and as said previously you end up loading the user anyway. If this could get merged into core, I am sure it would make many people pleased.
There are a number of modules that rely on \Drupal::currentUser() to get access information but without having a real user this can have unexpected functionality as it is implemented now.
I have rewritten the patch from #88 against the latest version of 8.3. I also added delta as an optional argument as suggest in #90.
Comment #93
berdirThanks for the reroll, don't forget to set issues to needs review when uploading a patch.
Keep in mind that code still shouldn't rely on the current user being a user entity, but loading the entity now is at least a load from static cache.
Comment #95
berdirThis removes the comment specific parts from this, lets see how many test fails are left.
Comment #96
jibranShouldn't this be other way around first check the is_array and then isset?
We should use delta here.
Comment #97
berdir1. Nope because the $delta key might not be set. This is just there to prevent matching it as string index as it's possible that levels could be a string, depending on how the entity is initialized.
2. yep.
Comment #98
richgerdesI added the delta from #96's 2.
Comment #99
richgerdesUploaded the wrong version of the patch the first time.
Comment #103
andypostRelated to #63 profiling
Comment #105
erik frèrejeanI've been running this patch in production for the past six months without problem, and seeing that all raised issues have been resolved I think this is ready to be RTBC'ed.
Comment #106
hchonovThe method should be defined in the interface
FieldableEntityInterface.I think, that the
elseifneeds the sameissetcheck like theif.What is the point of the second condition? If the first one is met, then the second one will always be true.
Same question as the previous one.
Comment #107
richgerdesThanks for the review. 2, 3, and 4 all sound like good suggestions. They have been applied in the attached patch.
For 1 however, I think there is a concern here with moving the function definition. Ultimately, moving it will create a potential backwards incompatibility. If there is a EntityType which implements the
FieldableEntityInterfacewithout extending theContentEntityBasethen they won't receive the new function, and code will break. While I hope this doesn't occur, its still possible and as a result has a negative effect since it may prevent sites (and modules) from upgrading to the latest version.Has this been done with interfaces previously without causing havoc? If not, its probably best to make a new interface to hold this definition. Thoughts?
Comment #108
richgerdesComment #111
avpadernoComment #112
aleevasReverted patch from #112 up to 8.9
Comment #113
aleevasComment #117
janlaureys commentedUpdated the patch so it applies in 9.2 and up again.
Comment #118
andypostLast patch is broken for current core
Comment #119
janlaureys commentedYeah I was a little quick to upload the patch before verifying it on 9.3.
Updated it again and now verified it applies on 9.3 and 9.4 as well.
Edit: I'm well out of my depth here. Can see all tests failing already. Apologies :(.
Comment #120
janlaureys commentedFixed the service definition and class constructor to also include the messenger service.
Comment #121
janlaureys commentedOkay I think I'm getting there. Actually ran the commit-code-check script this time and turned out fine.
Comment #123
casey commentedComment #124
ravi.shankar commentedAdded reroll of patch #121 on Drupal 9.5.x.
Comment #125
avpadernoComment #126
ravi.shankar commentedPatch #124 didn't pass the test.
Comment #127
berdirThe patches since #119 are missing the getFieldValue() method that needs to be restored from #117.
Comment #128
ravi.shankar commentedRestored the changes of patch #117 as per comment #127.
Comment #129
avpadernoComment #130
anchal_gupta commentedComment #131
berdirmake it multiple-value fields then as it doesn't like multi
Comment #132
ravi.shankar commentedAddressed typo of patch #128.
Comment #133
ravi.shankar commentedSorry didn't notice, patch #130.
Comment #136
andypostComment #137
eugene.britComment #138
medha kumariRerolled the patch #137 in Drupal 10.1.x.
Comment #139
avpadernoComment #140
mrinalini9 commentedRerolled patch #138, please review it.
Comment #141
mrinalini9 commentedFixing custom commands failure issues in patch #140, please review it.
Comment #142
avpadernoComment #143
ameymudras commentedComment #145
eugene.britRe-roll #137 for D10.0.x
Comment #147
znerol commentedNeeds a reroll for 11.x. I'll stick around for reviews here. Find me in the sky lounge at DDD vienna if there are any questions.
Also desperately needs an issue summary update.
Comment #148
avpadernoComment #149
znerol commentedComment #150
znerol commentedIssue summary: done.
Comment #151
znerol commentedThis is not true. There is an optional $delta argument.
Let's add type hints here.
Comment #152
_utsavsharma commentedRerolled for 11.x as mentioned in #147.
Tried to address pointer 2 in #151.
Comment #153
vsujeetkumar commentedIn #151.1, True $delta argument is optional, As per my understanding, Either we should update the comment like
or we can remove this line from the comment.
Comment #154
catchThese are two of very few database queries executed on an authenticated page request when dynamic page cache is a cache hit, if we load the user it will usually come from the cache.
See for example https://grafana.prod.cluster.tag1.io/explore?panes=%7B%22taO%22:%7B%22datasource%22:%22tempo%22,%22queries%22:%5B%7B%22query%22:%2269f4dbd9a3acdf0e49e11acc90cbae81%22,%22queryType%22:%22traceql%22,%22refId%22:%22A%22,%22datasource%22:%7B%22type%22:%22tempo%22,%22uid%22:%22tempo%22%7D,%22limit%22:20%7D%5D,%22range%22:%7B%22from%22:%22now-6h%22,%22to%22:%22now%22%7D%7D%7D&schemaVersion=1&orgId=1 this trace from the Umami authenticated performance test. Would only be a small optimization from that perspective but worth doing I think, so tagging with performance for that.
Comment #155
znerol commentedThis needs a reroll from #152 , and it should be converted to a merge request.
Comment #158
prem suthar commentedAs per the #155 re-rolled the patch #152 and created the Mr . also Addressed the #151 Points
Please review.
Comment #159
znerol commentedThanks @prem suthar, the MR looks right. Regrettably, there are two test failures in @core/modules/user/tests/src/Unit/Plugin/Core/Entity/UserTest.php@. I can reproduce them locally. The full output:
Not sure, maybe it is necessary to modify
User::getRoles()and use$this->getFieldValue('roles')instead of$this->get('roles'). Possibly @berdir could help here.Comment #160
berdirThe performance aspect of this is a tradeoff, that's also why this issue has been stuck so long.
Yes, we save two queries, but the other side of the coin is that to access the information we need from (status, roles) requires that we load the field definitions and create the field objects for it, which is much slower than the UserSession object.
(FWIW, core has been doing that anyway for years due to #3158130: Many calls to ContextRepository::getAvailableContexts() due to entity upcasting but that's fixed now).
Especially when profiling, that cost is clearly visible and it's hard to compare that with the cost of database queries. On high load sites, possibly with multiple servers and redis/memcache, that cost is easier to balance against extra database queries, but that's not the default setup.
I tried to address that with the getFieldValue() method. But that's complex and I'm not convinced it's entirely reliable and shouldn't be mixed up with this issue iMHO. So either we accept that downside of the change or we postpone it on an issue to improve that performance cost. Which might or might not ever happen.
Comment #161
andypostLooking at CI jobs I see no viable perf regression, so except fixing the last test it looks ready to go (good to get it in in early days on 11.2)
Comment #162
berdirWell yes, because there's a complex workaround in place to counter the performance problem. But even without that, it is very unlikely to have an impact on CI execution time. But it's been tested and shown a long time ago in comments around #60. I doubt things are fundamentally different now.
Comment #163
richgerdesI was able to get the test past the initial failure, however, since we now use an entity and not a session object, there are other dependencies, including the field definitions, which aren't loaded in the unit test scope. We could mock the full User Entity, but this seems overly complicated and costly. Alternatively, we could move the test to a kernel or functional test so we get access to the container and entity schema. Not sure what the core best practice recommendations are here though. I would think we would copy the required test scenarios to the new test and move it to the Kernal Space, if that's enough to bootstrap what we need.
Comment #165
catchJust ran into a memory leak with ContentEntityBase::get() so I've opened a new issue to add ::getFieldValue() independently of here. If we can decide that's actually worth adding with the now two use cases identified for it, it would then unblock this issue.
Comment #166
catchAfter look at that issue, I think it's theoretically possible to fix the memory leak in a different way, but that still leaves the use case here, so I've opened #3574012: Add a getFieldValue() method to bypass typed data overhead for specific use cases to add the method from here. Leaving the PP-1 on, just points somewhere else now.
@richgerdes moving the test to a kernel test seems reasonable.
Comment #167
berdirComment #168
berdirRebased after #3574012: Add a getFieldValue() method to bypass typed data overhead for specific use cases landed.
Testing with xdebug on a dynamic page cache hit, I also had to use getFieldValue() in getRoles(), but with that, I confirmed that we're not loading the field definitions anymore.
Comment #169
amateescu commentedLeft a note on the MR :)
Comment #170
berdirDoing some profiling, this is still a significant regression on a dynamic page cache hit, around 10% or so.
The reason is entity storage, which loads the active entity type and field storage definitions in the constructor and then also initializes stuff. None of that should happen there, although we do need some information from the entity type, but it doesn't need to be the active definition. We just need to know if the entity type supports static/persistent caching to return cached results.
$this->entityType is really awkward to handle if we'd try to do BC for this. because it's an existing property from the parent that we're updating, so we can't really use __get() or so on this.
Comment #171
catchWe might be able to handle this with property hooks, maybe.