Problem/Motivation

I've spent some time playing with D8's entity query. One of the really nice things you can do is add conditions on fields on referenced entities (via the entityreference) field.

As an example: User has a field additional_info which is a Node with a field date_of_birth. With entity queries, you can then:

$query = \Drupal::entityQuery('user');

// Filter by role.
$query->condition('roles', 'administrator');

// Filter by date of birth
$query->condition('additional_info.entity.date_of_birth', '2000-01-01');

This would return any administrators born on the 1st Jan 2000.

Obviously, dates of birth of users are much better stored on profiles. However, as there is no field on the user linking to the profile, it is impossible to do queries like this without using direct queries and building all the joins yourself.

In addition to this, a field would allow you to access the information via a user object:

$user = \Drupal\user\Entity\User::load(1);
$date_of_birth = $user->additional_info->entity->date_of_birth->value;

Both of these would be very advantageous, especially in the context of CRM where there is a lot of data about users and you need to be able to access and query it easily.

Proposed resolution

There a couple of routes we could solve this:

A. Switch to storing the link between users and profiles in an entity reference field on the user
This could result in lots of profiles and an inability to tell the difference between active and inactive
B. Store a cache of the active profiles in an entity reference field
This would provide an easy way to access active profiles, and we can complete ignore everything else (as we probably should

I would be in favour of B, though I mention A as I know there is a more general discussion around the active/inactive.

Remaining tasks

Decide if we're happy with this proposal and, if so, which route we want to go.

API changes

You would be able to access active profile information via the user object and query it as part of a user entity query.

Data model changes

Additionally storing a cache of the existing information in an entityreference field on the user.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

andrewbelcher created an issue. See original summary.

mglaman’s picture

Why not use the methods available the the Profile entity's storage handler?

/** @var \Drupal\profile\Entity\ProfileInterface[] $profiles */
      $profiles = $storage->loadMultipleByUser($this->getOwner(), $this->getType());

This allows me to load multiple based on the owner of the user by type. Or just run the query on the profile entity storage, with the user being one field condition and value another?

Edit: Example query

$query = \Drupal::entityQuery('profile')
  ->condition('uid', 1)
  ->condition('default', 1)
  ->condition('date_of_birth', '2000-0101');
$results = $query->execute();
andrewbelcher’s picture

So the massive win of storing a cache in an entity_reference field on the user is that you can use it as part of a general query of users and across multiple profiles. Say you have 2 profiles, one of contact details and one of membership and you want to be able to find all users of a certain age who have an inactive membership but logged into the site in the last month. The entity_reference cache of the 'main' profile(s) allows you to do that in a single query.

This stuff is a massive win for CRM stuff where you have a large number of field, potentially on multiple profiles and need to be able to get information about them. I know you can build reports in views, but some times you need to be able to access this information for programmatic purposes rather than just reporting, at which point entity query comes into it's own.

mglaman’s picture

Ok, now the use case makes sense. As a way to basically have a condition on fields across two different profile types. How would you control access to the field on the user, or proposed solution? I'd assume this is something that should not be editable in the UI as it is automatically populated as a backreference.

Also, should this be something the module itself provides, or implementations that require it - such as the CRM contributed module.

andrewbelcher’s picture

View access would apply the same conditions as regular view profile access, though via field access. Editing wouldn't be possible.

Very happy if you'd rather it went into CRM. Thought it might be a useful addition in general, though obviously CRM is the main winner.

mglaman’s picture

Status: Active » Postponed

So I find the idea great. However, I feel it brings about a lot of questions:

  • How do we name the field?
  • What about a field for each profile type?
  • What if the profile type supports multiple?
  • Do we allow it to be displayed on user form? Like admins to manually adjust?
  • What about active and inactive? Inactive represents an entity in "trash", preserved for other references

I think it'd be great to see this incubated in a CRM module, with your use case. Something that implements the presave hook to add the profile reference to the user, adds the field to user entity, etc.

I'm moving this to Postponed because I would love to come back to this later once the implementation has been flushed out. I'd even like to link to the CRM module's implementation as an example. I'm always in IRC, feel free to ping me if you would like!

andrewbelcher’s picture

Attached is a patch that does it how I would suggest it is done in profile. Bits that are good about it being in profile as opposed to elsewhere:

  1. We can ditch profile_user_view() and profile_entity_extra_field_info() as these are handled by the entity_reference field. The ER field also gives more options (title display/render move/display mode etc)
  2. ProfileType create/delete operations can be dealt with in ProfileType rather than hooks (a bit cleaner)
  3. Responding to changes in profiles can be done in Profile::postSave()/Profile::postDelete() rather than hook_entity_insert()/hook_entity_update()/hook_entity_delete(), which is cleaner and less to implement (::postSave() is insert and update).
  4. The actual processing can live in Profile and therefore ProfileInterface and so be relied upon by other modules.

Having said that, I'm going to leave this here and convert it to live in Decoupled User Authentication (a CRM module) - #2658264: Store active profiles in ER base fields on the user. If in the future you decide that you do want this in Profile, could you post to that issue/create an issue on decoupled auth to remove the functionality.

If you've got the motivation - would still gratefully receive any review/feedback on what I've done/how I've done it!

andrewbelcher’s picture

Actually, in doing the port, we may not even need the hook_install() as the profile types get inserted...

andrewbelcher’s picture

FYI - have just committed this to decoupled auth - #2658264: Store active profiles in ER base fields on the user.

mglaman’s picture

andrewbelcher, I've been bouncing back and forth on this a lot lately. Has the query and simplified rendering been a big win?

andrewbelcher’s picture

The querying particularly has been very helpful! My use case is CRM, where we have lots of information about users that we need to be able to query on!

Also, having the fields provides easy access to the profiles from a user entity which makes programmatic changes much simpler than having to query profiles.

mglaman’s picture

Okay, I think I'm going to move forward and commit this. Does it make working with Token easier? RE: #2785585: Make profile fields avalaible as tokens

andrewbelcher’s picture

I haven't tried doing anything with token, but I wouldn't be surprised if it's easier on the basis of navigating user -> profiles is much easier.

Anonymous’s picture

Any update on this? I would love this being committed.

mglaman’s picture

Vic Luijkx, it's been on the back burner. I'm considering it to be a possible beta blocker. It seems like it'd simplify a lot of developer experience and possibly our own code.

mglaman’s picture

Status: Postponed » Needs work

Okay, I think we should move forward with this. It makes rendering profiles on the user much more customizable. Removes hook_extra_field_info which just stinks.

The only "bummer" is for multiple profiles there wouldn't be an easy way to just get the default when there are multiple profiles.

andrewbelcher’s picture

If we implement our own field type extending entity reference we can override some of the methods to give us that functionality, though we will have to check that doesn't mess with the query integration as its possible that's hardcoded to the field type!

I'm happy to have a look at that and pick it up, though I'm fairly busy right this minute!

mglaman’s picture

I'm okay with using the Profile storage to load the default. Most use cases probably won't allow for multiple profiles.

rlmumford’s picture

Status: Needs work » Needs review
FileSize
11.03 KB

Updated the patch so that it applies and added some test coverage making sure that the fields get created and removed with the ProfileTypes and making sure that the field points to the default item.

Status: Needs review » Needs work

The last submitted patch, 19: 2654980-19.patch, failed testing.

The last submitted patch, 19: 2654980-19.patch, failed testing.

Anonymous’s picture

Patch #19 gives the following error:

can't find file to patch at input line 279
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|diff --git a/tests/src/Functional/ProfileDefaultTest.php b/tests/src/Functional/ProfileDefaultTest.php
|index 414384e..03f649d 100644
|--- a/tests/src/Functional/ProfileDefaultTest.php
|+++ b/tests/src/Functional/ProfileDefaultTest.php
--------------------------

Anonymous’s picture

Sorry, my mistake. I did not realise I was using alpha5 instead of the dev version. I just tested patch #19 which works great on sites without the profile module already installed and configured. However, running update.php after updating the module causes the following error:

User error: Drupal\Core\Database\DatabaseExceptionWrapper thrown while calling __toString on a Drupal\Core\StringTranslation\TranslatableMarkup object in /path/to/site/core/lib/Drupal/Core/Database/Connection.php on line 671: SQLSTATE[42S02]: Base table or view not found: 1146 Table 'db_name.user__profile_example_profile' doesn't exist: SELECT t.* FROM {user__profile_example_profile} t WHERE (entity_id IN (:db_condition_placeholder_0)) AND (deleted = :db_condition_placeholder_1) AND (langcode IN (:db_condition_placeholder_2, :db_condition_placeholder_3, :db_condition_placeholder_4)) ORDER BY delta ASC; Array ( [:db_condition_placeholder_0] => 1 [:db_condition_placeholder_1] => 0 [:db_condition_placeholder_2] => nl [:db_condition_placeholder_3] => und [:db_condition_placeholder_4] => zxx ) in Drupal\Core\StringTranslation\TranslatableMarkup->__toString() (line 20 of core/lib/Drupal/Component/Utility/ToStringTrait.php).

heshanlk’s picture

Status: Needs work » Needs review
FileSize
10.76 KB

Fix the unit test issues.

Status: Needs review » Needs work

The last submitted patch, 24: 2654980-24.patch, failed testing.

The last submitted patch, 24: 2654980-24.patch, failed testing.

heshanlk’s picture

Status: Needs work » Needs review
FileSize
12.44 KB

Status: Needs review » Needs work

The last submitted patch, 27: 2654980-26.diff, failed testing.

The last submitted patch, 27: 2654980-26.diff, failed testing.

heshanlk’s picture

Status: Needs work » Needs review
FileSize
11.72 KB

Status: Needs review » Needs work

The last submitted patch, 30: 2654980-26.diff, failed testing.

The last submitted patch, 30: 2654980-26.diff, failed testing.

csedax90’s picture

Is there any news about implementation?

mglaman’s picture

sedax, the main hold up is how do we handle profile types which allow multiple references? Or in that case do we just not support the referenced field?

niknak’s picture

Any new patch is ready ?

csedax90’s picture

@mglaman I think the best thing is to have a multiple user reference, maybe recalled through the profile ID

Ex.
[node:author:profile:customer:field_first_name]

mglaman’s picture

bojanz had an idea: can we use a computed base field? This way we avoid the storage and can still manipulate via an entity query. Would simplify populating backreferences to profiles on user accounts.

mglaman’s picture

I just read https://www.drupal.org/docs/8/api/entity-api/dynamicvirtual-field-values....

My thought is:

  • Add a profiles computed field
  • The computed field extends (or acts like) EntityReferenceFieldItemList to load profiles.
  • We need to see how we can allow entity queries still with this field. Not sure if marking it computed will throw query error for the field not existing
mglaman’s picture

Here's an updated go using a computer field. It's much nicer than using extra fields and allows granular control. Still, needs some more work. Also, need to experiment with querying. It adds a calculated entity reference field for each profile type.

Here's an example of configuring user display and output.

And since it's a calculated field, we should only need to clear the entity field manager discovery cache, no storage changes, to make Drupal aware of the fields.

mglaman’s picture

This patch fixes chaining with the computed field. So something like $user->customer_profiles->first()->entity->address; works as if a normal entity reference. Also adds a test.

Still need to figure out the query, if possible. Ideally if this doesn't work out for the query, it solves the whole extra fields issue.

mglaman’s picture

Yep. That does not work

Drupal\Core\Entity\Query\QueryException: 'customer_profiles' not found in Drupal\Core\Entity\Query\Sql\Tables->ensureEntityTable()

Despite all efforts, this happens when compiling query conditions, which is way before altered is allowed. Looks like we'll need to use non-computed field and update the user whenever a new profile is created. Not horrible.

andrewbelcher’s picture

So we've been running with essentially the version of code from #7 in Decoupled Auth module. I've just found an interesting issue with it though! There is a race condition, which can result in the values getting stored and then cleared.

My specific case is the registration form where we've added a profile field and a submit handler. Due to #2932258: Split notification out of RegisterForm::save we are saving the account/profile before RegisterForm::save runs so our profile is available for tokens in the notification email. However, due to RegisterForm::$entity being a different object to the user returned from Profile::getOwner (as it's reloaded), the fields are being set by the updateProfileFields, but then getting cleared by RegisterForm::save... D'oh.

Possible solutions are:
1) Be very careful about order of saves - seems very fragile so although it may solve my immediate issue, I don't like it as a general solution...
2) Use computed fields so this never arises - requires solving the querying issue...
3) Always update the profile fields on user presave - potential performance issues...
4) Possibly some kind of flag on the field so we can detect if a change has been made by something other than updateProfileFields and, if so, re-run it at user presave - like above but mitigating the performance issues a bit...
5) Possibly use a service that will statically store the combinations, which can then be called as part of user presave - another possibly approach to improve performance or a user presave...

daggerhart’s picture

Getting caught up on this issue and I think the idea of improving DX from the user object perspective would be great.

Concerning the storage of references to profiles, I do something similar on a D7 site but with an additional level of abstraction. I automatically create a "meta" profile for each user that they cannot access, this is where I store the references to each of their other profiles by profile type (each profile type is its own entref field). Maybe it's worth considering doing the same here?

If we use a profile to store all the references to other profiles, then the only thing we would need on the user to achieve the DX goal would be a single entref field to the user's meta profile. This would allow for similar api improvements requested in the issue description, and would also avoid the need to save the user due to profile changes since the user would only ever need a single reference to their "meta" profile that should never change.

Pseudo usage based on issue description, the user to meta profile entref field is named "profiles" in these examples:

  • $user = \Drupal\user\Entity\User::load(1);
    $date_of_birth = $user->profiles->entity->some_profile_type->entity->date_of_birth->value;
    
  • $query = \Drupal::entityQuery('user');
    $query->condition('roles', 'administrator');
    $query->condition('profiles.entity.some_profile_type.entity.date_of_birth', '2000-01-01');
    

I'm sure there are drawbacks to this approach, but can't speak to them knowledgeably from a performance perspective. Each query from a user to their profiles would require an additional join is an obvious one.

jwilson3’s picture

Matt's patch in #39, basically fixes the issue I reported here: #2946670: Add computed {$profile_type}_profiles fields to users, use them to render profile data on the user profile regardless of the other issues here, I'd love to see that patch get comitted, if for nothing more than to have real widget configuration abilities on user field display.

Then the *proper* solution here could back out that code and replace it with something else, if it ever gets written.

manuel.adan’s picture

Assigned: Unassigned » manuel.adan
FileSize
16.02 KB
689 bytes

Patch #40 no longer applies on latest -dev because of #2930966: Fix phpcs errors (short array syntax, other tweaks). Ported and working on it. Interdiff doesn't work, attached a simple diff file.

manuel.adan’s picture

With #45 (actually #40 ported), I got an exception in the user register form (with embed profile):

Drupal\Core\Database\InvalidQueryException: Query condition profile.uid IN () cannot be empty. in Drupal\Core\Database\Query\Condition->condition()
...

That happened for new users that have no user ID yet.

I also found that the new profile reference field looks like empty when accessing though the API. For instance, the isEmpty() method returns TRUE despite the user as one or more profiles. I reviewed the field item list, now:

  • ProfileEntityFieldItemList extends EntityReferenceFieldItemList
  • the base class getters are overridden to ensure that the item list value is populated before read it

Not sure if setters should be overridden as well, since it is a read-only field. It might produce some side effect. At the moment, changes performed in the field value simply are not persistent.

manuel.adan’s picture

Assigned: manuel.adan » Unassigned
Status: Needs work » Needs review

Status: Needs review » Needs work

The last submitted patch, 46: storing_the_active-2654980-46.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

jonathanshaw’s picture

I think that some of @manuel.adan's improvements might be no longer needed (according to https://www.drupal.org/docs/8/api/entity-api/dynamicvirtual-field-values...) because #2392845: Add a trait to standardize handling of computed item lists was committed to D8.4.

@mglaman said in #41 that in order to get querying to work, "we'll need to use non-computed field and update the user whenever a new profile is created." (which was @andrewbelcher's original proposal.)

I propose that we do two things:
1) add a base field with storage of all profiles, to help querying, which should be this issue.
2) add a configurable computed field that references a single profile (defined in the field settings). Because it is configurable, sitebuilders can add multiple instances and therefore control more finely which profiles are shown in a particular display and in which sequence. This should be a separate issue. We might even be able to figure out a widget for it to work on forms.

joachim’s picture

> A. Switch to storing the link between users and profiles in an entity reference field on the user
> B. Store a cache of the active profiles in an entity reference field

Option C: Allow entity queries to go via backreferences.

I've got a hunch there's a core issue for that...

joachim’s picture

Bingo: #2975750: Allow entity queries to query reverse relationships.

I was hoping there'd be a patch there, but sadly not.

andrewbelcher’s picture

I think the answer is to support entity query via computed fields, which #2962457: Handle computed fields in entity queries: throwing a helpful exception is better than a PHP fatal error talks about as a follow up.

bojanz’s picture

Status: Needs work » Closed (won't fix)

#2946670: Add computed {$profile_type}_profiles fields to users, use them to render profile data on the user profile has introduced a computed field per profile type. No need to do anything here.