Improve performance of getReferencableEntities()

Is your widget select list? If so, can't you change it to autocomplete?

Also, since ER is OO, you can always extend the SelectionHandler with your class, and change the function signature like in your patch.

Yes, indeed. Don't use a select widget if you have an unbounded list of things to select from.

I have a patch that implements getReferencableEntities for the node entity (which is going to be the most common referenced entity) and uses a db lookup for titles which saves half the memory usage in my case (from 200+ to 100). The taxonomy term class already overrides getReferencableEntities so it makes sense to me to allow the node class to override this function and save all of the entity_load calls.

Fixed bundle in $options

-Added check if (!empty($results[$nid])) { to prevent notice errors.

Fixed syntax error, added entity type filter

Related D8 issue #2342569: Decide on a strategy to handle huge lists of options

Patch file had some cruft seek in and was causing failed tests. Reroll of same patch.

I'm not opposed on the principle, but let's make this generic. If an entity type (1) doesn't specify a $info['entity label'] callback, (2) has a $info['entity keys']['label'] and (3) has a base table, we can support a partial loading for it, else we just default to the complete loading version.

This is about performance, not scalability. This will never scale, regardless of how much performance you get out of it.

Here is an updated patch that also makes user option lists scale.

Removed debug

This patch is looking good to me - setting to review to run tests.

Also, I added a patch for OG to have it use this at #2642124: Consider using entity type specialized entityreference selection classes.

This should fix the tests.

It seems that for the access checks for user references we either have to (a) drop this approach for users completely and go back to entity_load, (b) duplicate the user access logic from entity module to work with the uid only, or (c) provide an incomplete entity with just the properties the entity access check needs. I went with (c) here as it seemed the simplest option, but I am not particularly happy with it.

Hi guys,

We've had the same issue on our site, but with entity field that references nodes. Although patches above helped with performance we went further and investigated the underlying issue in Entity module. It appears that in order to get list of options for a select list, Entity module is doing node_load for each of the options. We created a patch to replace node_load with a simple db_select.

I think fixing this issue in Entity module is better since that API may be used in other modules as well. Could you please review the patch and share your thoughts? Thanks.

Here is the link:
https://www.drupal.org/node/2636332

Hi @temkin, just to let you know that this might be two different issues. I have applied your patch but it didn't give any performance boost. But the one here (https://www.drupal.org/node/2151631#comment-10709382) it did. (confirming that works for me, one more review passed).

I guess I could apply both patches since yours it focus in on the "getlabel" part. But what really made a difference for me was to avoid the "entity_load" function. Not sure if my case is very specific or other people will find the same kind of results.

Cheers!

I can confirm the patch https://www.drupal.org/node/2151631#comment-10709382 took away 40,392 calls to entity_load when applied to a site with 12,907 users

Stumbled upon this issue again on another site. Entity Reference field with a Select widget causes out of memory error on node edit page. With patch from #23 everything works fine.

I am using the views selection handler and it was very slow as well. I have adopted this patch for the views handler and shaved 30 seconds off my page request.

Rerolling #28 to fix issue with user entityreferences breaking the form. Users don't have a bundle entity key.

straight up reroll

Version 1.6 released

Has anyone tested this with the 1.6 release? Automated tests are passing nicely.

pushed this into 7.x-1.x dev

see
#3347507: Plan for 7.x-1.10

Great job everyone above,

this patch/fix also resolves a memory allocation error.

I noticed a server crash with memory allocation failure after consuming 4 gigabytes of ram.

The 1.7 release solves the bug thanks to the above patch, along with installing the latest entity module

#2927873: Fatal error, allowed memory size exhausted, after update to 7.x-1.5

https://www.drupal.org/project/entityreference/releases/7.x-1.7

When we used 7.x-1.6, we did not have any errors caused by this module. After updating this module to 7.x-1.7, we got this error:

EntityMalformedException: Missing bundle property on entity of type taxonomy_term. in entity_extract_ids() (line 8090 of
/.../includes/common.inc).

To get rid of this error, we load an entity inside the loop. Please see the attached patch.

@joseph.olstad, in addition to what @franceslui (I work with Frances, full disclosure) mentioned above there is also a security issue here too because entity_load() will add a security access check tag and translation tag capabilities, example for taxonomy terms:

/modules/taxonomy/taxonomy.module:1254

  protected function buildQuery($ids, $conditions = array(), $revision_id = FALSE) {
    $query = parent::buildQuery($ids, $conditions, $revision_id);
    $query->addTag('translatable');
    $query->addTag('taxonomy_term_access');

Which is true for other entity types as well.

So quick fix is revert, or consider the patch middle ground that solves the bundle issue where the entity key's bundle is not on the base table that was db_select'd.

For this patch, wouldn't it be better to check to see if the bundle property is missing first before loading?

I'll roll a new patch

New patch

actually, my patch is possibly not any better than patch 37, not sure if bundle will ever be set in this case.

@joelpittet, if you have access to this environment I'd like to run a test or two and inspect the $entity returned on line 150

in the case of the crashing, what is your $target_type variable value?
what also is the value of $entity ?

@joelpittet, if you have access to this environment I'd like to run a test or two and inspect the $entity variable returned on line 150

$query->execute() as $entity

what would be the value of $entity when this crashes?

what would be the value of $target_type when this crashes?

It would be great to find a better solution than reverting since this fixes a critical memory allocation error on my clients environment and it was reported above to fix memory allocation issues in other environments.

might not be possible to do so easily however maybe worth a bit of effort.

@joelpittet, @franceslui, will this patch work for you?

As for the "security issue", I'd like to see a real-world scenario for this use case. There's multiple levels of access checks and we're trying to balance out performance and prevent memory allocation errors.

Please review/test patch 43

@joelpittet

I did a little bit of debugging on my own environment and tested this code which assumes first taxonomy_term tid = 1 , then node nid =1.

  $entity_id = 1;
  $target_type = 'taxonomy_term';
  $loaded_entities = entity_load($target_type, array($entity_id));
  $entity = reset($loaded_entities);
  list($id,, $bundle) = entity_extract_ids('taxonomy_term', $entity);
  echo "\n";
  echo print_r(entity_extract_ids('taxonomy_term', $entity), TRUE);
  echo "\n";
  echo "id: " . $id;
  echo "\n";
  echo "bundle: " . $bundle;
  echo "\n";

  $target_type = 'node';
  $loaded_entities = entity_load($target_type, array($entity_id));
  $entity = reset($loaded_entities);
  list($id,, $bundle) = entity_extract_ids('node', $entity);
  echo "\n";
  echo print_r(entity_extract_ids('node', $entity), TRUE);
  echo "\n";
  echo "id: " . $id;
  echo "\n";
  echo "bundle: " . $bundle;
  echo "\n";

The output was as follows:

Array
(
    [0] => 1
    [1] => 
    [2] => tags
)

 id: 1
 bundle: tags

Array
(
    [0] => 1
    [1] => 1
    [2] => article
)

 id: 1
 bundle: article

What entity type is causing this problem on your end, I tested taxonomy_term and node?

@joelpittet @franceslui, please test patch 43

@joseph.olstad #43 entity_extract_ids is where the error is happening so that won't fix the problem.

@joseph.olstad Thanks for taking some of your weekend to look into this, much appreciated.

The crux of this is that db_select() was used to replace entity_load() so while db_select will be hugely faster, that is mostly due to the lack of hooks that could be fired off to do things to "load" the entity.

$entity = stdClass when db_select() builds it.
$entity = *EntityController class when entity_load() finishes with it.

And in our case taxonomy_term_data table doesn't store the bundle name, it stores vid, and needs those extra hooks to join the vocabulary table.

@franceslui, you probably need to add a empty check after the entity_load in case it was not loaded due to access control or something for #37

@joseph.olstad oh I think I see where you might be confused, the code we care about is the part at the end of the committed patch from #31

-      $entities = entity_load($target_type, array_keys($result));
-      foreach($entities as $entity) {
+
+      // Do not load the entire entity for performance reasons.
+      $entity_info = entity_get_info($target_type);
+      $query = db_select($entity_info['base table'], 'e');
+      $query->fields('e');
+      $query->condition('e.' . $entity_info['entity keys']['id'], array_keys($result), 'IN');
+
+      foreach($query->execute() as $entity) {

You see where there is no entity_load()

If this only gets a partial revert or better patch, maybe it should live over here #3348756: EntityMalformedException: Missing bundle property error when entity reference selection mode is set to Views: Filter by an entity reference view?

@joelpittet, please try this patch

#3348756-04: EntityMalformedException: Missing bundle property error when entity reference selection mode is set to Views: Filter by an entity reference view

alternatively, this approach may also resolve the issue for you:

#3348756-06: EntityMalformedException: Missing bundle property error when entity reference selection mode is set to Views: Filter by an entity reference view

If someone experiences what was described in #37, please try this patch:
#3348756-19: EntityMalformedException: Missing bundle property error when entity reference selection mode is set to Views: Filter by an entity reference view

If I understood correctly I believe that @joelpittet is working on an automated test to make sure that we fix this issue for good.

Marking this back as fixed as all the work is happening in the other issue mentioned in #53

Thanks to the UBC team, especially Joelpittet and @franceslui, 1.8 has the regression fix, a usability improvement and added test coverage.
https://www.drupal.org/project/entityreference/releases/7.x-1.8

here's the usability improvement:
#2571107: Improve the ENTER keydown behaviour of ER autocomplete

please try the latest release

https://www.drupal.org/project/entityreference/releases/7.x-1.9