Problem/Motivation

Some issues such as

#3007752: Entity usage list does not scale
#3015287: Allow usage records to be registered in background
#3056026: Remove orphan handling of paragraphs when parent exists
#2985265: Cache viewable results when generating usage page
#3002332: Track composite (e.g. paragraph) entities directly and only as their host
#2971131: Improve label handling on usage page
#2949952: Make it easier to retrieve full relationship chain

are examples that show how our current approach is not scalable, and also hacky in some scenarios.

Storing all 1-to-1 relationships in DB is great, but not using the same rules to display them on the UI is challenging. On the other hand, arguably most users of this module are not really interested in that flexibility, and only interested in "in what node is this piece of media being used (regardless of all paragraphs/blocks in between them)".

This issue is to explore a new architecture of the module to simplify things, and hopefully allow better scaling for large sites.

Note: This will obviously be done in a new 3.x branch, since important API-level aspects would change.

Proposed resolution

  • We introduce the concepts of top/middle/bottom level for entities
  • Only Top-level entities are tracked as source (by default only nodes, sites could override that)
  • Middle-level entities are disregarded entirely, but traversed when looking for targets
  • Bottom-level entities are all entities configured to have the "Usage" local task (tab) on their page. These are the only "target entities" we store information for in DB
  • We no longer care to store information about the fieldname, relationship method (plugin id), or count in DB. Now all a DB row tells us is that: "a (source, top-level) entity of type: A, with id: B, on its revision C and language D points to a (target, bottom-level) entity of type E with id: F". This "reference" from source and target may be direct or through any number of intermediate non-top / non-bottom entities.
  • All tracking calculation is deferred to a (usually) background process using DestructableInterface
  • We can now build the usage page on the UI using direct paged queries. The page will show usages grouped by source entity type (so we can easily join the entity table on the default revision) and only show records that point to the default revision
  • We no longer need to provide any specific views integration

Remaining tasks

Should be fixed here:

  1. We need to implement new instances of hook_entity_delete() and similar, to: a) Remove records from the DB when target (bottom) entities are deleted, b) Set the "needs regeneration" flag/warning when middle-level entities are deleted

Can be fixed in follow-ups:

  1. Having dropped the views integration, we might consider a custom field handler that would either show "Not being used", or a link to "Check usages"
  2. When a field / field storage is deleted, and also when any middle-level entity is deleted, we may end up with stale info in the DB. There may be several ways to approach this, it's probably best to discuss / test pros and cons of each in a follow-up.
  3. When an entity is used only in past revisions of a top-level entity, those "hidden" usages will not show up in the usage list. We need a mechanism to let users discover the past usages through the UI, if needed.
  4. Currently to expose new entity types to be tracked as source, sites need to write custom code and override \Drupal\entity_usage\EntityUsageSourceLevel::TOP_LEVEL_TYPES. It would be better to expose this to be configurable on the UI.

User interface changes and modifications on default behavior

  1. The usage page (when visitors click on the "Usage" tab) now only displays rows for top-level source entities.
  2. The usage page no longer displays columns for "Field Name", or "Used in".
  3. The usage page will now group source entities by their type, with a common pager for all groups. For example, if the "Number if items per group" (defined in the settings form) is set to 10, and Nodes and Users are configured to be tracked as top-level entities, then the first page will display a group of 10 rows for node sources, and another group of 10 rows for user sources. The next page would fetch the next 10 rows for both groups, and so on.
  4. When the module is first installed, node and media entities (if they exist) will have the "Usage" tab enabled by default. This will mean they are automatically tracked as targets (bottom-level) by default. Note: This does not apply to existing sites.
  5. By default only node entities are tracked as source.
  6. In some situations (for example after a field has been deleted), a warning in the status report page will be displayed, informing users that usage re-generation is needed. In order to do so and remove the message, users need to go to the Batch Update form and trigger a batch update of usage statistics.

API changes

No change is needed in tracking plugins, as long as they extended the \Drupal\entity_usage\EntityUsageTrackBase base class.

The changes below might affect custom or contrib code interacting with this module:

  1. The setting option usage_controller_items_per_page is now called usage_controller_items_per_group.
  2. The hook hook_entity_usage_block_tracking() no longer receives method, field_name, or count as parameters.
  3. The entity_usage DB table no longer has columns for method, field_name, or count
  4. The system now uses a state flag entity_usage_needs_regeneration to display a warning on the status report page when we detect stale data might exist
  5. The Drupal\entity_usage\EntityUpdateManager service now implements DestructableInterface, and during CRUD hooks we only register the operations that happened during the current request. All real usage tracking is deferred to the end of the request (normally in background), inside the \Drupal\entity_usage\EntityUpdateManager::destruct() method.
  6. The module no longer provides specific views integration. In other words, we no longer implement hook_views_data() or hook_views_data_alter().
  7. The methods: ::trackUpdateOnCreation(), ::trackUpdateOnEdition(), and ::trackUpdateOnDeletion() from EntityUpdateManager are now protected instead of public.
  8. The method \Drupal\entity_usage\EntityUsageInterface::registerUsage() is now only intended to _adding new records_ (instead of adding/updating existing). Its signature has changed since it now receives less arguments.
  9. A new \Drupal\entity_usage\EntityUsageInterface::deleteUsage() method is created to allow deleting a specific record from the DB.
  10. The method \Drupal\entity_usage\EntityUsageInterface::deleteByField() is removed, since we no longer have field information in DB
  11. All events dispatched by this module have been adjusted, since we no longer pass information about field_name, method, etc. Also, a new event is created when a specific DB record is deleted.
  12. The \Drupal\entity_usage\EntityUsageInterface::listSources() return value now no longer includes information about the field name, method, or count for the retrieved records.
  13. The \Drupal\entity_usage\EntityUsageInterface::listTargets() return value is now a simple associative array where keys are target entity types, and values are indexed arrays of target entity IDs.
  14. The (already) deprecated methods \Drupal\entity_usage\EntityUsageInterface::listUsage() and \Drupal\entity_usage\EntityUsageInterface::listReferencedEntities() were removed.
CommentFileSizeAuthor
#26 entity_usage-3060802-refactor-26-D9.patch198.44 KBomarlopesino
#11 3060802-11.patch184.68 KBmarcoscano
#6 interdiff-5-6.txt105.17 KBmarcoscano
#6 3060802-6.patch184.65 KBmarcoscano
#5 interdiff-4-5.txt6.14 KBmarcoscano
#5 3060802-5-do-not-test.patch97 KBmarcoscano
#4 3060802-4-do-not-test.patch96.39 KBmarcoscano
#4 interdiff-3-4.txt28.44 KBmarcoscano
#3 3060802-3-do-not-test.patch77.02 KBmarcoscano
#3 interdiff-2-3.txt27.56 KBmarcoscano
#2 3060802-2-do-not-test.patch51.07 KBmarcoscano

Issue fork entity_usage-3060802

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

About issue forks

Comments

marcoscano created an issue. See original summary.

marcoscano’s picture

marcoscano
Primary language Portuguese, Brazil
Location Barcelona, Spain
commented
Issue summary: View changes
StatusFileSize
new 3060802-2-do-not-test.patch51.07 KB

First shot as a POC, seems to work in manual testing (checking records in the DB).

Several things still pending:
- All changes to the controller
- Update hooks
- Update the tests
- Test / adjust batch update code
- Figure out what to do when a field or a middle-entity is deleted
- Test / adjust the views integration

marcoscano’s picture

marcoscano
Primary language Portuguese, Brazil
Location Barcelona, Spain
commented
StatusFileSize
new interdiff-2-3.txt27.56 KB
new 3060802-3-do-not-test.patch77.02 KB

Still todo:
- Pager on the controller
- New controller for the "More info" page (past revisions)
- Update hooks
- Update the tests
- Figure out what to do when a field or a middle-entity is deleted
- Test / adjust the views integration

marcoscano’s picture

marcoscano
Primary language Portuguese, Brazil
Location Barcelona, Spain
commented
StatusFileSize
new interdiff-3-4.txt28.44 KB
new 3060802-4-do-not-test.patch96.39 KB

About "New controller for the More Info page":

I'm less convinced a page using past revisions is useful when there is an usage in the default revision. It doesn't add any value to know that previous revisions also used it, and it is still being used in the current revision. So I'm actually removing that column from the table.

What would be useful, though, is to have a mechanism to let users know that there is a usage ONLY in a past revision of a node. But that's tricky, since that node isn't in the main usage page because... well... there's no usage in the default revision! Still need to think the best way to get around this.

Views integration:

Since we no longer have the "count" column in DB, I'm feeling we shouldn't have the views relationships anymore, since there is no point in including it. Site-builders can just create a link to the usage page and display for each row in their view something like "Check usage", which would point to the usage path for that entity.

If anything, we can provide a custom views field plugin that instead of always showing the link, display "No usages" when there are no usages recorded, but that can be a follow-up I think.

Field deletion / Middle-entity deletion:

This patch includes a very basic approach: whenever a field or field storage is deleted, if they were in a top or middle-level entity, we set a flag that will display an error in the status report, informing that bulk regeneration is needed.

This system can likely be improved to be smarter.

Still pending:
- Decide how to best display the "hidden" (past revisions-only) usages on the UI
- Update the tests

marcoscano’s picture

marcoscano
Primary language Portuguese, Brazil
Location Barcelona, Spain
commented
Title: Re-architecture the module in a simpler, opinionated and more performant approach » Refactor module architecture in a simpler, opinionated and more performant approach
Status: Active » Needs review
StatusFileSize
new 3060802-5-do-not-test.patch97 KB
new interdiff-4-5.txt6.14 KB

Setting to NR to hopefully have opinions about the new approach, even though there are no updates to tests yet.

I've added a message to the usage page that for now only informs users if "there are hidden usages", even though we aren't providing a way of telling the user what those hidden usages are.

marcoscano’s picture

marcoscano
Primary language Portuguese, Brazil
Location Barcelona, Spain
commented
StatusFileSize
new 3060802-6.patch184.65 KB
new interdiff-5-6.txt105.17 KB

Now with tests updated, and some bugs caught along the way.
tests++

marcoscano’s picture

marcoscano
Primary language Portuguese, Brazil
Location Barcelona, Spain
commented
Issue summary: View changes
marcoscano’s picture

marcoscano
Primary language Portuguese, Brazil
Location Barcelona, Spain
commented
Issue summary: View changes
marcoscano’s picture

marcoscano
Primary language Portuguese, Brazil
Location Barcelona, Spain
commented
Issue summary: View changes
brooke_heaton’s picture

brooke_heaton commented

Update hook 8301 fails for me with:

[error] The 'target_id_string' field specification does not define 'not null' as TRUE.

marcoscano’s picture

marcoscano
Primary language Portuguese, Brazil
Location Barcelona, Spain
commented
StatusFileSize
new 3060802-11.patch184.68 KB

Thanks @brooke_heaton ! It seems that I was testing this on new installs only, and not sure how that change got in there :). This patch fixes it, in any case (also re-rolled, since no interdiff)

Updb seems to work for me now:

Performing system_update_8701                                                                                                                                                                            [ok] 
Performing entity_usage_update_8300                                                                                                                                                                      [ok] 
Performing system_update_8702                                                                                                                                                                            [ok] 
Performing entity_usage_update_8301                                                                                                                                                                      [ok] 
Post updating entity_usage                                                                                                                                                                               [ok] 
Post updating menu_link_content                                                                                                                                                                          [ok] 
Post updating system                                                                                                                                                                                     [ok] 
Post updating system                                                                                                                                                                                     [ok] 
Post updating taxonomy                                                                                                                                                                                   [ok] 
Post updating taxonomy                                                                                                                                                                                   [ok] 
Post updating text                                                                                                                                                                                       [ok] 
Post updating views                                                                                                                                                                                      [ok] 
Post updating views                                                                                                                                                                                      [ok] 
Cache rebuild complete.                                                                                                                                                                                  [ok] 
Finished performing updates.                                                                                                                                                                             [ok] 
The following updates are pending:

system module : 
  8701 -   Remove the unused 'system.theme.data' from state. 
  8702 -   Add the 'revision_translation_affected' entity key. 

entity_usage module : 
  8300 -   Rename items_per_page into items_per_group. 
  8301 -   Recreate the entity usage table with the new schema. 

entity_usage module : 
  Re-generate entity_usage statistics on the 3.x branch.

menu_link_content module : 
  Update custom menu links to be revisionable.

system module : 
  Clear the menu cache.   @see https:www.drupal.orgprojectdrupalissues3044364
  Clear the library cache and ensure aggregate files are regenerated.

taxonomy module : 
  Add status with settings to all form displays for taxonomy entities.
  Update taxonomy terms to be revisionable.

text module : 
  Update text_with_summary fields and widgets to add summary required flags.

views module : 
  Define default values for limit operators settings in all filters.
  Rebuild cache to allow placeholder texts to be translatable.
marcoscano’s picture

marcoscano
Primary language Portuguese, Brazil
Location Barcelona, Spain
commented

I opened a 3.x branch of the module with #11, just so it's easier to keep track of changes.

rp7’s picture

rp7 commented

Great stuff being done here, nice work.

Since we no longer have the "count" column in DB, I'm feeling we shouldn't have the views relationships anymore, since there is no point in including it. Site-builders can just create a link to the usage page and display for each row in their view something like "Check usage", which would point to the usage path for that entity.

If anything, we can provide a custom views field plugin that instead of always showing the link, display "No usages" when there are no usages recorded, but that can be a follow-up I think.

Are there reasons we are not using Views for rendering the usage page? This would allow the usage page to be more customizable.

Currently on a project where the client doesn't want usages to be listed on a separate page, but rather as a block in the sidebar of the canonical page (/node/123). If there was views integration, this would be as simple as adding a block display.

Are there any hard-blockers for this? Willing to put effort into this.

rp7’s picture

rp7 commented

Another question: will the 2.x branch still be supported? If so, for how long?

marcoscano’s picture

marcoscano
Primary language Portuguese, Brazil
Location Barcelona, Spain
commented

@rp7 thanks for chiming in!

Are there reasons we are not using Views for rendering the usage page?

That's an interesting question :)
In the 2.x branch, that was impossible since that page needed some "massaging" of the results before building the page. It's true, though, that with this simplified approach in 3.x we may very well simplify it even further and use a view for that... Apart from introducing a hard dependency on views, I can't think of any other major drawback of this idea right now, so it could be worth exploring.

Another question: will the 2.x branch still be supported? If so, for how long?

Another good question. This 3.x branch is very much experimental for now, there's still a lot to figure out and edges to polish for it to be even usable. If everything works well, though, and we are happy with the improvements, my idea was to focus new features on 3.x only. On the other hand, I don't plan to force anybody to upgrade, since there are disruptive changes. So even if 3.x is successful, I envision 2.x still receiving major bug fixes / security fixes for the foreseeable future.

geek-merlin’s picture

geek-merlin
Primary language German
Location Freiburg, Germany
commented
Related issues: +#3090018: Proposal to refactor entity_usage
frob’s picture

frob
Primary language English
commented

Really liking this module. Is there any hope in splitting this module as a part of the experimental 3.x redesign? Right now it seems like it could make a useful API module which includes the plugin definitions and services and then more sub-modules which include things like views integration and UI.

frob’s picture

frob
Primary language English
commented

I noticed in #3090018: Proposal to refactor entity_usage they mention using a the D7 Tree module approach to simplifying nested references. Another approach that might be easier is to use the nested-set approach that the entity_hierarchy module is using.

omarlopesino made their first commit to this issue’s fork.

omarlopesino opened merge request !14

omarlopesino’s picture

omarlopesino
he/him
commented
Status: Needs review » Needs work

Hi. In a project, I need to see where the media are being used, and the 3.x resolution fits as we do want to ignore the entities referenced between the node and the media.

I've created an MR from the dev branch, rerolling patch #11.

I've found a problem, there was an infinite recursion produced in these circumstances:
- There is a content type with a paragraph that allows references to other nodes.
- A node is created with this paragraph.
- The node that is referenced in this paragraph, also has a reference to the other node.
- The entity usage of that node is recalculated
In those conditions, the entity reference track plugin would go fall into an infinite recursion as it will find a reference that points to the self node that is currently being recalculated.

The problem is fixed in this commit: https://git.drupalcode.org/issue/entity_usage-3060802/-/commit/663bcaaf3... , by not going deep into entities that are currently being calculated.

The module looks functional now but the tests need to be reviewed after the reroll, I will look into it. Any help is appreciated.

omarlopesino’s picture

omarlopesino
he/him
commented
Status: Needs work » Needs review

Now the tests are fixed, along with some corrections that were missing after the re-roll. Please review, thanks!

flyke’s picture

flyke commented

I cant apply patch #11 latest dev.

composer require 'drupal/entity_usage:2.x-dev@dev'

with this in the patches section in my composer.json:

"drupal/entity_usage": {
                "3060802 - patch": "https://www.drupal.org/files/issues/2019-07-04/3060802-11.patch"
            }

this gives error failed to apply patch.

If I manually try to apply the patch (downloaded it locally):
git apply -v --directory=web/modules/contrib/entity_usage 3060802-11.patch

then I can see that there ae parts of the patch that can be applied, but a lot that cant:

Checking patch web/modules/contrib/entity_usage/config/install/entity_usage.settings.yml...
Checking patch web/modules/contrib/entity_usage/config/schema/entity_usage.schema.yml...
Checking patch web/modules/contrib/entity_usage/entity_usage.api.php...
Checking patch web/modules/contrib/entity_usage/entity_usage.install...
Checking patch web/modules/contrib/entity_usage/entity_usage.module...
Checking patch web/modules/contrib/entity_usage/entity_usage.post_update.php...
Checking patch web/modules/contrib/entity_usage/entity_usage.services.yml...
error: while searching for:
  entity_usage.entity_update_manager:
    class: Drupal\entity_usage\EntityUpdateManager
    arguments: ['@entity_usage.usage', '@plugin.manager.entity_usage.track', '@config.factory']
  entity_usage.route_subscriber:
    class: Drupal\entity_usage\Routing\RouteSubscriber
    arguments: ['@entity_type.manager', '@config.factory']

error: patch failed: web/modules/contrib/entity_usage/entity_usage.services.yml:8
error: web/modules/contrib/entity_usage/entity_usage.services.yml: patch does not apply
Checking patch web/modules/contrib/entity_usage/entity_usage.views.inc...
Checking patch web/modules/contrib/entity_usage/src/Controller/ListUsageController.php...
error: while searching for:
use Drupal\Core\Access\AccessResult;
use Drupal\Core\Config\ConfigFactoryInterface;
use Drupal\Core\Controller\ControllerBase;
use Drupal\Core\Entity\EntityFieldManagerInterface;
use Drupal\Core\Entity\EntityInterface;
use Drupal\Core\Entity\EntityTypeManagerInterface;
use Drupal\Core\Entity\RevisionableInterface;
use Drupal\Core\Language\LanguageInterface;
use Drupal\Core\Link;
use Drupal\entity_usage\EntityUsageInterface;
use Symfony\Component\DependencyInjection\ContainerInterface;

/**

error: patch failed: web/modules/contrib/entity_usage/src/Controller/ListUsageController.php:5
error: web/modules/contrib/entity_usage/src/Controller/ListUsageController.php: patch does not apply
Checking patch web/modules/contrib/entity_usage/src/EntityUpdateManager.php...
Checking patch web/modules/contrib/entity_usage/src/EntityUsage.php...
error: while searching for:
        'target_id',
        'target_id_string',
        'target_type',
        'method',
        'field_name',
        'count',
      ])
      ->condition($source_id_column, $source_entity->id())
      ->condition('source_type', $source_entity->getEntityTypeId())
      ->condition('count', 0, '>')
      ->orderBy('target_id', 'DESC')
      ->execute();

    $references = [];
    foreach ($result as $usage) {
      $target_id_value = !empty($usage->target_id) ? $usage->target_id : $usage->target_id_string;
      $references[$usage->target_type][(string) $target_id_value][] = [
        'method' => $usage->method,
        'field_name' => $usage->field_name,
        'count' => $usage->count,
      ];
    }

    return $references;

error: patch failed: web/modules/contrib/entity_usage/src/EntityUsage.php:285
error: web/modules/contrib/entity_usage/src/EntityUsage.php: patch does not apply
Checking patch web/modules/contrib/entity_usage/src/EntityUsageInterface.php...
error: while searching for:
   *   The source entity to check for references.
   *
   * @return array
   *   A nested array with usage data. The first level is keyed by the type of
   *   the target entities, the second by the target id. The value of the second
   *   level contains all other information like the method used by the source
   *   to reference the target, the field name and the target language code.
   *
   * @see \Drupal\entity_usage\EntityUsageInterface::listSources()
   */
  public function listTargets(EntityInterface $source_entity);

  /**
   * Determines where an entity is used (deprecated).
   *
   * This method should not be used in new integrations, and is only provided
   * as BC-layer for existing implementations. Note however that the count
   * returned on 2.x will be different from the count returned on 1.x, once
   * now we track all revisions / translations independently.
   *
   * @param \Drupal\Core\Entity\EntityInterface $entity
   *   A target (referenced) entity.
   * @param bool $include_method
   *   (optional) Whether the results must be wrapped into an additional array
   *   level, by the reference method. Defaults to FALSE.
   *
   * @return array
   *   A nested array with usage data.The first level is keyed by the type of
   *   the source entity, the second by the referencing objects ID. The value of
   *   the second level contains the usage count, which will be summed for all
   *   revisions and translations tracked.
   *   Note that if $include_method is TRUE, the first level is keyed by the
   *   reference method, and the second level will continue as explained above.
   *
   * @deprecated in branch 2.x.
   *   Use \Drupal\entity_usage\EntityUsageInterface::listSources() instead.
   */
  public function listUsage(EntityInterface $entity, $include_method = FALSE);

  /**
   * Determines referenced entities (deprecated).
   *
   * This method should not be used in new integrations, and is only provided
   * as BC-layer for existing implementations. Note however that the count
   * returned on 2.x will be different from the count returned on 1.x, once
   * now we track all revisions / translations independently.
   *
   * @param \Drupal\Core\Entity\EntityInterface $entity
   *   A source entity.
   *
   * @return array
   *   A nested array with usage data.The first level is keyed by the type of
   *   the target entity, the second by the referencing objects ID. The value of
   *   the second level contains the usage count, which will be summed for all
   *   revisions and translations tracked.
   *
   * @deprecated in branch 2.x.
   *   Use \Drupal\entity_usage\EntityUsageInterface::listTargets() instead.
   */
  public function listReferencedEntities(EntityInterface $entity);

}

error: patch failed: web/modules/contrib/entity_usage/src/EntityUsageInterface.php:158
error: web/modules/contrib/entity_usage/src/EntityUsageInterface.php: patch does not apply
Checking patch web/modules/contrib/entity_usage/src/EntityUsageSourceLevel.php...
Checking patch web/modules/contrib/entity_usage/src/EntityUsageSourceLevelInterface.php...
Checking patch web/modules/contrib/entity_usage/src/EntityUsageTrackBase.php...
Checking patch web/modules/contrib/entity_usage/src/EntityUsageTrackInterface.php...
Checking patch web/modules/contrib/entity_usage/src/Events/EntityUsageEvent.php...
Checking patch web/modules/contrib/entity_usage/src/Events/Events.php...
Checking patch web/modules/contrib/entity_usage/src/Form/BatchUpdateForm.php...
error: while searching for:
use Drupal\Core\Entity\EntityTypeManagerInterface;
use Drupal\Core\Form\FormBase;
use Drupal\Core\Form\FormStateInterface;
use Symfony\Component\DependencyInjection\ContainerInterface;

/**

error: patch failed: web/modules/contrib/entity_usage/src/Form/BatchUpdateForm.php:5
error: web/modules/contrib/entity_usage/src/Form/BatchUpdateForm.php: patch does not apply
Checking patch web/modules/contrib/entity_usage/src/Form/EntityUsageSettingsForm.php...
Hunk #1 succeeded at 251 (offset 2 lines).
Hunk #2 succeeded at 283 (offset 2 lines).
Checking patch web/modules/contrib/entity_usage/src/Plugin/EntityUsage/Track/EntityReference.php...
Checking patch web/modules/contrib/entity_usage/tests/modules/entity_usage_test/entity_usage_test.module...
Checking patch web/modules/contrib/entity_usage/tests/src/FunctionalJavascript/BatchUpdateTest.php...
Checking patch web/modules/contrib/entity_usage/tests/src/FunctionalJavascript/ConfigEntityTrackingTest.php...
error: while searching for:
    // We should be at /webform/contact/usage.
    $this->assertContains("/webform/contact/usage", $session->getCurrentUrl());
    $assert_session->elementContains('css', 'main table', 'Node that points to a webform');
    $assert_session->elementContains('css', 'main table', 'Related Webforms');
  }

  /**

error: patch failed: web/modules/contrib/entity_usage/tests/src/FunctionalJavascript/ConfigEntityTrackingTest.php:148
error: web/modules/contrib/entity_usage/tests/src/FunctionalJavascript/ConfigEntityTrackingTest.php: patch does not apply
Checking patch web/modules/contrib/entity_usage/tests/src/FunctionalJavascript/ConfigurationFormTest.php...
error: while searching for:
use Drupal\field\Entity\FieldStorageConfig;
use Drupal\media\Entity\Media;
use Drupal\node\Entity\Node;
use Drupal\Tests\media\Functional\MediaFunctionalTestCreateMediaTypeTrait;

/**

error: patch failed: web/modules/contrib/entity_usage/tests/src/FunctionalJavascript/ConfigurationFormTest.php:7
error: web/modules/contrib/entity_usage/tests/src/FunctionalJavascript/ConfigurationFormTest.php: patch does not apply
Checking patch web/modules/contrib/entity_usage/tests/src/FunctionalJavascript/DynamicEntityReferenceTest.php...
Checking patch web/modules/contrib/entity_usage/tests/src/FunctionalJavascript/EmbeddedContentTest.php...
error: while searching for:
use Drupal\field\Entity\FieldStorageConfig;
use Drupal\node\Entity\Node;
use Drupal\Tests\entity_usage\Traits\EntityUsageLastEntityQueryTrait;

/**
 * Basic functional tests for the usage tracking of embedded content.

error: patch failed: web/modules/contrib/entity_usage/tests/src/FunctionalJavascript/EmbeddedContentTest.php:6
error: web/modules/contrib/entity_usage/tests/src/FunctionalJavascript/EmbeddedContentTest.php: patch does not apply
Checking patch web/modules/contrib/entity_usage/tests/src/FunctionalJavascript/IntegrationTest.php...
Checking patch web/modules/contrib/entity_usage/tests/src/FunctionalJavascript/ListControllerTest.php...
Checking patch web/modules/contrib/entity_usage/tests/src/FunctionalJavascript/ParagraphsTest.php...
error: while searching for:
use Drupal\field\Entity\FieldStorageConfig;
use Drupal\media\Entity\Media;
use Drupal\Tests\entity_usage\Traits\EntityUsageLastEntityQueryTrait;
use Drupal\Tests\media\Functional\MediaFunctionalTestCreateMediaTypeTrait;
use Drupal\Tests\paragraphs\FunctionalJavascript\ParagraphsTestBaseTrait;
use Drupal\user\Entity\Role;

error: patch failed: web/modules/contrib/entity_usage/tests/src/FunctionalJavascript/ParagraphsTest.php:6
error: web/modules/contrib/entity_usage/tests/src/FunctionalJavascript/ParagraphsTest.php: patch does not apply
Checking patch web/modules/contrib/entity_usage/tests/src/FunctionalJavascript/RevisionsTranslationsTest.php...
Checking patch web/modules/contrib/entity_usage/tests/src/FunctionalJavascript/ViewsTest.php...
Checking patch web/modules/contrib/entity_usage/tests/src/Kernel/EntityUsageTest.php...
Hunk #1 succeeded at 85 (offset 3 lines).
error: while searching for:
    /** @var \Drupal\Core\Entity\EntityInterface $source_entity */
    $source_entity = $this->testEntities[1];
    $source_vid = ($source_entity instanceof RevisionableInterface && $source_entity->getRevisionId()) ? $source_entity->getRevisionId() : 0;
    $field_name = 'body';
    $this->injectedDatabase->insert($this->tableName)
      ->fields([
        'target_id' => $target_entity->id(),

error: patch failed: web/modules/contrib/entity_usage/tests/src/Kernel/EntityUsageTest.php:106
error: web/modules/contrib/entity_usage/tests/src/Kernel/EntityUsageTest.php: patch does not apply
Checking patch web/modules/contrib/entity_usage/tests/src/Traits/PostRequestTrackingTrait.php...
omarlopesino’s picture

omarlopesino
he/him
commented

Those errors are fixed in the MR. I should have hidden the patches which won't work, my bad.

@flyke the merge request attached to the task solves all that problems, may you apply a patch from that MR and check again?

brooke_heaton’s picture

brooke_heaton commented

The MR patch is not working for me against "drupal/entity_usage": "2.x-dev@dev". Patch fails:
https://git.drupalcode.org/project/entity_usage/-/merge_requests/14.patch

omarlopesino’s picture

omarlopesino
he/him
commented
StatusFileSize
new entity_usage-3060802-refactor-26-D9.patch198.44 KB

I don't know why, but the plain diff does not fail against the 8.x-2.x branch, please use it instead.

In any case, I've attached the patch complementing the MR.

omarlopesino’s picture

omarlopesino
he/him
commented
anybody’s picture

anybody
Primary language German
Location Porta Westfalica
commented

As this module is used by 26,105 installations, several other modules and even seems quite helpful as Drupal Core (entity_reference) addition, could this perhaps be pushed forward by any community initiative or something like that?

@marcoscano this is still assigned to you, are you working on this actively?

marcoscano’s picture

marcoscano
Primary language Portuguese, Brazil
Location Barcelona, Spain
commented
Assigned: marcoscano » Unassigned

I am not currently working on this, forgot to un-assign the issue.

I still think though that there are challenges with both sides.

On one hand, the 2.x (currently supported branch) has scalability issues as pointed out in the issues in the issue summary. On the other hand, the idea with the refactoring here changes significantly the approach, still has a few edges that need polishing, and is much more opinionated, which means it might not be a good fit for all projects.

Personally, I would love to pursue this 3.x approach in the long run, but before diving deeper into it, I would love to hear that it's not only a "theoretical" solution, but actually meets the needs of real projects out there.

brooke_heaton’s picture

brooke_heaton commented

@marcoscano - I'm late to the party here but have a site that has an immense number of entity reference fields and this module seemed rather crucial. Can you elaborate on what you mean by 'opinionated' and in what way? What types of projects would suffer from a different approach and what types of projects would benefit?

marcoscano’s picture

marcoscano
Primary language Portuguese, Brazil
Location Barcelona, Spain
commented

@brooke_heaton the issue description goes in more detail, but the most important part is that we would be disregarding all intermediate entities in a relationship chain, and only store in the DB information about the origin and the final target. So for instance in a relationship chain such as node -> paragraph -> paragraph -> media, only the relationship between the node and the media item is stored. In a paragraphs scenario this isn't a big deal, since paragraphs have no standalone representation in Drupal, but who knows what sites might be doing, so far the module was not special-casing types of entities, and we just blindly stored all 1-to-1 relationships in DB. With the new approach we start looking at the entities themselves that are part of a relationship and decide which ones we store and which one we don't, that's why it's more opinionated.

acbramley’s picture

acbramley commented

The main issue with the proposed solution I see is that some entity types may be a mix of top, mid, or bottom level depending on what the entity itself is being used for.

Let's take a scenario (which is in use in a current client project that heavily uses entity_usage) with Node, Managed link (from linky module) and Block content entity types.

Nodes can reference Linky entities via entity reference fields and embedded wysiwyg links (via linkit).
Nodes can reference blocks via layout builder, or embedded in wysiwyg content (via entity_embed)
Blocks can reference Linky entities via entity reference fields and embedded wysiwyg links.

The chain in this scenario seems to be Node = Top, Block = Middle, Linky = Bottom which will work for the majority of use cases.

However, what about blocks that aren't embedded via layout builder? Such as blocks embedded via entity_embed or blocks that just appear on the site in other locations. Those need to be treated as Top level entities. How would this new architecture denote when a block is a middle or a top level?

We could say reusable blocks are always top level, and inline blocks are always mid level, however what about when a reusable block is embedded via entity_embed? You would want that treated as mid (or maybe top, or maybe both?). This is where it could become too complex/opinionated.

This is just one example using common entity types, but there are endless possibilities.

berdir’s picture

berdir
Primary language German
Location Switzerland
commented

I can't answer for @marcoscano, but I think #3002332: Track composite (e.g. paragraph) entities directly and only as their host from me did influence the plan here. Also note that I didn't actually look at the patch yet, sadly.

I don't think your use cases are that complicated. Yes, the module will be "opinionated", but it's opinions are simple and a "middle entity" is only such in explicit, clear cases:

* Entity Reference Revisions has an entity type flag for what it calls composite entities, which mostly means paragraphs (there might be others, not sure). No questions there, paragraphs are designed as embedded, *single-use* entities.
* Content Blocks in core have a reusable flag, if that's set it is also only used in a single place in a single entity, so it can be considered a middle entity. any other block embed or reference is not.

The decision will be with the plugins and maybe a hook or so customize it.

TMGMT has similar logic for example for composite entities (does not support layout builder yet I think), which works very well for us.

Right now, entity_usage with paragraphs is quite complicated to use, as you need to join/loop through an undefined amount of layers and implement this composite logic at query time. This would massively simplify that. For example, TMGMT right now also doesn't support looking up suggested/related entities to translate through paragraphs, and with this, we could optionally support entity_usage to cover that and all the other things that entity_usage has plugins for. We also have a custom integration that shows embedded media entities so users can translate them, that would get so much simpler and support more use cases.

acbramley’s picture

acbramley commented

@Berdir keen to see how it turns out :)

partdigital’s picture

partdigital commented

One suggested approach that we've been using on our project to handle entity usage:

We created a service that accepts a top entity along with a specification. It would then traverse through the tree and only store the results that we needed based on that specification. As we traversed the tree we would also store the location of each item so that once the usage was captured we could easily traverse that set with methods like getParent(), getChild(), getSibling() etc.

For example, our API looks like this:

We define a specification. It's basically just an array but it could be made into a plugin/config entity and given a name. So that you could define meaningful traversal specifications for your project. You can also simply generate a "default" specification by observing what fields and entity types there are on the site. Though I've usually found it more useful to be more explicit somehow. 

$spec = [
      'page' => [
        'entity_type' => 'node',
        'bundle' => 'page',
        'fields' => [
          'field_entity_reference' => [],
        ],
      ],
      'article' => [
        'entity_type' => 'node',
        'bundle' => 'article',
        'fields' => ['field_entity_reference' => []],
      ],
    ];

We then pass that specification into a method. 

$collection = $service->getReferencedEntitiesCollection($parentEntity, $spec);

Now we can do things like this:

// Get first level of children.
$collection->getChildren();

// Get all children recursively
$collection->getAllChildren();

// Get the immediate parent if the child is known.
$collection->getParent($entity);

// Get all the parents of a known child. 
$collection->getAllParents($entity);

// Get all siblings 
$collection->getSiblings($parent, $entity);

This is very fast because we store the entity id and its location in the set (basically an index). See the example below. The key is the location and the value is the entity id.

[
  '0' => 6
  '0:0' => 4
  '0:1' => 3
  '1' => 5
  '1:0' => 3
  '1:1' => 2
  '1:1:0' => 1
  '2' => 8
  '3' => 9
  '4' => 10
];

To get this working with the broader entity usage, you could:

  • Cache/store each set for each top entity.
  • Create a relationship between the top entity and each child entity so that it's easy to find the set. So in the example above you'd have 10 records.

The api might look like this: 

// Finds the top entity and all its sets with this child. Let's say it's just one collection. 
$collection = $service->findCollection($childEntity);

// This then gets its immediate parent (not the top parent)
$collection->getParent($childEntity);

Just food for thought as you're working on this :)

acbramley’s picture

acbramley commented

Is the plan to still go ahead with this 3.x branch? I see there's now a 4.x branch using entity_track. Surely we should consolidate efforts on a single new architecture?

We use this module pretty heavily on one of our client projects and they've recently asked for features such as filtering the Usage list by current/previous revision so I'm happy to help the efforts in order to unlock so of those more complex features.

marcoscano’s picture

marcoscano
Primary language Portuguese, Brazil
Location Barcelona, Spain
commented

Thanks all who have been providing feedback and ideas to this issue. Apologies for not replying earlier 🙏

@acbramley thanks! I will take any help available :)

Currently I would say that both 3.x and 4.x branch are very much experimental and shouldn't be used on prod. Development on 3.x stalled at some point because I didn't feel good being the only one moving this idea forward (being this such a disruptive architectural change). Then at some point in time @seanb and @askibinski came up with the idea of splitting the API into a generic layer to "track things", and then make Entity Usage just be a consumer of that API, which makes sense to me, but we didn't fully make the switch into this new 4.x branch, and the development kind of stalled.

Yes, I think at this point it makes sense to envision the refactoring mentioned here on top of the 4.x branch. In order for that to happen, I would say that a rough roadmap could be:

ET = Entity Track
EU = Entity Usage

0- [NEEDS WORK] Fix tests in D10 / Switch to GitlabCI #3408387: Fix tests in HEAD for D10
1- [ALMOST DONE ?] review the current code / update the branches with latest commits on EU (entity_usage) 2.x and ensure we have feature parity between ET 1.x + EU 4.x and EU 2.x
- This was kind of OK as of Dec 2022 with #3324787: Update 4.x branch and #3324797: Update with entity_usage 2.x changes but we'd need to review latest bug-fixes since then.
2- [NEEDS REVIEW] ensure that the test coverage of ET 1.x and EU 4.x combined is equivalent of what we have in EU 2.x
- This probably happened as part of the above issues as well, but we'd need to double-check we are not losing test coverage in the switch
3- [NEEDS WORK] ensure we have an upgrade path for existing users on EU 2.x #3326110: Create an upgrade path for EU 2.x -> ET 1.x + EU 4.x
4- [DONE ?] have some real world experience / feedback of ET 1.x + EU 4.x
- I know of one reasonably-sized project that is using ET+EU on prod for a couple years now, but it would be great to get more alpha testers out there if we can.

After this, I believe we could tag a EU 4.0.0-beta1 and mark it as recommended branch instead of 2.x.

Then, it would likely make sense to revisit the refactor from this issue and simplify everything in a 5.x branch probably?

I am OK going forward with this plan and welcome everyone that is able/willing to participate.

Thanks!

claudiu.cristea’s picture

claudiu.cristea
Primary language Romanian
Location Arad 🇷🇴
commented

As there's no move here, and because I badly needed something like the refactoring has envisioned, I had to create a new module which is more or less based on this idea. Enter Track Usages.

Here are some key differences:

  • Track Usages only registers relations between source (top-level) entities and target entities. Traversable (middle-level) entities are traversed but not registered as distinct records/rows. This allows the database to scale, as with Entity Usage, very often the usage table becomes unmaintainable.
  • The configuration is stored in config entities, meaning you can record usages for different scenarios. For instance, you may want to track the relation between nodes and their files, but for a different scope, you may want to track record the relation between node and taxonomy terms. For each scope, you will define a different configuration.
  • This module doesn't offer any UI to end users, its only business is to offer the usages

Posted this comment for those who might be interested.

PS: I needed this kind of functionality to achieve the scope of File Visibility module

eelkeblok’s picture

eelkeblok
he/him
Primary language Dutch
Location Netherlands 🇳🇱
commented

As a new and interesting twist, the project page for Entity Track says

This module is now obsolete
The latest changes in the Entity Usage module have mode this module obsolete. Entity usage now has much more generic support for generic tracking plugins.

However, I am unsure what changes those are. Looking for a way to speed up the indexing process.

vasike’s picture

vasike
Location Ramnicu Valcea
commented
Status: Needs review » Needs work

I tried @claudiu.cristea solution
and I added some MRs for some issues
https://www.drupal.org/project/issues/track_usage
Maybe they could help with some solutions on some projects ... for some people "here"

Also this doesn't look ready for review ... it still "Needs work" ... imho