Problem/Motivation

Note: this may not be the best place for it, but I'd hate to make such a small module for this.

Nearly every Drupal site that I work with has entities which are imported from an external source and I have to leverage some sort of mapping. Many times this is done via Migrate and it's mapping tables. Sometimes it is not and I end up following the same trick: entity field to store the remote ID and storage methods to find synced/unsynced entities.

I'd like to build an API for finding entities based on a source identifier.

Proposed resolution

  • Create a new field type that so that the schema can have an index set. It should also have two properties so that we can reference the source ID and a version, if the remote source supports versions/revisions. This way local entities can be revisioned alongside the source data. The source ID is the main property, so it can work without the version value.
  • A new entity key called source to specify the field which stores the source ID.
  • A new entity repository service that allows loading an entity based on a source ID, just like loadEntityByUuid. Except it uses the source key.

Remaining tasks

Get a "go" or "no go" of scope for this in the Entity module.

User interface changes

None.

API changes

Addition of a way to look up entities based on a source identifier.

Data model changes

None, since it's an opt-in process for developers.

Issue fork entity-3269526

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

mglaman created an issue. See original summary.

mglaman’s picture

Based on #3255187: Drop support for Drupal 8 this should go in its own module. Maybe entity_remote_source

mglaman’s picture

Status: Active » Closed (won't fix)
aaronmchale’s picture

If such a module is created, please link it here, I may be interested in using it.

mglaman’s picture

It'll definitely be created, I just need to think of a proper module namespace for it. I don't want it to be entity_remote_source as it implies that it may do more than provide a "pointer" API for remote associations. I currently have entity_source_id but that's not exactly settled. I have it locally, tested, etc. I just need to be happy with a name and push it.

aaronmchale’s picture

I'll have a think as well, maybe ping you on Slack if I come up with any good names 👍

mglaman’s picture

Awesome! I'll ping you as well if anything comes to mind.

mglaman’s picture

Assigned: Unassigned » mglaman
Status: Closed (won't fix) » Active

I'm re-opening this and going to push it forward here. I'll have an MR next week.

mglaman’s picture

Status: Active » Needs review

Okay, I've pushed the code and tests. Tests give us 100% reported coverage for the field type and repository.

I've added a default widget which allows to read-only of the field values, for debugging purposes. I need to create a field formatter. But it's ready for review.

mglaman’s picture

I'm generating 100,000 nodes with a UUID field and this field to provide some query performance notes.

Also, should provide a generate sample values module

  /**
   * {@inheritdoc}
   */
  public static function generateSampleValue(FieldDefinitionInterface $field_definition) {
    $uuid = \Drupal::service('uuid');
    $values['id'] = $uuid->generate();
    $values['version'] = $uuid->generate();
    return $values;
  }
mglaman’s picture

Base fields added to nodes

$fields['source_uuid'] = BaseFieldDefinition::create('uuid')
  ->setReadOnly(TRUE);
$fields['source_id'] = BaseFieldDefinition::create('entity_source_id')
  ->setReadOnly(TRUE);

UUID field item

EXPLAIN SELECT * from node_field_data WHERE node_field_data.source_uuid = '5f81bdae-2f34-48f0-91d1-eb97fbd418f6';
+------+-------------+-----------------+-------+--------------------------------+--------------------------------+---------+-------+------+-------+
| id   | select_type | table           | type  | possible_keys                  | key                            | key_len | ref   | rows | Extra |
+------+-------------+-----------------+-------+--------------------------------+--------------------------------+---------+-------+------+-------+
|    1 | SIMPLE      | node_field_data | const | node_field__source_uuid__value | node_field__source_uuid__value | 131     | const |    1 |       |
+------+-------------+-----------------+-------+--------------------------------+--------------------------------+---------+-------+------+-------+
1 row in set (0.000 sec)

SourceItem

EXPLAIN SELECT * from node_field_data WHERE node_field_data.source_id__id = 'a4b2dccc-ee92-45c2-9c15-8aba87b8b72e';
+------+-------------+-----------------+------+----------------------------------+----------------------------------+---------+-------+------+-----------------------+
| id   | select_type | table           | type | possible_keys                    | key                              | key_len | ref   | rows | Extra                 |
+------+-------------+-----------------+------+----------------------------------+----------------------------------+---------+-------+------+-----------------------+
|    1 | SIMPLE      | node_field_data | ref  | node_field__source_id__source_id | node_field__source_id__source_id | 258     | const |    1 | Using index condition |
+------+-------------+-----------------+------+----------------------------------+----------------------------------+---------+-------+------+-----------------------+
1 row in set (0.000 sec)

EXPLAIN SELECT * from node_field_data WHERE node_field_data.source_id__id = 'a4b2dccc-ee92-45c2-9c15-8aba87b8b72e' AND node_field_data.source_id__version = '20ef49af-78be-4a89-8fc0-c9a0c78de4a8';
+------+-------------+-----------------+------+----------------------------------+----------------------------------+---------+-------------+------+-----------------------+
| id   | select_type | table           | type | possible_keys                    | key                              | key_len | ref         | rows | Extra                 |
+------+-------------+-----------------+------+----------------------------------+----------------------------------+---------+-------------+------+-----------------------+
|    1 | SIMPLE      | node_field_data | ref  | node_field__source_id__source_id | node_field__source_id__source_id | 516     | const,const |    1 | Using index condition |
+------+-------------+-----------------+------+----------------------------------+----------------------------------+---------+-------------+------+-----------------------+
1 row in set (0.001 sec)

mglaman’s picture

Since code statements wrap and don't overflow, I've added the data into the MR https://git.drupalcode.org/project/entity/-/merge_requests/11

mglaman’s picture

Status: Needs review » Needs work

Chatted with @eiriksm. He brought up shoring up the testing more.

For instance, let's say someone already uses a textfield with modified schema to have an index and then decides to use this field. We should verify the following code still works:

$term->set('remote_id', $remote_id)
$entity_query->condition('remote_id', $remote_id)

Since we have the main property name set, it should just work. But let's get that tested!