Maybe with a combination of:

CommentFileSizeAuthor
#11 interdiff_5_11.txt7.42 KBsteven jones
#11 entity_share-automatic_synchronization-WIP-3080629-11.patch32.01 KBsteven jones
#5 entity_share-automatic_synchronization-WIP-3080629-5.patch33.39 KBgrimreaper

Comments

Grimreaper created an issue. See original summary.

fathershawn’s picture

fathershawn
Primary language English
Location New York
CreditAttribution: fathershawn at Digital Pulp commented

Perhaps! - My client wants the update to happen via automation, not just to be notified, however. I will checkout Consumers though to see if it helps with implementation. This line interests me though:

or sending only the CRUDed entity.

At the start our system will have a small number of sites, but if it scales, I either have to throttle the pings from server to client that trigger a pull or change to push to prevent what someone else named a "thundering herd" of requests from client to server.

grimreaper’s picture

grimreaper
Primary language French
Location France 🇫🇷
CreditAttribution: grimreaper at Smile commented
Assigned: Unassigned » grimreaper

  • Grimreaper authored 229450f on 8.x-2.x 
    Issue #3080629 by Grimreaper: Create queue helper service in...
  • Grimreaper authored a7a76f7 on 8.x-2.x
    Issue #3080629 by Grimreaper: Add missing "declare(strict_types = 1);"
grimreaper’s picture

grimreaper
Primary language French
Location France 🇫🇷
CreditAttribution: grimreaper at Smile commented
Assigned: grimreaper » Unassigned
Status: Active » Needs review
StatusFileSize
new entity_share-automatic_synchronization-WIP-3080629-5.patch33.39 KB

And here is a patch for this feature...

Exhausting.

Finally I didn't use the consumers module and I had chosen to rely on the existing entity_share_async sub-module.

What does the patch do?

  • Adds an endpoint in the entity_share_async module to be able to enqueue entities to be synced later
  • Adds a new sub module "entity share notifier"

So:

  • entity_share_async => entity_share_client
  • entity_share_notifier => entity_share_server
  • On the server website, enable entity_share_notifier
  • On the client website, enable entity_share_async
  • On the server website, configure subscriber entities, fill the info to connect to the client website with a user that have the new permission "Access async endpoint".
  • Select which channels to notify
  • Add or edit an entity which entity type, bundle and langcode match one or more channels
  • On the client website, on the pull form you will e able to see enqueued entities
  • On the client website, and see that your entities had been synced

This is a WIP patch for the following reasons:

  • it needs automated tests
  • it needs feedbacks
  • There is still a TODO: Needs to test that the entity is on the channel because otherwise for example, if you have multiple channels on the same entity type, bundle, langcode, but different filters, the entity will be marked for syncing for each channel and so there will be warning logged in entity_share_async if trying to import an entity not found in a channel and you can have undesired results. FOr example if you want to configure a channel filtered on a boolean to distinguish content to sync automatically or not, this filter will not be taken into account.

I am also feeling that entity share is reaching slowly a point of unmaintainability with all those services cross dependent. #3060694: Rework service and tests is becoming critical.

Thanks for the review and feedbacks!

mrpauldriver’s picture

mrpauldriver CreditAttribution: mrpauldriver commented

I am very new to entity_share, so please excuse me if I am missing something obvious.

The subscriber form requires a Remote ID

The remote ID used on the subscriber website to pull this server.

How to determine the ID to use?

grimreaper’s picture

grimreaper
Primary language French
Location France 🇫🇷
CreditAttribution: grimreaper at Smile commented

Hello,

The remote ID is the machine name of the remote website config entity on your client website.

mrpauldriver’s picture

mrpauldriver CreditAttribution: mrpauldriver commented

Thanks. Is there an easy way discovering this, rather diving into yml files?

grimreaper’s picture

grimreaper
Primary language French
Location France 🇫🇷
CreditAttribution: grimreaper at Smile commented

I don't know your setup, but if you don't have access to the client website BO, yes, only looking at yml files is the solution.

This question is out of the scope of the issue queue.

grimreaper’s picture

grimreaper
Primary language French
Location France 🇫🇷
CreditAttribution: grimreaper at Smile commented
Version: 8.x-2.x-dev » 8.x-3.x-dev
steven jones’s picture

steven jones CreditAttribution: steven jones as a volunteer commented
StatusFileSize
new entity_share-automatic_synchronization-WIP-3080629-11.patch32.01 KB
new interdiff_5_11.txt7.42 KB

Here's an updated patch that takes the excellent work in #5 and extends it to add in the new import config stuff, and also allows it to work with entities where language translation is not available.

grimreaper’s picture

grimreaper
Primary language French
Location France 🇫🇷
CreditAttribution: grimreaper at Smile commented

Hello @Steven Jones,

Thanks a lot for the updated patch! This will greatly help when going back on this issue.

Currently not the priority, so not reviewing/testing.

Regards,

grimreaper’s picture

grimreaper
Primary language French
Location France 🇫🇷
CreditAttribution: grimreaper at Smile commented
Status: Needs review » Postponed

Hello,

Changing the status into "Postponed", waiting for https://www.drupal.org/project/entity_share_websub to have a release to test.

grimreaper’s picture

grimreaper
Primary language French
Location France 🇫🇷
CreditAttribution: grimreaper at Smile commented
Issue tags: +Europe2020
grimreaper’s picture

grimreaper
Primary language French
Location France 🇫🇷
CreditAttribution: grimreaper at Smile commented
Related issues: +#3192103: Channel subscription

I have compared again patch from this issue then Entity Share Websub and I think the only point preventing me from marking this issue as won't fix/works as designed is that Entity Share Websub is missing #3192103: Channel subscription.

Otherwise I think Entity Share Websub has a better handling of this use case.

grimreaper’s picture

grimreaper
Primary language French
Location France 🇫🇷
CreditAttribution: grimreaper at Smile commented 
  1. +++ b/modules/entity_share_notifier/src/Form/EntityShareSubscriberForm.php
@@ -0,0 +1,153 @@
+    $form['basic_auth'] = [

    If this is an auth method like in the remote config entity, this shouls use the same mecanism now that authentication is pluggable. 

  2. +++ b/modules/entity_share_notifier/src/HookHandler/EntityHookHandler.php
@@ -0,0 +1,228 @@
+      ->getStorage('channel')

    Filter the channels based on the langcode, entity type and bundle before loading the channels if possible to avoid loading hundreds of entities depending on the websites. 

  3. +++ b/modules/entity_share_notifier/src/HookHandler/EntityHookHandler.php
@@ -0,0 +1,228 @@
+      // TODO. Will require to add a new field on channel to store the user to

    Or this will require to have an array of channel_id, user_id in the subscriber config entity instead of an array of channel IDS. 

  4. +++ b/modules/entity_share_notifier/src/HookHandler/EntityHookHandler.php
@@ -0,0 +1,228 @@
+      ->loadMultiple();

    Possible to add a condition on the channels based on $channels_to_notify? To avoid loading unnecessary entities. 

  5. +++ b/modules/entity_share_notifier/src/HookHandler/EntityHookHandler.php
@@ -0,0 +1,228 @@
+      $http_client = $this->clientFactory->fromOptions([

    Should we require that a subscriber should reference a remote to benefit from the authentication system? And so we can use the remote manager service to get the HTTP client.

beerendlauwers’s picture

beerendlauwers CreditAttribution: beerendlauwers at Netvlies commented

EDIT: I noticed you were talking about another part of the code. This part goes inside the channels loop.

I wrote some code a long while ago for the channel filtering. This goes before $channels_to_notify[] = $channel->id();.

It's pretty rough, of course. Ideally, the operators would plug in to an entityQuery, but I'm not sure if the semantics of JSON:API and entityQuery coincide. Alternatively, we could have a plugin for each filter operator and write the needed logic there.

      $filters = $channel->get('channel_filters');

      $passedFilters = true;
      foreach ($filters as $filter) {
        $directField = $filter['path'];
        if ($entity->hasField($directField)) {

          // Only checks first value right now.
          $firstValueFromFilter = trim((string)$filter['value'][0]);

          if ($filter['operator'] === '=') {
            $fieldValue = trim($entity->get($directField)->getString());

            if ($fieldValue !== $firstValueFromFilter) {
              $passedFilters = false;
            }
          } elseif ($filter['operator'] === 'CONTAINS') {
            if (!Arrays::any($entity->get($directField)->getValue(), function (
              $element
            ) use ($firstValueFromFilter) {
              if (isset($element['value'])) {
                $fieldValue = $element['value'];
              }

              return $fieldValue === $firstValueFromFilter;
            })) {
              $passedFilters = false;
            }
          }
        }
      }

      if (!$passedFilters) {
        continue;
      }
beerendlauwers’s picture

beerendlauwers CreditAttribution: beerendlauwers at Netvlies commented

For number 5, we could load it from the remote_id:

      $remote_id = $subscriber->get('remote_id');
      $remote_config_id = $subscriber->get('remote_config_id');
      $remote = $this->entityTypeManager->getStorage('remote')->load($remote_id);

      if (!($remote instanceof RemoteInterface)) {
          throw new \LogicException("Could not load remote with ID $remote_id");
      }

      $http_client = $remote->getHttpClient(true);

This could also be done in the constructor if you feel that's cleaner.