Entity Synchronization

The Entity Synchronization module provides a framework for exchanging data between a Drupal application and one or more systems, internal or external, Drupal or not.

This is an API/Framework module; it does not provide functionality out of the box and, until more features are added, it requires some custom code to make it work for you.

Multiple remotes

Entity Synchronization can be set up to exchange data with multiple remote systems. Remote systems can be other Drupal sites, proprietary APIs or external systems such as CRM, ERP, CMS, or even mailing systems. It can also be configured to transform and synchronize data between different parts of the same Drupal application, or to migrate data from Drupal 7 to Drupal 11.

Modular architecture

Synchronizations can be defined as Drupal configuration entities, that together with an API client define the method of data exchange. API client adapters can be used, for example, to:

• Import or export entities via API calls.
• Import or export data to CSV files or spreadsheets.
• Send or receive data via a message queue.

Ecosystem

There's a number of modules that already provide integrations and other features building on top of Entity Synchronization. They can serve as guides to implementing your own integration.

• Acumatica
• Commerce Amazon MWS
• Entity Synchronization CSV
• Entity Synchronization Mailgun
• Entity Synchronization OData
• Microsoft Dataverse

Feature overview

Import operations

• Import an individual remote entity and map it to an existing local entity or create a new one.
• Import a list of remote entities and map them to existing local entities or create new ones.
• Import an individua local entity i.e. fetch updates from its corresponding remote entity.
• Import a list of local entities i.e. fetch updates from their corresponding remote entities (in progress).

Managed imports

Entity Synchronization does the heavy lifting for you. You can schedule an import to run on cron and the module will do the following, depending on configuration:

• Run a full import of all entities.
• Import a list of recently created or updated entities.
• Run partial consecutive imports of recently created or updated entities, with an import starting where the previous import stopped. This helps to prevent problems that can arise with importing hundreds of thousand or millions of entities at once.
• A basic locking mechanism that prevents concurrent imports, as that can cause problems in many cases.

Export operations

• Export an individual local entity and map it to an existing remote entity or create a new one.
• Export a list of local entities and map them to existing remote entities or create new ones (in progress).
• Export an individual remote entity i.e. send updates from its corresponding local entity (in progress).
• Export a list of remote entities i.e. send updates from their corresponding local entities (in progress).

Managed Exports

Entity Synchronization can automatically queue local entities to be exported whenever they are created or updated. Automatic field change detection so that exports are triggered only when relevant fields change.

Import/export loops

We have worked on ways to prevent import/export loops i.e. an export changes the entity on the remote, which triggers an import and update the local entity, which then triggers an export etc.

Field mapping

The data in exchange are essentially data stored in entity fields. Field mapping between local and remote entities can be defined as configuration, with data transformation done via plugins, custom callbacks. Pipelines of data transformations with validation rules are supported as well.

As many entities as desired can be configured to be imported and or exported, each with its own field mapping. It is possible to set up one remote entity to be split and mapped to two local entities and vice versa e.g. a remote User entity is partly mapped to a local User entity and partly to a local Profile entity.

Dependent synchronizations

In some cases you want to make sure that some entities are imported before others. For example, a taxonomy term should be imported before an entity that references that term. We've got your back! Entity Synchronization supports dependent imports/exports where an operation triggers or is triggered by another. Currently, this works with event subscribers. Some more work is required though to make it just configuration.

Session management

The Entity Synchronization Session submodule provides a foundation for managing the connections to the remote systems e.g. using the OAuth2 protocol.

Asynchronous operations

The Entity Synchronization Async submodule provides tools for connecting to remote systems that work asynchronosly e.g. put requests on a message bus and require you to collect the results later.

On the Drupal side, operations can be executed either asynchronously via queues, or at the end of requests that resulted in entity changes - so that end users are not affected by the execution time.

Why not Migrate

In our experience, the Migrate module has been architected with one-time imports of data into Drupal sites in mind, even though it can be pushed to cover some more cases. The Entity Synchronization module tries to solve the problem of continuous, potentially real-time data exchange between multiple systems and that includes both imports and exports. I can cover the cases that the Migrate module covers as well.

Development status

The module has been in use for years on a number of enterprise production environments; it is stable and reliable. A stable release will be made after more test coverage and documentation has been added.

Contributing

As a developer, you can submit merge requests. The module follows the Krystal coding standards that adds a set of rules on top of the Drupal coding standards mainly to improve readability.

If you are not a developer, you can help with testing and with issues in the issue queue. If you are interested in sponsoring this module to add more features, contact the module maintainers.