Entity Synchronization

The Entity Synchronization module provides a framework for exchanging data between Drupal and one or more remote systems.

This is an API/Framework module; it does not provide functionality out of the box and, until more features are added, it requires some custom code to make it work for you.

Multiple remotes

Entity Synchronization can be set up to exchange data with multiple remote systems. Remote systems can be other Drupal sites, proprietary APIs or external systems such as CRM and ERP systems.

It is essentially tailored for the enterprise that needs to exchange data between multiple systems. It can be used though by other modules to easily create modules that integrate Drupal with other systems.

Pluggable architecture - providers

Providers can be written in contrib or custom modules to define the method of data exchange. We have currently used it with a proprietary API provider (adapter) in an enterprise setting, but contrib modules can be written to implement providers for the following cases, for example:

• An SDK that makes calls to an API.
• Provider that imports/exports data from/to files.
• Provider that sends/receives data via a message queue.

Import operations

• Import an individual remote entity and map it to an existing local entity or create a new one.
• Import a list of remote entities and map them to existing local entities or create new ones.
• Import an individua local entity i.e. fetch updates from its corresponding remote entity.
• Import a list of local entities i.e. fetch updates from their corresponding remote entities (in progress).

Managed imports

Entity Synchronization does the heavy lifting for you. You can schedule an import to run on cron and the module will do the following, depending on configuration:

• Run a full import of all entities.
• Import a list of recently created or updated entities.
• Run partial consecutive imports of recently created or updated entities, where an import starts where the previous import stopped. This helps to prevent problems that can arise with importing hundreds of thousand or millions of entities at once.
• A basic locking mechanism that prevents concurrent imports, as that can cause problems in many cases.

Export operations

• Export an individual local entity and map it to an existing remote entity or create a new one.
• Export a list of local entities and map them to existing remote entities or create new ones (in progress).
• Export an individual remote entity i.e. send updates from its corresponding local entity (in progress).
• Export a list of remote entities i.e. send updates from their corresponding local entities (in progress).

Managed Exports

Entity Synchronization can automatically queue local entities to be exported whenever they are created or updated.

Import/export loops

We have worked on ways to prevent import/export loops i.e. an export changes the entity on the remote, which triggers an import and update the local entity, which then triggers an export etc. Working, but some more work could make the system more reliable.

Field mapping

The data in exchange are essentially data stored in entity fields. Field mapping between local and remote entities can be defined as configuration. Custom callbacks can handle special field types (plugins in progress).

As many entities as desired can be configured to be imported and or exported, each with its own field mapping. It is possible to set up one remote entity to be split and mapped to two local entities and vice versa e.g. a remote User entity is partly mapped to a local User entity and partly to a local Profile entity.

Dependent synchronizations

In some cases you want to make sure that some entities are imported before others. For example, a taxonomy term should be imported before an entity that references that term. We've got your back! Entity Synchronization supports dependent imports/exports where an operation triggers or is triggered by another. Some more work is required though to make it just configuration.

Configuration

Entity Synchronization endeavours to do all the heavy lifting for you; write a client adapter for your provider (if one doesn't exist), configure, done. More work and testing is required to reach that ideal state though.

Why not Migrate

In our experience, the Migrate module has been architected with one-time imports of data into Drupal sites in mind, even though it can be pushed to cover some more cases. The Entity Synchronization module tries to solve the problem of continuous, potentially real-time data exchange between multiple systems and that includes both imports and exports.

Development status

The module is currently tested in an enterprise setting and it is fairly stable; however, more work and testing is required to reach a stable release and there may be small API changes until an beta release is made.

A fair amount of unit tests has been written, even though we would like to increase code coverage to test all cases.

Note on branches

Projects and contrib modules using Entity Synchronization for the first time should start with branch 4.x

Contributing

The module follows the Krystal coding standards that adds a set of rules on top of the Drupal coding standards mainly to improve readability.

If you are not a developer you can open support issues or test existing issues in the issue queue.