See #2748609: [meta] Preserving auto-increment IDs on migration is fragile.

Problem/Motivation

Preserving serial IDs in the upgrade process (e.g., source site nid 53 => destination site nid 53) means that if any content with the same ID is manually created on the destination side before upgrade, it will be overwritten with data from the source. In case users don't take proper precautions before performing a migration, we could try to detect at import time if the incoming source data would overwrite pre-existing non-migrated content.

Proposed resolution

Add a process plugin to vulnerable migrations which checks to see if the ID being migrated already exists on the destination side and was not migrated (is not in the migration map table). If the source entity would overwrite a non-migrated entity, throw an error.

  1. Maintain data integrity
    Destination nodes will not be overwritten. Some source records will not be imported.
  2. No surprises
    If someone hasn’t read the docs, they may be surprised that they only got a partial migration.
  3. Provide a path forward
    Difficult path forward - manually migrate the content that got rejected?
  4. Preserve URLs (e.g., node/17).
    Only for the content which was successfully imported. If you’ve manually created, say, node/17, then node/17 on the destination site will show different content from node/17 on the source site.
  5. Minimize technical debt.
    A fairly straight-forward process plugin.
  6. Minimize effort to implement.
    A fairly straight-forward process plugin.

Remaining tasks

Implement it.

User interface changes

None.

API changes

None.

Data model changes

None.

Comments

mikeryan created an issue. See original summary.

cilefen’s picture

cilefen CreditAttribution: cilefen commented
Priority: Normal » Major

@xjm, @alexpott, @effulgentsia, @lauriii, @catch and I discussed this issue at a recent meeting and agreed that since the parent is critical, we would make each potential solution major priority.

rakesh.gectcr’s picture

rakesh.gectcr
he/him
Primary language Malayalam
Location Manchester
CreditAttribution: rakesh.gectcr as a volunteer and at Valuebound commented
Assigned: Unassigned » rakesh.gectcr

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.0-alpha1 will be released the week of July 31, 2017, which means new developments and disruptive changes should now be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

rakesh.gectcr’s picture

rakesh.gectcr
he/him
Primary language Malayalam
Location Manchester
CreditAttribution: rakesh.gectcr as a volunteer and at Valuebound commented

Isn't similar to https://www.drupal.org/node/2876085 or duplicate?

quietone’s picture

quietone CreditAttribution: quietone at Acro Commerce commented

Similar in intent, provide useful information to the user about ID conflicts.

#2876085: Before upgrading, audit for potential ID conflicts does that by auditing migrations before they are run. I've been testing that via the UI and it will display a message if there is existing content on the destination site. Whereas, here the proposed resolution is to add a process plugin that will throw and error if there is an ID conflict.

I do wonder if the work on the other issue has anything to inform or change this issue.

rakesh.gectcr’s picture

rakesh.gectcr
he/him
Primary language Malayalam
Location Manchester
CreditAttribution: rakesh.gectcr as a volunteer and at Valuebound for Acro Commerce commented

@quietone, Thank you for the confirmation.

rakesh.gectcr’s picture

rakesh.gectcr
he/him
Primary language Malayalam
Location Manchester
CreditAttribution: rakesh.gectcr as a volunteer and for Acro Commerce commented

Just confirming, EntityExists process plugin we already have it. Can't we use that, instead of creating a new one?

If we are creating a new one, what should be the name?

rakesh.gectcr’s picture

rakesh.gectcr
he/him
Primary language Malayalam
Location Manchester
CreditAttribution: rakesh.gectcr as a volunteer and for Acro Commerce commented
Status: Active » Needs review
rakesh.gectcr’s picture

rakesh.gectcr
he/him
Primary language Malayalam
Location Manchester
CreditAttribution: rakesh.gectcr as a volunteer and for Acro Commerce commented

I am also seeing that, there can be some difference Because we are planning to search migration map table and Entity exists. Still need some more confirmation and clarification to convince.

quietone’s picture

quietone CreditAttribution: quietone at Acro Commerce commented

Yes, I think entity_exists will do the job.

heddn’s picture

heddn
Primary language English
Location Nicaragua
CreditAttribution: heddn at MTech, LLC, Acro Commerce commented

I think this is duplicate of #2876085: Before upgrading, audit for potential ID conflicts. Should this be closed duplicate?

cilefen’s picture

cilefen CreditAttribution: cilefen at Institute for Advanced Study commented

Not as per #6.

heddn’s picture

heddn
Primary language English
Location Nicaragua
CreditAttribution: heddn at MTech, LLC, Acro Commerce commented

We have 'entity_exists', which returns the ID or FALSE and 'log' process plugins. We also have skip_on_empty. If you chain these together, then it will skip and you can log things. So, I think this could be closed as outdated. Since not all of these things existed when this was opened.

heddn’s picture

heddn
Primary language English
Location Nicaragua
CreditAttribution: heddn at MTech, LLC, Acro Commerce commented
Priority: Major » Normal
Status: Needs review » Active

Dropping priority here. With the critical blocker in core now, this isn't as important. If it is necessary, the building blocks for doing this already exist. And as there is no patch, changing status.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.0-alpha1 will be released the week of January 17, 2018, which means new developments and disruptive changes should now be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.6.x-dev » 8.7.x-dev

Drupal 8.6.0-alpha1 will be released the week of July 16, 2018, which means new developments and disruptive changes should now be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

quietone’s picture

quietone CreditAttribution: quietone commented
Assigned: rakesh.gectcr » Unassigned

@rakesh.gectcr, Hi! How are you? I'm doing some triage and since you haven't gotten to this I am going to un-assign you.

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.