Problem/Motivation

It is typical and normal that a site cannot migrate into D8 all in one migration. Most of the site is migrated on one date. Then a few days later a final cut-over is declared and all content that wasn't migrated previously is caught up. This isn't all that hard to do. Run drush ms and see what things are unprocessed and run those individual migrations another time.

There are two problems with this:

1. With the user picture migration, the source database does not have entity IDs, so newly created IDs on the destination site can then conflict with other migrations that do (like files), this has a dedicated issue at #2826047: 6-8 user picture migration must create new fids, which can conflict with fids used by other migrations.

2. With nodes and similar entities, after the initial migration, it's possible for entities to be created both on the source and destination sites. When this happens, it's impossible then to correctly preserve IDs - since the same numeric ID represents a different entity.

Audiences to consider

  1. Non-technical users of the core UI
  2. Drush novices - simply running an upgrade command
  3. Technical users doing custom migrations

Requirements to consider

  1. Maintain data integrity
  2. No surprises
  3. Provide a path forward
  4. Preserve URLs (e.g., node/17).
  5. Minimize technical debt.
  6. Minimize effort to implement.

Proposed resolution

Possible blockers to things are:

#2818147: Use Migration process plugin in *all* places
#2890690: MigrateLookup plugin has inconsistent return values.
#2891073: MigrationLookup doesn't create stub when there's multiple migrations.
#2890844: Migration Lookup plugin does not return multiple values when matched

Possible solutions

Pre-requisite - thorough documentation on drupal.org, and a clear description on the preparatory screen of the core UI, of the risks associated with maintaining IDs, and describing approaches - avoid adding manual content until after all migration is done, doing a custom migration which does not preserve IDs, ... This is necessary regardless of any additional technical approaches.
Documentation only
We rely entirely on documenting the issues.

  1. Maintain data integrity
    Risk: People don’t pay attention to the documentation and screw themselves.
  2. No surprises
    Risk: If they’ve ignored the documentation, the results will be very surprising.
  3. Provide a path forward
    In the documentation, if they go back and read it.
  4. Preserve URLs (e.g., node/17).
    Yes.
  5. Minimize technical debt.
    No new code to maintain.
  6. Minimize effort to implement.No new code to implement.

Pre-upgrade audit See #2876085: Before upgrading, audit for potential ID conflicts
Run a process before the actual import which identifies conflict risks - cases where an ID in the content to be imported matches an ID that already exists in the destination.

  1. Maintain data integrity
    If this is a required step, the user can choose whether to proceed and overwrite any conflicts, or take an alternative approach (per the docs).
  2. No surprises
    They are told exactly what content could be affected by conflicts.
  3. Provide a path forward
    Yes.
  4. Preserve URLs (e.g., node/17).
    Yes.
  5. Minimize technical debt.
    Audit code should be reasonably isolated and maintainable.
  6. Minimize effort to implement.
    We’d need to first do what the import process normally does, identify and instantiate the necessary migrations. For each migration, we need to identify the destination ID field, see if that field is mapped in the process section, look for any of destination ID values which are not in the migration’s map table, and see if any of *those* IDs are in the source. Oh, but we need to consolidate this across all migrations to the same entity type… Ugh.

Manipulate auto-increment values See #2876086: Manipulate auto-increment values to avoid conflicts
The assumption here is that the migration is run on a clean (content-free) site, and each migration when complete sets “the” table for the destination entity to have an auto-increment a “safe” distance beyond the highest migrated ID.

  1. Maintain data integrity
    Maintains integrity, with the small risk of underestimating how far to bump the auto-increment (too much content added on the source system).
  2. No surprises
    Yes.
  3. Provide a path forward
    No path forward needed.
  4. Preserve URLs (e.g., node/17).
    Yes.
  5. Minimize technical debt.
    API addition to database system. Complex and fragile implementation (see below).
  6. Minimize effort to implement.
    Needs a database schema API for retrieving/setting auto-increment. For each migration, we need to identify the destination ID field, see if it’s being mapped, and figure out what table contains the auto-increment ID.

Identify conflicts during import See #2876090: Warn if migrated content will overwrite manually-created content
Add a process plugin to vulnerable migrations which checks to see if the ID being migrated already exists on the destination side and was not migrated (is not in the migration map table). If the source entity would overwrite a non-migrated entity, throw an error.

  1. Maintain data integrity
    Destination nodes will not be overwritten. Some source records will not be imported.
  2. No surprises
    If someone hasn’t read the docs, they may be surprised that they only got a partial migration.
  3. Provide a path forward
    Difficult path forward - manually migrate the content that got rejected?
  4. Preserve URLs (e.g., node/17).
    Only for the content which was successfully imported. If you’ve manually created, say, node/17, then node/17 on the destination site will show different content from node/17 on the source site.
  5. Minimize technical debt.
    A fairly straight-forward process plugin.
  6. Minimize effort to implement.
    A fairly straight-forward process plugin.

Remaining tasks

User interface changes

API changes

Data model changes

CommentFileSizeAuthor
#15 2748609.patch4.52 KBcatch

Comments

heddn created an issue. See original summary.

heddn’s picture

Issue summary: View changes
mikeryan’s picture

Title: File migration breaks user profile migration (or vice versa) » Preserving auto-increment IDs on migration is fragile

The root problem is that preserving IDs is inherently fragile. For any entity (or other data with auto-incrementing integer IDs), if a single item is created on the D8 site other than by copying it over directly with its ID, a collision risk is created. In this instance, entirely within the default migration process itself, since the D6 user picture migration creates file entities that did not have corresponding fids on D6, a subsequent file migration will collide with those auto-incremented fids. The problem could just as easily occur manually - after initial migration you create a user account on the D8 side for, say, a consultant helping with the site-build, and when you run an incremental update some new user added on the D6/D7 side will collide with that account, overwriting it. Or, you try to get a head-start on site-building and add a page - boom.

So, what can be done about this?

  1. Document that if you plan on doing incremental migrations, DO NOT create stuff in D8 (although that doesn't address the original problem here, which is within/among the migrations themselves).
  2. Set ridiculously high auto-increment values on the D8 tables so preserved IDs won't collide with new ones.
  3. ?
heddn’s picture

Some logic exists in DedupeBase. But it defaults to adding _0, _1, _2, etc to a string value. Rather, what we want is to dedupe by adding to a serial entity id. And convert 100 => 101, 102, 103, etc. Instead of 100_0, 100_1, 100_2, etc.

While perhaps not the best in performance, if we passed in the actual fid as a starting point, the exists check in theory wouldn't have to go very far. Since content entry of new stuff probably hasn't created all *that* much new content. This, of course, falls down if many thousands of users and their profile photo are migrated.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.0-beta1 was released on August 3, 2016, which means new developments and disruptive changes should now be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

catch’s picture

Set ridiculously high auto-increment values on the D8 tables so preserved IDs won't collide with new ones.

Could we set this after the migrations maybe? So set the autoincrement to $current_autoincrement + max($total_entities / 10, 100) or something.

Then if you migrate 100 entities, you'd have autoincrement id of 200, and if you migrate 1 million, 1,100,000. Not pretty but would solve most cases here.

alexpott’s picture

I also think we need to add comments to the core migrations so that if people copy the migrations to build custom migrations they know what they are letting themselves in for if they don't comment out the ID migrations in things like core/modules/node/migration_templates/d7_node.yml. Something like:

diff --git a/core/modules/node/migration_templates/d7_node.yml b/core/modules/node/migration_templates/d7_node.yml
index b763534..6036d46 100644
--- a/core/modules/node/migration_templates/d7_node.yml
+++ b/core/modules/node/migration_templates/d7_node.yml
@@ -6,6 +6,8 @@ deriver: Drupal\node\Plugin\migrate\D7NodeDeriver
 source:
   plugin: d7_node
 process:
+  # If you are using this file to build a custom migration consider commenting
+  # out the nid and vid migrations to allow incremental migrations.
   nid: nid
   vid: vid
   langcode:

Because I've seen projects that didn't know to do this.

mikeryan’s picture

Issue tags: +Migrate critical

Given we have two migrate-criticals depending on this, marking this as well.

mikeryan’s picture

The current proposal (from catch) is:

Could we set this after the migrations maybe? So set the autoincrement to $current_autoincrement + max($total_entities / 10, 100) or something.

Per catch's reply at https://www.drupal.org/node/2818155#comment-11756847:

Right but I think bumping the auto-increment ID before the user picture starts is a different fix to bumping it after other ID-to-ID migrations run - it'd be specific to this one.

So, if we were to do this, what would be the approach? What are the challenges?

  1. How do we portably query and modify the auto-increment value across database engines? We would need to add new public APIs to the database schema API, wouldn't we?
  2. Precisely when and where would the modification be made? In a POST_IMPORT event handler for the key migratons? It will need to be smart enough to do this only after the full completion of the first full import (keep UI batching in mind here). And, of course, it needs to recognize when IDs are being preserved, and what the primary table and its ID column are - this knowledge, I think, would need to be pretty much hard-coded into the affected migrations.
  3. As suggested in #2818155: Add comments to d6_file stating the conflicts with d6_user_picture_file above, d6_user_picture_file would need to do this before rather than after (or, the d6_file post-import bump would need to account for potential user pictures).
alexpott’s picture

I'm pretty sure that #2818155: Add comments to d6_file stating the conflicts with d6_user_picture_file means that we need to set the auto-increments prior to being the migration. They way anything extraneous created during a migration will get a non-clashing ID.

mikeryan’s picture

So, even if we are to pursue trying to manipulate auto-increment values dynamically, implementing that is going to take some time. For now, let's at least do:

Document that if you plan on doing incremental migrations, DO NOT create stuff in D8 (although that doesn't address the original problem here, which is within/among the migrations themselves).

Child issue #2824671: Document dangers of preserving IDs on migration created for this.

heddn’s picture

I've cross-posted in #2826742: Expose migration types in Migrate UI that it could possibly help lessen the impact of this problem by giving some more flexibility on when you have to hit the button on migrating over content.

catch’s picture

Priority: Major » Critical

Going to bump this to critical independently since it's one of the last known data integrity issues with migrate.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.0-alpha1 will be released the week of January 30, 2017, which means new developments and disruptive changes should now be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

catch’s picture

Issue summary: View changes
Status: Active » Needs review
StatusFileSize
new4.52 KB

Here's a proof of concept implementing #6.

Adds a method to Database\driver\mysql\Schema to allow setting the autoincrement on a table. This will need a new interface, and postgres and sqlite implementations if the idea itself is OK, and test coverage in the database tests. The postgres equivalent is ALTER SEQUENCE - we already have an ALTER SEQUENCE when we're renaming tables/columns, so the logic can likely be figured out from that.

The database API bits may need splitting out to another issue before this one is committed, but I think we should keep things here for now since it's very unlikely we'll have another use case in core apart from this.

Adds methods to SqlContentEntityStorage.php to set the autoincrement on the base and revision tables. This is a bit specific but couldn't really think of anywhere else to put that logic - we don't want the table names leaking out of here.

Makes EntityContentBase migrate plugin implement ImportAwareInterface, then queries the highest ID in the base/revision tables and sets the autoincrement there. This seemed the most appropriate place to do it to me, but defer to migrate folks if there's somewhere better. Also I didn't try to do it conditionally on whether the migration is preserving IDs or anything yet.

Obviously very rough but should be enough for discussion at least. The ContentEntityBase kernel test passes, but I didn't run any others.

alexpott’s picture

Issue tags: +Triaged D8 critical

Discussed with @catch, @cilefen, @effulgentsia and @xjm. We agreed that this should be a critical issue since regardless of the experimental nature of migrate this is a data integrity issue that can result in data loss. We also agreed that @catch's solution looks like a good basis for an acceptable work-around since a complete solution is likely to be impossible.

catch’s picture

Bit of self-review.

  1. +++ b/core/lib/Drupal/Core/Database/Driver/mysql/Schema.php
    @@ -598,6 +598,17 @@ public function fieldExists($table, $column) {
     
    +  /**
    +   * Set the autoincrement value on a table.
    +   */
    +  public function setAutoIncrement($table, $value) {
    +    if (!is_numeric($value)) {
    +      throw \InvalidArgumentException('AUTO_INCREMENT may only be set to an integer.');
    +    }
    

    Needs proper validation, is_numeric() stops SQL injection but otherwise not useful, but do we want to allow '100' here or not? Could probably just do is_int() since it's a new method.

    Do we just add this to the interface and do a hard break for contrib drivers, or new interface and instanceof when it's used?

  2. +++ b/core/lib/Drupal/Core/Entity/Sql/SqlContentEntityStorage.php
    @@ -1682,6 +1682,22 @@ public function countFieldData($storage_definition, $as_bool = FALSE) {
     
       /**
    +   * Set the autoincrement value on the base table.
    +   */
    +  public function setBaseTableAutoIncrement($value) {
    +    $schema_handler = $this->database->schema();
    +    $schema_handler->setAutoIncrement($this->baseTable, $value);
    +  }
    +
    +  /**
    +   * Set the autoincrement value on the revision table.
    +   */
    +  public function setRevisionTableAutoIncrement($value) {
    +    $schema_handler = $this->database->schema();
    +    $schema_handler->setAutoIncrement($this->revisionTable, $value);
    +  }
    +
    

    Don't really like this here, but don't have a better idea where to put it. Ideas welcome. This is a fairly hard bc break once we add it to the interface, which makes this likely not possible for a patch release.

  3. +++ b/core/modules/migrate/src/Plugin/migrate/destination/EntityContentBase.php
    @@ -262,6 +264,51 @@ public function rollback(array $destination_identifier) {
    +    $result =  $this->storage->getQuery()
    +      ->sort($id_key, 'DESC')
    +      ->range(0, 1)
    +      ->execute();
    

    At least with MySQL you can get the autoincrement value from INFORMATION_SCHEMA, and I was going to implement something like this at first. However:

    1. Getting the highest ID is usually close enough here, if there's been some deletions and the autoincrement is higher, do we care? Only problem is if there's a rollback or something but that shouldn't happen just before this runs since it's post-import.

    2. Doing the entity query here saves adding cross-database support in the database layer and adding another method or two to the entity storage, so keeps the API surface lower.

  4. +++ b/core/modules/migrate/src/Plugin/migrate/destination/EntityContentBase.php
    @@ -262,6 +264,51 @@ public function rollback(array $destination_identifier) {
    +      $min = $last_id / 10;
    

    This needs to round...

  5. +++ b/core/modules/migrate/src/Plugin/migrate/destination/EntityContentBase.php
    @@ -262,6 +264,51 @@ public function rollback(array $destination_identifier) {
    +      $this->storage->setBaseTableAutoIncrement($auto_increment);
    

    This should be configurable in $settings so you can just set it to 1000 or 10000 or whatever you need for custom migrations.

alexpott’s picture

Discussed a bit with @catch. One thought I had was that it'd be great to be able to set the auto-increment values before starting all the migrations - this way we'd be able to ensure that even if dependencies are incorrect then any new records created by migrations will have a higher ID than a migrated record. The problem though is that we don't know how to get this information from the source plugin prior to running migrations. We'd need both the ID and VID for many entities - basically any auto-increment column. Perhaps there's another way we can ensure migrations that create new records are delayed until the "real" migrations are done. If we can do this then problems like #2826047: 6-8 user picture migration must create new fids, which can conflict with fids used by other migrations just can't occur.

mikeryan’s picture

Version: 8.4.x-dev » 8.3.x-dev
Status: Needs review » Needs work

Some notes:

  1. We don't want to do this more than absolutely necessary. For one thing, if the destination ID is not being set by the migration, messing with the auto-increment is unnecessary and should be avoided.
  2. Entity IDs are not necessarily integer auto-increment columns, correct? We need to check for that.
  3. When a migration is batched in the UI, the post-import event is invoked for each batch - we'd like to only do this on the last batch for a given migration. We can check allRowsProcessed()...
heddn’s picture

re #19.2: D8=>D8 or D8=>D9 will definitely have issue with non-integer entity ids. I'm not aware of any D6/D7 contrib modules that use a non-integer id, however custom entities are fair-game.

heddn’s picture

Component: migration system » database system
Status: Needs work » Needs review
Issue tags: +Needs subsystem maintainer review

Flagging as needs subsystem review to see if this POC makes sense. If after review, re-assign back to migrate.

mikeryan’s picture

Version: 8.3.x-dev » 8.4.x-dev
Component: database system » migration system
Issue summary: View changes
Status: Needs review » Needs work
Issue tags: -Needs subsystem maintainer review

We (me, heddn, phenproxima, xjm, gaborhojtsy, cilefen, Maxocub, moshe, catch) had a meeting at DrupalCon yesterday to check in on the progress of migrate towards stability in 8.4.x, and this was the main issue discussed. Based on those (as well as previous and subsequent) discussions, I've updated the issue summary to describe the alternatives we have and the implications of each measured against what we're trying to accomplish (starting with "maintain data integrity"). This is a quick dump and surely incomplete (and reflecting my own biases, I'm sure) - the others involved in the discussion should review and modify as needed.

quietone’s picture

@mikeryan, thanks. As usual, your writing is very clear to me.

I wanted to see the options in a chart, so I put it all in LibreCalc. Then, I figured, maybe someone else would like to see it as well, so I copied it to a sheet on the Migrate triage. It is pretty basic, but did what i wanted.

catch’s picture

Issue summary: View changes

Updated the issue summary just to note the database API addition wouldn't really be a bc break at all. We'd have to do an instanceof check and leave the bug unfixed for contrib database drivers until they implement it, but no API break as such.

heddn’s picture

Issue tags: +Baltimore2017

Tagging

heddn’s picture

@quietone, thanks for putting together the spreadsheet. After looking at it, it seems that perhaps: "Pre-upgrade audit" has the most going for it. So I guess I'd put my vote on it.

catch’s picture

fwiw I disagree with the technical debt assessment on the autoincrement one - it's one method on database drivers, an event implementation, and probably a line in settings.php to configure it - not nothing, but could be a lot worse.

heddn’s picture

So, while I like the idea of a pre-upgrade audit... I think we should make auto-increment serial ids and identifying conflicts as options.

I cannot get over the "Ewww" factor of auto-incrementing the serial ids. But in some cases, that *will* be the solution. Example: d.o. upgrade.

But in other cases, I'd just want to identify the uids and nids that are in conflict so I can manually intervene and copy/paste the 2 users that were created to administer the new D8 system and the 3 blog posts that got created on the new system.

I vote for full flexibility. All the options. We get "Identify conflicts" as part of a pre-upgrade audit. Per #24, adding that option is not too hard. Can we do everything and make pre-audit the default first step of a migration in the UI and add it as a feature to migrate_tools?

mikeryan’s picture

I vote for full flexibility. All the options.

In the long run, yes. But what's the minimal subset to declare migrate "stable" in 8.4.0?

mikeryan’s picture

Title: Preserving auto-increment IDs on migration is fragile » [meta] Preserving auto-increment IDs on migration is fragile
Category: Bug report » Plan

We can indeed address all three - our current feeling is that the audit should be our 8.4.x stability goal, and the others can be addressed later. We will add child issues for each of those three approaches.

mikeryan’s picture

Status: Needs work » Active

Child issues created. The audit, which we consider the essential piece for 8.4.x stability, is Critical/Migrate-critical, while the other two are Normal and may be addressed after 8.4.0.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.0-alpha1 will be released the week of July 31, 2017, which means new developments and disruptive changes should now be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

webchick’s picture

Issue tags: +Migrate API

Inventing a new tag to illustrate that this is a Migrate critical specifically blocking Migrate API's stability. (Please advise if this is wrong.)

heddn’s picture

masipila’s picture

Hi,

I ran into the autoincrement issue in a slightly different context than what was used as an example in the issue summary. I'll write my scenario here so that we can discuss if this kind of scenario should also be added to the docs.

Context:
1. I'm migrating Curling.fi from D7 to D8.

2. I have a custom module for the online result service. See https://www.curling.fi/en/competitions/leagues/11897/games
https://www.curling.fi/en/competitions/leagues/11897/standings

3. The datamodel in D7 is a mixture of Node content types (league, team, player) and custom objects (division, group, game). The custom objects have their own custom database tables with autoincrement ids.

4. In D8, I refactored the whole module so that now also divisions, groups and games and Node content types.

In plaing English: when I run the migration for divisions / groups / games, I obviously can't use the legacy division / group / game ID as nid because that would collide with an existing nid. This means that each division / group / game will get the next free nid when I migrate them.

Now, if this migration was done before I executed some of my D7-D8 node migrations, the new division / group / game nids might collide with nids that were already used on D7.

I resolved this so that I'm manually bumping the autoincrement to high enough number so that there will be no collisions.

Would this example be something that would be good to add to some docs? I can write a more generic description if you point me to which doc this could be a good addition.

Cheers,
Markus

mwebaze’s picture

Hi,

would writing a process plugin that gets the last known id for a particular node for a specific node_type be helpful? I have implemented it this way however I am pains trying to figure out the best way to uniquely identify a node without using the nid. Is this possible?

heddn’s picture

If this is a meta, and all the children are opened, does this need to stay at Critical?

andypost’s picture

heddn’s picture

Issue tags: -Migrate API +Migrate Drupal UI

Now that #2876085: Before upgrading, audit for potential ID conflicts has landed, we need to make clear that the migrate API parts of this critical are accomplished. #2687843: Add back incremental migrations through the UI still needs to be accomplished, but that is part of migrate drupal ui, not of the API module.

Re-tagging this issue to make that more clear.

xjm’s picture

Issue tags: -Triaged D8 critical

From that issue from @catch:

I think we need to leave the meta issue open (if not blocking) to keep looking at ways to stop people getting into this situation i the first place (like set autoincrement).

So we still need to keep this open, but we can probably downgrade it from critical. The framework and release managers will check this over and downgrade it if appropriate (maybe next week).

Great work on this everyone!

gabesullice’s picture

@xjm asked me to take a second look at this issue and consider whether it should still be marked as "critical".

While I don't see that there is a solution that actually meets requirements 1 and 3 (maintain data integrity & provide a path forward), I think that it can be downgraded.

This issue was marked critical only because it was critical to #2859225: [policy, no patch] Mark Migrate (not Migrate Drupal) as Stable, but it is now impossible to unknowingly lose data integrity.

Therefore, I think this should be downgraded to "major" because it no longer is an "instability" of the module and all the proposals for paths forward are unlikely to cause BC breaks. However, it is still a major issue because users still cannot migrate sites while preserving data integrity if they have potential conflicts (which is highly likely for migrations of any substantial size).

heddn’s picture

Priority: Critical » Major
Issue tags: -Migrate critical

Small correction: The size of the dataset doesn't matter. It just depends if the site contributors decide during a migration they want to enter new data into the new Drupal 8 site before the final migration is run. On very small sites, this might not be acceptable. While the opposite could easily be true. Size doesn't matter. Entering data into the new site before the final migration is complete is the important distinction.

With that said, I'm going to lower to major.

gabesullice’s picture

Small correction: The size of the dataset doesn't matter.

That was poor wording on my part. I should have said "complexity" not "size"

IOW: Any site with user photos and files, or content translation, or sites that need to turn something that wasn't previously a node into a node as in #35, etc., will not be able to migrate without custom efforts.

catch’s picture

IOW: Any site with user photos and files, or content translation, or sites that need to turn something that wasn't previously a node into a node as in #35, etc., will not be able to migrate without custom efforts.

Apart from some known, unresolved bugs with content translation migrations, all of these are fine without a custom migration as long as you migrate your existing content first, create any new entities afterwards (either post-migration or via migration dependencies), and don't create any further new content on the original site that you need to migrate later - for example user pictures are created after all other files in the 6-8 migration.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.0-alpha1 will be released the week of January 17, 2018, which means new developments and disruptive changes should now be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

alberto56’s picture

Until this issue is resolved, I have documented an example of how to work around this using very high auto_increment values as suggested above, including a workaround for #1209466: Users table not auto-increment for uid?, on a project I use to track an example of migrations from D6 and D7 to D8. This can project can be found on Github at https://github.com/dcycle/d6_to_d8_migration_example/tree/7 and the documentation can be found in the "Instructions to avoiding ID collisions during incremental migrations" section of the README file.

heddn’s picture

Issue summary: View changes

Made this meta more actionable by linking to the sub-children.

Version: 8.6.x-dev » 8.7.x-dev

Drupal 8.6.0-alpha1 will be released the week of July 16, 2018, which means new developments and disruptive changes should now be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

mpp’s picture

We're trying to solve the problem of unaliased URLs.

Initially we didn't have any id mapping in our project but as it's being used allover in core we supposed it was a best practice.
As a surplus we'd need a lot less lookups because we already know the id.

We're well aware we'll only get one shot at this migration as the second someone/thing creates an entity the counters will be broken.

Any chance on #15 ever getting in or will it be more likely that the id mappings will disappear?

heddn’s picture

You'll need to set the auto increment to a very high number manually.

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

donquixote’s picture

Set ridiculously high auto-increment values on the D8 tables so preserved IDs won't collide with new ones.

I think this is the solution, although I would replace "ridiculously high" with "sufficiently high".
If it is too high, then it will look weird in urls and other places.

Imo this should happen immediately on site install, OR immediately after the respective database table has been created.
E.g. with hook_modules_installed().

This allows for a scenario when some entities are going to be created _before_ any migrations run, and possibly before any migrate module is enabled.

So, if we do this outside of migrate, then how can we know which auto-increment counter is "sufficiently high"?

  1. The site builder / developer has to provide hard-coded auto-increment ids, e.g. in a custom yml file.
  2. The system has access to the migrate source database, before migrations are enabled. In this case, it can determine suitable auto-increment ids based on a formula applied to the original ids.
  3. Like 1., but the yml file can be generated and updated by migrate, on the developer's local machine.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

dinarcon’s picture

Hi, I have released a new module to help with setting the AUTO_INCREMENT value for tables when using the MySQL driver https://www.drupal.org/project/auto_increment_alter An article talking about this topic will be available at https://udrupal.com/auto-increment-alter-documentation Comments and feedback are very much appreciated.

quietone’s picture

Issue tags: -Migrate Drupal UI +Migrate UI
quietone’s picture

xjm credited cilefen.

xjm credited effulgentsia.

xjm’s picture

Adding triage credits per @alexpott in #16.

Version: 11.x-dev » main

Drupal core is now using the main branch as the primary development branch. New developments and disruptive changes should now be targeted to the main branch.

Read more in the announcement.