Problem/Motivation
It is typical and normal that a site cannot migrate into D8 all in one migration. Most of the site is migrated on one date. Then a few days later a final cut-over is declared and all content that wasn't migrated previously is caught up. This isn't all that hard to do. Run drush ms and see what things are unprocessed and run those individual migrations another time.
There are two problems with this:
1. With the user picture migration, the source database does not have entity IDs, so newly created IDs on the destination site can then conflict with other migrations that do (like files), this has a dedicated issue at #2826047: 6-8 user picture migration must create new fids, which can conflict with fids used by other migrations.
2. With nodes and similar entities, after the initial migration, it's possible for entities to be created both on the source and destination sites. When this happens, it's impossible then to correctly preserve IDs - since the same numeric ID represents a different entity.
Audiences to consider
- Non-technical users of the core UI
- Drush novices - simply running an upgrade command
- Technical users doing custom migrations
Requirements to consider
- Maintain data integrity
- No surprises
- Provide a path forward
- Preserve URLs (e.g., node/17).
- Minimize technical debt.
- Minimize effort to implement.
Proposed resolution
Possible blockers to things are:
#2818147: Use Migration process plugin in *all* places
#2890690: MigrateLookup plugin has inconsistent return values.
#2891073: MigrationLookup doesn't create stub when there's multiple migrations.
#2890844: Migration Lookup plugin does not return multiple values when matched
Possible solutions
Pre-requisite - thorough documentation on drupal.org, and a clear description on the preparatory screen of the core UI, of the risks associated with maintaining IDs, and describing approaches - avoid adding manual content until after all migration is done, doing a custom migration which does not preserve IDs, ... This is necessary regardless of any additional technical approaches.
Documentation only
We rely entirely on documenting the issues.
- Maintain data integrity
Risk: People don’t pay attention to the documentation and screw themselves. - No surprises
Risk: If they’ve ignored the documentation, the results will be very surprising. - Provide a path forward
In the documentation, if they go back and read it. - Preserve URLs (e.g., node/17).
Yes. - Minimize technical debt.
No new code to maintain. - Minimize effort to implement.No new code to implement.
Pre-upgrade audit See #2876085: Before upgrading, audit for potential ID conflicts
Run a process before the actual import which identifies conflict risks - cases where an ID in the content to be imported matches an ID that already exists in the destination.
- Maintain data integrity
If this is a required step, the user can choose whether to proceed and overwrite any conflicts, or take an alternative approach (per the docs). - No surprises
They are told exactly what content could be affected by conflicts. - Provide a path forward
Yes. - Preserve URLs (e.g., node/17).
Yes. - Minimize technical debt.
Audit code should be reasonably isolated and maintainable. - Minimize effort to implement.
We’d need to first do what the import process normally does, identify and instantiate the necessary migrations. For each migration, we need to identify the destination ID field, see if that field is mapped in the process section, look for any of destination ID values which are not in the migration’s map table, and see if any of *those* IDs are in the source. Oh, but we need to consolidate this across all migrations to the same entity type… Ugh.
Manipulate auto-increment values See #2876086: Manipulate auto-increment values to avoid conflicts
The assumption here is that the migration is run on a clean (content-free) site, and each migration when complete sets “the” table for the destination entity to have an auto-increment a “safe” distance beyond the highest migrated ID.
- Maintain data integrity
Maintains integrity, with the small risk of underestimating how far to bump the auto-increment (too much content added on the source system). - No surprises
Yes. - Provide a path forward
No path forward needed. - Preserve URLs (e.g., node/17).
Yes. - Minimize technical debt.
API addition to database system. Complex and fragile implementation (see below). - Minimize effort to implement.
Needs a database schema API for retrieving/setting auto-increment. For each migration, we need to identify the destination ID field, see if it’s being mapped, and figure out what table contains the auto-increment ID.
Identify conflicts during import See #2876090: Warn if migrated content will overwrite manually-created content
Add a process plugin to vulnerable migrations which checks to see if the ID being migrated already exists on the destination side and was not migrated (is not in the migration map table). If the source entity would overwrite a non-migrated entity, throw an error.
- Maintain data integrity
Destination nodes will not be overwritten. Some source records will not be imported. - No surprises
If someone hasn’t read the docs, they may be surprised that they only got a partial migration. - Provide a path forward
Difficult path forward - manually migrate the content that got rejected? - Preserve URLs (e.g., node/17).
Only for the content which was successfully imported. If you’ve manually created, say, node/17, then node/17 on the destination site will show different content from node/17 on the source site. - Minimize technical debt.
A fairly straight-forward process plugin. - Minimize effort to implement.
A fairly straight-forward process plugin.
Remaining tasks
User interface changes
API changes
Data model changes
| Comment | File | Size | Author |
|---|---|---|---|
| #15 | 2748609.patch | 4.52 KB | catch |
Comments
Comment #2
heddnComment #3
mikeryanThe root problem is that preserving IDs is inherently fragile. For any entity (or other data with auto-incrementing integer IDs), if a single item is created on the D8 site other than by copying it over directly with its ID, a collision risk is created. In this instance, entirely within the default migration process itself, since the D6 user picture migration creates file entities that did not have corresponding fids on D6, a subsequent file migration will collide with those auto-incremented fids. The problem could just as easily occur manually - after initial migration you create a user account on the D8 side for, say, a consultant helping with the site-build, and when you run an incremental update some new user added on the D6/D7 side will collide with that account, overwriting it. Or, you try to get a head-start on site-building and add a page - boom.
So, what can be done about this?
Comment #4
heddnSome logic exists in DedupeBase. But it defaults to adding
_0, _1, _2, etcto a string value. Rather, what we want is to dedupe by adding to a serial entity id. And convert 100 => 101, 102, 103, etc. Instead of 100_0, 100_1, 100_2, etc.While perhaps not the best in performance, if we passed in the actual fid as a starting point, the exists check in theory wouldn't have to go very far. Since content entry of new stuff probably hasn't created all *that* much new content. This, of course, falls down if many thousands of users and their profile photo are migrated.
Comment #6
catchCould we set this after the migrations maybe? So set the autoincrement to $current_autoincrement + max($total_entities / 10, 100) or something.
Then if you migrate 100 entities, you'd have autoincrement id of 200, and if you migrate 1 million, 1,100,000. Not pretty but would solve most cases here.
Comment #7
alexpottI also think we need to add comments to the core migrations so that if people copy the migrations to build custom migrations they know what they are letting themselves in for if they don't comment out the ID migrations in things like
core/modules/node/migration_templates/d7_node.yml. Something like:Because I've seen projects that didn't know to do this.
Comment #8
mikeryanGiven we have two migrate-criticals depending on this, marking this as well.
Comment #9
mikeryanThe current proposal (from catch) is:
Per catch's reply at https://www.drupal.org/node/2818155#comment-11756847:
So, if we were to do this, what would be the approach? What are the challenges?
Comment #10
alexpottI'm pretty sure that #2818155: Add comments to d6_file stating the conflicts with d6_user_picture_file means that we need to set the auto-increments prior to being the migration. They way anything extraneous created during a migration will get a non-clashing ID.
Comment #11
mikeryanSo, even if we are to pursue trying to manipulate auto-increment values dynamically, implementing that is going to take some time. For now, let's at least do:
Child issue #2824671: Document dangers of preserving IDs on migration created for this.
Comment #12
heddnI've cross-posted in #2826742: Expose migration types in Migrate UI that it could possibly help lessen the impact of this problem by giving some more flexibility on when you have to hit the button on migrating over content.
Comment #13
catchGoing to bump this to critical independently since it's one of the last known data integrity issues with migrate.
Comment #15
catchHere's a proof of concept implementing #6.
Adds a method to Database\driver\mysql\Schema to allow setting the autoincrement on a table. This will need a new interface, and postgres and sqlite implementations if the idea itself is OK, and test coverage in the database tests. The postgres equivalent is ALTER SEQUENCE - we already have an ALTER SEQUENCE when we're renaming tables/columns, so the logic can likely be figured out from that.
The database API bits may need splitting out to another issue before this one is committed, but I think we should keep things here for now since it's very unlikely we'll have another use case in core apart from this.
Adds methods to SqlContentEntityStorage.php to set the autoincrement on the base and revision tables. This is a bit specific but couldn't really think of anywhere else to put that logic - we don't want the table names leaking out of here.
Makes EntityContentBase migrate plugin implement ImportAwareInterface, then queries the highest ID in the base/revision tables and sets the autoincrement there. This seemed the most appropriate place to do it to me, but defer to migrate folks if there's somewhere better. Also I didn't try to do it conditionally on whether the migration is preserving IDs or anything yet.
Obviously very rough but should be enough for discussion at least. The ContentEntityBase kernel test passes, but I didn't run any others.
Comment #16
alexpottDiscussed with @catch, @cilefen, @effulgentsia and @xjm. We agreed that this should be a critical issue since regardless of the experimental nature of migrate this is a data integrity issue that can result in data loss. We also agreed that @catch's solution looks like a good basis for an acceptable work-around since a complete solution is likely to be impossible.
Comment #17
catchBit of self-review.
Needs proper validation, is_numeric() stops SQL injection but otherwise not useful, but do we want to allow '100' here or not? Could probably just do is_int() since it's a new method.
Do we just add this to the interface and do a hard break for contrib drivers, or new interface and instanceof when it's used?
Don't really like this here, but don't have a better idea where to put it. Ideas welcome. This is a fairly hard bc break once we add it to the interface, which makes this likely not possible for a patch release.
At least with MySQL you can get the autoincrement value from INFORMATION_SCHEMA, and I was going to implement something like this at first. However:
1. Getting the highest ID is usually close enough here, if there's been some deletions and the autoincrement is higher, do we care? Only problem is if there's a rollback or something but that shouldn't happen just before this runs since it's post-import.
2. Doing the entity query here saves adding cross-database support in the database layer and adding another method or two to the entity storage, so keeps the API surface lower.
This needs to round...
This should be configurable in $settings so you can just set it to 1000 or 10000 or whatever you need for custom migrations.
Comment #18
alexpottDiscussed a bit with @catch. One thought I had was that it'd be great to be able to set the auto-increment values before starting all the migrations - this way we'd be able to ensure that even if dependencies are incorrect then any new records created by migrations will have a higher ID than a migrated record. The problem though is that we don't know how to get this information from the source plugin prior to running migrations. We'd need both the ID and VID for many entities - basically any auto-increment column. Perhaps there's another way we can ensure migrations that create new records are delayed until the "real" migrations are done. If we can do this then problems like #2826047: 6-8 user picture migration must create new fids, which can conflict with fids used by other migrations just can't occur.
Comment #19
mikeryanSome notes:
Comment #20
heddnre #19.2: D8=>D8 or D8=>D9 will definitely have issue with non-integer entity ids. I'm not aware of any D6/D7 contrib modules that use a non-integer id, however custom entities are fair-game.
Comment #21
heddnFlagging as needs subsystem review to see if this POC makes sense. If after review, re-assign back to migrate.
Comment #22
mikeryanWe (me, heddn, phenproxima, xjm, gaborhojtsy, cilefen, Maxocub, moshe, catch) had a meeting at DrupalCon yesterday to check in on the progress of migrate towards stability in 8.4.x, and this was the main issue discussed. Based on those (as well as previous and subsequent) discussions, I've updated the issue summary to describe the alternatives we have and the implications of each measured against what we're trying to accomplish (starting with "maintain data integrity"). This is a quick dump and surely incomplete (and reflecting my own biases, I'm sure) - the others involved in the discussion should review and modify as needed.
Comment #23
quietone commented@mikeryan, thanks. As usual, your writing is very clear to me.
I wanted to see the options in a chart, so I put it all in LibreCalc. Then, I figured, maybe someone else would like to see it as well, so I copied it to a sheet on the Migrate triage. It is pretty basic, but did what i wanted.
Comment #24
catchUpdated the issue summary just to note the database API addition wouldn't really be a bc break at all. We'd have to do an instanceof check and leave the bug unfixed for contrib database drivers until they implement it, but no API break as such.
Comment #25
heddnTagging
Comment #26
heddn@quietone, thanks for putting together the spreadsheet. After looking at it, it seems that perhaps: "Pre-upgrade audit" has the most going for it. So I guess I'd put my vote on it.
Comment #27
catchfwiw I disagree with the technical debt assessment on the autoincrement one - it's one method on database drivers, an event implementation, and probably a line in settings.php to configure it - not nothing, but could be a lot worse.
Comment #28
heddnSo, while I like the idea of a pre-upgrade audit... I think we should make auto-increment serial ids and identifying conflicts as options.
I cannot get over the "Ewww" factor of auto-incrementing the serial ids. But in some cases, that *will* be the solution. Example: d.o. upgrade.
But in other cases, I'd just want to identify the uids and nids that are in conflict so I can manually intervene and copy/paste the 2 users that were created to administer the new D8 system and the 3 blog posts that got created on the new system.
I vote for full flexibility. All the options. We get "Identify conflicts" as part of a pre-upgrade audit. Per #24, adding that option is not too hard. Can we do everything and make pre-audit the default first step of a migration in the UI and add it as a feature to migrate_tools?
Comment #29
mikeryanIn the long run, yes. But what's the minimal subset to declare migrate "stable" in 8.4.0?
Comment #30
mikeryanWe can indeed address all three - our current feeling is that the audit should be our 8.4.x stability goal, and the others can be addressed later. We will add child issues for each of those three approaches.
Comment #31
mikeryanChild issues created. The audit, which we consider the essential piece for 8.4.x stability, is Critical/Migrate-critical, while the other two are Normal and may be addressed after 8.4.0.
Comment #33
webchickInventing a new tag to illustrate that this is a Migrate critical specifically blocking Migrate API's stability. (Please advise if this is wrong.)
Comment #34
heddnThis is a blocker for #2859225: [policy, no patch] Mark Migrate (not Migrate Drupal) as Stable
Comment #35
masipila commentedHi,
I ran into the autoincrement issue in a slightly different context than what was used as an example in the issue summary. I'll write my scenario here so that we can discuss if this kind of scenario should also be added to the docs.
Context:
1. I'm migrating Curling.fi from D7 to D8.
2. I have a custom module for the online result service. See https://www.curling.fi/en/competitions/leagues/11897/games
https://www.curling.fi/en/competitions/leagues/11897/standings
3. The datamodel in D7 is a mixture of Node content types (league, team, player) and custom objects (division, group, game). The custom objects have their own custom database tables with autoincrement ids.
4. In D8, I refactored the whole module so that now also divisions, groups and games and Node content types.
In plaing English: when I run the migration for divisions / groups / games, I obviously can't use the legacy division / group / game ID as nid because that would collide with an existing nid. This means that each division / group / game will get the next free nid when I migrate them.
Now, if this migration was done before I executed some of my D7-D8 node migrations, the new division / group / game nids might collide with nids that were already used on D7.
I resolved this so that I'm manually bumping the autoincrement to high enough number so that there will be no collisions.
Would this example be something that would be good to add to some docs? I can write a more generic description if you point me to which doc this could be a good addition.
Cheers,
Markus
Comment #36
mwebaze commentedHi,
would writing a process plugin that gets the last known id for a particular node for a specific node_type be helpful? I have implemented it this way however I am pains trying to figure out the best way to uniquely identify a node without using the nid. Is this possible?
Comment #37
heddnIf this is a meta, and all the children are opened, does this need to stay at Critical?
Comment #38
andypostYep, it still critical cos blocks #2687843-76: Add back incremental migrations through the UI
Comment #39
heddnNow that #2876085: Before upgrading, audit for potential ID conflicts has landed, we need to make clear that the migrate API parts of this critical are accomplished. #2687843: Add back incremental migrations through the UI still needs to be accomplished, but that is part of migrate drupal ui, not of the API module.
Re-tagging this issue to make that more clear.
Comment #40
xjmFrom that issue from @catch:
So we still need to keep this open, but we can probably downgrade it from critical. The framework and release managers will check this over and downgrade it if appropriate (maybe next week).
Great work on this everyone!
Comment #41
gabesullice@xjm asked me to take a second look at this issue and consider whether it should still be marked as "critical".
While I don't see that there is a solution that actually meets requirements 1 and 3 (maintain data integrity & provide a path forward), I think that it can be downgraded.
This issue was marked critical only because it was critical to #2859225: [policy, no patch] Mark Migrate (not Migrate Drupal) as Stable, but it is now impossible to unknowingly lose data integrity.
Therefore, I think this should be downgraded to "major" because it no longer is an "instability" of the module and all the proposals for paths forward are unlikely to cause BC breaks. However, it is still a major issue because users still cannot migrate sites while preserving data integrity if they have potential conflicts (which is highly likely for migrations of any substantial size).
Comment #42
heddnSmall correction: The size of the dataset doesn't matter. It just depends if the site contributors decide during a migration they want to enter new data into the new Drupal 8 site before the final migration is run. On very small sites, this might not be acceptable. While the opposite could easily be true. Size doesn't matter. Entering data into the new site before the final migration is complete is the important distinction.
With that said, I'm going to lower to major.
Comment #43
gabesulliceThat was poor wording on my part. I should have said "complexity" not "size"
IOW: Any site with user photos and files, or content translation, or sites that need to turn something that wasn't previously a node into a node as in #35, etc., will not be able to migrate without custom efforts.
Comment #44
catchApart from some known, unresolved bugs with content translation migrations, all of these are fine without a custom migration as long as you migrate your existing content first, create any new entities afterwards (either post-migration or via migration dependencies), and don't create any further new content on the original site that you need to migrate later - for example user pictures are created after all other files in the 6-8 migration.
Comment #46
alberto56 commentedUntil this issue is resolved, I have documented an example of how to work around this using very high auto_increment values as suggested above, including a workaround for #1209466: Users table not auto-increment for uid?, on a project I use to track an example of migrations from D6 and D7 to D8. This can project can be found on Github at https://github.com/dcycle/d6_to_d8_migration_example/tree/7 and the documentation can be found in the "Instructions to avoiding ID collisions during incremental migrations" section of the README file.
Comment #47
heddnMade this meta more actionable by linking to the sub-children.
Comment #49
mpp commentedWe're trying to solve the problem of unaliased URLs.
Initially we didn't have any id mapping in our project but as it's being used allover in core we supposed it was a best practice.
As a surplus we'd need a lot less lookups because we already know the id.
We're well aware we'll only get one shot at this migration as the second someone/thing creates an entity the counters will be broken.
Any chance on #15 ever getting in or will it be more likely that the id mappings will disappear?
Comment #50
heddnYou'll need to set the auto increment to a very high number manually.
Comment #55
donquixote commentedI think this is the solution, although I would replace "ridiculously high" with "sufficiently high".
If it is too high, then it will look weird in urls and other places.
Imo this should happen immediately on site install, OR immediately after the respective database table has been created.
E.g. with hook_modules_installed().
This allows for a scenario when some entities are going to be created _before_ any migrations run, and possibly before any migrate module is enabled.
So, if we do this outside of migrate, then how can we know which auto-increment counter is "sufficiently high"?
Comment #61
dinarcon commentedHi, I have released a new module to help with setting the
AUTO_INCREMENTvalue for tables when using the MySQL driver https://www.drupal.org/project/auto_increment_alter An article talking about this topic will be available at https://udrupal.com/auto-increment-alter-documentation Comments and feedback are very much appreciated.Comment #62
quietone commentedComment #63
quietone commentedComment #66
xjmAdding triage credits per @alexpott in #16.