[meta] Migrations from Drupal 8 to Drupal 8 and Drupal 8 to Drupal 9

I could not wait for the 8-8 migration, and do not know how to implement it.
Is there any 8.x sources as an example? such as node or user source.
If there has an example, so we can study it and coding the migration for other modules.

No work has yet begun, or is anywhere near beginning, on D8 sources - we're still busy nailing down the D6 and D7 migration paths, hence this is a Postponed Plan rather than an Active Task. My expectation is that we'll do this initially in contrib (like the D5 migration support) with an eye towards merging into core in a later 8.x release.

Note that while the migrations themselves, and narrow source plugins for specific cases, will most likely be developed first in contrib, I think a configuration source plugin is broadly useful enough (and a necessary enabler for the contrib work) to go into 8.1 core: #2621500: Configuration source plugin.

This can be active for the next development minor (currently 8.2.x).

Also, I think this is major.

If we follow the existing pattern, we will also need a D8 test fixture. Or even better, more than one, to cover different use cases.

Irregardless of how many there are, I think they need to be built via the UI with the intention of making them once and only once. I mean, with all the modules, settings, data and modifications needed. We should not have to modify the dump for each and every migration. It is time consuming and prone to error. The dumps should only need minimal attention, such as updates.

And probably improve the tools, so that things we test, like timestamps, don't get changed every time a patch is made after using the UI. Or to automatically change text, like the maintenance text, to begin with 'Drupal 8' so we can confirm it migrated to the new destination correctly.

8-to-8 should replace the exception thrown in SqlContentEntityStorageSchema::onEntityTypeUpdate

(Just trying to fix that this issue has about the most unsearchable title.)

Is this a challenge to find an issue with a more unsearchable title?

We started working on something for a project we have, put it on d.o now: https://www.drupal.org/project/migrate_drupal_d8

Not extensively tested yet, we're still developing it.

Since the plan is that Drupal 9 will basically be Drupal 8.x with deprecated code/features removed, it seems we won't need an upgrade path to Drupal 9, right? The pain of migrating to a new Drupal instance to achieve an upgrade should be a one-time thing (or at least, not something we'll need to do again for a couple of major releases). So, while it will be good to have D8 sources to support custom D8->D* migrations, I don't think that rises to Major priority.

Oh, this is definitely major. The whole policy relies on having D8 sources that are part of the code that gets deprecated and upgraded. Finishing D6/7 migrations of course needs to come first, but this is a key part of making upgrades easier in the future. Edit: already we hear from people that being able to do migrations within D8 would save a lot of pain for experimental modules and so forth. We had a meeting a couple weeks ago with experimental module owners and one of the key takeaways was a need for migrations.

That's an interesting topic that goes far beyound this issue I think.

Pure API deprecations are fairly easy, we have a new API, both do the same we eventually remove the old, done.

Deprecating configuration or storage is a lot more complicated. I honestly have no clue yet how we're supposed to that. The configuration of rest.settings to configuration entities was one example that we've already done in 8.x and honestly it was/is a mess, broke modules like rest_ui and so on. If we'd been strict/honest about BC in rest.module, that would have never been allowed to happen the way it did. (Not blaming anyone, but that's how it is).

We'd need a sync in both ways, so that the data/configuration can be found in both the old and the new place, but what if the new structure can't even be fully representated in the old structure? Or if it's a change that by defintion conflicts with the old way? so many "interesting" problems.

But I'm not sure if we need migrations because as written, the data needs to be in both places from the beginning, so the only thing would be to automatically run migrations in update functions and I doubt that is easier than just writing code directly.

Right, sorry, the bit about migrations within a D8 site for experimental was just an aside. The main point is that D8 sources are a requirement for the new upgrade and deprecation model to actually work, because otherwise we're stuck with legacy database tables forever again and are back to the pain of D6 upgrades. The expectation will still be that you create a new D9 site and migrate into it; the change is just that you don't have to wait 1-2 years for contrib modules or rewrite all your custom code. Migrate is an essential piece of the puzzle for us.

Does this mean that the expectation is if we have a BC change in data model (database table), at the point we implement the new model we leave the data represented in the old model until D9? My expectation was that data would be upgraded in place as we go if we have such an instance - as Berdir suggests, maintaining two data models at once seems painful.

I think it may be hard to picture how this works in practice until we hit a concrete example (are there any such changes in the works now?).

We now have a d8 source for content entities. What is left in this? Let's update the issue summary on what is left or is important.

I've opened #2945181: Beef up Config D8 source plugin. I'm switching this into a meta. I think we still need a plan or next steps of what is needed for this.

See we don't have any plans to release 9 soon. And we now have a D8=>D8 migrate path, I'm going to drop priority on this meta. There's no urgency on it right now.

This seems the most applicable for asking some questions about supporting D6 (and D7) when D9 is released.

• Do we still need to support an upgrade path from D6 (and D7) when D9 is released?
• Do we need to support upgrading from D8 using an external DB connection or is in-place upgrading from D8 a viable option?

The later question speaks directly to the statements made in several large Drupal developer gatherings by Dries and others that D8 is the last major Drupal change. From here on out, D9.0.0 and D8.Final will be API equal except for BC removals. If that statement is true, then we /could/ require anyone wanting to upgrade to D9 first upgrade to D8.Final, then upgrade a second time to D9.0.0, then a third time to 9.x.x, or whatever is the latest release at the time.

The upside to not supporting external DB connections upgrades (maybe in contrib?) is that we drastically reduce the code-base core. We wouldn't need to keep CCK/Field plugins. Our Sql based source plugins in Migrate Drupal probably mostly disappear. We don't need to write a DB equivalent of the Entity API for pulling in "all nodes", etc.

The downside, we don't really have a very seamless upgrade path for legacy sites still on D6 or even D7.

Does the existing code need updates/changes to keep that support (ie. is it in the way of new API introductions, etc.)? Or is the main concern with keeping that code that there may be bugs, security issues, etc. that mean more maintenance burden compared to less code?

My concern is we have D6, D7 db support. If we keep that in D9, then we probably need to have parity and add D8 db support. Adding D8 while still supporting early then very much adds to the support burden. Keeping field plugins in D6/D7 is one thing, but then extending it another time into D8 just keeps adding to the code-base. D6 hasn't been supported for a while and we've given ample time to upgrade to D8.

I'm resigned to keeping all of it, but perhaps we need to consider how to segregate the D6 and D7 code out a little more so at some point we can remove it. Or move it to contrib. Much of migrate is BC shims and with what we are wanting to do in #3004480: Migrate in D9, anything we can do to remove legacy support will make support easier and allow us to remove more BC code.

Re: Drupal 6 support, even if its slated to be removed in Drupal 9, all the work on it would need to be done with refactoring everything else since it is part of Drupal 8 now. That could be avoided if Drupal 6 support is marked deprecated, then it could use deprecated APIs in Drupal 8 but that means when its removed in Drupal 9 and moved to contrib, it would already not be Drupal 9 compatible. The only way to make it Drupal 9 compatible in contrib when Drupal 9 ships is to keep it up to date in core (even if its marked deprecated).

So actually removing it would only save the work that is anticipated to happen *after* Drupal 9 is released then?

Just wondering... If we're able to drop PHP 5 support mid-cycle (i.e., after 8.0 and before 9.0), why can't we also drop D6 migration support mid-cycle? Suppose we made 8.7 the last version to have D6 migrations, then for any D6 sites that haven't yet migrated, is it unreasonable for them to migrate to 8.7 first, then update.php to later versions of 8?

#33++

Well, our minor releases are supposed to be backwards compatible. We unfortunately(?) cannot fabricate a world where PHP 5 is still supported and receives security fixes so that is an outside circumstance we are adopting to to keep our users secure. Does such circumstance exist for Drupal 6 migration support?

I think backwards compatibility is for the ongoing operation of a site, including its administrative tools. But a migration from an unsupported version of Drupal is not part of ongoing site operation. In other words, from the perspective of a site, so long as they can go from D6 -> 8.7 -> Current, isn't that effectively backwards compatible?

If you are working on an incremental migration of a Drupal 6 site to 8.x and then before you are done with that and update to 8.x+1 which drops support for such migrations, then it does affect your ongoing operation yes.

Or you are a contributed module that extends the D6 migrations in core, and you stop being able to operate with Drupal 8.x+1, that is also not backwards compatible.

I could make a compelling argument that since we have 12 month security support... upgrading to the latest version isn't needed. Especially if we took 8.last-1 as the last to support D6. There would be at least 6 months of security support overlap + 8.last-1 isn't meant as a landing place but as a temporary place during the migration.

I discussed this issue with on a call with @alexpott, @catch, Gábor Hojtsy, @plach, and @webchick, and here's where I think we ended up:

• We should assume that update.php will work from 8.LAST to 9.0. Therefore, D8 migration sources are not a requirement for opening the Drupal 9 branch.
• D8 migration sources would be necessary to get in at some point in time (e.g., to support 8 -> 10 migrations) and nice to get in sooner than later (e.g., if it was in now, people could do 8 -> 8 migrations for site re-architectures such as migrating a site from multi-site to domain module).
• For BC reasons, D6 migration sources need to remain in the 8.x branch for the rest of its lifetime. Sorry :(
• D6 migrations sources can be removed from the 9.x branch as soon as that branch is opened.
• We can begin deprecating D6 migrations sources and the APIs they rely on whenever it's desired to do so (starting now). D6 migration tests that start triggering deprecation notices as a result of doing that can be annotated with @group legacy to make the test pass.
• If people working on this are willing to try getting D8 migration sources into 8.x (while D6 migration sources are also there), that would be awesome. If it's too difficult, then they can go into 9.x only (after D6 sources are removed).

It's of course possible that we as a group didn't understand the full problem space here, or that I got something wrong in writing this up, so please point out if anything here doesn't seem right. Thanks.

@effulgentsia: thanks, created #3010983: Deprecate Drupal 6 and Drupal 7 migrations and move to contrib to track that as a distinct issue. Will update #2608062: [META] Requirements for tagging Drupal 9.0.0-alpha1 accordingly.

The actual D8 source plugin that works with a DB connection issue is #2992746: Support for external databases in `content_entity` migration source?. There is also contrib module https://www.drupal.org/project/migrate_drupal_d8, which might be what we use as the base.

Not a requirement for Drupal 9 release, so removing from its parent.

#14 Berdir

We started working on something for a project we have, put it on d.o now: https://www.drupal.org/project/migrate_drupal_d8

It using database tables to migrate data, but instead it could use entity query factory and iterators over result
this way source may provide upgrades from revisionable and translatable layouts for entities or abstract definition with typed data

A lot of people are already using JSON:API for D8 -> D8 migrations. It seems to me that improving support for JSON:API as a migration source fits with the "API first" approach.

If we support JSON:API well enough, then we will not need source plugins for D8+. Anyone with an older version of D8 should be able to upgrade to 8.7, then enable JSON:API, then migrate to the current version.

Updated the IS.

At the migrate meeting early this morning it was decided to pursue using the JSON:API, #3150949: Add a migration source plugin for JSON:API

Is there anything that needs to be done here?

We have source plugins for D8/D9, which are being used successfully. Drupal 9 has been released. And there is one child issue and the usual practice is to close when only one issue is left.

@quietone There's only one child issues, but there are two issues in the issue summary that aren't child issues. So … it's not clear to me what the state/next steps here are 🤓😅

@catch, that would be #3114899: Future of migrate in core in a post Drupal 7 world