New import API for major version upgrades (was migrate module)

one more point is that we need to be able to preserve ids in the D8 site. thats possible already for users and nodes but not for comments and terms and so on. we need to add this capability for those and drop integer IDs for others (text formats, vocabs, ..).

i think that ultimately this is proposal is a good idea. our system is so pluggable (field_sql_storage) and alterable that it is going to become impossible to write an one upgrade path for everyone. at that point, this is our backup plan.

We'd need to move Migrate module into core for this to happen. Otherwise, we'd end up with a poorly maintained upgrade functionality, which might not even be ready when Drupal core is ready. As a matter of fact, the Joomla! project and community has this separation and heavily suffers from it. Many Joomla users are considering to switch to Drupal due to the problematic major version upgrade path and because it's a well-known fact that Drupal has an excellent upgrade/update system.

Knowing the awesomeness of Migrate 2, I think this is a very good idea! It would make core upgrades a lot more powerful, I certainly see the potential for smoother migration of more complex sites.

One thing though, wouldn't this basically be a migration between two sites so to speak (e.g. D7 to D8), instead of SQL queries changing the tables of one? How would that work? Automatically install D8 within the same database as D7 using a prefix, then migrate D7 data to the clean D8 tables? How about contrib, I suppose they would have to provide migration definitions for core upgrades while sticking to hook_update_N() for module upgrades within one version.

The deployment guys might be interested in this too. Solving the configuration and deployment problem is one of the goals for Drupal 8, and Migrate 2 should have a bit to offer there.

Subscribe

subscribe

I am so on board with this approach I have said it in my Drupal 8 talks, this is the way forward, absolutely, totally, no question.

A few points:

1. Batch API support for running migrations could be added without too much trouble. It hasn't been a priority for Moshe and I, but see #1017246: UI for running migrations.
2. The Migration class contains the support for the basic data transfer model - objects moving one at a time, maintaining mappings from old IDs to new IDs. It is derived from MigrationBase, and by deriving directly from MigrationBase you can run arbitrary operations as part of the migration process. For example, schema updates.
3. On the negative side, the Migration class moves one object at a time, calling Drupal APIs (node_save() etc.) for each one - for a large site, this can take hours.
4. But, perhaps an alternative implementation on top of MigrationBase could provide an optimized path for bulk updates.

As an aside, I'd rather see a features-like approach to schema updates - just change the schema declaration and have core automatically make the necessary changes. You'd have to explicitly make changes like renaming columns/tables, but for the most common operations (adding columns/tables/indexes, tweaking a column definition) why should you have to write code? I imagined there must be a feature request out there for this, but I can't find one...

I've had the pleasure of using Migrate 2.x on a project, it's an interesting approach to consider when it comes to core.

I think core should have only a small set of migrate API to allow module to evolve in contrib

$this->addFieldMapping('subscribe', NULL)
     ->defaultValue('q0rban');

Suppose this defaults should be alterable by features module for example

Subscribing.

Subscribing.

Subscribing as well. I did this for a site from D6 to D7 and migrate was a wonderful experience (I had a similar experience of running site upgrades in the past except it was from 4.6 to 6 which was...a LOT of planning and careful work). In total support of such an approach.

I've been using (and developing) http://drupal.org/project/uprpc. Because it's geared specifically to Drupal-to-Drupal, fewer mappings are needed.

Joachim has an interesting idea. The advantage of RPC is that the D7 site is running and so it can hand over data structures and we just need to import them -- and an export / import API is a goal anyways. Two birds with one stone?

I think standardized export/import is a good goal but we are too far away right now. Need to push everything into fields/entities and get the WS part of WSCCI done. For D7=>D8, raw DB source is the best way, IMO.

However D7 has no export so -- this is a no go.

About this possible D8 core feature, I've proposed on the Migrate issue queue (#1234528: Include a module with classes to migrate content from Drupal 6 to Drupal 7) to have it first as D7 contrib., with a submodule of Migrate, for D6 -> D7 upgrades. However, the issue is closed for Migrate because they were considering already to start a new separate project for this functionality, possibly in coming months.

And about D5 -> D6, there is now Up RPC (comment #16).

I learned that Joomla provides a migration for major upgrades just like we propose here. See http://docs.joomla.org/Migrating_from_Joomla_1.5_to_Joomla_1.6.

We probably need a list of issues which are needed to make this simple. The biggest one I can think of is that we need to get all entities to standardize into an entity_save() pattern so that they can share the ->is_new property and thus we can preserve IDs when migrating.

As for unit testing, we could use simpletest or Drush's phpuni- based test framework. Either of those is capable of counting the rows in the source DB and assuring that they match, along with spot checking several entities of each type.

I'm thinking that migration of configuration is a different issue and not as complex, especially given the Config initiaitive.

@sun: I understand you reasons, comment #2, for why this needs to be in core to make sure it is properly taken care of, based on how the community currently does things.

An alternative would be to add a new classification between core and contribs, called "core tools", "core contribs" or something like that. These projects would be treated more or less on the same level as core, but with a few exceptions. They would not have string or API freeze as core has for example.

Projects such as this, as well as Upgrade Manager, that is not needed for a sites daily functionality, would then be able to be developed and improved as needed during a majors lifetime, but at the same time be separated from core so users easily know the difference.

Another advantage would be that sites only need to install them when/if they are needed.

subscribe

Subscribe

Throwing this on Dries's plate for feedback.

subscribe

Subscribe.

Considering how in concrete terms Migrate would join core, some random thoughts:

1. In core, there's no need for it to be a module - it's just an API, and can go in includes.
2. I'd like to see #1184944: Make entities classed objects, introduce CRUD support go in first - per my comment, that should significantly simplify things.
3. How much of the Migrate API do we need to fulfill the upgrade scenario?
4. How much more would we need to have a general-purpose Import API?

Subscribe. As mentioned in #5, it's great to have a new site built while being able to pull fresh content from the previous one (usually still live) on demand and selectively.

Cross-linking http://drupal.org/sandbox/frjo/1332996 which I ran across on Twitter. The description given was a D6 => D7 upgrade path in Migrate2. Haven't investigated the code at all.

Almost a year has passed since my comment on this issue, and I'm even more convinced this is the right idea.
Not only is it cleaner, but having rollbacks and incremental migrations is gold.
Install D8, point it to a D7 site, tell it to do it's thing. Deploy.

This would be so awesome. Especially for larger sites that were too complex to be able to use the upgrade path provided by drupal in D6 and D7. I hate that it is so, but for all the larger sites I have ever upgraded - I ended up doing an actual migration. This was for a number of reasons - one being the idea that "I might as well just do a little clean-up while I am at it (sigh)", but even without that, the upgrade path was not covering all content.

@bojanz:

I fully agree with you in that ditching upgrade support between major releases and replace it with something based on the Migrate module is the way forward.

Looking back at the last 3-6 months before 7.0 was released, a lot of the work then where around sorting out the upgrade path (new issues are still reported btw...). However, looking critical on it, the only thing that upgrade path supports is upgrading one Drupal Core major to another. I can't think of any site that doesn't have contrib modules that also needs to be available for the new major before an upgrade can even be considered.

Then looking at the mother-of-all-upgrades that Drupal 8 very well will be if only half of the core initiatives goes in, the upgrade support to D8 will without question both delay the launch and require a lot of unnecessary precious development and testing resources that instead can be used for far more useful things.

Also, Drupal 8 will change everything in the way content is both managed and presented so I question if any website even will benefit from upgrading instead of rewriting to really make use of all the new goodies.

How long will upgrade support delay the release of Drupal 8? Lets be honest and admit it will be quite some time.

On the other hand, by focusing on building a kick-ass migrating tool for the content, site owners can instead focus on building a completely new site that takes full advantage of all the new goodies. The migration process will then be able to not only get all their existing content over, but also tailor it to really take advantage of these new capabilities.

Lastly, I also see migrating and content staging (being able to move content between different servers) has a lot in common. This combining those two is probably also a very good idea, they for use have a lot of overlapping logic.

So, lets look at if it really is worth the effort and limited resources that will be required to have upgrade support as a requirement!

Maybe its time for is to look at major new Drupal version in reality isn't anything but a community controlled fork of the previous major version?

I don't know if there is a dependency on entity clean-ups, but I would love for this to get rolling. It would be great to have some idea on the process by Drupalcon so we can work on this for the sprints etc.

What is the best way to start looking at the upgrade process? Is an IRC meeting what we need? Or some collaborative document?

What do we need to do to get this moving?

@catch: There is now a sandbox by Migrate module maintainer mikeryan: Drupal-to-Drupal data migration. I think it's intended to prepare a new contrib module based on Migrate, initially for D6 -> D7 upgrades.

It would be great to have this working in contrib, to move the new upgrade/migrate functionality into core before Drupal 8 feature freeze: December 1st, 2012.

There are also other sandboxes for Drupal upgrades with Migrate:

• BTMash's REDCAT Migration (D6 -> D7)
• frjo's xdeb migration (D6 -> D7)
• meba's Drupal 5 to 7 Migrate Example (D5 -> D7, but no public code for this sandbox: #1096472: Push the code to git.d.o)

Related issues:

• #1234528: Include a module with classes to migrate content from Drupal 6 to Drupal 7
• #1159130: Create source for Drupal 6 CCK sites
• #1338648: Migrating directly from D5 to D7

Related articles:

• BTMash's Migrate more with less! Or how I learned to love and create dynamic migrations, describing a D6 -> D7 upgrade with Migrate module.
• quicksketch's Direct Drupal 5 to Drupal 7 Migration in 24hrs, with a custom module and a couple of comments on using Migrate for this.

(Modified issue title, major updates -> major upgrades, according to the Content style guide).

I haven't actually populated that sandbox yet, I'll try to do that this weekend.

I have a scope question - do we want to create a narrow upgrade API just for this purpose, or aim for a more general import API? The former would need pretty much all of the latter anyway... Thinking also of namespacing, rather than "migrate" I would call the API we put in core "upgrade" or "import".

As for the core migrate API, there are a number of things I'd like to differently but would break existing modules, been saving them for V3 of the API. So, maybe the first concrete step here is to open an 8.x-3.x branch of migrate and start working there (with a parallel 8.x-3.x branch of migrate_d2d, once I commit it to the sandbox).

I agree - the namespacing will be confusing if it is "migrate". I vote for "import".

General import API FTW, if it's doable. That's basically the first problem people converting to Drupal encounter is how to get their stuff into their shiny new Drupal site. :)

"Migrate in core"++++++++++

@bojanz:

Not sure that would actually be the best option, especially with the constrictions on core features once a .0 release exists.

Basic functionality that does not lock improvements needs would be great to have in core yes, but I would much rather see that features such as Migrate are placed in a new category between core and contrib. They would have all the rules, regulation and community maintainers as for core, expect for two things - They would be free of the api/string freeze core has.

Modules in this category are those that are not essential to run a Drupal site, or important tools that are needed for specific tasks. Migration and content staging is great examples of this. Another is the Update Manager for example.

Neither of those tasks are essential to running a Drupal site, but they are important features/tool for many users. They also share a need that core actually prevents, namely that they all need to be further developed between majors that is very different from other core features.

This category could also be used to promote contrib modules that one day will be moved to core.

Oki, risking to go too much OT here so I stop...

Well, modules are still free to iterate in contrib after a stable release. For example, http://drupal.org/project/simpletest has a 7.x-2.0 release with features that the Drupal 7 core version doesn't have. And http://drupal.org/project/project_browser is doing interesting interation on Update Manager through the contrib cycle.

Let's not make core more useless just so we can iterate. :)

Just so I understand clearly then, would the plan be for core to strictly have an import api and then have solutions in contrib to actually support the ability to bring in the content or is the plan to support everything? I'm trying to understand the full plan in how data (at the very least, Drupal Core data) will get itself mapped out in D8 core. With standard sql storage on the source end, I can see it being possible to have everything in core; Since entities have a proper crud api in D8, the setup for the migration destinations should be quite a deal simpler as well. I make the assumption based off using the entityapi based entity destination include that is in commerce migrate for product entities and it working with without any issues with another type of entity (registration) after a few trivial modifications. However, the source mapping could be problematic; there are sites like the examiner that use sql storage and nosql storage for data; would core have an expectation to handle such scenarios? Also, what would be supported? I can see this supporting a import from Drupal 7 to 8; but would D6 to D8 be supported? How about with D9/D10 in the future?

Finally, your post talks about how modules can interate in contrib after a stable release; I guess the assumption would then be that contrib modules like switcheroo (http://drupal.org/project/switcheroo) to completely facilitate core module overrides are ok to use?

The one other way I can think of doing all this would be via having primary migrations which try to map whatever core data it can over from D(n-x) to Dn. So if fields, can be mapped, great. And so on and so forth. And developers would write secondary migration code (or use a module like feeds) to complete the mapping of any extra data that didn't make it through over into the site. A situation like this could help facilitate a situation from some that use nosql storage of content and provide something from contrib to work inline with the work in core?

I understand this issue as having the core of Migrate in Drupal core. And no reason not to. Contrib can extend and provide extra parts as usual.
(Additional handlers, extended classes, etc).
Case in point: We have Queue API in core, but we also have Advanced queue in contrib.

Yes, core import capabilities should be of course extendable by contrib. For example, D8 core can include the general import API discussed in this issue, and ready to use support to import at least D7 core. Support for more sources can be extended by D8 contrib: specific classes and mappings to import D7 contrib, possibly also D6 core and contrib, and most popular CMS, etc.: WordPress, Joomla, vBulletin, Blogger, Typo3... See also Migrate Extras for contrib import support.

I understood that it should be extendable in the contrib space much like how migrate already is (and completely agree) :) I was confused on the scope of what ends up in core (understand migrate as a general import api in core; but a d7 -> d8 upgrade/migrate module to be packaged in as well?). I can imagine a person who has a 'complex' D7 setup (contrib fields, relations, media, views etc) thinking 'oh, this will migrate everything!' only later realizing that isn't the case so I just want things to be clear.

A few words on the broad structure I have in mind.

The classes in the current Migrate's includes folder would form the core Import API.

The classes in Migrate's plugins/destinations folder (the destination folder) would be part of the respective modules they support. Actually, I'm hoping that all entities can be handled by a single destination class (instead of the separate MigrateDestinationNode etc. classes), which will help tremendously, we'll just need to add destination classes for any non-entity data we want to migrate.

sql.inc and sqlmap.inc from plugins/sources would be part of the core Import API, support Database API sources and the map/message tables for tracking. I'm not sure which (if any) of the other source plugins we'd want to pull into core - probably XML (at least MigrateSourceXML, there are multiple XML implementations), with the rest staying in the contrib Migrate module.

The migrate_d2d module (to be committed to my sandbox soon) would be the basis of the upgrade support, which is of course the original motivation for this issue. Maybe that goes into an 'upgrade' namespace...

Would some version of the migrate_ui submodule go in core? We can table it until we get the API in and see how much we miss the UI, for now let's assume that stays in contrib.

Other than the extra source plugins and migrate_ui, we should of course look at what other pieces would stay in contrib. E.g., the issue* stuff on field mappings (for helping document the migration development process), making sure that the core API does support these extensions.

re: contrib upgrades - Contrib modules would be expected to support the Import API for the upgrade process themselves, I should think. Not core's responsibility...

One thing that was brought up early in this issue but hasn't received much attention since then is the user experience.

Although we can allow people who have special requirements to manually install a separate Drupal 8 instance on their own (and trigger the migration from there), that doesn't seem like an acceptable experience for most sites; the standard way of just visiting update.php and clicking the buttons needs to still work also.

Here are some thoughts on how we could do that and wind up with a win-win:

1. When you visit update.php, it modifies the database credentials in memory to add a random prefix (exactly like simpletest does).
2. With those new database credentials, it calls install_drupal() and installs the Drupal 8 site. This is unlikely to work at the moment, but the work to make it possible has a lot of overlap with #1215104: Use the non-interactive installer in WebTestBase::setUp(), so two birds with one stone.
3. Then it runs the migration.
4. Once everything is OK, it moves the old (Drupal 7) database tables out of the way, by running db_rename_table() to give them a random prefix, and then it also runs db_rename_table() to get rid of the prefixes on the new (Drupal 8) ones. At that point the site is running Drupal 8... The old Drupal 7 tables can be deleted at the site admin's leisure (maybe once they confirm in the UI that they are willing to accept the upgrade?).

BTMash wrote:

Since entities have a proper crud api in D8, the setup for the migration destinations should be quite a deal simpler as well. I make the assumption based off using the entityapi based entity destination include that is in commerce migrate for product entities and it working with without any issues with another type of entity (registration) after a few trivial modifications.

Probably, the work on #1346204: [meta] Drupal 8 Entity API improvements, including #1184944: Make entities classed objects, introduce CRUD support, is going to simplify imports, as suggested by mikeryan (#30, #49), although it looks like the full benefit could come for D8->D9; moshe weitzman commented (#19):

I think standardized export/import is a good goal but we are too far away right now. Need to push everything into fields/entities and get the WS part of WSCCI done. For D7=>D8, raw DB source is the best way, IMO.

BTMash wrote:

However, the source mapping could be problematic; there are sites like the examiner that use sql storage and nosql storage for data; would core have an expectation to handle such scenarios? (...) And developers would write secondary migration code (or use a module like feeds) to complete the mapping of any extra data that didn't make it through over into the site. A situation like this could help facilitate a situation from some that use nosql storage of content and provide something from contrib to work inline with the work in core?

Migrate can also manage sources such as XML, CSV, JSON, etc., although it's especially popular for SQL database sources, while Feeds seems to be the most popular import module for sources such as RSS/Atom feeds, CSV files, and OPML files, and it's also the most direct competitor of the more basic Aggregator in core. Also, many people are using Feeds because #1123538: UI for defining migrations is not yet implemented for Migrate 2. Well, maybe Feeds and Migrate could be complementary import solutions for users and developers.

David_Rothstein wrote:

One thing that was brought up early in this issue but hasn't received much attention since then is the user experience.
Although we can allow people who have special requirements to manually install a separate Drupal 8 instance on their own (and trigger the migration from there), that doesn't seem like an acceptable experience for most sites; the standard way of just visiting update.php and clicking the buttons needs to still work also.

There is also another possibility, Aegir's migrate/upgrade with the click of a button, although I think it works better for minor updates, for example migrating a site from an old D7.10 platform to a new D7.12 platform (and then probably removing the old platform after the migration). For major upgrades of complex sites the somewhat similar Migrate way seems better, maybe including the automated option for more standard sites that you propose (ok for database migration, not sure about file replacement). However, having the old site as a backup for a time is an advantage of this issue approach, to prevent problems after major upgrades.

Just read @Crell's excellent Backward compatible APIs post. It explained a lot of things for me and made me even more convinced that we really should ditch upgrade support between majors and instead focus on content migration.

I simply fail to find even one remotely good reason for why it should be kept.

Aside from the technical implementation, I'd like to make sure to manage expectations, early.

In light of #1333898: Support unstable2unstable upgrades? [policy, no patch] and many other D8 issues in the queue right now, I somewhat fail to see how a migrate-based upgrade process would ever be able to eliminate the module updates landing for HEAD right now.

That is, because some of them revealed critical update/upgrade path (fatal) errors. In most cases, the only way to actually get to know of them ahead of time was to actually update D8 to D8, which inherently requires a module update that performs the required changes.

If we wouldn't have implicitly tested that update path on developer sites, we would not have known of those upgrade errors at all. We wouldn't have noticed until we had attempted to do a stable release (which means that arbitrary people are trying to upgrade, in the same way like core developers, with varying configurations, as being the case while developing on D8).

In turn, writing proper updates for D8 saved us from critical (as in: very very wonky to figure out many months after the fact) upgrade path failures. Figuring out the concrete root cause would hardly be possible. That action mostly relies on the recent commit log. This success of fixing upgrade path bugs early makes me very hesitant and strongly question any proposal that aims for "ditching the update/upgrade path altogether."

In addition to that, the migrate approach won't ever replace the update path within a major version (contrary to upgrade paths across major versions), since the required base infrastructure for doing so would be way too big. (for example, on a site with millions or billions of node, comment, or user records, it might take days to even perform the initial copy of original data that needs to be updated.)

In short: I'd like to see a more concrete proposal for how we're going to manage the update path and upgrade path in the future - without deferring the entire fatal errors to later.

That said, please don't get irritated by this comment. I like the migrate approach. And even if it would turn out that the approach is not able to cope with or replace our current update.php, I'd definitely love to see it as an alternative approach.

I agree with sun for pretty much all #54

I'd definitely love to see it as an alternative approach.

@sun, @David_Rothstein, @pounard: I also think that, given the different use cases, two options are better than one; we could have them for now, and see if people use one or both.

The already existing one (hook_update()) is mandatory because you cannot seriously ask people to write migrate implementation for minor updates or normal maintainance. The later proposal aka migrate-like solution in core would probably help but only as an API for people who wish to use it.

I just did a quick search for 7.x bug reports with the word "upgrade" in them, it returned 16 pages, nearly 800 issues. A similar search for 8.x returned returned 4 pages.

For both 7.x and 8.x there are 5 pages with still open issues (fixed filtered out)

Now, some of these issues are most likely for update problems (point releases), but scanning through a few of the pages tells me the majority is related to problem with major->new major upgrading.

Still, this tells us quite a lot about the scope of the work related, and necessary, for supporting major->new manor upgrading.

I haven't done an upgrade myself, only been part of a team doing a D6->d7 migration (because the new site was a complete rebuild to take advantage of new features), but I have read quite a few posts and articles from skilled and experienced Drupal users giving it a go.

None of those started with "I clicked upgrade and like magic my Drupal 6 site was transformed into a perfect Drupal 7 site". No, instead they described a process that not only required a long wait for the right modules, and themes to be ported, and of course support upgrading, decisions about what could be "lost" in the upgrade, but also a ton of repeated testing required on a copy of the site, etc etc etc...

I am quite convinced that I don't have enough skills to be able to deal with everything involved, even with the help of the upgrade-help modules mentioned in this report.

The experience I have though, tells me that upgrading to Drupal 8 (if that still is supported) is something that I don't want to do anyways. With all the new features and everything, why would I want to transform my Drupal 7 sites to work the same in Drupal 8 when I can use everything learned, take advantage of all the new features, build a much improved new site and then migrate the old content?

• How many do you know that has successfully upgraded a D6 site to D7 and been happy with the result?
• How much time did they had to spend on it?
• How long did they have to wait until enough of the contribs where ported so they actually could try performing the upgrade?
• What issues where they faced with during the upgrade process?

But most importantly, what skills and experience was required to perform the upgrade to get an as complete, and issue free, D7 version of the site as possible?

Who is doing these upgrades?

Looking at the usage statistics for Drupal shows me that since the release of Drupal 7, Drupal 6 usage has gone down from 355k to 300k (including an unusually big 21k drop the last week). We also know that only sites that pings home are included in that statistics. Therefore I believe its safe to say that a lot of development and test sites are included in those numbers. Thus a large part of that drop is simply due to users switching from developing/testing on D6 to D7. I know I haven't had a D6 running more than a couple of times the last year for example.

So, why is there still so many sites on D6?

Then we also need to ask us:

• How much time are core developers spending on maintaining the major->new major upgrade path?
• How much time are contrib maintainers spending on supporting it in their projects?

Combining those numbers I am convinced we are talking about tens of thousands of hours for skilled developers, plus a massive amount of hours for people who have spent time testing upgrading, bug fixes...

Also - Do you know anyone that really enjoy and look forward to work on it?

How would the Migrate module look if just 10% of that time had been spent on it instead of trying to fix the upgrade path?

So, instead of deciding to keep upgrading or not we need to face the facts and simply ask us:

Is it worth keeping it or can the massive amount of time and valuable resources be used on something better?

#39 mikeryan: I haven't actually populated that sandbox http://drupal.org/sandbox/mikeryan/1234554 yet, I'll try to do that this weekend.

How's that going?

@kika: This weekend, I promise;). I spent last weekend on migrate and wordpress_migrate...

So, in terms of moving this forward, a couple of thoughts/questions:

1. Where to do the initial work - a sandbox, or the (not-yet-existent) Migrate 8.x-3.x-dev branch? I kind of picture doing the initial work in the Migrate branch, then at a stable point moving the central bits to core while leaving value-added bits in Migrate.

2. Do we just port the existing code, or step back a bit and first design a new class hierarchy based on the overall architecture, then cut-and-paste in the code that still fits? There are any number of improvements that could be made - e.g., using interfaces instead of abstract classes - not to mention ye olde protected/public debate (warning: chx magnet;).

OK, migrate_d2d is now up in my sandbox.

For DrupalCon attendees, I've scheduled a BOF for Thursday at 1PM to discuss moving forward on this.

BOF summary (plus some additional post-BOF thoughts) at http://groups.drupal.org/node/219784.

FYI, we are near to committing a new plugin system for core. That would influence the code here for sure.

Well, the plugin system has been committed by Dries yesterday to D8.

Also, Mike Ryan is working now on Migrate's #1123538: UI for defining migrations. This could help with one of the main obstacles for wider Drupal adoption, the complex work often required for major upgrades, especially of medium to large sites, distributions, etc.

This task is still a normal issue with 63 followers. However, given that adding this second upgrade path (migration) as an option in D8 core could have strategic significance for a better Drupal's user experience during major upgrades, should this task be a normal or a major issue? At this time or later?

Priority levels of issues
Drupal 8 feature freeze: December 1st, 2012

Mike is making slow progress on a new import API which would be used here. See http://drupal.org/sandbox/mikeryan/1234554

Also note I scribbled out some thoughts about the D8 API at http://groups.drupal.org/node/219784#comment-780673.

Thank you very much to Mike Ryan for upgrading his sandbox to full D7 module: Drupal-to-Drupal data migration.

About the Import and Drupal to Drupal APIs for D8 core, reading the State of Drupal 2011 Survey, with over 3,000 responses (see Dries' article, and results [pdf]), something remarkable is that they seem to be some of the most requested for Drupal 8 by the community:

What are Drupal's biggest opportunities? [10 answers]
1. Replacing legacy platforms: 38.4%
(...)

What are your most wanted *end-user* features for Drupal 8? [16 answers]
1. Better media/asset handling: 33.2%
2. Mobile support: 27.1%
3. A WYSIWYG editor: 27.0%
4. Better performance: 26.4%
5. Automatic upgrade functionality for core: 24.0%
(...)

What are your most wanted *developer* features for Drupal 8? [16 answers]
1. Native HTML 5/CSS3 support: 40.7%
2. Configuration management (exportables): 35.3%
3. Better internal APIs (easier to use, consistency): 27.1%
4. Content import/export tools in core: 25.2%
(...)

It seems this would possibly make, not only an appropriate major issue, but also a good core initiative.

New "official" core initiatives 6 weeks away from feature freeze don't make a lot of sense, but that should absolutely not stop someone from taking this on. There are 74 people following this issue, the code to make it happen is in at project, Let's Do It!™ :D

I'd be comfortable with migrate module if and only if:

1) For the initial patch to be committable, I'd like to see us write migration classes that replace all update functions in Drupal 8 to date. I don't want a regression in the upgrade path.

2) We maintain an upgrade path all throughout the development cycle. I don't want to wait until code freeze to start working on the upgrade path.

3) We ship with a UI that people can use to upgrade their sites. Extremely large sites may have to use Drush, but small sites should be able to be upgraded without Drush.

:(

However, if not the Import and Drupal to Drupal core APIs for D8, a possibility could be just including the new Drupal-to-Drupal data migration, now in contrib, as an optional core module to get some visibility for users, not affecting the rest of core, and of course having also the normal upgrade path to comply with Dries' #74.

The evolving migration/upgrade classes could be added later in contrib, complementing the Drupal to Drupal optional core module, which would be in the category of "small, sef-contained" improvements that Dries wrote about yesterday:

"At the same time, there are a lot of small, non-destabilizing features that would make Drupal 8 better. (...) To help with this goal, catch and I have discussed a plan for allowing some features to continue to be committed to core up until RC1, providing we are under thresholds. To help guide us towards release, we plan to reduce the critical task/bug thresholds by one per week, starting the week of code freeze (...) Primarily, this means small, self-contained (...)

Code freeze and thresholds (2013-02-25)

Major upgrades are a really strategic matter for Drupal adoption and, you know, one of the main reasons (apart from ease of use, visual design...) of people who choose other platforms such as WordPress, etc.

Whether there is still a little chance for D8 or we have to wait a few years, we could let decide one of the committers when this is completely clear.

Here are the reasons that I believe this cannot happen for D8:

- We cannot depend on contrib, so we'd need to move Migrate into core. Migrate is not ported to Drupal 8.
- Replacing the entire D7 -> D8 upgrade path is not "a small, non-destabilizing feature". It's a very large, far-reaching feature that needs a lot of testing.
- None of Dries' conditions are met, and all of them require a substantial amount of work to be met -- more work, IMO, than we have time remaining for during this cycle.

I will not change the version back, but I do think this needs to be bumped to D9.

Of course the APIs are for D9. I was suggesting a much simpler possibility, a rather isolated, optional core module, disabled by default, as just a first small step which is possible for D8. As said, not replacing but having the normal upgrade path as well, the only one that can met #74 for now.

It's just thinking about the users. Naturally, we who discuss this do not have any need for it. We know the new Drupal to Drupal contrib module, currently for D7. But for most users it's completely unknown, and major upgrades of non-small Drupal sites are still something to be afraid of. Having this optional alternative right in core could help change this. Documentation could have some effect as well, but usually much smaller.

If there is any real chance for this first step in D8, then more specific issues including upgrade to D8, etc. could be opened. Even this simpler way is indeed difficult, given that the module maintainer is one lone hero, overwhelmed by other works. So, more views are needed about possible ways to follow, this suggestion or something different.

If Migrate is already the de facto migration path to D8, why put it in core, especially at this stage when 1/ D8 is feature freeze 2/ there is no D8 version of Migrate yet.

Upgrade between major Drupal versions is always a problem. However, when D8 is feature freeze, we should focus on stabilizing it. Frankly I don't think non-small D7 sites will be migrated to D8 right after the date when D8 is released. New D8 sites, yes, but not upgrade.

If we want to help mikeryan and don't want to delay D8, then file new issues in Migrate and D2D Migrate queues.

Edit: cross post.

No, I haven't done concrete work on a D8 version of Migrate yet. I'm currently working on a front-end for migrate_d2d that, as with each new major initiative I've tackled, is exposing basic flaws in the API that would need to be addressed to make it core-worthy. My plan is to focus on D8 after finishing this work, but it's clearly too late at this point to get this into core - I regret that I didn't take more initiative to make the time to dive into D8 earlier. It is what it is - as a practical matter, contrib Migrate has become a frequent D6->D7 upgrade path and I anticipate this will be even more true for D7->D8.

The best driver for getting Migrate 8.x done would be another Examiner.com - a big site bold enough to start moving to D8 now as Examiner.com did during the D7 development cycle. What I would love to see (and work on) would be an initiative to migrate drupal.org to D8 - wouldn't it be great to see d.o running D8 on release day?

FWIW, we've used the migrate_remote for live incremental (not upgrade, but day-to-day data pushes) Drupal-to-Drupal data migrations on a network of very big Drupal Commerce sites.

It is currently a sandbox at https://drupal.org/sandbox/fgm/1138504

Resurrecting for D8 per Dries' Drupalcon announcement, more to come.

Related: #2098717: Name chx and mikeryan the migrate maintainers

Marking for summary update.

So, I didn't make it to Drupalcon and don't have Dries' precise announcement on hand, but some comments based on what I've been hearing:

1. The idea is to provide a one-click upgrade from at least D7 and D6 to D8, based on today's D7 migrate and migrate_d2d modules.
2. What would go into core would be minimally enough of the Migrate module framework to support that upgrade scenario - the rest will go into contrib.
3. It's also important (at least to me;) that the core D8 migration API be rich enough that the contrib Migrate module can maintain all its current functionality, as well as move forward with richer UI support, by cleanly extending the core API. So, I at least would like to make sure whatever bits of migrate are necessary to support this get into core, even if they aren't all technically necessary for the upgrade scenario. The upgrade scenario is still priority #1, of course.
4. One important difference between the traditional upgrade process and a migration-based one is that the existing upgrade process manipulates the "legacy" database in-place, while the migration framework is based on copying from one database to another. We need a good amount of thought about how this will work in practice.

More specific technical thoughts:

1. Dependency injection & CMI! The existing framework relies far too heavily on defining stuff in class constructors, requiring you to extend classes to do most anything. If the Migration class in particular takes sources/destinations/field mappings through dependency injection, and all this configuration is maintained in CMI, it all becomes much more flexible, and in particular a lot more can be done without writing actual code.
2. Anonymous functions will be nice to replace callbacks (at least). Besides that, we can add more general and consistent means of manipulating data along the way (alternatives to prepareRow/callbacks/prepare/complete). And better use of exception handling (I'm looking at you, prepareRow).
3. The purpose of the existing MigrationBase class is largely to support making non-migration (i.e., non copy-data-row-by-row) tasks part of a migration process. I haven't looked at the task management in D8, but hopefully we can leverage that instead. If it supports grouping of tasks, then throw MigrateGroup in here as well.
4. Migration - the big one, we'll need to go through feature-by-feature and figure what belongs in core, and what the contrib Migrate module can add on.
5. Source classes - we'll just need the SQL (PDO-based) class in core, I think. chx just might want to add MongoDB;).
6. Destination classes - the existing entity-based classes (node, user, comment, ...) should collapse down to a single entity destination class. And, a CMI destination class would be added to absorb migrated configuration data.
7. Handlers should become plugins, I think.

That's from the base migrate side, I'll come back later with thoughts on migrate_d2d. Just should note that it only migrates content currently, so we'll need classes for moving configuration (including in particular content type and field definitions) over.

My experience with migrations, including upgrades to file servers in an earlier life, has been that the safest way to perform such a migration is often to do it in multiples steps

• stop running the normal services of the original code base. In our case, that typically means stopping normal web service. Professional sites will let they reverse proxy serve from cache (possibly even outdated, this is supported by e.g. Varnish) during the update
• on the original code base, run a freeze package which exposes all data in a format usable by the migration process and terminates operation by placing the data(base) in a situation where no further change is allowed unless the data is thawed either by a migration cancel or by the next phase
• replace the code base by the new one, performing a fresh deployment with boot-only data, and thaw/import the frozen data to fill the new schema
• remove the import package
• restore normal operation, possibly after ensuring replication, cache warming...

Thanks to D8 "decoupled" nature, many components can be used with D6/D7 and could be used for the freeze package.

In-progress notes from our Prague discussion on the topic are here:
https://docs.google.com/a/acquia.com/document/d/1oc3MTse2mI9iG1v3t-CKT76...

Just to throw in my own $0.02 here:

First of all, having chx and mikeryan as the maintainers is awesome!

However, whatever else happens, lets make sure the Migrate / Migrate_d2d can continue to improve and advance between major core versions. mikeryan, moshe, drewish and all the other maintainers have been doing awesome work with Migrate.

Given how rapidly these modules have been improving and expanding in contrib, I *really* don't want to see them negatively affected by an API freeze, and prevented from implementing significant improvements between major versions due to core policies.

Regarding core improvement, we are going to rely on plugins heavily and make sure the alter hook is set so that migrate can replace plugins with

function migrate_migrate_source_info_alter(&$definitions) {
  $definitions['drupal7_node'] = $definitions['contrib_drupal7_node'];
  unset($definitions['contrib_drupal7_node']);
  // or not annotate the contrib class and $definitions['drupal7_node']['class'] = 'Drupal\migrate\some\class';
}

to replace the drupal7_node core source class with the contrib one.

Maybe there are ideas in https://github.com/docteurklein/php-etl that we would want to use.

While Migrate is definitely an ETL tool, we want to use as much code possible and that's easier if we stay with the migrate codebase. Discussion is now at http://groups.drupal.org/imp ; all relevant links are also there.

linky

out of the sandbox and into core... #2125717: Migrate in core: patch #1

Which is the duplicate issue?

Just coming from #1921822-27: Take account of drupal_hash_salt during migration path from 7.x and assume that this has been fixed.

Can I use this module to migrate the data from old Drupal (

4.5

) to new version of Drupal?
If you know any other module or method to migrate content and configuration of old Drupal to new Drupal would really be great
Thanks in advance

This issue was closed nine years ago, if you are looking for help to upgrade your fifteen-year-old site please look at one of these forums:

• Upgrading Drupal
• Converting to Drupal

comments are now disabled