I had to do an upgrade of a very large and complex Drupal 5 site to Drupal 6, and ended up with a process of upgrading to Drupal 6, manually tweaking things, exporting to features, then rinsing and repeating to the point where all I had to do was upgrade, then enable the features, but this took a fair bit of work and features doesn't cover everything.
On the other hand I had the pleasure of spending a bit of time with migrate module during the migration of a large site to D7 and migrate module was a much nicer experience:
* It can be run from drush.
* It gives you nice status updates as it goes along.
* You can rollback.
* You can do incremental migrations from a live site.
* It uses Drupal APIs for the incoming content.
* You can set your Drupal 7 site up with a nice install profile before running the migration so everything's ready.
* You can post new content to your Drupal 7 site, alongside doing the migration from the old one, with a bit of extra work.
None of these things are true for update.php, except maybe the drush bit.
I'm not sure exactly how this would look, but a few ideas:
- Instead of hook_update_N(), we set up migrate classes for core (and contrib modules with data to migrate) that takes the Drupal 6 data and uses Drupal 7 API functions to import it.
- With exportables in Drupal 8, we'd want to have some method of upgrading Drupal 7 exportables (probably from contrib) to Drupal 8. Something like this has existed for Views upgrades although I've never used it.
- For large sites, they may not want to go the exportables route, and just build the site in Drupal 8, then pull in the old content.
- It would be entirely possible to have migrate classes from Drupal 6 content too, making it easier to skip major versions.
- We'd have the incoming classes all ready for people migrating from other platforms.
- We'd still need to use hook_update_N() for alpha/beta/rc upgrades - but there shouldn't be too many schema updates once we get to that point.
Main issues:
- this is potentially a much smoother workflow if you have a large upgrade to do.
- it's potentially much more confusing for novice users upgrading their sites, although there ought to be a lot less that can go wrong (since we'd never actually alter the data in the Drupal 7 site).
- Drush is great, but for people not using drush we need a UI. Drupal 6 migrate module had one, but I'm not sure if there's a plan to add that to the Drupal 7 version.
Thoughts?
Comments
Comment #1
moshe weitzman CreditAttribution: moshe weitzman commentedone more point is that we need to be able to preserve ids in the D8 site. thats possible already for users and nodes but not for comments and terms and so on. we need to add this capability for those and drop integer IDs for others (text formats, vocabs, ..).
i think that ultimately this is proposal is a good idea. our system is so pluggable (field_sql_storage) and alterable that it is going to become impossible to write an one upgrade path for everyone. at that point, this is our backup plan.
Comment #2
sunWe'd need to move Migrate module into core for this to happen. Otherwise, we'd end up with a poorly maintained upgrade functionality, which might not even be ready when Drupal core is ready. As a matter of fact, the Joomla! project and community has this separation and heavily suffers from it. Many Joomla users are considering to switch to Drupal due to the problematic major version upgrade path and because it's a well-known fact that Drupal has an excellent upgrade/update system.
Comment #3
ximo CreditAttribution: ximo commentedKnowing the awesomeness of Migrate 2, I think this is a very good idea! It would make core upgrades a lot more powerful, I certainly see the potential for smoother migration of more complex sites.
One thing though, wouldn't this basically be a migration between two sites so to speak (e.g. D7 to D8), instead of SQL queries changing the tables of one? How would that work? Automatically install D8 within the same database as D7 using a prefix, then migrate D7 data to the clean D8 tables? How about contrib, I suppose they would have to provide migration definitions for core upgrades while sticking to
hook_update_N()
for module upgrades within one version.The deployment guys might be interested in this too. Solving the configuration and deployment problem is one of the goals for Drupal 8, and Migrate 2 should have a bit to offer there.
Comment #4
mikeryanSubscribe
Comment #5
catch@ximo: Yes it'd be the same model for contrib too. Obviously there are issues of dependencies etc., but I think migrate already handles that (although it might not map to upgrades 1-1, but it's in place to at least some extent).
In terms of workflow, you'd install the Drupal 8 site, add the Drupal 7 database credentials, then without writing to the D7 site at all, you can migrate the content over - the Drupal 7 site can still be serving visitors and adding content if necessary too.
Once you're happy, you then move the D7 site out of the way and stick the D8 one there instead, or change vhost config, or whatever you need to do to switch. This way the Drupal 7 site is always available to roll back to as well - there's no question of not having taken a backup etc.
@sun: I'd have no problem with migrate module being in core and agree this would be necessary for it to be at all viable. Where it gets a bit sticky is the drush support/dependency.
Comment #6
plachsubscribe
Comment #7
chx CreditAttribution: chx commentedI am so on board with this approach I have said it in my Drupal 8 talks, this is the way forward, absolutely, totally, no question.
Comment #8
mikeryanA few points:
As an aside, I'd rather see a features-like approach to schema updates - just change the schema declaration and have core automatically make the necessary changes. You'd have to explicitly make changes like renaming columns/tables, but for the most common operations (adding columns/tables/indexes, tweaking a column definition) why should you have to write code? I imagined there must be a feature request out there for this, but I can't find one...
Comment #9
bojanz CreditAttribution: bojanz commentedI've had the pleasure of using Migrate 2.x on a project, it's an interesting approach to consider when it comes to core.
Comment #10
andypostI think core should have only a small set of migrate API to allow module to evolve in contrib
Comment #11
q0rban CreditAttribution: q0rban commentedComment #12
andypostSuppose this defaults should be alterable by features module for example
Comment #13
Leeteq CreditAttribution: Leeteq commentedSubscribing.
Comment #14
ksenzeeSubscribing.
Comment #15
BTMash CreditAttribution: BTMash commentedSubscribing as well. I did this for a site from D6 to D7 and migrate was a wonderful experience (I had a similar experience of running site upgrades in the past except it was from 4.6 to 6 which was...a LOT of planning and careful work). In total support of such an approach.
Comment #16
joachim CreditAttribution: joachim commentedI've been using (and developing) http://drupal.org/project/uprpc. Because it's geared specifically to Drupal-to-Drupal, fewer mappings are needed.
Comment #17
catchFor configuration, it looks like CMI will use JSON or YAML (doesn't really matter what the format is in terms of this issue).
Assuming there is some kind of configuration upgrade path, it will mean taking config from the database and exporting it to that format - in a way that can be read back into D8.
That seems like more or less the same problem if this is done in a Drupal 8 hook_update_N() vs. a module to be enabled in D7.
Comment #18
chx CreditAttribution: chx commentedJoachim has an interesting idea. The advantage of RPC is that the D7 site is running and so it can hand over data structures and we just need to import them -- and an export / import API is a goal anyways. Two birds with one stone?
Comment #19
moshe weitzman CreditAttribution: moshe weitzman commentedI think standardized export/import is a good goal but we are too far away right now. Need to push everything into fields/entities and get the WS part of WSCCI done. For D7=>D8, raw DB source is the best way, IMO.
Comment #20
chx CreditAttribution: chx commentedHowever D7 has no export so -- this is a no go.
Comment #21
juan_g CreditAttribution: juan_g commentedAbout this possible D8 core feature, I've proposed on the Migrate issue queue (#1234528: Include a module with classes to migrate content from Drupal 6 to Drupal 7) to have it first as D7 contrib., with a submodule of Migrate, for D6 -> D7 upgrades. However, the issue is closed for Migrate because they were considering already to start a new separate project for this functionality, possibly in coming months.
And about D5 -> D6, there is now Up RPC (comment #16).
Comment #22
moshe weitzman CreditAttribution: moshe weitzman commentedI learned that Joomla provides a migration for major upgrades just like we propose here. See http://docs.joomla.org/Migrating_from_Joomla_1.5_to_Joomla_1.6.
We probably need a list of issues which are needed to make this simple. The biggest one I can think of is that we need to get all entities to standardize into an entity_save() pattern so that they can share the ->is_new property and thus we can preserve IDs when migrating.
As for unit testing, we could use simpletest or Drush's phpuni- based test framework. Either of those is capable of counting the rows in the source DB and assuring that they match, along with spot checking several entities of each type.
I'm thinking that migration of configuration is a different issue and not as complex, especially given the Config initiaitive.
Comment #23
tsvenson CreditAttribution: tsvenson commented@sun: I understand you reasons, comment #2, for why this needs to be in core to make sure it is properly taken care of, based on how the community currently does things.
An alternative would be to add a new classification between core and contribs, called "core tools", "core contribs" or something like that. These projects would be treated more or less on the same level as core, but with a few exceptions. They would not have string or API freeze as core has for example.
Projects such as this, as well as Upgrade Manager, that is not needed for a sites daily functionality, would then be able to be developed and improved as needed during a majors lifetime, but at the same time be separated from core so users easily know the difference.
Another advantage would be that sites only need to install them when/if they are needed.
Comment #24
rickmanelius CreditAttribution: rickmanelius commentedsubscribe
Comment #25
naxoc CreditAttribution: naxoc commentedSubscribe
Comment #26
webchickThrowing this on Dries's plate for feedback.
Comment #27
pillarsdotnet CreditAttribution: pillarsdotnet commentedComment #28
gddsubscribe
Comment #29
renat CreditAttribution: renat commentedSubscribe.
Comment #30
mikeryanConsidering how in concrete terms Migrate would join core, some random thoughts:
Comment #31
NicolasH CreditAttribution: NicolasH commentedSubscribe. As mentioned in #5, it's great to have a new site built while being able to pull fresh content from the previous one (usually still live) on demand and selectively.
Comment #32
webchickCross-linking http://drupal.org/sandbox/frjo/1332996 which I ran across on Twitter. The description given was a D6 => D7 upgrade path in Migrate2. Haven't investigated the code at all.
Comment #33
bojanz CreditAttribution: bojanz commentedAlmost a year has passed since my comment on this issue, and I'm even more convinced this is the right idea.
Not only is it cleaner, but having rollbacks and incremental migrations is gold.
Install D8, point it to a D7 site, tell it to do it's thing. Deploy.
Comment #34
naxoc CreditAttribution: naxoc commentedThis would be so awesome. Especially for larger sites that were too complex to be able to use the upgrade path provided by drupal in D6 and D7. I hate that it is so, but for all the larger sites I have ever upgraded - I ended up doing an actual migration. This was for a number of reasons - one being the idea that "I might as well just do a little clean-up while I am at it (sigh)", but even without that, the upgrade path was not covering all content.
Comment #35
tsvenson CreditAttribution: tsvenson commented@bojanz:
I fully agree with you in that ditching upgrade support between major releases and replace it with something based on the Migrate module is the way forward.
Looking back at the last 3-6 months before 7.0 was released, a lot of the work then where around sorting out the upgrade path (new issues are still reported btw...). However, looking critical on it, the only thing that upgrade path supports is upgrading one Drupal Core major to another. I can't think of any site that doesn't have contrib modules that also needs to be available for the new major before an upgrade can even be considered.
Then looking at the mother-of-all-upgrades that Drupal 8 very well will be if only half of the core initiatives goes in, the upgrade support to D8 will without question both delay the launch and require a lot of unnecessary precious development and testing resources that instead can be used for far more useful things.
Also, Drupal 8 will change everything in the way content is both managed and presented so I question if any website even will benefit from upgrading instead of rewriting to really make use of all the new goodies.
How long will upgrade support delay the release of Drupal 8? Lets be honest and admit it will be quite some time.
On the other hand, by focusing on building a kick-ass migrating tool for the content, site owners can instead focus on building a completely new site that takes full advantage of all the new goodies. The migration process will then be able to not only get all their existing content over, but also tailor it to really take advantage of these new capabilities.
Lastly, I also see migrating and content staging (being able to move content between different servers) has a lot in common. This combining those two is probably also a very good idea, they for use have a lot of overlapping logic.
So, lets look at if it really is worth the effort and limited resources that will be required to have upgrade support as a requirement!
Maybe its time for is to look at major new Drupal version in reality isn't anything but a community controlled fork of the previous major version?
Comment #36
catchWhat do we need to do to get this moving? Is it dependent on entity clean-ups, or could we start looking at how the upgrade process might be structured now?
Comment #37
naxoc CreditAttribution: naxoc commentedI don't know if there is a dependency on entity clean-ups, but I would love for this to get rolling. It would be great to have some idea on the process by Drupalcon so we can work on this for the sprints etc.
What is the best way to start looking at the upgrade process? Is an IRC meeting what we need? Or some collaborative document?
Comment #38
juan_g CreditAttribution: juan_g commented@catch: There is now a sandbox by Migrate module maintainer mikeryan: Drupal-to-Drupal data migration. I think it's intended to prepare a new contrib module based on Migrate, initially for D6 -> D7 upgrades.
It would be great to have this working in contrib, to move the new upgrade/migrate functionality into core before Drupal 8 feature freeze: December 1st, 2012.
There are also other sandboxes for Drupal upgrades with Migrate:
Related issues:
Related articles:
(Modified issue title, major updates -> major upgrades, according to the Content style guide).
Comment #39
mikeryanI haven't actually populated that sandbox yet, I'll try to do that this weekend.
I have a scope question - do we want to create a narrow upgrade API just for this purpose, or aim for a more general import API? The former would need pretty much all of the latter anyway... Thinking also of namespacing, rather than "migrate" I would call the API we put in core "upgrade" or "import".
As for the core migrate API, there are a number of things I'd like to differently but would break existing modules, been saving them for V3 of the API. So, maybe the first concrete step here is to open an 8.x-3.x branch of migrate and start working there (with a parallel 8.x-3.x branch of migrate_d2d, once I commit it to the sandbox).
Comment #40
naxoc CreditAttribution: naxoc commentedI agree - the namespacing will be confusing if it is "migrate". I vote for "import".
Comment #41
webchickGeneral import API FTW, if it's doable. That's basically the first problem people converting to Drupal encounter is how to get their stuff into their shiny new Drupal site. :)
Comment #42
bojanz CreditAttribution: bojanz commented"Migrate in core"++++++++++
Comment #43
tsvenson CreditAttribution: tsvenson commented@bojanz:
Not sure that would actually be the best option, especially with the constrictions on core features once a .0 release exists.
Basic functionality that does not lock improvements needs would be great to have in core yes, but I would much rather see that features such as Migrate are placed in a new category between core and contrib. They would have all the rules, regulation and community maintainers as for core, expect for two things - They would be free of the api/string freeze core has.
Modules in this category are those that are not essential to run a Drupal site, or important tools that are needed for specific tasks. Migration and content staging is great examples of this. Another is the Update Manager for example.
Neither of those tasks are essential to running a Drupal site, but they are important features/tool for many users. They also share a need that core actually prevents, namely that they all need to be further developed between majors that is very different from other core features.
This category could also be used to promote contrib modules that one day will be moved to core.
Oki, risking to go too much OT here so I stop...
Comment #44
webchickWell, modules are still free to iterate in contrib after a stable release. For example, http://drupal.org/project/simpletest has a 7.x-2.0 release with features that the Drupal 7 core version doesn't have. And http://drupal.org/project/project_browser is doing interesting interation on Update Manager through the contrib cycle.
Let's not make core more useless just so we can iterate. :)
Comment #45
BTMash CreditAttribution: BTMash commentedJust so I understand clearly then, would the plan be for core to strictly have an import api and then have solutions in contrib to actually support the ability to bring in the content or is the plan to support everything? I'm trying to understand the full plan in how data (at the very least, Drupal Core data) will get itself mapped out in D8 core. With standard sql storage on the source end, I can see it being possible to have everything in core; Since entities have a proper crud api in D8, the setup for the migration destinations should be quite a deal simpler as well. I make the assumption based off using the entityapi based entity destination include that is in commerce migrate for product entities and it working with without any issues with another type of entity (registration) after a few trivial modifications. However, the source mapping could be problematic; there are sites like the examiner that use sql storage and nosql storage for data; would core have an expectation to handle such scenarios? Also, what would be supported? I can see this supporting a import from Drupal 7 to 8; but would D6 to D8 be supported? How about with D9/D10 in the future?
Finally, your post talks about how modules can interate in contrib after a stable release; I guess the assumption would then be that contrib modules like switcheroo (http://drupal.org/project/switcheroo) to completely facilitate core module overrides are ok to use?
The one other way I can think of doing all this would be via having primary migrations which try to map whatever core data it can over from D(n-x) to Dn. So if fields, can be mapped, great. And so on and so forth. And developers would write secondary migration code (or use a module like feeds) to complete the mapping of any extra data that didn't make it through over into the site. A situation like this could help facilitate a situation from some that use nosql storage of content and provide something from contrib to work inline with the work in core?
Comment #46
bojanz CreditAttribution: bojanz commentedI understand this issue as having the core of Migrate in Drupal core. And no reason not to. Contrib can extend and provide extra parts as usual.
(Additional handlers, extended classes, etc).
Case in point: We have Queue API in core, but we also have Advanced queue in contrib.
Comment #47
juan_g CreditAttribution: juan_g commentedYes, core import capabilities should be of course extendable by contrib. For example, D8 core can include the general import API discussed in this issue, and ready to use support to import at least D7 core. Support for more sources can be extended by D8 contrib: specific classes and mappings to import D7 contrib, possibly also D6 core and contrib, and most popular CMS, etc.: WordPress, Joomla, vBulletin, Blogger, Typo3... See also Migrate Extras for contrib import support.
Comment #48
BTMash CreditAttribution: BTMash commentedI understood that it should be extendable in the contrib space much like how migrate already is (and completely agree) :) I was confused on the scope of what ends up in core (understand migrate as a general import api in core; but a d7 -> d8 upgrade/migrate module to be packaged in as well?). I can imagine a person who has a 'complex' D7 setup (contrib fields, relations, media, views etc) thinking 'oh, this will migrate everything!' only later realizing that isn't the case so I just want things to be clear.
Comment #49
mikeryanA few words on the broad structure I have in mind.
The classes in the current Migrate's includes folder would form the core Import API.
The classes in Migrate's plugins/destinations folder (the destination folder) would be part of the respective modules they support. Actually, I'm hoping that all entities can be handled by a single destination class (instead of the separate MigrateDestinationNode etc. classes), which will help tremendously, we'll just need to add destination classes for any non-entity data we want to migrate.
sql.inc and sqlmap.inc from plugins/sources would be part of the core Import API, support Database API sources and the map/message tables for tracking. I'm not sure which (if any) of the other source plugins we'd want to pull into core - probably XML (at least MigrateSourceXML, there are multiple XML implementations), with the rest staying in the contrib Migrate module.
The migrate_d2d module (to be committed to my sandbox soon) would be the basis of the upgrade support, which is of course the original motivation for this issue. Maybe that goes into an 'upgrade' namespace...
Would some version of the migrate_ui submodule go in core? We can table it until we get the API in and see how much we miss the UI, for now let's assume that stays in contrib.
Other than the extra source plugins and migrate_ui, we should of course look at what other pieces would stay in contrib. E.g., the issue* stuff on field mappings (for helping document the migration development process), making sure that the core API does support these extensions.
Comment #50
mikeryanre: contrib upgrades - Contrib modules would be expected to support the Import API for the upgrade process themselves, I should think. Not core's responsibility...
Comment #51
David_Rothstein CreditAttribution: David_Rothstein commentedOne thing that was brought up early in this issue but hasn't received much attention since then is the user experience.
Although we can allow people who have special requirements to manually install a separate Drupal 8 instance on their own (and trigger the migration from there), that doesn't seem like an acceptable experience for most sites; the standard way of just visiting update.php and clicking the buttons needs to still work also.
Here are some thoughts on how we could do that and wind up with a win-win:
Comment #52
juan_g CreditAttribution: juan_g commentedBTMash wrote:
Probably, the work on #1346204: [meta] Drupal 8 Entity API improvements, including #1184944: Make entities classed objects, introduce CRUD support, is going to simplify imports, as suggested by mikeryan (#30, #49), although it looks like the full benefit could come for D8->D9; moshe weitzman commented (#19):
BTMash wrote:
Migrate can also manage sources such as XML, CSV, JSON, etc., although it's especially popular for SQL database sources, while Feeds seems to be the most popular import module for sources such as RSS/Atom feeds, CSV files, and OPML files, and it's also the most direct competitor of the more basic Aggregator in core. Also, many people are using Feeds because #1123538: UI for defining migrations is not yet implemented for Migrate 2. Well, maybe Feeds and Migrate could be complementary import solutions for users and developers.
David_Rothstein wrote:
There is also another possibility, Aegir's migrate/upgrade with the click of a button, although I think it works better for minor updates, for example migrating a site from an old D7.10 platform to a new D7.12 platform (and then probably removing the old platform after the migration). For major upgrades of complex sites the somewhat similar Migrate way seems better, maybe including the automated option for more standard sites that you propose (ok for database migration, not sure about file replacement). However, having the old site as a backup for a time is an advantage of this issue approach, to prevent problems after major upgrades.
Comment #53
tsvenson CreditAttribution: tsvenson commentedJust read @Crell's excellent Backward compatible APIs post. It explained a lot of things for me and made me even more convinced that we really should ditch upgrade support between majors and instead focus on content migration.
I simply fail to find even one remotely good reason for why it should be kept.
Comment #54
sunAside from the technical implementation, I'd like to make sure to manage expectations, early.
In light of #1333898: Support unstable2unstable upgrades? [policy, no patch] and many other D8 issues in the queue right now, I somewhat fail to see how a migrate-based upgrade process would ever be able to eliminate the module updates landing for HEAD right now.
That is, because some of them revealed critical update/upgrade path (fatal) errors. In most cases, the only way to actually get to know of them ahead of time was to actually update D8 to D8, which inherently requires a module update that performs the required changes.
If we wouldn't have implicitly tested that update path on developer sites, we would not have known of those upgrade errors at all. We wouldn't have noticed until we had attempted to do a stable release (which means that arbitrary people are trying to upgrade, in the same way like core developers, with varying configurations, as being the case while developing on D8).
In turn, writing proper updates for D8 saved us from critical (as in: very very wonky to figure out many months after the fact) upgrade path failures. Figuring out the concrete root cause would hardly be possible. That action mostly relies on the recent commit log. This success of fixing upgrade path bugs early makes me very hesitant and strongly question any proposal that aims for "ditching the update/upgrade path altogether."
In addition to that, the migrate approach won't ever replace the update path within a major version (contrary to upgrade paths across major versions), since the required base infrastructure for doing so would be way too big. (for example, on a site with millions or billions of node, comment, or user records, it might take days to even perform the initial copy of original data that needs to be updated.)
In short: I'd like to see a more concrete proposal for how we're going to manage the update path and upgrade path in the future - without deferring the entire fatal errors to later.
That said, please don't get irritated by this comment. I like the migrate approach. And even if it would turn out that the approach is not able to cope with or replace our current update.php, I'd definitely love to see it as an alternative approach.
Comment #55
pounardI agree with sun for pretty much all #54
Comment #56
juan_g CreditAttribution: juan_g commented@sun, @David_Rothstein, @pounard: I also think that, given the different use cases, two options are better than one; we could have them for now, and see if people use one or both.
Comment #57
pounardThe already existing one (hook_update()) is mandatory because you cannot seriously ask people to write migrate implementation for minor updates or normal maintainance. The later proposal aka migrate-like solution in core would probably help but only as an API for people who wish to use it.
Comment #58
catchThe proposal has never been to use migrate API for minor updates, only for major version upgrades.
I'm not sure about sun's comment. Many of the upgrade issues that have bee found on developer sandboxes have been things like stale data about file paths in the registry and similar between the 7.x and 8.x sites. Those upgrade issues would completely disappear for major upgrades if we move to migrate - since we'd be migrating data + configuration and absolutely nothing else. Developer sandboxes very rarely if ever catch actual data migration issues, since for those by definition you need a decent amount of data.
Comment #59
tsvenson CreditAttribution: tsvenson commentedI just did a quick search for 7.x bug reports with the word "upgrade" in them, it returned 16 pages, nearly 800 issues. A similar search for 8.x returned returned 4 pages.
For both 7.x and 8.x there are 5 pages with still open issues (fixed filtered out)
Now, some of these issues are most likely for update problems (point releases), but scanning through a few of the pages tells me the majority is related to problem with major->new major upgrading.
Still, this tells us quite a lot about the scope of the work related, and necessary, for supporting major->new manor upgrading.
I haven't done an upgrade myself, only been part of a team doing a D6->d7 migration (because the new site was a complete rebuild to take advantage of new features), but I have read quite a few posts and articles from skilled and experienced Drupal users giving it a go.
None of those started with "I clicked upgrade and like magic my Drupal 6 site was transformed into a perfect Drupal 7 site". No, instead they described a process that not only required a long wait for the right modules, and themes to be ported, and of course support upgrading, decisions about what could be "lost" in the upgrade, but also a ton of repeated testing required on a copy of the site, etc etc etc...
I am quite convinced that I don't have enough skills to be able to deal with everything involved, even with the help of the upgrade-help modules mentioned in this report.
The experience I have though, tells me that upgrading to Drupal 8 (if that still is supported) is something that I don't want to do anyways. With all the new features and everything, why would I want to transform my Drupal 7 sites to work the same in Drupal 8 when I can use everything learned, take advantage of all the new features, build a much improved new site and then migrate the old content?
But most importantly, what skills and experience was required to perform the upgrade to get an as complete, and issue free, D7 version of the site as possible?
Who is doing these upgrades?
Looking at the usage statistics for Drupal shows me that since the release of Drupal 7, Drupal 6 usage has gone down from 355k to 300k (including an unusually big 21k drop the last week). We also know that only sites that pings home are included in that statistics. Therefore I believe its safe to say that a lot of development and test sites are included in those numbers. Thus a large part of that drop is simply due to users switching from developing/testing on D6 to D7. I know I haven't had a D6 running more than a couple of times the last year for example.
So, why is there still so many sites on D6?
Then we also need to ask us:
Combining those numbers I am convinced we are talking about tens of thousands of hours for skilled developers, plus a massive amount of hours for people who have spent time testing upgrading, bug fixes...
Also - Do you know anyone that really enjoy and look forward to work on it?
How would the Migrate module look if just 10% of that time had been spent on it instead of trying to fix the upgrade path?
So, instead of deciding to keep upgrading or not we need to face the facts and simply ask us:
Is it worth keeping it or can the massive amount of time and valuable resources be used on something better?
Comment #60
kika CreditAttribution: kika commentedHow's that going?
Comment #61
mikeryan@kika: This weekend, I promise;). I spent last weekend on migrate and wordpress_migrate...
So, in terms of moving this forward, a couple of thoughts/questions:
1. Where to do the initial work - a sandbox, or the (not-yet-existent) Migrate 8.x-3.x-dev branch? I kind of picture doing the initial work in the Migrate branch, then at a stable point moving the central bits to core while leaving value-added bits in Migrate.
2. Do we just port the existing code, or step back a bit and first design a new class hierarchy based on the overall architecture, then cut-and-paste in the code that still fits? There are any number of improvements that could be made - e.g., using interfaces instead of abstract classes - not to mention ye olde protected/public debate (warning: chx magnet;).
Comment #62
mikeryanOK, migrate_d2d is now up in my sandbox.
Comment #63
mikeryanFor DrupalCon attendees, I've scheduled a BOF for Thursday at 1PM to discuss moving forward on this.
Comment #64
mikeryanBOF summary (plus some additional post-BOF thoughts) at http://groups.drupal.org/node/219784.
Comment #65
moshe weitzman CreditAttribution: moshe weitzman commentedFYI, we are near to committing a new plugin system for core. That would influence the code here for sure.
Comment #66
juan_g CreditAttribution: juan_g commentedWell, the plugin system has been committed by Dries yesterday to D8.
Also, Mike Ryan is working now on Migrate's #1123538: UI for defining migrations. This could help with one of the main obstacles for wider Drupal adoption, the complex work often required for major upgrades, especially of medium to large sites, distributions, etc.
Comment #67
juan_g CreditAttribution: juan_g commentedThis task is still a normal issue with 63 followers. However, given that adding this second upgrade path (migration) as an option in D8 core could have strategic significance for a better Drupal's user experience during major upgrades, should this task be a normal or a major issue? At this time or later?
Priority levels of issues
Drupal 8 feature freeze: December 1st, 2012
Comment #68
moshe weitzman CreditAttribution: moshe weitzman commentedMike is making slow progress on a new import API which would be used here. See http://drupal.org/sandbox/mikeryan/1234554
Comment #69
mikeryanAlso note I scribbled out some thoughts about the D8 API at http://groups.drupal.org/node/219784#comment-780673.
Comment #72
juan_g CreditAttribution: juan_g commentedThank you very much to Mike Ryan for upgrading his sandbox to full D7 module: Drupal-to-Drupal data migration.
About the Import and Drupal to Drupal APIs for D8 core, reading the State of Drupal 2011 Survey, with over 3,000 responses (see Dries' article, and results [pdf]), something remarkable is that they seem to be some of the most requested for Drupal 8 by the community:
It seems this would possibly make, not only an appropriate major issue, but also a good core initiative.
Comment #73
webchickNew "official" core initiatives 6 weeks away from feature freeze don't make a lot of sense, but that should absolutely not stop someone from taking this on. There are 74 people following this issue, the code to make it happen is in at project, Let's Do It!™ :D
Comment #74
Dries CreditAttribution: Dries commentedI'd be comfortable with migrate module if and only if:
1) For the initial patch to be committable, I'd like to see us write migration classes that replace all update functions in Drupal 8 to date. I don't want a regression in the upgrade path.
2) We maintain an upgrade path all throughout the development cycle. I don't want to wait until code freeze to start working on the upgrade path.
3) We ship with a UI that people can use to upgrade their sites. Extremely large sites may have to use Drush, but small sites should be able to be upgraded without Drush.
Comment #75
cweagans:(
Comment #76
juan_g CreditAttribution: juan_g commentedHowever, if not the Import and Drupal to Drupal core APIs for D8, a possibility could be just including the new Drupal-to-Drupal data migration, now in contrib, as an optional core module to get some visibility for users, not affecting the rest of core, and of course having also the normal upgrade path to comply with Dries' #74.
The evolving migration/upgrade classes could be added later in contrib, complementing the Drupal to Drupal optional core module, which would be in the category of "small, sef-contained" improvements that Dries wrote about yesterday:
Major upgrades are a really strategic matter for Drupal adoption and, you know, one of the main reasons (apart from ease of use, visual design...) of people who choose other platforms such as WordPress, etc.
Whether there is still a little chance for D8 or we have to wait a few years, we could let decide one of the committers when this is completely clear.
Comment #77
cweagansHere are the reasons that I believe this cannot happen for D8:
- We cannot depend on contrib, so we'd need to move Migrate into core. Migrate is not ported to Drupal 8.
- Replacing the entire D7 -> D8 upgrade path is not "a small, non-destabilizing feature". It's a very large, far-reaching feature that needs a lot of testing.
- None of Dries' conditions are met, and all of them require a substantial amount of work to be met -- more work, IMO, than we have time remaining for during this cycle.
I will not change the version back, but I do think this needs to be bumped to D9.
Comment #78
juan_g CreditAttribution: juan_g commentedOf course the APIs are for D9. I was suggesting a much simpler possibility, a rather isolated, optional core module, disabled by default, as just a first small step which is possible for D8. As said, not replacing but having the normal upgrade path as well, the only one that can met #74 for now.
It's just thinking about the users. Naturally, we who discuss this do not have any need for it. We know the new Drupal to Drupal contrib module, currently for D7. But for most users it's completely unknown, and major upgrades of non-small Drupal sites are still something to be afraid of. Having this optional alternative right in core could help change this. Documentation could have some effect as well, but usually much smaller.
If there is any real chance for this first step in D8, then more specific issues including upgrade to D8, etc. could be opened. Even this simpler way is indeed difficult, given that the module maintainer is one lone hero, overwhelmed by other works. So, more views are needed about possible ways to follow, this suggestion or something different.
Comment #79
catchI believe moshe and Mike Ryan were working on an 8.x port of migrate but I don't know what happened with it.
I agree with #77 and would add that unless we're able to completely replace the major upgrade path with migrate there's no way this can happen in core since we can't maintain both at the same time. It's bad enough maintaining one upgrade path in core - we have 8.x release blocking upgrade path bugs /still/ in Drupal 7 let alone what's in 8.x. And while it's sad that it's going to be the hook_update_N() version rather than the migrate version for 8.x this issue was stalled for a long time then there was no visible work on it.
If people work on a migrate module 7-8 upgrade path in contrib (and nothing stops that), then hopefully that'll get promoted as the better way to do it.
In terms of backporting to from 9.x to 8.x, I guess that would be feasible without breaking APIs (certainly it won't affect existing sites - they either updated or they didn't). Where keeping this at 8.x fails on the criteria of Dries' blog post is that this would be a multiple-issue project with lots of major/critical follow-ups that would push out the 8.x release date - unless someone does an amazing job of convincing everyone that's not the case (i.e. a perfectly working patch that's better than the existing 8.x upgrade path already) then that makes it a 9.x issue.
Comment #80
jcisio CreditAttribution: jcisio commentedIf Migrate is already the de facto migration path to D8, why put it in core, especially at this stage when 1/ D8 is feature freeze 2/ there is no D8 version of Migrate yet.
Upgrade between major Drupal versions is always a problem. However, when D8 is feature freeze, we should focus on stabilizing it. Frankly I don't think non-small D7 sites will be migrated to D8 right after the date when D8 is released. New D8 sites, yes, but not upgrade.
If we want to help mikeryan and don't want to delay D8, then file new issues in Migrate and D2D Migrate queues.
Edit: cross post.
Comment #81
mikeryanNo, I haven't done concrete work on a D8 version of Migrate yet. I'm currently working on a front-end for migrate_d2d that, as with each new major initiative I've tackled, is exposing basic flaws in the API that would need to be addressed to make it core-worthy. My plan is to focus on D8 after finishing this work, but it's clearly too late at this point to get this into core - I regret that I didn't take more initiative to make the time to dive into D8 earlier. It is what it is - as a practical matter, contrib Migrate has become a frequent D6->D7 upgrade path and I anticipate this will be even more true for D7->D8.
The best driver for getting Migrate 8.x done would be another Examiner.com - a big site bold enough to start moving to D8 now as Examiner.com did during the D7 development cycle. What I would love to see (and work on) would be an initiative to migrate drupal.org to D8 - wouldn't it be great to see d.o running D8 on release day?
Comment #82
fgmFWIW, we've used the migrate_remote for live incremental (not upgrade, but day-to-day data pushes) Drupal-to-Drupal data migrations on a network of very big Drupal Commerce sites.
It is currently a sandbox at https://drupal.org/sandbox/fgm/1138504
Comment #83
mikeryanResurrecting for D8 per Dries' Drupalcon announcement, more to come.
Related: #2098717: Name chx and mikeryan the migrate maintainers
Comment #84
marvil07 CreditAttribution: marvil07 commentedMarking for summary update.
Comment #85
mikeryanSo, I didn't make it to Drupalcon and don't have Dries' precise announcement on hand, but some comments based on what I've been hearing:
More specific technical thoughts:
That's from the base migrate side, I'll come back later with thoughts on migrate_d2d. Just should note that it only migrates content currently, so we'll need classes for moving configuration (including in particular content type and field definitions) over.
Comment #86
fgmMy experience with migrations, including upgrades to file servers in an earlier life, has been that the safest way to perform such a migration is often to do it in multiples steps
Thanks to D8 "decoupled" nature, many components can be used with D6/D7 and could be used for the freeze package.
Comment #87
xjmIn-progress notes from our Prague discussion on the topic are here:
https://docs.google.com/a/acquia.com/document/d/1oc3MTse2mI9iG1v3t-CKT76...
Comment #88
brianV CreditAttribution: brianV commentedJust to throw in my own $0.02 here:
First of all, having chx and mikeryan as the maintainers is awesome!
However, whatever else happens, lets make sure the Migrate / Migrate_d2d can continue to improve and advance between major core versions. mikeryan, moshe, drewish and all the other maintainers have been doing awesome work with Migrate.
Given how rapidly these modules have been improving and expanding in contrib, I *really* don't want to see them negatively affected by an API freeze, and prevented from implementing significant improvements between major versions due to core policies.
Comment #89
chx CreditAttribution: chx commentedRegarding core improvement, we are going to rely on plugins heavily and make sure the alter hook is set so that migrate can replace plugins with
to replace the drupal7_node core source class with the contrib one.
Comment #90
catchMoving to task. Not sure how to handle critical migration path stuff that doesn't actually block the release of 8.0 so leaving at 'major' for now.
Comment #91
cosmicdreams CreditAttribution: cosmicdreams commentedMaybe there are ideas in https://github.com/docteurklein/php-etl that we would want to use.
Comment #92
chx CreditAttribution: chx commentedWhile Migrate is definitely an ETL tool, we want to use as much code possible and that's easier if we stay with the migrate codebase. Discussion is now at http://groups.drupal.org/imp ; all relevant links are also there.
Comment #92.0
chx CreditAttribution: chx commentedlinky
Comment #93
YesCT CreditAttribution: YesCT commentedout of the sandbox and into core... #2125717: Migrate in core: patch #1
Comment #94
mgiffordWhich is the duplicate issue?
Just coming from #1921822-27: Take account of drupal_hash_salt during migration path from 7.x and assume that this has been fixed.
Comment #95
manjuphoenix CreditAttribution: manjuphoenix commentedCan I use this module to migrate the data from old Drupal (
4.5
) to new version of Drupal?
If you know any other module or method to migrate content and configuration of old Drupal to new Drupal would really be great
Thanks in advance
Comment #96
DamienMcKennaThis issue was closed nine years ago, if you are looking for help to upgrade your fifteen-year-old site please look at one of these forums:
comments are now disabled