Work taking place in https://www.drupal.org/project/migrate_upgrade for now

Problem/Motivation

Currently, the plan is that the UI for upgrades to Drupal 8 from Drupal 6/7 will be initially developed in contrib, to be pulled into core when "ready" (a decision will be made before RC whether it will be made available in 8.0.0). This UI is under development as migrate_upgrade, and at Drupalcon Austin Bojhan and webchick sat down with me to review the current state of that effort. From that session came a punchlist of suggested changes to the UI as it is now. I'll be adding those issues in the sandbox as children of this meta issue - the meta is here in core for visibility, since this work will eventually end up in core. Note that many of these will be Novice issues, for those looking to get started contributing.

For historical background - previous discussion on the UI leading to its current implementation - see #2200379: Upgrade UI.

How to find upgrade page

Appearance/layout of upgrade page

Workflow

Feedback

Comments

mikeryan’s picture

mikeryan’s picture

Issue summary:View changes
mikeryan’s picture

Issue summary:View changes
mikeryan’s picture

Issue summary:View changes
mikeryan’s picture

Issue summary:View changes
mikeryan’s picture

Issue summary:View changes
mikeryan’s picture

Issue summary:View changes
mikeryan’s picture

Issue summary:View changes
mikeryan’s picture

Issue summary:View changes
mikeryan’s picture

For context, attached are screenshots of the interface as reviewed with Bojhan and webchick - all the current child issues are relative to this version of the UI.

mikeryan’s picture

Issue summary:View changes
mikeryan’s picture

Issue summary:View changes

That's it, all the known child issues at this time have been added.

mikeryan’s picture

So, out of all these child issues, the biggest one is #2282021: Consolidate upgrade process to one form/one click. I'm thinking of simplifying it even further, narrowing the scope of the migrate_upgrade functionality to just a one-click full configuration-and-content import (or maybe allowing a configuration import and subsequent content import). Please weigh in at #2282021: Consolidate upgrade process to one form/one click.

Thanks.

benjy’s picture

Would be nice to have an update on this issue of the current state of the UI, big outstanding issues etc.

mikeryan’s picture

Issue summary:View changes

Added context issue.

mikeryan’s picture

Sorry I missed that question - basically, the UI has been working to the extent the underlying framework has for several months, with occasional chasing-HEAD breakage. I haven't tried it for a few weeks so it might well be broken at the moment, but catching up with recent core changes should take care of that. The open issues aren't really major - either cleanup/improving reuse, or providing more/better feedback - the biggest thing I think is the context issue. Should the migration readiness assessment being discussed be integrated with this upgrade form, or separate? Should it be in contrib - in migrate_plus, or perhaps coder_upgrade?

benjy’s picture

Also, after talking with webchick I thought i'd do an initial review, i forked migrate_upgrade into a sandbox and i've done some general clean-up here: https://www.drupal.org/sandbox/benjy/2448261

@mikeryan, not sure if you want me to post it into one big patch in the migrate upgrade issue queue or if you want me to break it up. The diff is here: http://privatepaste.com/458f93e589

Also, I can confirm, all is working with HEAD as of the last day or two.

mikeryan’s picture

I would like to see it broken up, I see a few different things going on there it'd be easier to review separately:

  1. Database driver setup refactoring.
  2. Version detection changes.
  3. Batch results logging.
  4. Miscellaneous rearrangement and simple coding standard stuff (e.g., use of $this->t()).

On a quick once-over I have some comments/questions, but it'll be easier to discuss in more specific issues.

Thanks!

benjy’s picture

Sorry, I should have mentioned there is an overview on the sandbox overview page.

iantresman’s picture

Just to add my support for a user interface (UI). Most people running Drupal sites are not programmer/developers, and without a UI, will find migration very difficult, if not impossible.

I envisage an option to specify a Drupal 6/7 mySql hostname, username and password, and the migration system to investigate what may be migrated, followed by an option to select data to migrate.

I believe the node import module provides a UI to import/migrate from CSV files.

webchick’s picture

Priority:Normal» Major
Issue tags:+Migrate critical

Escalating to major, and also creating the "Migrate critical" tag since this is actually critical (but not critical enough to block D8's release).

#2030501: [meta] Ensure that Drupal 6 sites have a functional upgrade path to either Drupal 7 or 8 before Drupal 6 loses security support is a D8 critical tracking this.

mikeryan’s picture

An update - right now, the main thing I want to get into the UI under development as migrate_upgrade before submitting to core is #2463909: Migrations should support non-installed default configurations (templates) - been busy pursuing other basic core API issues, but I'll try to get my focus back on that one. Actually, the next thing I plan to do is #2504759: Implement drush migrate-upgrade command, which will make it easier to develop/test the template support.

webchick’s picture

Status:Active» Needs review
Issue tags:+Needs usability review, +needs accessibility review

So while there's no patch here yet, https://www.drupal.org/project/migrate_upgrade actually exists and is functional. (I tried it myself last week on both mine and Dries's blogs :)).

If we want to get this in before RC1 (which really only makes sense once the other migrate criticals are resolved), we need to do some reviews of it ASAP from a UX/a11y POV. Tagging.

webchick’s picture

Testing instructions + screenshot at https://www.drupal.org/node/2257723

mikeryan’s picture

Quick question - what should the module name be for the UI? The existing contrib migrate_upgrade module contains both UI and a drush command, and at least for the short term (until we follow through with submitting it to the drush project) continue to exist for the drush. Should the upgrade UI in core be migrate_upgrade_ui? Or perhaps migrate_drupal_ui?

andypost’s picture

is this UI depends on migrate_drupal or migrate module itself?

mikeryan’s picture

Yes - it's a UI to run the migrations defined in migrate_drupal.

Pending any objections, right now I'm doing it as migrate_drupal_ui...

mikeryan’s picture

mikeryan’s picture

Oh, on the migrate_drupal_ui name - I just want to point out a counter-argument to using that, which is that while migrate_drupal provides a general framework for Drupal-to-Drupal migrations, this UI is narrowly focused on the straight, complete upgrade use case - a general-purpose UI for doing custom Drupal-to-Drupal migrations will go into contrib (D8 version of migrate_d2d).

webchick’s picture

I would just stick it in the Migrate Drupal module directly, personally. That might not be popular. :)

benjy’s picture

I would just stick it in the Migrate Drupal module directly, personally. That might not be popular. :)

I don't mind that idea if we could wrap it in a permission.

Otherwise migrate_upgrade_ui is probably better for contrib and migrate_drupal_ui better for cores more specific use case.

mikeryan’s picture

Issue tags:+Needs tests
StatusFileSize
new23.54 KB
PASSED: [[SimpleTest]]: [PHP 5.5 MySQL] 107,595 pass(es).
[ View ]

Here's a patch adding the upgrade UI to core as module migrate_drupal_ui. Obviously needs tests, UX work, etc., but it is functional. I've done some cleanup relative to the contrib migrate_upgrade version - the most visible change is providing more feedback while running the migrations.

Heading off the first question a code reviewer will have looking at this patch in isolation - what's with this MigrationCreationTrait and why isn't it simply implemented within MigrateUpgradeForm? The reason is the exact same functionality is needed by the migrate-upgrade drush command (currently in the contrib migrate_upgrade module, ultimately to be added to drush itself) - rather than having the drush command (or, for that matter, any alternate upgrade front ends people might want to build) reinvent the wheel, it should be reusable. We could make it a class instead - maybe even a service - feedback welcome.

andypost’s picture

  1. +++ b/core/modules/migrate_drupal_ui/migrate_drupal_ui.info.yml
    @@ -0,0 +1,10 @@
    +dependencies:
    +  - migrate
    +  - migrate_drupal

    +++ b/core/modules/migrate_drupal_ui/src/Controller/MigrateController.php
    @@ -0,0 +1,24 @@
    +    return $this->redirect('dblog.overview');

    so it depends on migrate_drupal and dblog

  2. +++ b/core/modules/migrate_drupal_ui/migrate_drupal_ui.install
    @@ -0,0 +1,16 @@
    +function migrate_drupal_ui_install() {
    +  $url = Url::fromUri('base:upgrade')->toString();
    +  drupal_set_message(t("The Migrate Drupal UI module has been enabled. Proceed to the <a href='@url'>upgrade form</a>.", array('@url' => $url)));

    useless

  3. +++ b/core/modules/migrate_drupal_ui/migrate_drupal_ui.module
    @@ -0,0 +1,19 @@
    + * @file
    + * Alert administrators before starting the import process.

    module do different

  4. +++ b/core/modules/migrate_drupal_ui/src/Form/MigrateUpgradeForm.php
    @@ -0,0 +1,164 @@
    +/*
    +    $form['files'] = array(
    +      '#type' => 'details',

    tbd

mikeryan’s picture

+++ b/core/modules/migrate_drupal_ui/migrate_drupal_ui.info.yml
@@ -0,0 +1,10 @@
+package: Core

Just a note that per https://www.drupal.org/node/2313651#comment-10269591, on the next roll of this patch let's set the package to Experimental.

phenaproxima’s picture

I think the Migrate Upgrade module should be rolled straight into Migrate Drupal, not as a submodule. It has no other use beyond Migrate Drupal. It was always intended to work with Migrate Drupal. IMO there's no point hiding the UI while exposing Migrate Drupal's API (such as it is); hide it all or don't.

mikeryan’s picture

Well, the migrate_drupal framework will be useful in contrib (migrate_d2d) to support custom Drupal-to-Drupal migration paths. That being said, I'm imagining migrate_d2d may end up using the core upgrade form anyway (altering it to be the first step of a wizard instead of a standalone form), so having the upgrade UI automatically present when migrate_drupal is enabled isn't necessarily unnecessary...

webchick’s picture

Part #1/N of my review. :)

Ok testing this patch, along with #2532534: Migration Files for Drupal 7 Comments against the database from webchick.net. This is a fairly bare-bones D7 site with no additional contribs other than Omega for the theme and Mollom for the spam fighting. It has been upgraded since Drupal 5, I believe. It also has some cruft (one time I tested advpoll module out, installed wp once or twice), but since most real-world databases will also have cruft, I think that's ok. ;)


First, I got very confused because I couldn’t find the module to turn it on. ;) That’s because it doesn’t yet have the “Core (Experimental)” package like the other Migration modules do.

RECOMMENDATION Change package to “Core (Experimental)”

Once that’s done, it looks like this:

Migrate, Migrate Drupal, and Migrate Drupal UI modules all listed under Core (Experimental) package on modules page.

Turning on Migrate Drupal UI automatically enables the Migrate and Migrate Drupal modules. So far so good!

Upon enabling it you get the following message at the top of the modules page, directing you to the upgrade form at /upgrade.

The Migrate Drupal UI module has been enabled. Please proceed to the upgrade form.

The only problem with that is if you missed that message in that moment, or if someone else enabled this module before you got there, there’s no way to figure out what you’re supposed to do. There’s no help page (link just says “No help is available for module Migrate Drupal UI.” ), and there’s also no “configure” link on the module from the modules page to jump me right there.

COMMIT BLOCKER Need a hook_help(), like all Drupal modules. Part of the docs gate.
RECOMMENDATION Add a “configure” property to the .info file that points off to /upgrade.

Once you meander over to /upgrade, you see this:

Form with fields for source DB and files directory location.

First, it asks for your source DB credentials, then the location of your public files directory.

OBSERVATION Oh sweet, you can enter the location of a local file directory, too. That probably makes a lot more sense than http://, given that would both a) greatly slow down the migration with 100s or 10000s of HTTP requests and b) pummel your live site, if that’s the address you gave, which seems to be what it’s asking for.

RECOMMENDATION I think remove the default here, and switch the help text around to mention the local files directory option first.

OBSERVATION What about private files? Mike pointed out that for D6 there was no distinction between private/public files, so one file destination will get both. D7 though has separate directories, and for that there’s #2547125: D7 file migration should allow selecting public/private/all files. Tagged that one as a Migrate D7 critical.

RECOMMENDATION Add a version selector to the form (at the top, most likely): Drupal 6 or Drupal 7. If Drupal 7 is selected, toggle visibility of a “Private files” text field. (This selector could also be form_altered() by contrib to provide for example a Drupal 5 selection as well, or used as a “rip-cord” in the event that D7 is not ready when we want to support D6 “officially”; just hard-code the field to D6 and let contrib alter other options in.)

RECOMMENDATION Remove the word “public” from that field’s help text if D6 is selected, since it’ll actually import all files in that case.

RECOMMENDATION More minor, but “Source site files/path” sounds a bit overly technical. I would call this “Source Files” (for the details group, to match “Source Database”) and “Files directory” for the textbook label.

Going to run through an actual migration next.

yoroy’s picture

Initial thoughts based on webchicks screenshots above:

Since this is a set of modules for a very specific and one-time scenario, that message with the link to the upgrade form should probably be persistent. Similar to how it works when you're in maintenance mode.

I think we need more careful and consistent wording, it's a mix of 'upgrade' and 'migrate' at the moment.

hass’s picture

Add a version selector to the form (at the top, most likely): Drupal 6 or Drupal 7. If Drupal 7 is selected, toggle visibility of a “Private files” text field.

We may better connect to the source database and detect the source version automatically. May add one extra step or use ajax.

webchick’s picture

Here's part #2/N, aka, "What happens when things go wrong." :)


Ok, heeeere we go! Testing a D7 -> D8 migration now.

First, I fill in my values:

Database/file values

Incidentally, if you fill in total garbage here, the error messages are quite obtuse:

Raw SQL errorRECOMMENDATION It would be ideal if we could re-use the same error handling as the installer, which prints much user-friendlier messages.

OBSERVATION There doesn't seem to be any validation around the file path field? Or at least I wasn't able to trigger it despite typing in garbage values for both local filepath and http:// address (that could've been due to "migrate already happened" errors tho, see below... but in any case we should probably validate the form first before we proceed).

This instantiates a Batch API process, like so:

Batch API indicates various things are being imported.

So far, so good.

Then, you hit a problem, such as:

Lots of PHP problems

Example output:

Missing bundle entity, entity type node_type, entity id story. (/Users/webchick/Sites/8.x/core/lib/Drupal/Core/Entity/EntityType.php:793)
Missing bundle entity, entity type comment_type, entity id comment_node_advpoll_binary. (/Users/webchick/Sites/8.x/core/lib/Drupal/Core/Entity/EntityType.php:793)
Missing bundle entity, entity type comment_type, entity id comment_node_advpoll_ranking. (/Users/webchick/Sites/8.x/core/lib/Drupal/Core/Entity/EntityType.php:793)
Missing bundle entity, entity type comment_type, entity id comment_node_story. (/Users/webchick/Sites/8.x/core/lib/Drupal/Core/Entity/EntityType.php:793)
Missing bundle entity, entity type comment_type, entity id comment_node_page. (/Users/webchick/Sites/8.x/core/lib/Drupal/Core/Entity/EntityType.php:793)
Missing bundle entity, entity type node_type, entity id advpoll_binary. (/Users/webchick/Sites/8.x/core/lib/Drupal/Core/Entity/EntityType.php:793)
Missing bundle entity, entity type node_type, entity id advpoll_ranking. (/Users/webchick/Sites/8.x/core/lib/Drupal/Core/Entity/EntityType.php:793)
Missing bundle entity, entity type node_type, entity id story. (/Users/webchick/Sites/8.x/core/lib/Drupal/Core/Entity/EntityType.php:793)
Missing bundle entity, entity type node_type, entity id story. (/Users/webchick/Sites/8.x/core/lib/Drupal/Core/Entity/EntityType.php:793)

RECOMMENDATION One annoying thing off the bat is that I can see there are errors, but in almost all cases they disappear faster than I can copy/paste or screenshot them. (something about php_code, something about files, something about story). The output upon moving to each step gets overrridden. Ideally, the messages would just append together and I could save the entire thing as a log of what happened (this is what update.php used to do).

RECOMMENDATION These things are errors, so should be styled as errors, so they're not missed.

Once the migration is complete, you’re automatically redirected to your home page with a status message on how you did:

Indicates success/fail of migrationsRECOMMENDATION Here we’re using “Import” which is a third synonymous word with Migrate/Upgrade. Agreed with yoroy; let’s definitely get the terminology cleaned up.

The "Review detailed migration log" link takes me to the normal database log, with the messages filtered down to migrate_drupal_ui automatically (nice):

Database log showing migrate errorsCOMMIT BLOCKER However, the big problem is that none of the PHP errors I saw from earlier are included here. :( Only the high-level info on what passed/failed. I tried resetting the db log filters, thinking these were possibly “php” errors and thus wouldn’t show up in the filtered view, but nothing there either. Migrate is somehow eating these PHP errors from the log..?

COMMIT BLOCKER When a migration fails, the message is “Import of Drupal 7 comments failed”. Well, thanks a lot. ;) How did it fail, and where do I start debugging?

OBSERVATION Some of these seem to report false positives. The row at the top of my report, “Imported Drupal 7 node title label” does not have a corresponding failure row, yet obviously did not work because my site still has no nodes. :)

So then you might think to yourself, "Ok, self. Let's apply some more migrate patches and try that migration again!" Oh, except you can't. You get this when you hit the "Perform upgrade" button again:

'Migration' entity with ID 'd7_comment' already existsCOMMIT BLOCKER We need a way to either re-do the migration or to clear it so you can start from scratch again.

RECOMMENDATION That error message is also super obtuse. Can we get that cleaned up to be more human-friendly? :) "Comments were already imported. Clear database and try again." or similar.

webchick’s picture

Two other notes to capture from IRC:

1) The migrate UI is currently associated with "administer site configuration" but on many sites that's given out kind of willy-nilly. We should most likely base it around "Administer software updates" instead. (Generally-speaking, only User 1 should ever be using this.)

2) While untested (I'll test this as soon as I can get something to actually migrate ;)), the likely thing that happens when using this UI is that any existing content/users/etc. you have are going to be overridden with those from the old site. For example, user 1 was admin, now it's "root." Node 1 was "test" now it's "Welcome to our site."

Given that, we probably need to do a couple of things:

- Big, fat, undeniable red warning on this page (maybe even make "Perform upgrade" red instead of "friendly blue") if site has existing content, and what the consequences of that will be.
- Possibly in the installer, make this an explicit choice: "are you installing D8 to build from, or are you installing D8 to migrate to?" (This would be a separate issue, since there's a lot to contend with there.)

webchick’s picture

Just a quick edit to remove display of those screenshots so you can still see the patch.

mikeryan’s picture

Status:Needs review» Needs work

Anything marked "done" here is in the code for the next patch, more complex things will come.

so it depends on migrate_drupal and dblog

Good point - for the moment, I'm adding the explicit dependency to the .info.yml file. But, is requiring dblog a good idea?

RECOMMENDATION Change package to “Core (Experimental)”

Done.

RECOMMENDATION Need a hook_help(), like all Drupal modules. Part of the docs gate.

hook_help() was there, but not for the help.page.migrate_drupal_ui route. Added some initial help text there.

RECOMMENDATION Add a “configure” property to the .info file that points off to /upgrade.

Hmm, it is there and working for me (at least with the changes I've made so far, none of which seemed specifically applicable)?

RECOMMENDATION I think remove the default here, and switch the help text around to mention the local files directory option first.

Done.

RECOMMENDATION Add a version selector to the form (at the top, most likely): Drupal 6 or Drupal 7. If Drupal 7 is selected, toggle visibility of a “Private files” text field. (This selector could also be form_altered() by contrib to provide for example a Drupal 5 selection as well, or used as a “rip-cord” in the event that D7 is not ready when we want to support D6 “officially”; just hard-code the field to D6 and let contrib alter other options in.)

Will do. Note that actually handling private files for D7 depends on #2547125: D7 file migration should allow selecting public/private/all files.

RECOMMENDATION Remove the word “public” from that field’s help text if D6 is selected, since it’ll actually import all files in that case.

Done for the moment, D7 variant to be added as part of the above.

RECOMMENDATION More minor, but “Source site files/path” sounds a bit overly technical. I would call this “Source Files” (for the details group, to match “Source Database”) and “Files directory” for the textbook label.

Done.

Since this is a set of modules for a very specific and one-time scenario, that message with the link to the upgrade form should probably be persistent. Similar to how it works when you're in maintenance mode.

I'll look into how that's done. Related thought - should we require the site to be in maintenance mode before allowing the upgrade operation?

I think we need more careful and consistent wording, it's a mix of 'upgrade' and 'migrate' at the moment.

So, as discussed on IRC - this form is specifically for a direct, simple upgrade from an earlier version of Drupal to Drupal 8, which is a special case of the general concept of migration. Other scenarios (including customized Drupal-to-Drupal migrations) will be supported in contrib and stick to the "migrate" language. So, my original thought was to make sure all user-facing text here uses the "upgrade" language - but, it is built on the migrate and migrate_drupal modules (which support the wider "migrate" use case), and any errors that bubble up from those modules will be using the "migrate" language, which could be confusing. Once we've got the framework nailed down such errors should be rare, but they will happen.

We may better connect to the source database and detect the source version
automatically. May add one extra step or use ajax.

We're doing the automatic detection now. The issue there is that only D7 needs that private file directory specified, and it's not practical to enable this with ajax from the db credentials. We also really want to keep this to a single form if we can. So, an explicit version selection doesn't seem overly burdensome, all things considered - the detection code will still be there to sanity-check the selection.

And one other thing, upgraded array() instances to [].

Now, on to the newer comments...

webchick’s picture

Incidentally, I think we should push the "whether or not to do D6/D7 as an explicit selector and/or how to show the public/private text fields for D7" issue over to #2547125: D7 file migration should allow selecting public/private/all files. That's likely to need some discussion, so we shouldn't hold this UI up which I was able to just successfully use to migrate webchick.net's content over to D8 (with a few migrate bug fix patches). YEAH! :D

mikeryan’s picture

RECOMMENDATION It would be ideal if we could re-use the same error handling as the installer, which prints much user-friendlier messages.

Will look into how the installer nicifies the messages.

OBSERVATION There doesn't seem to be any validation around the file path field? Or at least I wasn't able to trigger it despite typing in garbage values for both local filepath and http:// address (that could've been due to "migrate already happened" errors tho, see below... but in any case we should probably validate the form first before we proceed).

So, file_exists() on a local filepath and... get_headers() on an HTTP URL? Or, anything with ://, no reason you couldn't put the files on ftp://...

RECOMMENDATION One annoying thing off the bat is that I can see there are errors, but in almost all cases they disappear faster than I can copy/paste or screenshot them. (something about php_code, something about files, something about story). The output upon moving to each step gets overrridden. Ideally, the messages would just append together and I could save the entire thing as a log of what happened (this is what update.php used to do).

The problem here in circumstances like this is the volume - the modern batch API puts the messages above the progress bar, which will get shoved below the fold in no time. I'm sure there's a way to move the progress bar above though, but I don't like the idea of unbounded messages being saved and displayed, it can get crazy if some module has an "Undefined index" notice triggered by every node you migrate (all too common in my D7 experience).

RECOMMENDATION These things are errors, so should be styled as errors, so they're not missed.

Will look at that, MigrateMessageCapture will need to get the message levels.

COMMIT BLOCKER However, the big problem is that none of the PHP errors I saw from earlier are included here. :( Only the high-level info on what passed/failed. I tried resetting the db log filters, thinking these were possibly “php” errors and thus wouldn’t show up in the filtered view, but nothing there either. Migrate is somehow eating these PHP errors from the log..?

They're probably in the infamous message tables, we need a way to expose them here in a structured way... or just pipe them to dblog.

COMMIT BLOCKER When a migration fails, the message is “Import of Drupal 7 comments failed”. Well, thanks a lot. ;) How did it fail, and where do I start debugging?

Good question. The reason it failed should have been emitted right before that message (almost certainly because a migration it was dependent on failed). Not sure why you didn't see "Migration d7_comment did not meet the requirements: [details here]" as the log message immediately before that one.

OBSERVATION Some of these seem to report false positives. The row at the top of my report, “Imported Drupal 7 node title label” does not have a corresponding failure row, yet obviously did not work because my site still has no nodes. :)

"Drupal 7 node title label" did most likely succeed - that's (if I'm not mistaken) is just about setting the title label on node types, it doesn't migrate the nodes themselves.

Anyway, clearly messaging in general needs attention - what should go on the batch page, what should go into the dblog, how to expose the further detail from migrate's message tables....

Bojhan’s picture

Mike one way to work on the messaging a bit more is to have them listed here, perhaps as a screenshot with the quoted text. Its quite hard to try and trigger all the situations in which messages could appear to help you in writing better messages.

I notice some of the spacing in webchicks screenshots are off, is this simply artefact of the screenshot or does it not vertically align?

mikeryan’s picture

Mike one way to work on the messaging a bit more is to have them listed here, perhaps as a screenshot with the quoted text. Its quite hard to try and trigger all the situations in which messages could appear to help you in writing better messages.

Some of the kinds of messages seen in Angie's examples are:

  • Simple progress messages. On the batch page as the migration is running you see things like "Imported Drupal 7 field configuration" and "Currently importing Drupal 7 filter format configuration". In the dblog (displayed at the end) you see "Importing Drupal 7 node title label" (this seems kind of pointless) and "Imported Drupal 7 node title label".
  • If a migration as a whole fails, something like "Import of Drupal 7 menu links failed" will appear as the process is running, and in dblog at the end.
  • The SQLSTATE "Unknown database" error is of course being thrown by the Database API, apparently install.php intercepts such errors and makes them look better.
  • "Missing bundle entity, entity type node_type, entity id story. (/Users/webchick/Sites/8.x/core/lib/Drupal/Core/Entity/EntityType.php:793)" is being thrown by the entity system. Since migration touches all sorts of APIs, pretty much any error that entity/configuration/field/etc. systems can generate can be exposed during migration.
  • Not seen here - if a migration depends on another migration which did not complete, there should be a message about that like
    <?php
            $this
    ->t('Migration @id did not meet the requirements. @message @requirements', array(
             
    '@id' => $this->migration->id(),
             
    '@message' => $e->getMessage(),
             
    '@requirements' => $e->getRequirementsString(),
            )),
    'error');
    ?>

    I don't know why that didn't show up on the batch page or in the log. I do know the message itself needs work.

I notice some of the spacing in webchicks screenshots are off, is this simply artefact of the screenshot or does it not vertically align?

Do you have a specific example? They look like I would expect.

Going back to bits I missed from Angie:

COMMIT BLOCKER We need a way to either re-do the migration or to clear it so you can start from scratch again.

RECOMMENDATION That error message is also super obtuse. Can we get that cleaned up to be more human-friendly? :) "Comments were already imported. Clear database and try again." or similar.

This form actually is performing two steps - it generates the appropriate migrations based on the source site and destination site configurations (e.g., if the book module isn't enabled on both sites, no book migration config entity will be created), and then it invokes the import operation on those generated migrations. So, we could theoretically offer to delete the existing migrations first, or to overwrite them, but what happens with any partial content/configuration already imported? Content could be rolled back, but no one's keeping track of the original configuration. At any rate, in what circumstances would the results of running it again differ? In this case, with the application of patches, but I don't think recovering from running with bleeding-edge patches is a use-case we need to cover. Indeed, I would argue that this is a backup-your-database-and-be-prepared-to-restore-it operation - it is a major version upgrade, after all.

The migrate UI is currently associated with "administer site configuration" but on many sites that's given out kind of willy-nilly. We should most likely base it around "Administer software updates" instead. (Generally-speaking, only User 1 should ever be using this.)

Done.

While untested (I'll test this as soon as I can get something to actually migrate ;)), the likely thing that happens when using this UI is that any existing content/users/etc. you have are going to be overridden with those from the old site. For example, user 1 was admin, now it's "root." Node 1 was "test" now it's "Welcome to our site."

Well, at the moment user 1 won't get touched, but we need to figure out how to handle it: #2560637: Improve handling of uid 1 during migration

Big, fat, undeniable red warning on this page (maybe even make "Perform upgrade" red instead of "friendly blue") if site has existing content, and what the consequences of that will be.

Backup all your shit! Now!

(dang, style="color: red" doesn't work)

Possibly in the installer, make this an explicit choice: "are you installing D8 to build from, or are you installing D8 to migrate to?" (This would be a separate issue, since there's a lot to contend with there.)

Yes, we discussed this in IRC - if the answer is "upgrading", we would install migrate_drupal_ui (or whatever it's called) and redirect straight to /upgrade at the end of the install process. Here's another consideration, however - if, say, that book module is enabled on the source site but not in D8, book hierarchies will not be migrated. It was an explicit decision early on that the upgrade process would not automatically enable modules in D8, but we do need a space somewhere for the site administrator to have the opportunity to do it. And they need info on what modules are at risk of not coming over, which we can't give them until they've submitted the credentials on the upgrade form...

mikeryan’s picture

So, along with the question of giving administrators information on what stuff won't migrate until the relevant D8 modules are installed, there's https://www.drupal.org/node/2562073#comment-10289183. With the PHP module out of core, we will be converting php_code filters to filter_null, meaning that content using a format with the php_code filter will not display. Big WTF for anyone using php_code who didn't carefully read the release notes.

I know we'd like to have the single form, single click, but I'm feeling more and more we need to have a confirmation page that does some analysis of the source and destination sites before executing the migration. This page might:

  1. List all source site modules that will be imported based on the current configuration.
  2. List any D8 modules not enabled for which there is source data to import (and allow them to be enabled directly from this page?).
  3. If the php_code filter is in use, warn that content using it will not display without either installing the contrib php module, or editing it after migration to set a different format and remove any PHP.

Thoughts?

webchick’s picture

Well, something like that sounds absolutely fantastic to me.

So page 1: Just your DB credentials / File locations.

page 2: "Howdy, partner. It looks like you're running Drupal 7, with 46 contributed modules. You should know the following things about your site:

- EVERYTHING YOU HOLD DEAR WILL BE OVERWRITTEN. MAKE SURE YOU HAVE BACKUPS.

- PHP Code blah blah
- Foo, Bar, Baz modules don't have migrations, their stuff won't be moved over.
- something else.

[Proceed!]

mikeryan’s picture

I think the best way to move forward on the various feedback/ideas is to work on them in the migrate_upgrade project, where they can be done in separate issues/patches, and the improvements made immediately available to people doing upgrades today. It should be fairly simple to script turning that module into a core patch (mv the files, do a few renames, rewrite some namespaces), so at appropriate checkpoints I can post an updated patch here.

webchick’s picture

Issue summary:View changes
Status:Needs work» Postponed

Yep, agreed.

mikeryan’s picture

OK, I've backported all the work I've done here (plus a little more message tweaking) to migrate_upgrade). Next steps:

  1. Create individual issues for outstanding topics above, parented to this issue.
  2. Write a script for regenerating this core patch from the current migrate_upgrade module.
  3. Use that script at appropriate checkpoints to update the patch in this issue.