Problem/Motivation

With Drupal 8.4.x as of 9 Mar 2017, if you try to migrate a multilingual Drupal 6 site using CCK and several i18n/translation modules, you will run into some problems. [earlier 8.x versions -- there were more problems -- see original summary section below].

The main difficulty remaining is taxonomy. If you are using i18ntaxonomy to translate taxonomy terms:
- The taxonomy translations will not come through
- Node translations in foreign languages tagged with the taxonomy will not come through tagged at all

I also had a problem with one of the vocabularies, that the vocabulary, terms, and nodes came through, and the nodes have the tags on them, but when you go to the tag page, the node isn't listed.

There were also several other problems with the migration; see comment #14 for details... issues will be filed.

Proposed resolution

No idea.

Remaining tasks

Unsure.

User interface changes

Migration will work better.

API changes

Data model changes

Original report (outdated)

I recently tried out the experimental Migrate, Migrate Drupal, and Migrate UI modules for the first time, on an existing Drupal 6 site (I know, it should have been done long ago. But.).

I ran into a problem where in some nodes having multiple revisions, the latest revision was imported with an empty Body field. The older revisions of the same nodes are fine.

I looked in the database, and the node_revision__body table is missing an entry for the latest revision (it has entries for the earlier revisions), and the node__body has no entry at all for that node ID.

There were some errors in the migrate log, which looked like this:

Source ID 951: Unable to get entity

(with no other information, but they occurred during the time when the import was getting node revisions). However, there was not a message that matched the node ID or revision ID of one of the nodes I saw this problem on. There were quite a few with this problem, but not all nodes with revisions had the problem.

I have no idea how to debug this or what other information I can provide... but it happened. It may be limited to one content type, which happened to have the machine name "article" on my old site -- at least, I have only seen it on Article content so far, but I haven't looked through the other content yet. I deleted the Standard install profile's Article and Page content types before I did the migration. Page content seems to have come through without any problem. ???

Files: 
CommentFileSizeAuthor
#2 d6_migration_empty-2793947-2.patch726 bytesmikeryan

Comments

jhodgdon created an issue. See original summary.

mikeryan’s picture

I've tried adding revisions in the dev copy of my D6 site, reverting them, etc., but can't reproduce this with 8.2.x (the migration system hasn't had any 8.3.x-specific changes yet).

Source ID 951: Unable to get entity

Given this is happening during revision migration, I have to believe it's related in some way - even if it's not happening on the nodes you're seeing the missing bodies on, I'd bet it shares a root cause. Looking at the code, during a node revision migration, this would most likely happen due to an attempt to retrieve the node the revision is tied to (side note: there's no reason we shouldn't include the node ID it was attempting to retrieve in the message, which would be helpful here). Well, just for the hell of it, I'm attaching a (completely untested) patch to do that...

Anyway, in this message 951 is the D6 vid of the revision - can you see what the nid is, and make sure that was properly migrated into D8?

Apart from that:

  1. Any "interesting" contrib or custom modules on either the D6 or D8 sides? Particularly workflow modules, which may do interesting things with revisions...
  2. Did you create any content manually on the D8 side before running the migration? If so, maybe there are collisions between source and destination nids and vids...
jhodgdon’s picture

Thanks for investigating!

So... in answer to your questions...
1. No modules on either side related to workflow/revisions. On the D8 side, when I ran the migration, there were no contrib modules at all, actually. I have since added only Pathalogic.

2. There was zero content on the destination d8 side before I did the migration. All I did was:
- Install with Standard profile.
- Uninstall a few unneeded Core modules, such as Search and History
- Delete the provided Article and Page content types that come with the Standard install profile

3. Everything seems to have been migrated successfully, except for most of the Article content items missing their body in the latest revision. Three of the 25 articles were OK, and the other 22 were missing bodies in the latest revisions; the older ones had their revisions.

4. I also took a look at the vid 951. It is a different content type, and an older revision of that node (not the most current one). Let's see. On the old site, nid 666 has three revisions: 950, 951, and 954. On the new site, ... it looks like that node was indeed not migrated, assuming that the nid would be the same. Oops. I did not have any error messages about that node being missed.

jhodgdon’s picture

As another note, that node in d6 is the Spanish translation of an English node. Something weird may be happening due to that?

mikeryan’s picture

Ah, translations... We have minimal support so far for translation migration, just nodes themselves, not revisions (see #2746541: Migrate D6 node revision translations to D8). Note that in D8, translations are not separate nodes, but rather revisions of the primary node, so it is expected that nid 666 as a translation will not appear in D8 - the Spanish translation should now appear as a translation on the English node.

Are the nodes with the missing bodies translations, or do they have translations?

mikeryan’s picture

Or, perhaps a better way to phrase the question - are the missing-body nodes different from the successful nodes in some translation-related way?

jhodgdon’s picture

OK, this is morphing into two separate issues.

a) The articles with missing bodies in the latest revision. This content type is untranslated. In D6, they are content type "article" (which I created there), and that content type was created for me by the migration.

An example is nid 13, which in D6 has revisions with vid 45, 49, 78, and 79. All revisions have an empty language code field (which in D8 was migrated into "und"). The bodies of the first three revisions came through fine in the migration, but the body of the final revision is empty (there is an entry in the node_revision table in D8 for vid 79, and the other fields are fine, but node_revision__body does not have an entry for that revision).

Hm. I had thought that all but 3 of the articles were OK, but I actually think that none of them was OK in the migration.

b) The translated nodes that had error messages during revision import. These are a separate content type, which is the only one that is translated on my site. And it sounds like you know about the node revision problem with translations. In any case... let's see. Node 666 in D6 is a translation of node 664. And it looks like node 664 came through successfully with an English and a Spanish translation. The Spanish one has no revision history, but it seems to have the latest revision. So, that's OK.

c) Other nodes had no problems. For instance, I have another content type called "news_item" and one called "page". None of them had language codes or translations in D6, but it was only "article" nodes that had import problems.

jhodgdon’s picture

Hm. Actually, I take it back. I thought that only my articles were affected. Actually, quite a bit of the content didn't come through with the body fields -- I just found some more that were not articles that also had blank bodies in the latest revision.

So, I'm not sure what to do here. My blocks, menus, views, etc. didn't come through very well either (or not at all in the case of the views, which is apparently a known problem)... I was hoping to at least get the content migrated, but it may actually save me time to do it manually with copy/paste, rather than migrating and then trying to figure out what worked and what was broken. :(

I thought we were not going to end-of-life Drupal 6 until we had a viable migration strategy -- this doesn't seem very viable.

Sorry. I'm not ranting at you Mike -- you've done a LOT to get it to this point! Just wondering about the overall strategy from the project maintainers... I know no one wanted to maintain Drupal 6 (myself included), but if there's no way to migrate it to 8 that actually works, even just the content, for a site like mine (relatively uncomplicated, just a few CCK fields)... I'm not feeling very good about this decision to end-of-life Drupal 6. It may be feasible on a relatively small site to use copy/paste to migrate, but... not great for anything else.

quietone’s picture

Issue tags: +migrate-d6-d8

Adding tag.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.0-alpha1 will be released the week of January 30, 2017, which means new developments and disruptive changes should now be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Gábor Hojtsy’s picture

@jhdogdon pointed to this issue for bugs with content translation migration but as per #7 there is no unknown new issue with it and as per #8 there may be but its very unclear to me. Reading the comments I could not collect what is going on, reading the issue summary it is not up to date anymore. :/

jhodgdon’s picture

OK, I still have the D6 database available. I'll try a migration into a clean latest-8.x-dev site and see what happens. I eventually abandoned the effort of migrating my d6 site and just did it manually (copy/paste). Which was very sad.

Gábor Hojtsy’s picture

I agree its said, but these tests are how we figure out the bugs, so thanks for doing it :)

jhodgdon’s picture

OK, I did a test. TL;DR -- I will post a summary in the next comment.

Summary of my D6 site... not a very big site:

- 6 content types using CCK:
-- article: fields of type text, file. Not translated. 23 nodes. 84 revisions total. Has comments.
-- download: Fields of type text, file, integer. Not translated. 11 nodes. 88 revisions. Has comments.
-- download_external: Fields of type text, integer, link. Not translated. 6 nodes. 18 revisions.
-- hodgblog: Fields of type file (one is image), taxonomy. Translated using core Content Translation + CCK Translation. The taxonomy vocabulary for this one is also translated. Also has comments. 80 nodes (40 in English and 40 Spanish translations). 151 revisions.
-- news_item: Has taxonomy. Not translated. Also has comments. 74 nodes. 141 revisions.
-- page: No fields (except the native D6 body field). Not translated. 20 nodes. 113 revisions.
There are 388 comments total.

- Several custom blocks
- One custom menu, plus links in Primary Links
- 4 views with several displays (pretty standard stuff, like a blog teasers page, recent articles, taxonomy list for the blog)
- One user

Here's what I did to test migrating this site today:
- Downloaded the 8.4.x-dev tarball from project/drupal
- Installed with Standard install profile, without update manager, using the UI
- Uninstalled a few core modules I didn't need, for speed (like search, history, help, tour), from the UI
- Deleted the Page and Article content types, from the UI
- Installed the 4 multilingual modules, plus Migrate, Migrate Drupal, and Migrate UI. Note: Did not add Spanish language.
- Cleared the dblog

Now I am ready to migrate:
- When I installed the Migrate UI module, it pointed me to go to /upgrade, so I did that.
- Read through the instructions, double checked that the field modules I should need are installed (like Image, File)
- Clicked Continue, entered database and file directory information.
- [PROBLEM] It tells me that a LOT of stuff that I think should be included is missing from migrate, such as filefield, imagefield, link, number, and date. Most of these are in Core now. I don't have any date fields. I checked a few of the missing D6 projects (filefield, imagefield, link), and didn't see anything about any d8 modules I could enable that would provide migration tips. So, proceeded.

Lots of errors (in no particular order):

a) [PROBLEM] Files not existing. This is because the directory I entered ended in a slash. Oops. I was not warned, and the slash was not removed. That is really dumb. I'll file an issue about that, if there isn't one already.

b) [PROBLEM]

Source ID node,full,hodgblog,field_images: Failed to lookup field type array ( 0 => 'filefield', 1 => 'hidden', ) in the static map.

I have no idea what that means. Maybe it's related to the image files being missing.

c) [PROBLEM]

Notice: Undefined index: value in Drupal\field\Plugin\migrate\process\d6\FieldInstanceSettings->transform() (line 37 of /opt/www/d8migrate/core/modules/field/src/Plugin/migrate/process/d6/FieldInstanceSettings.php) 

I'll file an issue about that, if there isn't one already.

d) [PROBLEM]

Notice: Undefined index: d6_i18n_user_settings in Drupal\migrate_drupal_ui\Form\MigrateUpgradeForm->buildConfirmForm() (line 1046 of /opt/www/d8migrate/core/modules/migrate_drupal_ui/src/Form/MigrateUpgradeForm.php) 

I'll file an issue about that.

e) [PROBLEM]

 	Source ID 951: Unable to get entity

These seem to be while migrating URL aliases was active (according to the Migrate UI in progress). Perhaps I had some aliases linked to nodes I deleted? Unsure. The last alias in my url_alias table in the d6 database is ID 690, amd the last node is ID 704, so I don't know what these IDs are.

f) [PROBLEM]

SQLSTATE[42S02]: Base table or view not found: 1146 Table 'poplarware_d6.profile_fields' doesn't exist: SELECT COUNT(*) AS expression FROM (SELECT 1 AS expression FROM @profile_fields pf) subquery; Array ( ) 

Then the next message is

 	Operation on User profile field instance configuration failed

I don't know why it was looking for profile_fields but I don't have that table. I'll file an issue.

g)

Notice: Undefined index: allowed_html in Drupal\filter\Plugin\migrate\process\FilterSettings->transform() (line 45 of /opt/www/d8migrate/core/modules/filter/src/Plugin/migrate/process/FilterSettings.php) 

This is due to a missing text filter module that I had in D6.

I wasn't really warned very well about this problem... but to fix it, or at least get around it, I had to go to Configuration / Text formats and save the Full HTML and Filtered HTML text formats, to remove the null filters that didn't exist. Without doing that, content is not shown if it has those text formats.

OK. So let's see what the result is, as far as content.

1. [PROBLEM] Taxonomy. It looks like the English terms came through, but the translations of the translated vocabulary did not get migrated. The d6 site was using the i18ntaxonomy module to translate terms.

[PROBLEM] -- Another problem: although the taxonomy terms are shown on the nodes that are tagged with the two taxonomies, when I go to the term pages, the corresponding nodes are not shown there, for one of the two taxonomies. Ironically, the translated one is working (showing the English nodes), but the untranslated one isn't.

2. Nodes:
- The untranslated content items came through fine, with revisions. The display order of all of them is very weird (comments are above body), and images/files are missing (see above).
- The translated content items came through mostly fine, but with no revision history (I think no revisions on translations is a known issue and/or by design current, so that is OK).
[PROBLEM] However, the Spanish translations have no taxonomy terms on them (English or Spanish). They definitely do in the D6 site.

3. Custom Blocks: seem to have come through. They're all dumped into the main area of the page. Oh well.

Gábor Hojtsy’s picture

Taxonomy translation migrations are still missing, tracked at #2225781: Migrate D6 i18n taxonomy vocabularies and #2784371: Migrate D6 i18n taxonomy term language (but not yet translations) for D6 and will then get need to get ported to D7 too.

jhodgdon’s picture

Issue summary: View changes

So, in summary: With 8.4.x, the only i18n migration problem seems to be the taxonomy:
- The taxonomy terms translated with the d6 contrib i18ntaxonomy module did not come through as translated.
- The nodes in Spanish that were tagged with these terms did not come through tagged at all (with the English or Spanish versions of the tags).
- The other vocabulary, which wasn't translated: nodes came through tagged, but they don't show up on the taxonomy page for those tags.

There were several other migrate problems that are separate issues. I'll research/file issues.

Meanwhile, updating summary here.

jhodgdon’s picture

Title: D6 migration - empty revision bodies » D6 migration testing with i18n

New issue title...

Summary of issues related to the latest test (comment #14):

a) Translations of taxonomy did not come through. Existing issues:
#2225781: Migrate D6 i18n taxonomy vocabularies
#2784371: Migrate D6 i18n taxonomy term language (but not yet translations)

b) Translated nodes with taxonomy terms didn't get the taxonomy terms attached. Filed new issue:
#2859297: Migrate d6 translated nodes with translated taxonomy (I do not think it is part of the above two issues.)

c) Translated node revisions not migrated. Existing issues:
#2746541: Migrate D6 node revision translations to D8

d) Taxonomy pages... To review, on the news content type, the taxonomy terms are on the nodes, but the taxonomy page doesn't show the nodes. I just investigated this. The problem is that the default taxonomy term view has a filter that restricts to the current page's language. So, if I go to the /term/12 page, the language is English. But aside from my hodgblog nodes, all of the migrated nodes came through with language code "und", not "en". Is this a bug, or by design? It makes for a somewhat broken site.

e) Various errors and messages seen during the migrate process. Issues:
(new issue) #2859304: Migrate UI - lots of irrelevant warnings at start
(existing issue -- added comment) #2562845-20: Validate the file path field before executing migration?
(existing issue) #2827175: Undefined index: none FieldInstanceSettings.php:37
(new issue) #2859310: Migrate UI: Got undefined index errors in dblog from MigrateUpgradeForm
(new issue) #2859315: SQL error from profile_fields when migrating d6 to d8 without Profile module

I think that this error noted in #14:
Source ID 951: Unable to get entity
was probably related to the translation revisions or translation taxonomy terms, because the only ID 951 I can find in my d6 database is a revision ID for one of the translated nodes. So ... not making a new issue for that error. Hopefully it's one of the above.

jhodgdon’s picture

So aside from #14 (d) I think we can close this issue. Any thoughts on whether that should be a new issue, an existing issue, or it's not an issue even though it seems wrong to me?

jhodgdon’s picture

Status: Active » Closed (duplicate)

I decided to file #2859359: Migrating content with missing language is problematic for the remaining issue -- it may be closed as Won't Fix, but at least it has been recorded.

This issue is now a duplicate of all of those other issues listed on #17 and here, and should be closed.