Follow-up to #2225681: Migrate D6 i18n blocks translated strings

Problem/Motivation

A migration for custom block (D6 boxes) translated strings is needed. The translations for the title and body of custom blocks is in the i18n strings module.

Proposed resolution

Write a custom source plugin. Note the translation for the title and body of custom blocks are on different rows in the locales_target source. The source plugin needs to keep track of what item is being translated on a row.
And, of course, tests.

CommentFileSizeAuthor
#96 interdiff.txt1.07 KBquietone
#96 2909444-96.patch12.34 KBquietone
#90 interdiff-88-90.txt1.14 KBquietone
#90 2909444-90.patch13.4 KBquietone
#88 interdiff.txt1.31 KBquietone
#88 2909444-88.patch13.38 KBquietone
#81 interdiff.txt3.03 KBquietone
#81 2909444-81.patch13.26 KBquietone
#75 interdiff.txt1.25 KBquietone
#75 2909444-75.patch13.3 KBquietone
#73 interdiff.txt4.38 KBquietone
#73 2909444-73.patch13.3 KBquietone
#70 2909444-70.patch13.06 KBjofitz
#70 interdiff-69-70.txt927 bytesjofitz
#69 2909444-69.patch12.98 KBjofitz
#65 interdiff.txt855 bytesquietone
#65 2909444-65.patch12.96 KBquietone
#63 2909444-63.patch12.77 KBquietone
#63 interdiff.txt4.97 KBquietone
#62 interdiff.txt561 bytesquietone
#62 2909444-62.patch12.95 KBquietone
#61 interdiff.txt1.2 KBquietone
#61 2909444-61.patch12.99 KBquietone
#59 interdiff.txt1.81 KBquietone
#59 2909444-59.patch11.42 KBquietone
#56 interdiff.txt4.05 KBquietone
#56 2909444-56.patch13.5 KBquietone
#47 2909444-47.patch13.76 KBjofitz
#47 interdiff-44-47.txt6.66 KBjofitz
#44 2909444-44.patch13.45 KBquietone
#36 interdiff.txt5.26 KBquietone
#36 2909444-36.patch16.13 KBquietone
#34 interdiff-32-34.txt2.22 KBjofitz
#34 2909444-34.patch17.62 KBjofitz
#32 interdiff.txt5.87 KBquietone
#32 2909444-32.patch17.58 KBquietone
#31 interdiff.txt5.09 KBquietone
#31 2909444-31.patch15.43 KBquietone
#30 2909444-27.patch15.04 KBquietone
#28 interdiff.txt3.56 KBquietone
#25 interdiff.txt4.8 KBquietone
#25 2909444-25.patch15.02 KBquietone
#19 interdiff-17-19.txt1.02 KBjofitz
#19 2909444-19.patch14.67 KBjofitz
#17 interdiff-2909444-14-16.txt469 bytesrakesh.gectcr
#17 2909444-16.patch15.22 KBrakesh.gectcr
#14 interdiff-2909444-11-14.txt352 bytesrakesh.gectcr
#14 2909444-14.patch15.21 KBrakesh.gectcr
#11 2909444-11.patch14.66 KBjofitz
#11 interdiff-7-11.txt2.45 KBjofitz
#7 2909444-7.patch12.21 KBjofitz
#7 interdiff-4-7.txt3.52 KBjofitz
#4 2909444-4.patch12.01 KBjofitz
#4 interdiff-2-4.txt20.77 KBjofitz
#2 2909444-2.patch12.02 KBmaxocub

Comments

maxocub created an issue. See original summary.

maxocub’s picture

Status: Active » Needs review
StatusFileSize
new12.02 KB

This is a split of the patch from #2225681: Migrate D6 i18n blocks translated strings for easier reviews and commit. No change has been made to the remaining code.

maxocub’s picture

Status: Needs review » Needs work

1.

+++ b/core/modules/content_translation/migration_templates/d6_custom_block_translation.yml
@@ -0,0 +1,39 @@
+label: Custom blocks

Custom block translations?

2.

+++ b/core/modules/content_translation/src/Plugin/migrate/process/d6/i18nTranslation.php
@@ -0,0 +1,31 @@
+ * Processes an i18n translation.

Can we add some documentation & code example? Like that we expect that the untranslated value is the first item in the array and that the translated is second?

3.

+++ b/core/modules/content_translation/src/Plugin/migrate/process/d6/i18nTranslation.php
@@ -0,0 +1,31 @@
+    list($untranslated, $translated) = $value;

Can we check that $value is actualy composed of 2 strings?

4.

+++ b/core/modules/content_translation/src/Plugin/migrate/source/d6/BoxTranslation.php
@@ -0,0 +1,99 @@
+      ->fields('lt', ['lid', 'translation', 'language']);
+    $query->orderBy('b.bid');

Nit: Can be chained, I think.

5.

+++ b/core/modules/content_translation/src/Plugin/migrate/source/d6/BoxTranslation.php
@@ -0,0 +1,99 @@
+    if (empty($this->idMap->lookupDestinationIds([
+      'bid' => $bid,
+      'language' => $language,
+    ]))
+    ) {

I'm not sure what the standard says, but I think this would be more readable on one line.

6.

+++ b/core/modules/content_translation/src/Plugin/migrate/source/d6/BoxTranslation.php
@@ -0,0 +1,99 @@
+    $query = $this->select('i18n_strings', 'i18n')
+      ->fields('i18n', ['lid']);
+    $query
+      ->condition('i18n.property', $property2)
+      ->condition('i18n.objectid', $bid);
...
+    $query->condition('lt.language', $language);
+    $query->addField('lt', 'translation');

This can all be chained.

7.

+++ b/core/modules/content_translation/tests/src/Kernel/Migrate/d6/MigrateCustomBlockContentTranslationTest.php
@@ -0,0 +1,71 @@
+    ConfigurableLanguage::createFromLangcode('en')->save();
+    ConfigurableLanguage::createFromLangcode('fr')->save();
+    ConfigurableLanguage::createFromLangcode('zu')->save();

No need to manualy create each languages, just need to add 'language' to the executed migrations below.

8.

+++ b/core/modules/content_translation/tests/src/Kernel/Plugin/migrate/source/d6/BoxTranslationTest.php
@@ -0,0 +1,134 @@
+ * Tests i18n block title source plugin.

This needs a better comment, since it's not only the title that is migrated/tested.

9.

--- /dev/null
+++ b/core/modules/content_translation/migration_templates/d6_custom_block_translation.yml

--- /dev/null
+++ b/core/modules/content_translation/src/Plugin/migrate/source/d6/BoxTranslation.php

--- /dev/null
+++ b/core/modules/content_translation/tests/src/Kernel/Migrate/d6/MigrateCustomBlockContentTranslationTest.php

--- /dev/null
+++ b/core/modules/content_translation/tests/src/Kernel/Plugin/migrate/source/d6/BoxTranslationTest.php

I'm not sure that content_translation is the right place for this code. I think it should go in block_content, like the node translation migrations are in the node module.

jofitz’s picture

Status: Needs work » Needs review
StatusFileSize
new20.77 KB
new12.01 KB
  1. Changed label.
  2. Added simple documentation and example
  3. @todo Can you explain what you mean, please?
  4. Chained
  5. Combined onto one line.
  6. Chained
  7. Removed lines and added 'language' migration.
  8. Tweaked comment.
  9. Moved files from content_translation to block_content
jofitz’s picture

Status: Needs review » Needs work

Setting back to Needs Work for someone to address #3.3.

maxocub’s picture

About #3.3:

If someone mis-configure this plugin by doing something like

foo:
  plugin: i18n_translation
  source: bar

or like

foo:
  plugin: i18n_translation
  source:
    - bar

the list() won't work and we should throw an exception.

jofitz’s picture

Status: Needs work » Needs review
StatusFileSize
new3.52 KB
new12.21 KB

Correct the namespace in the moved files.
Address #3.3.

The last submitted patch, 4: 2909444-4.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

Status: Needs review » Needs work

The last submitted patch, 7: 2909444-7.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

maxocub’s picture

Oh, and one last thing, the i18n_translation process plugin will need a unit test.

jofitz’s picture

Status: Needs work » Needs review
StatusFileSize
new2.45 KB
new14.66 KB

Added a unit test for the i18n_translation process plugin.

I haven't been able to get to the bottom of the remaining test failure - I can't understand why this fails when MigrateCustomBlockContentTranslationTest passes. My only suspicion is that 'language' should be one of the required migration_dependencies for d6_custom_block_translation, but I am unable to test that on my local build.

Status: Needs review » Needs work

The last submitted patch, 11: 2909444-11.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

rakesh.gectcr’s picture

Assigned: Unassigned » rakesh.gectcr
rakesh.gectcr’s picture

Status: Needs work » Needs review
StatusFileSize
new15.21 KB
new352 bytes
migration_dependencies:
   required:
     - block_content_type
+    - language
 provider:
   - block_content
   - migrate_drupal

Passed the test in my local, Will give it a try

maxocub’s picture

I think this 'language' dependency should be optional since this migration file is not only used in multilingual migrations.

maxocub’s picture

Issue tags: +Vienna2017

Also adding the vienna tag for our kanban.

rakesh.gectcr’s picture

StatusFileSize
new15.22 KB
new469 bytes

Oops, Here we go,

maxocub’s picture

Status: Needs review » Needs work

Damn, I didn't look at it enough in my last review, but now I think that this language dependency should really be in the d6_custom_block_translation.yml file and it should be a required dependency. Sorry about that, rakesh.gectcr!

jofitz’s picture

Status: Needs work » Needs review
StatusFileSize
new14.67 KB
new1.02 KB

Moved the migration dependency as recommended by @maxocub.

maxocub’s picture

Assigned: rakesh.gectcr » Unassigned
Status: Needs review » Reviewed & tested by the community

All concerns have been addressed, I think this is ready to be RTBCed.

quietone’s picture

@rakesh.gectcr, @Jo Fitzgerald and @maxocub thanks for getting this to RTBC. It was a goal of mine to get this done in Vienna. And it happened without me doing anything. Love the community.

maxocub’s picture

@quietone should be credited on this issue since she wrote the original patch in #2225681: Migrate D6 i18n blocks translated strings, which was split in two issues.

webchick’s picture

Assigned: Unassigned » phenaproxima
Status: Reviewed & tested by the community » Needs review

@phenaproxima says he would like to review this.

quietone’s picture

Status: Needs review » Needs work
  1. +++ b/core/modules/content_translation/tests/src/Unit/process/i18nTranslationTest.php
    @@ -0,0 +1,72 @@
    +namespace Drupal\Tests\content_translation\Unit\process;
    

    This should be Drupal\Tests\content_translation\Unit\migrate\process

  2. +++ b/core/modules/content_translation/tests/src/Unit/process/i18nTranslationTest.php
    @@ -0,0 +1,72 @@
    +  public function testI18nTranslationTest($value, $expected) {
    

    Should have parameter definitions for $value and $expected in the doc block

  3. +++ b/core/modules/content_translation/tests/src/Unit/process/i18nTranslationTest.php
    @@ -0,0 +1,72 @@
    +  public function testI18nTranslationInvalidTest($value) {
    

    Ditto.

  4. +++ b/core/modules/content_translation/tests/src/Unit/process/i18nTranslationTest.php
    @@ -0,0 +1,72 @@
    \ No newline at end of file
    

    Add a newline.

This needs to be moved back to content_translation. It was split in #105 of #2225681: Migrate D6 i18n blocks translated strings.

The node translation migrations can be in the node module because it uses the same source plugin as non-translated node. But for blocks, a new source plugin is needed because both the block table and the i18n_blocks table are required.

When block is enabled on the destination site this block translation migration will be discovered but the source site may not have translations. Without the existence of all the source tables the migration will generate an error about 'base table or view not found'. And for those with a single language site this will be confusing and just plain wrong. This can be avoided by moving the block translation code to content_translation. That way the block translation migration is only discovered when content_translation is enabled, which is needs to be to translate blocks.

Therefor this should be moved back to content_translations.

quietone’s picture

Status: Needs work » Needs review
StatusFileSize
new15.02 KB
new4.8 KB

Made the changes above, without moving the files to content_translation. Just want to wait to see if someone disagrees.

phenaproxima’s picture

Status: Needs review » Needs work
  1. +++ b/core/modules/block_content/migration_templates/d6_custom_block_translation.yml
    @@ -0,0 +1,40 @@
    +  constants:
    +    type: basic
    

    This completely assumes that the destination site has installed Standard. It's a safe enough assumption, but I wonder if we shouldn't resolve this dynamically in a hook_migration_plugins_alter() implementation, and set the constant value to the first available block content type that has a body field. If none are available, the entire migration should be made unavailable. That way, this migration will work with profiles that are not Standard.

  2. +++ b/core/modules/block_content/migration_templates/d6_custom_block_translation.yml
    @@ -0,0 +1,40 @@
    +    plugin: i18n_translation
    

    So...this migration is in block_content, but it's relying on a plugin that exists in content_translation? That seems dicey.

  3. +++ b/core/modules/block_content/src/Plugin/migrate/source/d6/BoxTranslation.php
    @@ -0,0 +1,94 @@
    +    // Add in the property, which is either title or body.
    +    $query->leftJoin('i18n_strings', 'i18n', 'i18n.objectid = b.bid');
    +    $query->isNotNull('i18n.lid');
    

    We are assuming that the i18n_strings table exists here. We should probably be smarter about that.

    Maybe one option is to override initializeIterator() and return an empty ArrayIterator if these tables are not existing in the source database.

  4. +++ b/core/modules/block_content/src/Plugin/migrate/source/d6/BoxTranslation.php
    @@ -0,0 +1,94 @@
    +    $query->leftJoin('locales_target', 'lt', 'lt.lid = i18n.lid');
    

    I'm not sure we should assume that locales_target exists.

  5. +++ b/core/modules/block_content/src/Plugin/migrate/source/d6/BoxTranslation.php
    @@ -0,0 +1,94 @@
    +    if (empty($this->idMap->lookupDestinationIds(['bid' => $bid, 'language' => $language]))) {
    +      $row->setSourceProperty('migrate', TRUE);
    +    }
    

    What's happening here? Can it be documented in a comment?

  6. +++ b/core/modules/block_content/src/Plugin/migrate/source/d6/BoxTranslation.php
    @@ -0,0 +1,94 @@
    +    $row->setSourceProperty('body_untranslated', Unicode::truncate($row->getSourceProperty('options/attributes/title'), 255));
    

    Er...why is the body being derived from options/attributes/title?

  7. +++ b/core/modules/block_content/src/Plugin/migrate/source/d6/BoxTranslation.php
    @@ -0,0 +1,94 @@
    +    return [
    +      'bid' => $this->t('The block numeric identifier.'),
    +      'language' => $this->t('Language for this field.'),
    +      'title' => $this->t('Block title translation.'),
    +      'body' => $this->t('Block body translation.'),
    +    ];
    

    There are several interesting properties being defined in prepareRow() that we should document here...

  8. +++ b/core/modules/block_content/tests/src/Kernel/Plugin/migrate/source/d6/BoxTranslationTest.php
    @@ -0,0 +1,134 @@
    +    // The expected results.
    +    $tests[0]['expected_results'] = [
    

    Nit: No need for "The expected results" comment, since the array key makes it pretty obvious :)

  9. +++ b/core/modules/content_translation/src/Plugin/migrate/process/d6/i18nTranslation.php
    @@ -0,0 +1,48 @@
    +    if ($translated) {
    +      return $translated;
    +    }
    +    else {
    +      return $untranslated;
    +    }
    

    Nit: We could condense this to return $translated ?: $untranslated

  10. +++ b/core/modules/content_translation/tests/src/Unit/migrate/process/i18nTranslationTest.php
    @@ -0,0 +1,81 @@
    +  /**
    +   * {@inheritdoc}
    +   */
    +  protected function setUp() {
    +    parent::setUp();
    +  }
    

    We can remove this.

  11. +++ b/core/modules/content_translation/tests/src/Unit/migrate/process/i18nTranslationTest.php
    @@ -0,0 +1,81 @@
    +   * @expectedException \Drupal\migrate\MigrateException
    +   * @expectedExceptionMessage You need to specify both the translated and untranslated values on the i18n_translation plugin.
    

    Apparently we no longer use annotations for these; we need to call $this->setExpectedException() in the test method.

maxocub’s picture

@quietone: Thanks for the clarification, I had not realized that, so I agree that this should go in content_trnaslation.

quietone’s picture

Status: Needs work » Needs review
StatusFileSize
new3.56 KB

26.2. See my comment in #24 explaining why these files should be moved back to content_translation.

This patch only moves the files to content_translation.

phenaproxima’s picture

Status: Needs review » Needs work

This patch only moves the files to content_translation.

Er...no patch is attached to #28?

quietone’s picture

Status: Needs work » Needs review
StatusFileSize
new15.04 KB

Here is the patch, mysteriously missing, from #28.

Oh, and even with the wrong comment number. Did I not get enough sleep?

quietone’s picture

StatusFileSize
new15.43 KB
new5.09 KB

5. Removed this, looks like leftover from a different approach.
7. Improved.
8. Fixed
9. Sure can, fixed.
10. Fixed
11. Fixed
And added some documentation to i18nTranslation.php process plugin.

Still to do: 1, 3, 4, and 6.

quietone’s picture

StatusFileSize
new17.58 KB
new5.87 KB

1. Actually, this isn't needed. This migration is only adding translated strings of certain fields. The block itself exists and is not being changed.
3 and 4. If InitializeIterator returns an empty interator then nothing is migrated and the user has no idea why. Because of that I tried something different. The source plugin checks if the tables exist, and if not, throws an exception.
6. Where did that come from? Fixed to get the body from, well, the body.

Also, while working on this I stepped through the source plugin and found a bug. This line, $property2 = ($property == 'title') ? 'body' : 'title'; assumes that the property will always be title or body. But there was a row where the property was 'name' and the translation was a taxonomy string. Because of that the query has a new condition, that 'type'='block'.

I think that finishes all the items in #26. Now, lets see what the testbot finds.

Status: Needs review » Needs work

The last submitted patch, 32: 2909444-32.patch, failed testing. View results

jofitz’s picture

Assigned: phenaproxima » Unassigned
Status: Needs work » Needs review
StatusFileSize
new17.62 KB
new2.22 KB

Check the migrateMessages rather than expecting an exception (inspired by Drupal\Tests\migrate_drupal\Kernel\dependencies\MigrateDependenciesTest).

phenaproxima’s picture

Status: Needs review » Needs work

I found some relatively minor things, but overall this looks good. Looooovely test coverage!

  1. +++ b/core/modules/content_translation/migrations/d6_custom_block_translation.yml
    @@ -0,0 +1,39 @@
    +  constants:
    +    type: basic
    

    type is no longer used, so we can remove it from here.

  2. +++ b/core/modules/content_translation/migrations/d6_custom_block_translation.yml
    @@ -0,0 +1,39 @@
    +  id:
    +    plugin: migration
    +    migration: d6_custom_block
    +    source:
    +      - bid
    

    The plugin should be migration_lookup, not migration, and we should skip the row if it returns null.

  3. +++ b/core/modules/content_translation/migrations/d6_custom_block_translation.yml
    @@ -0,0 +1,39 @@
    +  info:
    +    plugin: i18n_translation
    +    source:
    +      - title_untranslated
    +      - title_translated
    

    I don't know if we actually need to change this, but we don't need i18n_translation to be its own plugin. We could do something like this:

    -
      plugin: callback
      source:
        - title_translated
        - title_untranslated
      callable: array_filter
    -
      plugin: callback
      callable: reset
    

    This will filter out the values that don't exist, and return the first one that does. Same thing that i18n_translation does.

    The advantage being that we don't have to write and test yet another process plugin to do simple one-line logic. (This is why I think declarative programming is a mistake in a system like Migrate, but that's another discussion.)

  4. +++ b/core/modules/content_translation/migrations/d6_custom_block_translation.yml
    @@ -0,0 +1,39 @@
    +  'body/format':
    +    plugin: migration
    +    migration: d6_filter_format
    +    source: format
    

    We should use the migration_lookup plugin, not migration. And should we skip the row, or set a default value, if it fails?

  5. +++ b/core/modules/content_translation/src/Plugin/migrate/source/d6/BoxTranslation.php
    @@ -0,0 +1,107 @@
    +    if (($this->getDatabase()->schema()->tableExists('i18n_strings')) &&
    +      $this->getDatabase()->schema()->tableExists('locales_target')) {
    

    Can we call $this->getDatabase()->schema() once and reuse the return value, just to clean this up and make it more readable?

  6. +++ b/core/modules/content_translation/src/Plugin/migrate/source/d6/BoxTranslation.php
    @@ -0,0 +1,107 @@
    +    else {
    +      throw new MigrateException("Missing one or both translation tables, 'i8n_strings' and 'locales_target'");
    +    }
    

    This is more elegant -- I like it.

  7. +++ b/core/modules/content_translation/src/Plugin/migrate/source/d6/BoxTranslation.php
    @@ -0,0 +1,107 @@
    +    $property2 = ($property == 'title') ? 'body' : 'title';
    

    I don't like "$property2" as a variable name. Is there any chance we could rename it to something a little more descriptive?

quietone’s picture

Status: Needs work » Needs review
StatusFileSize
new16.13 KB
new5.26 KB

1. Fixed
2. Fixed
3. I'd like to not have that process plugin too. But I didn't see a way to do it in the pipeline. And the suggested code, in the patch, fails. Any other idea?
4. Fixed but didn't add a skip row because since block migration has already fun, I figure, this will already exist.
5. Fixed
6. Yes, I agree.
7. Yes, it is a yucky name. Changed to $translation.

Status: Needs review » Needs work

The last submitted patch, 36: 2909444-36.patch, failed testing. View results

quietone’s picture

+++ b/core/modules/content_translation/tests/src/Kernel/Plugin/migrate/source/d6/BoxTranslationTest.php
--- /dev/null
+++ b/core/modules/content_translation/tests/src/Unit/migrate/process/i18nTranslationTest.php

Oh, this needs to be removed too.

But still the process pipeline modified as suggest in #35.3, with this error.

1) Drupal\Tests\content_translation\Kernel\Migrate\d6\MigrateCustomBlockContentTranslationTest::testCustomBlockContentTranslation
Parameter 1 to reset() expected to be a reference, value given
maxocub’s picture

Could we add an optional configuration to the callback process plugin so we can pass variables by reference?

Something like:

  -
    plugin: callback
    callable: reset
    pass_by_reference: true

And in Callback.php:

  if (!empty($this->configuration['pass_by_reference'])) {
    $value = call_user_func_array($this->configuration['callable'], [&$value]);
  }
  else {
    $value = call_user_func($this->configuration['callable'], $value);
  }
phenaproxima’s picture

Huh, yeah. I can see why that might have failed. How about this, then?

# Filter out the empty values, preferring the translated version if it exists.
-
  plugin: callback
  callable: array_filter
  source:
    - body_translated
    - body_untranslated
# Re-key the array to guarantee that, if it has any values left, index 0 exists.
-
  plugin: callback
  callable: array_values
# Extract index 0.
-
  plugin: extract
  index: [0]
Could we add an optional configuration to the callback process plugin so we can pass variables by reference?

I don't believe so; I don't think call_user_func() supports passing by reference. call_user_func_array() might, but reference-ness is a messy business anyway and I'd prefer to avoid it, as I can imagine it having unintended side effects.

maxocub’s picture

Yeah, I prefer the solution in #40 than adding support to pass by reference.

quietone’s picture

I don't see how this will work when both the translated and untranslated values for a given property exist in the row.

phenaproxima’s picture

The key is the array_filter.

Given this array: ["the translated value", "the untranslated value"]

If I call array_filter() on that, it will look exactly the same. The translated value will be first. So the extract plugin will pluck that value from the array.

But if there's no translated value: ["", "The untranslated value"]...

Then array_filter() will knock out the first (empty) item, and extracting the first item in the array will give me the untranslated one. Which is what we want, since there was no translated value.

I don't see how it won't work, personally.

quietone’s picture

Status: Needs work » Needs review
StatusFileSize
new13.45 KB

Removed the now unused process plugin test. The pipeline now uses array_filter and current.

    -
      plugin: callback
      source:
        - title_translated
        - title_untranslated
      callable: array_filter
    -
      plugin: callback
      callable: current

RE my comment in #42, I completely misread the suggestion in #40, missing the use of array_values. So was confused about the array indices, which I then totally failed to convey in my comment.

quietone’s picture

+++ b/core/modules/content_translation/src/Plugin/migrate/source/d6/BoxTranslation.php
@@ -0,0 +1,106 @@
+    if ($schema->tableExists('i18n_strings') && $schema->tableExists('locales_target')) {

If this is being added here, there should be a follow up to add this to any existing i18n translation source plugin that uses these tables.

phenaproxima’s picture

Status: Needs review » Needs work

This keeps looking better and better to me. One final round of fixes, and then we can mark this RTBC.

  1. +++ b/core/modules/content_translation/src/Plugin/migrate/source/d6/BoxTranslation.php
    @@ -0,0 +1,106 @@
    +class BoxTranslation extends DrupalSqlBase {
    +  /**
    +   * {@inheritdoc}
    +   */
    

    Nit: There should be an empty line after the opening brace of the class.

  2. +++ b/core/modules/content_translation/src/Plugin/migrate/source/d6/BoxTranslation.php
    @@ -0,0 +1,106 @@
    +    // Set untranslated values as defaults.
    +    $row->setSourceProperty('title_untranslated', $row->getSourceProperty('info'));
    +    $row->setSourceProperty('body_untranslated', $row->getSourceProperty('body'));
    

    Rather than do this, we should just add the fields in query(): $query->addField('b', 'body', 'body_untranslated');

  3. +++ b/core/modules/content_translation/src/Plugin/migrate/source/d6/BoxTranslation.php
    @@ -0,0 +1,106 @@
    +    $query = $this->select('i18n_strings', 'i18n')
    +      ->fields('i18n', ['lid'])
    +      ->condition('i18n.property', $translation)
    +      ->condition('i18n.objectid', $bid);
    +    $query->leftJoin('locales_target', 'lt', 'i18n.lid = lt.lid');
    +    $query->condition('lt.language', $language)
    +      ->addField('lt', 'translation');
    

    This seems potentially quite useful for other i18n migrations. Can we maybe move this into a trait, or new base class extending DrupalSqlBase, as a helper method? This could also fulfill @quietone's request in #45, and spare us a follow-up issue.

  4. +++ b/core/modules/content_translation/src/Plugin/migrate/source/d6/BoxTranslation.php
    @@ -0,0 +1,106 @@
    +      'body' => $this->t('Block body.'),
    +      'info' => $this->t('Admin title of the block/box.'),
    

    As noted before, let's get rid of these.

  5. +++ b/core/modules/content_translation/tests/src/Kernel/Migrate/d6/MigrateCustomBlockContentTranslationMissingTest.php
    @@ -0,0 +1,47 @@
    +  public static $modules = [
    

    Should be protected.

  6. +++ b/core/modules/content_translation/tests/src/Kernel/Migrate/d6/MigrateCustomBlockContentTranslationMissingTest.php
    @@ -0,0 +1,47 @@
    +   * Tests the Drupal 6 i18n custom block strings to Drupal 8 migration when the
    +   * source translation tables are missing.
    

    This violates coding standards. The short description needs to be one line. We can add more detail in subsequent paragraphs.

  7. +++ b/core/modules/content_translation/tests/src/Kernel/Migrate/d6/MigrateCustomBlockContentTranslationMissingTest.php
    @@ -0,0 +1,47 @@
    +    $this->assertEquals($this->migrateMessages['error'], [new FormattableMarkup('Migration failed with source plugin exception: @e', ['@e' => "Missing one or both translation tables, 'i8n_strings' and 'locales_target'"])]);
    

    Nit: Can we assert $this->migrateMessages['error'][0], rather than wrap the FormattableMarkup in an array?

  8. +++ b/core/modules/content_translation/tests/src/Kernel/Migrate/d6/MigrateCustomBlockContentTranslationTest.php
    @@ -0,0 +1,68 @@
    +  public static $modules = [
    

    Should be protected.

  9. +++ b/core/modules/content_translation/tests/src/Kernel/Migrate/d6/MigrateCustomBlockContentTranslationTest.php
    @@ -0,0 +1,68 @@
    +    /** @var BlockContent $block */
    

    Nit: Needs to be the fully qualified class name.

  10. +++ b/core/modules/content_translation/tests/src/Kernel/Plugin/migrate/source/d6/BoxTranslationTest.php
    @@ -0,0 +1,133 @@
    +  public static $modules = ['content_translation', 'migrate_drupal'];
    

    Should be protected.

jofitz’s picture

Status: Needs work » Needs review
StatusFileSize
new6.66 KB
new13.76 KB
  1. Done.
  2. Done (and edited the test appropriately).
  3. I have not exported the potentially useful code into a trait or base class - perhaps that could be a follow-up?
  4. Done.
  5. Why should this be protected?
  6. Done.
  7. Done (and added assertCount() to maintain equivalent tests).
  8. See 5.
  9. Done.
  10. See 5.
quietone’s picture

3. Moved that to a followup, #2918295: Move i18n query to a trait
5, 8, 10. Was discuss in #2225681: Migrate D6 i18n blocks translated strings, #137->#149. It is being done in a follow up created, #2918296: Change all instances of $modules to protected.

That covers all the todos here.

phenaproxima’s picture

Status: Needs review » Reviewed & tested by the community

Full steam ahead.

quietone’s picture

Just noting that the migrations are in the correct directory, 'migrations'.

catch’s picture

Status: Reviewed & tested by the community » Needs review
+++ b/core/modules/content_translation/tests/src/Kernel/Migrate/d6/MigrateCustomBlockContentTranslationMissingTest.php
@@ -0,0 +1,50 @@
+    ]);
+    $this->assertEquals($this->migrateMessages['error'][0], new FormattableMarkup('Migration failed with source plugin exception: @e', ['@e' => "Missing one or both translation tables, 'i8n_strings' and 'locales_target'"]));
+    $this->assertCount(1, $this->migrateMessages['error']);

One thing with this. If the source tables are missing, then i18n_strings probably isn't installed on the source site, in which case there's nothing to migrate. So why do we treat this as an error?

quietone’s picture

In general, we don't check if the tables exist (although there are a few cases, and I don't recall which ones). Why is this on special?

Edit: I misread your comment, so this answer doesn't make much sense.

quietone’s picture

Issue tags: +blocker

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.4 was released on January 3, 2018 and is the final full bugfix release for the Drupal 8.4.x series. Drupal 8.4.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.5.0 on March 7, 2018. (Drupal 8.5.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.5.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

maxocub’s picture

Status: Needs review » Needs work
+++ b/core/modules/content_translation/src/Plugin/migrate/source/d6/BoxTranslation.php
@@ -0,0 +1,103 @@
+ * @MigrateSource(
+ *   id = "d6_box_translation",
+ *   source_module = "block"
+ * )

Shouldn't the source_module be i18nblocks?

I was wondering the same thing as in #51 which has not really been answered I think.

If the source_module was i18nblocks, then would it still be necessary to check if the tables exist?

quietone’s picture

StatusFileSize
new13.5 KB
new4.05 KB

Yes, it should be i18nblocks but that is separate from testing the existence of the translation tables.

However, removed the test for the existence of the translation tables. That isn't done in any of the other translation migrations that use those tables. If anyone wants to discuss that can be done in a separate issue and not hold up any of the i18n issues?

quietone’s picture

Status: Needs work » Needs review

NR for testing

Status: Needs review » Needs work

The last submitted patch, 56: 2909444-56.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

quietone’s picture

Status: Needs work » Needs review
StatusFileSize
new11.42 KB
new1.81 KB

This is failing because it no longer checks if the source tables exists and when I removed that I didn't remove the corresponding test. Note that adding checks for the existence of the tables was suggested in #28.3

Why remove this? If the source is Drupal it is assumed that if the relevant source_module is installed on the source then all the necessary tables are installed. In this case, when i18nblocks is installed on Drupal 6 several tables are created. They are i18n_blocks, i18n_strings, i18n_variable, languages, locales_source and locales_target.

Status: Needs review » Needs work

The last submitted patch, 59: 2909444-59.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

quietone’s picture

Status: Needs work » Needs review
StatusFileSize
new12.99 KB
new1.2 KB

Move i18nblocks to the available upgrade paths.

quietone’s picture

StatusFileSize
new12.95 KB
new561 bytes

remove unused use statement.

quietone’s picture

StatusFileSize
new4.97 KB
new12.77 KB

Renamed the the source properties to be a bit simpler.

quietone’s picture

Status: Needs review » Needs work

this needs the code to prevent processing duplicate rows.

quietone’s picture

Status: Needs work » Needs review
StatusFileSize
new12.96 KB
new855 bytes

Added the code to prevent processing duplicate rows.

quietone’s picture

Status: Needs review » Needs work

i18nmenulink broke Postgres and was fixed by alexpott here. Those changes will most likely need to be added here, and to the other i18n issues

rakesh.gectcr’s picture

jofitz’s picture

Kicked off a Postgres test, OOI.

jofitz’s picture

Status: Needs work » Needs review
StatusFileSize
new12.98 KB

Patch no longer applies. Re-rolling before doing anything else.

jofitz’s picture

StatusFileSize
new927 bytes
new13.06 KB

Added a similar Postgres fix to #2225587: Migrate D6 i18n menu links.

quietone’s picture

@Jo Fitzgerald, awesome, thx. Seems you are a star too!

phenaproxima’s picture

Status: Needs review » Needs work

I am not finding very much to complain about with this patch. It's dense, but I see why it needs to be dense. Ideally we'd be able to reduce that "density", but there's only so much we can do, really.

  1. +++ b/core/modules/content_translation/migrations/d6_custom_block_translation.yml
    @@ -0,0 +1,48 @@
    +    -
    +      plugin: callback
    +      callable: current
    

    This should probably be 'reset', to reset the internal pointer to the first item in the array.

  2. +++ b/core/modules/content_translation/migrations/d6_custom_block_translation.yml
    @@ -0,0 +1,48 @@
    +  'body/format':
    +    plugin: migration_lookup
    +    migration: d6_filter_format
    +    source: format
    

    Do we want to skip the row if lookup fails?

  3. +++ b/core/modules/content_translation/src/Plugin/migrate/source/d6/BoxTranslation.php
    @@ -0,0 +1,101 @@
    +    // Build a query based on 18n_strings table where each row has the
    

    Typo in 'i18n_strings' :)

  4. +++ b/core/modules/content_translation/src/Plugin/migrate/source/d6/BoxTranslation.php
    @@ -0,0 +1,101 @@
    +    $query->addField('b', 'info', 'title');
    

    Can we have a comment as to why this is being done? I understand it, but I'd like a comment for posterity.

  5. +++ b/core/modules/content_translation/src/Plugin/migrate/source/d6/BoxTranslation.php
    @@ -0,0 +1,101 @@
    +    // so PostgreSQL can make the join
    

    Needs a period at the end of the sentence.

  6. +++ b/core/modules/content_translation/src/Plugin/migrate/source/d6/BoxTranslation.php
    @@ -0,0 +1,101 @@
    +    // If this row has been migrated it is a duplicate then skip it.
    +    if ($this->idMap->lookupDestinationIds(['bid' => $bid, 'language' => $language])) {
    +      return FALSE;
    +    }
    

    This is very "heavy". Is there any way we could do this in the process pipeline? If not, no big deal...but ideally we could do it there, rather than here.

  7. +++ b/core/modules/content_translation/src/Plugin/migrate/source/d6/BoxTranslation.php
    @@ -0,0 +1,101 @@
    +    $translation = ($property == 'title') ? 'body' : 'title';
    

    Should use === here.

  8. +++ b/core/modules/content_translation/src/Plugin/migrate/source/d6/BoxTranslation.php
    @@ -0,0 +1,101 @@
    +      ->fields('i18n', ['lid'])
    

    We are never using the lid except as a join condition, so do we need to retrieve it?

  9. +++ b/core/modules/content_translation/tests/src/Kernel/Migrate/d6/MigrateCustomBlockContentTranslationTest.php
    @@ -0,0 +1,68 @@
    +    $this->assertTrue(REQUEST_TIME <= $block->getChangedTime() && $block->getChangedTime() <= time());
    

    This should be two separate calls to the assertGreaterOrEqual() (or similar) methods.

  10. +++ b/core/modules/content_translation/tests/src/Kernel/Migrate/d6/MigrateCustomBlockContentTranslationTest.php
    @@ -0,0 +1,68 @@
    +    $block = BlockContent::load(1)->getTranslation('zu');
    

    No need to reload the block; I think you could just do $block = $block->getTranslation('zu').

quietone’s picture

Status: Needs work » Needs review
StatusFileSize
new13.3 KB
new4.38 KB

1. Reset doesn't work here. Changing results in:

1) Drupal\Tests\content_translation\Kernel\Migrate\d6\MigrateCustomBlockContentTranslationTest::testCustomBlockContentTranslation
Parameter 1 to reset() expected to be a reference, value given

2. I don't know filters well at all but other migrations that use d6_filter_format do not include a skip on empty after it in the process pipeline and that includes d6_custom_block, on which this is based.
3, 4, 5, Fixed
6. Yes, this is a bit unusual but it is difficult to determine if this is a duplicate row and I would just as soon do it as early as possible. This is the same technique used in MenuLinkTranslation. Do you have a better idea?
7. Fixed
8. Needs testing
9. Fixed
10. Fixed

Edit: Update 1.

Status: Needs review » Needs work

The last submitted patch, 73: 2909444-73.patch, failed testing. View results

quietone’s picture

Status: Needs work » Needs review
StatusFileSize
new13.3 KB
new1.25 KB

Forgot to retest after the changes made to #72.8. the intention here was to only get rows where lt.lid was not NULL instead i18n_strings was being used. That is now corrected and the 'isNotNull' is moved to where it is easier to see.

phenaproxima’s picture

Status: Needs review » Reviewed & tested by the community

It's a happy-maker. Complicated, but a happy-maker. Fasten seat belts and prepare for landing.

gábor hojtsy’s picture

quietone’s picture

Retested and added a test for 8.6.x

quietone’s picture

Status: Reviewed & tested by the community » Needs review

Sorry, I think this needs a pause and a double check that the source plugins and migrations are in the desired directory. This has a different structure than the menu links translation it needs to be double checked.

heddn’s picture

Assigned: Unassigned » quietone

Looks like quietone wants to review this. Assigning.

quietone’s picture

StatusFileSize
new13.26 KB
new3.03 KB

This moves the files so that all files that were in content_translation are now in block_content, except the migration itself. This make it the same as the existing menu_links_translation, with the addition of destination_module to the migration so that the UI shows content_translation as the destination module.

quietone’s picture

Assigned: quietone » Unassigned

Ready for review

masipila’s picture

Status: Needs review » Reviewed & tested by the community

@phenaproxima set this to RTBC in #76.

According to the interdiff in #81, the only change since #76 is the change from content_translation to block_content and related namespace changes. Arguments in #81 for this change make sense to me.

Back to RTBC.

Cheers,
Markus

gábor hojtsy’s picture

Status: Reviewed & tested by the community » Needs work

Thanks all! I was reviewing the code and "The translation" struck me as an odd field. It looks like the same as the title translation(?) Do we need that and if so, can we give it a more specific title?

+      'translation' => $this->t('The translation.'),
+      'title_translated' => $this->t('Block title translation.'),
+      'body' => $this->t('Block body translation.'),
masipila’s picture

Assigned: Unassigned » masipila

assigning to myself.

heddn’s picture

masipila’s picture

Assigned: masipila » Unassigned

I discussed #84 with @Gábor Hojtsy in Slack to understand his concern.

1. The point here is that the locales_target.translation contains the translated string value, which is either translated title or translated body in this case. We have both those as separate fields so the field 'translation' seems to be redundant.

2. Second point that Gábor was pointing out was that the field names are not consistent as we have 'title_translated' and 'body', should probably be 'title_translated' and 'body_translated'.

I was thinking to do this today but I'm prioritizing the investigation of the i18n taxonomy term migration issue so unassigning myself from this.

Cheers,
Markus

quietone’s picture

Status: Needs work » Needs review
StatusFileSize
new13.38 KB
new1.31 KB

Thanks all.

So this updates prepareRow() to ensure that the property for both translation always exists on the row (title_translated and body_translated). One will always be NULL but the property will be set, if someone needs it. The fields() method is adjusted for that change. The 'translation' property is still on the row because I haven't thought of a better name and I am about to go out. (I hope my haste hasn't produced a bad patch).

masipila’s picture

Status: Needs review » Needs work

1. Body

@@ -83,10 +88,10 @@
       'body' => $this->t('Block body translation.'),
-      'body_untranslated' => $this->t('Block body.'),
+      'body_translated' => $this->t('Block body.'),

Are the field name and description correct here? Should 'body' be 'Block body' and 'body_translated' be 'Block body translation'?

2. Regarding the 'translation'.
Answering to myself and Gábor in #84 and #87. The query first reads the value from 'translation' and then we do the voodoo to determine if it was the title or body translation and based on this, populate the value to the correct field. So the answer is no, we can't remove this field from the query. However, finding descriptive names is always difficult and 'The translation' is prone to WTF moments. I propose that we have this description for it: Translation for title or body, depending on context. Use title_translated or body_translated.

3. Nit, not related to the content of the patch: It would be nice if the interdiff file name would indicate the issue number and the patch numbers that it interdiffs. The URL for this interdiff for example is https://www.drupal.org/files/issues/2018-06-16/interdiff_4.txt i.e. it doesn't give any indication which interdiff this was. When reviewing multiple issues, it would make it easier for reviewer if the file name in this case would be interdiff-2909444-81-88.txt

Cheers,
Markus

quietone’s picture

Status: Needs work » Needs review
StatusFileSize
new13.4 KB
new1.14 KB

89.1 Fixed
89.2 "Translation for title or body, depending on context. Use title_translated or body_translated." is accurate but long. Perhaps too long? And we know the context, the translation is the 'The translation of the value of "property"'. Would that shorter version be sufficient? Still seems difficult to be succinct. I've opted to use the shorter version in this patch but it is only a suggestion.
89.3 I used to always number the interdiff as you suggest, I guess I got lazy.

masipila’s picture

Hi!

This is ready once the tests pass. I added PostgreSQL and SQLite tests.

Markus

masipila’s picture

And the tests turned greencas expected.

We have test coverage, I have tested this manually, several contributors have reviewed the code and all feedback has been addressed. RTBC.

Markus

masipila’s picture

Status: Needs review » Reviewed & tested by the community

Helps if I actually change the status as well...

gábor hojtsy’s picture

Status: Reviewed & tested by the community » Needs work

Does not apply anymore.

masipila’s picture

Issue tags: +Needs reroll

Needs reroll

quietone’s picture

StatusFileSize
new12.34 KB
new1.07 KB

And here is the reroll. It was simple, the changes in the MigrateUpgrade tests have already been committed in #2225681: Migrate D6 i18n blocks translated strings.

quietone’s picture

Status: Needs work » Needs review

NR for testbot.

Status: Needs review » Needs work

The last submitted patch, 96: 2909444-96.patch, failed testing. View results

quietone’s picture

Version: 8.5.x-dev » 8.6.x-dev
Status: Needs work » Needs review
Issue tags: -Needs reroll

I was working off 8.6.x so the patch doesn't apply to 8.5.x and I think this should be 8.6 anyway.

phenaproxima’s picture

Status: Needs review » Reviewed & tested by the community

Restoring RTBC.

gábor hojtsy’s picture

Adjusting credits.

gábor hojtsy’s picture

Status: Reviewed & tested by the community » Fixed

Committed e64090e and pushed to 8.6.x. Thanks!

  • Gábor Hojtsy committed a79df57 on 8.6.x
    Issue #2909444 by quietone, Jo Fitzgerald, rakesh.gectcr, maxocub,...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.