Closed (fixed)
Project:
Drupal core
Version:
8.6.x-dev
Component:
migration system
Priority:
Major
Category:
Task
Assigned:
Unassigned
Issue tags:
Reporter:
Created:
17 Sep 2017 at 15:15 UTC
Updated:
12 Jul 2018 at 12:34 UTC
Jump to comment: Most recent, Most recent file
Comments
Comment #2
maxocub commentedThis is a split of the patch from #2225681: Migrate D6 i18n blocks translated strings for easier reviews and commit. No change has been made to the remaining code.
Comment #3
maxocub commented1.
Custom block translations?
2.
Can we add some documentation & code example? Like that we expect that the untranslated value is the first item in the array and that the translated is second?
3.
Can we check that $value is actualy composed of 2 strings?
4.
Nit: Can be chained, I think.
5.
I'm not sure what the standard says, but I think this would be more readable on one line.
6.
This can all be chained.
7.
No need to manualy create each languages, just need to add 'language' to the executed migrations below.
8.
This needs a better comment, since it's not only the title that is migrated/tested.
9.
I'm not sure that content_translation is the right place for this code. I think it should go in block_content, like the node translation migrations are in the node module.
Comment #4
jofitzComment #5
jofitzSetting back to Needs Work for someone to address #3.3.
Comment #6
maxocub commentedAbout #3.3:
If someone mis-configure this plugin by doing something like
or like
the list() won't work and we should throw an exception.
Comment #7
jofitzCorrect the namespace in the moved files.
Address #3.3.
Comment #10
maxocub commentedOh, and one last thing, the i18n_translation process plugin will need a unit test.
Comment #11
jofitzAdded a unit test for the i18n_translation process plugin.
I haven't been able to get to the bottom of the remaining test failure - I can't understand why this fails when MigrateCustomBlockContentTranslationTest passes. My only suspicion is that 'language' should be one of the required migration_dependencies for d6_custom_block_translation, but I am unable to test that on my local build.
Comment #13
rakesh.gectcrComment #14
rakesh.gectcrPassed the test in my local, Will give it a try
Comment #15
maxocub commentedI think this 'language' dependency should be optional since this migration file is not only used in multilingual migrations.
Comment #16
maxocub commentedAlso adding the vienna tag for our kanban.
Comment #17
rakesh.gectcrOops, Here we go,
Comment #18
maxocub commentedDamn, I didn't look at it enough in my last review, but now I think that this language dependency should really be in the
d6_custom_block_translation.ymlfile and it should be a required dependency. Sorry about that, rakesh.gectcr!Comment #19
jofitzMoved the migration dependency as recommended by @maxocub.
Comment #20
maxocub commentedAll concerns have been addressed, I think this is ready to be RTBCed.
Comment #21
quietone commented@rakesh.gectcr, @Jo Fitzgerald and @maxocub thanks for getting this to RTBC. It was a goal of mine to get this done in Vienna. And it happened without me doing anything. Love the community.
Comment #22
maxocub commented@quietone should be credited on this issue since she wrote the original patch in #2225681: Migrate D6 i18n blocks translated strings, which was split in two issues.
Comment #23
webchick@phenaproxima says he would like to review this.
Comment #24
quietone commentedThis should be Drupal\Tests\content_translation\Unit\migrate\process
Should have parameter definitions for $value and $expected in the doc block
Ditto.
Add a newline.
This needs to be moved back to content_translation. It was split in #105 of #2225681: Migrate D6 i18n blocks translated strings.
The node translation migrations can be in the node module because it uses the same source plugin as non-translated node. But for blocks, a new source plugin is needed because both the block table and the i18n_blocks table are required.
When block is enabled on the destination site this block translation migration will be discovered but the source site may not have translations. Without the existence of all the source tables the migration will generate an error about 'base table or view not found'. And for those with a single language site this will be confusing and just plain wrong. This can be avoided by moving the block translation code to content_translation. That way the block translation migration is only discovered when content_translation is enabled, which is needs to be to translate blocks.
Therefor this should be moved back to content_translations.
Comment #25
quietone commentedMade the changes above, without moving the files to content_translation. Just want to wait to see if someone disagrees.
Comment #26
phenaproximaThis completely assumes that the destination site has installed Standard. It's a safe enough assumption, but I wonder if we shouldn't resolve this dynamically in a hook_migration_plugins_alter() implementation, and set the constant value to the first available block content type that has a body field. If none are available, the entire migration should be made unavailable. That way, this migration will work with profiles that are not Standard.
So...this migration is in block_content, but it's relying on a plugin that exists in content_translation? That seems dicey.
We are assuming that the i18n_strings table exists here. We should probably be smarter about that.
Maybe one option is to override initializeIterator() and return an empty ArrayIterator if these tables are not existing in the source database.
I'm not sure we should assume that locales_target exists.
What's happening here? Can it be documented in a comment?
Er...why is the body being derived from options/attributes/title?
There are several interesting properties being defined in prepareRow() that we should document here...
Nit: No need for "The expected results" comment, since the array key makes it pretty obvious :)
Nit: We could condense this to
return $translated ?: $untranslatedWe can remove this.
Apparently we no longer use annotations for these; we need to call $this->setExpectedException() in the test method.
Comment #27
maxocub commented@quietone: Thanks for the clarification, I had not realized that, so I agree that this should go in content_trnaslation.
Comment #28
quietone commented26.2. See my comment in #24 explaining why these files should be moved back to content_translation.
This patch only moves the files to content_translation.
Comment #29
phenaproximaEr...no patch is attached to #28?
Comment #30
quietone commentedHere is the patch, mysteriously missing, from #28.
Oh, and even with the wrong comment number. Did I not get enough sleep?
Comment #31
quietone commented5. Removed this, looks like leftover from a different approach.
7. Improved.
8. Fixed
9. Sure can, fixed.
10. Fixed
11. Fixed
And added some documentation to i18nTranslation.php process plugin.
Still to do: 1, 3, 4, and 6.
Comment #32
quietone commented1. Actually, this isn't needed. This migration is only adding translated strings of certain fields. The block itself exists and is not being changed.
3 and 4. If InitializeIterator returns an empty interator then nothing is migrated and the user has no idea why. Because of that I tried something different. The source plugin checks if the tables exist, and if not, throws an exception.
6. Where did that come from? Fixed to get the body from, well, the body.
Also, while working on this I stepped through the source plugin and found a bug. This line,
$property2 = ($property == 'title') ? 'body' : 'title';assumes that the property will always be title or body. But there was a row where the property was 'name' and the translation was a taxonomy string. Because of that the query has a new condition, that 'type'='block'.I think that finishes all the items in #26. Now, lets see what the testbot finds.
Comment #34
jofitzCheck the migrateMessages rather than expecting an exception (inspired by Drupal\Tests\migrate_drupal\Kernel\dependencies\MigrateDependenciesTest).
Comment #35
phenaproximaI found some relatively minor things, but overall this looks good. Looooovely test coverage!
type is no longer used, so we can remove it from here.
The plugin should be migration_lookup, not migration, and we should skip the row if it returns null.
I don't know if we actually need to change this, but we don't need i18n_translation to be its own plugin. We could do something like this:
This will filter out the values that don't exist, and return the first one that does. Same thing that i18n_translation does.
The advantage being that we don't have to write and test yet another process plugin to do simple one-line logic. (This is why I think declarative programming is a mistake in a system like Migrate, but that's another discussion.)
We should use the migration_lookup plugin, not migration. And should we skip the row, or set a default value, if it fails?
Can we call $this->getDatabase()->schema() once and reuse the return value, just to clean this up and make it more readable?
This is more elegant -- I like it.
I don't like "$property2" as a variable name. Is there any chance we could rename it to something a little more descriptive?
Comment #36
quietone commented1. Fixed
2. Fixed
3. I'd like to not have that process plugin too. But I didn't see a way to do it in the pipeline. And the suggested code, in the patch, fails. Any other idea?
4. Fixed but didn't add a skip row because since block migration has already fun, I figure, this will already exist.
5. Fixed
6. Yes, I agree.
7. Yes, it is a yucky name. Changed to $translation.
Comment #38
quietone commentedOh, this needs to be removed too.
But still the process pipeline modified as suggest in #35.3, with this error.
Comment #39
maxocub commentedCould we add an optional configuration to the callback process plugin so we can pass variables by reference?
Something like:
And in Callback.php:
Comment #40
phenaproximaHuh, yeah. I can see why that might have failed. How about this, then?
I don't believe so; I don't think call_user_func() supports passing by reference. call_user_func_array() might, but reference-ness is a messy business anyway and I'd prefer to avoid it, as I can imagine it having unintended side effects.
Comment #41
maxocub commentedYeah, I prefer the solution in #40 than adding support to pass by reference.
Comment #42
quietone commentedI don't see how this will work when both the translated and untranslated values for a given property exist in the row.
Comment #43
phenaproximaThe key is the array_filter.
Given this array:
["the translated value", "the untranslated value"]If I call array_filter() on that, it will look exactly the same. The translated value will be first. So the extract plugin will pluck that value from the array.
But if there's no translated value:
["", "The untranslated value"]...Then array_filter() will knock out the first (empty) item, and extracting the first item in the array will give me the untranslated one. Which is what we want, since there was no translated value.
I don't see how it won't work, personally.
Comment #44
quietone commentedRemoved the now unused process plugin test. The pipeline now uses array_filter and current.
RE my comment in #42, I completely misread the suggestion in #40, missing the use of array_values. So was confused about the array indices, which I then totally failed to convey in my comment.
Comment #45
quietone commentedIf this is being added here, there should be a follow up to add this to any existing i18n translation source plugin that uses these tables.
Comment #46
phenaproximaThis keeps looking better and better to me. One final round of fixes, and then we can mark this RTBC.
Nit: There should be an empty line after the opening brace of the class.
Rather than do this, we should just add the fields in query():
$query->addField('b', 'body', 'body_untranslated');This seems potentially quite useful for other i18n migrations. Can we maybe move this into a trait, or new base class extending DrupalSqlBase, as a helper method? This could also fulfill @quietone's request in #45, and spare us a follow-up issue.
As noted before, let's get rid of these.
Should be protected.
This violates coding standards. The short description needs to be one line. We can add more detail in subsequent paragraphs.
Nit: Can we assert $this->migrateMessages['error'][0], rather than wrap the FormattableMarkup in an array?
Should be protected.
Nit: Needs to be the fully qualified class name.
Should be protected.
Comment #47
jofitzComment #48
quietone commented3. Moved that to a followup, #2918295: Move i18n query to a trait
5, 8, 10. Was discuss in #2225681: Migrate D6 i18n blocks translated strings, #137->#149. It is being done in a follow up created, #2918296: Change all instances of $modules to protected.
That covers all the todos here.
Comment #49
phenaproximaFull steam ahead.
Comment #50
quietone commentedJust noting that the migrations are in the correct directory, 'migrations'.
Comment #51
catchOne thing with this. If the source tables are missing, then i18n_strings probably isn't installed on the source site, in which case there's nothing to migrate. So why do we treat this as an error?
Comment #52
quietone commentedIn general, we don't check if the tables exist (although there are a few cases, and I don't recall which ones). Why is this on special?
Edit: I misread your comment, so this answer doesn't make much sense.
Comment #53
quietone commentedThis is blocking #2918295: Move i18n query to a trait.
Comment #55
maxocub commentedShouldn't the source_module be i18nblocks?
I was wondering the same thing as in #51 which has not really been answered I think.
If the source_module was i18nblocks, then would it still be necessary to check if the tables exist?
Comment #56
quietone commentedYes, it should be i18nblocks but that is separate from testing the existence of the translation tables.
However, removed the test for the existence of the translation tables. That isn't done in any of the other translation migrations that use those tables. If anyone wants to discuss that can be done in a separate issue and not hold up any of the i18n issues?
Comment #57
quietone commentedNR for testing
Comment #59
quietone commentedThis is failing because it no longer checks if the source tables exists and when I removed that I didn't remove the corresponding test. Note that adding checks for the existence of the tables was suggested in #28.3
Why remove this? If the source is Drupal it is assumed that if the relevant source_module is installed on the source then all the necessary tables are installed. In this case, when i18nblocks is installed on Drupal 6 several tables are created. They are i18n_blocks, i18n_strings, i18n_variable, languages, locales_source and locales_target.
Comment #61
quietone commentedMove i18nblocks to the available upgrade paths.
Comment #62
quietone commentedremove unused use statement.
Comment #63
quietone commentedRenamed the the source properties to be a bit simpler.
Comment #64
quietone commentedthis needs the code to prevent processing duplicate rows.
Comment #65
quietone commentedAdded the code to prevent processing duplicate rows.
Comment #66
quietone commentedi18nmenulink broke Postgres and was fixed by alexpott here. Those changes will most likely need to be added here, and to the other i18n issues
Comment #67
rakesh.gectcrComment #68
jofitzKicked off a Postgres test, OOI.
Comment #69
jofitzPatch no longer applies. Re-rolling before doing anything else.
Comment #70
jofitzAdded a similar Postgres fix to #2225587: Migrate D6 i18n menu links.
Comment #71
quietone commented@Jo Fitzgerald, awesome, thx. Seems you are a star too!
Comment #72
phenaproximaI am not finding very much to complain about with this patch. It's dense, but I see why it needs to be dense. Ideally we'd be able to reduce that "density", but there's only so much we can do, really.
This should probably be 'reset', to reset the internal pointer to the first item in the array.
Do we want to skip the row if lookup fails?
Typo in 'i18n_strings' :)
Can we have a comment as to why this is being done? I understand it, but I'd like a comment for posterity.
Needs a period at the end of the sentence.
This is very "heavy". Is there any way we could do this in the process pipeline? If not, no big deal...but ideally we could do it there, rather than here.
Should use === here.
We are never using the lid except as a join condition, so do we need to retrieve it?
This should be two separate calls to the assertGreaterOrEqual() (or similar) methods.
No need to reload the block; I think you could just do
$block = $block->getTranslation('zu').Comment #73
quietone commented1. Reset doesn't work here. Changing results in:
2. I don't know filters well at all but other migrations that use d6_filter_format do not include a skip on empty after it in the process pipeline and that includes d6_custom_block, on which this is based.
3, 4, 5, Fixed
6. Yes, this is a bit unusual but it is difficult to determine if this is a duplicate row and I would just as soon do it as early as possible. This is the same technique used in MenuLinkTranslation. Do you have a better idea?
7. Fixed
8. Needs testing
9. Fixed
10. Fixed
Edit: Update 1.
Comment #75
quietone commentedForgot to retest after the changes made to #72.8. the intention here was to only get rows where lt.lid was not NULL instead i18n_strings was being used. That is now corrected and the 'isNotNull' is moved to where it is easier to see.
Comment #76
phenaproximaIt's a happy-maker. Complicated, but a happy-maker. Fasten seat belts and prepare for landing.
Comment #77
gábor hojtsyComment #78
quietone commentedRetested and added a test for 8.6.x
Comment #79
quietone commentedSorry, I think this needs a pause and a double check that the source plugins and migrations are in the desired directory. This has a different structure than the menu links translation it needs to be double checked.
Comment #80
heddnLooks like quietone wants to review this. Assigning.
Comment #81
quietone commentedThis moves the files so that all files that were in content_translation are now in block_content, except the migration itself. This make it the same as the existing menu_links_translation, with the addition of destination_module to the migration so that the UI shows content_translation as the destination module.
Comment #82
quietone commentedReady for review
Comment #83
masipila commented@phenaproxima set this to RTBC in #76.
According to the interdiff in #81, the only change since #76 is the change from content_translation to block_content and related namespace changes. Arguments in #81 for this change make sense to me.
Back to RTBC.
Cheers,
Markus
Comment #84
gábor hojtsyThanks all! I was reviewing the code and "The translation" struck me as an odd field. It looks like the same as the title translation(?) Do we need that and if so, can we give it a more specific title?
Comment #85
masipila commentedassigning to myself.
Comment #86
heddnComment #87
masipila commentedI discussed #84 with @Gábor Hojtsy in Slack to understand his concern.
1. The point here is that the locales_target.translation contains the translated string value, which is either translated title or translated body in this case. We have both those as separate fields so the field 'translation' seems to be redundant.
2. Second point that Gábor was pointing out was that the field names are not consistent as we have 'title_translated' and 'body', should probably be 'title_translated' and 'body_translated'.
I was thinking to do this today but I'm prioritizing the investigation of the i18n taxonomy term migration issue so unassigning myself from this.
Cheers,
Markus
Comment #88
quietone commentedThanks all.
So this updates prepareRow() to ensure that the property for both translation always exists on the row (title_translated and body_translated). One will always be NULL but the property will be set, if someone needs it. The fields() method is adjusted for that change. The 'translation' property is still on the row because I haven't thought of a better name and I am about to go out. (I hope my haste hasn't produced a bad patch).
Comment #89
masipila commented1. Body
Are the field name and description correct here? Should 'body' be 'Block body' and 'body_translated' be 'Block body translation'?
2. Regarding the 'translation'.
Answering to myself and Gábor in #84 and #87. The query first reads the value from 'translation' and then we do the voodoo to determine if it was the title or body translation and based on this, populate the value to the correct field. So the answer is no, we can't remove this field from the query. However, finding descriptive names is always difficult and 'The translation' is prone to WTF moments. I propose that we have this description for it: Translation for title or body, depending on context. Use title_translated or body_translated.
3. Nit, not related to the content of the patch: It would be nice if the interdiff file name would indicate the issue number and the patch numbers that it interdiffs. The URL for this interdiff for example is https://www.drupal.org/files/issues/2018-06-16/interdiff_4.txt i.e. it doesn't give any indication which interdiff this was. When reviewing multiple issues, it would make it easier for reviewer if the file name in this case would be interdiff-2909444-81-88.txt
Cheers,
Markus
Comment #90
quietone commented89.1 Fixed
89.2 "Translation for title or body, depending on context. Use title_translated or body_translated." is accurate but long. Perhaps too long? And we know the context, the translation is the 'The translation of the value of "property"'. Would that shorter version be sufficient? Still seems difficult to be succinct. I've opted to use the shorter version in this patch but it is only a suggestion.
89.3 I used to always number the interdiff as you suggest, I guess I got lazy.
Comment #91
masipila commentedHi!
This is ready once the tests pass. I added PostgreSQL and SQLite tests.
Markus
Comment #92
masipila commentedAnd the tests turned greencas expected.
We have test coverage, I have tested this manually, several contributors have reviewed the code and all feedback has been addressed. RTBC.
Markus
Comment #93
masipila commentedHelps if I actually change the status as well...
Comment #94
gábor hojtsyDoes not apply anymore.
Comment #95
masipila commentedNeeds reroll
Comment #96
quietone commentedAnd here is the reroll. It was simple, the changes in the MigrateUpgrade tests have already been committed in #2225681: Migrate D6 i18n blocks translated strings.
Comment #97
quietone commentedNR for testbot.
Comment #99
quietone commentedI was working off 8.6.x so the patch doesn't apply to 8.5.x and I think this should be 8.6 anyway.
Comment #100
phenaproximaRestoring RTBC.
Comment #101
gábor hojtsyAdjusting credits.
Comment #102
gábor hojtsyCommitted e64090e and pushed to 8.6.x. Thanks!