Closed (fixed)
Project:
Drupal core
Version:
8.0.x-dev
Component:
migration system
Priority:
Normal
Category:
Task
Assigned:
Unassigned
Reporter:
Created:
9 Oct 2014 at 19:06 UTC
Updated:
3 Dec 2020 at 12:13 UTC
Jump to comment: Most recent, Most recent file
Comments
Comment #1
ultimikeComment #2
densolis commentedAssigned task to myself.
Comment #3
densolis commentedMoved comment to original case entry. (see above)
Comment #4
ultimikeComment #5
ultimikeComment #6
densolis commentedBoth the date_default_timezone and configurable_timezones are exposed on the d6 form:
/admin/settings/date-time. The prompts exposed on the corresponding form on D8.I wrote the migration for user-configurable time zone, but the default time zone is going to require plug in so that we can convert from + or - Greenwich time to a time zone.
I have attached a patch of what I have so far.
Comment #7
ultimikeDennis - your patch is 11MB - that's probably not right. I'm hiding it for now. Let's touch base on IRC later to figure out what went wrong...
Thanks,
-mike
Comment #8
densolis commentedMike,
oops - yeah, 11mb might be a big big. I figured out what I did wrong and re-rolled the patch and this time I read the contents of the patch. So here it is again.
I'll be working on the plug in later today.
Dennis
Comment #9
ultimikeDennis,
Just a couple of small things to fix while you're still working on this patch:
Extra white space at end of line.
Need newline at end of file.
Thanks,
-mike
Comment #10
densolis commentedComment #11
densolis commentedThe simple test runs fine. However, when I run the migration using drush, the values from D6 are not correctly migrated to D8.
Comment #12
densolis commentedMike,
Here is what I have so far. As I said above, the simple test runs fine, but I am having problems getting the migration to migrate the data from D6 to D8. I will try again with a fresh D8 site.
Here is my patch so far.
Comment #13
svendecabooterHi,
Tested your patch, and it seems to work fine with my test D6 & D8 installs.
I tested a few different configurations on the D6 site, and they migrated to D8 as expected.
Attached a new version of your patch with a few typos in comments fixed.
Comment #14
densolis commentedthe migrate and simple test are working for this patch. There are some issues with the php timezone_name_from_abbr function. It does not return a named time zone for each offset. I have attached a table with the following columns:
There have been bug in the php function timezone_name_from_abbr since 2008. It has been fixed, broken, fixed, and broken again.
I forgot to attached the time zone spread sheet. It is in comment 18 below.
Comment #15
densolis commentedsvendecabooter,
What were the typo? Never mind - I just saw them.
Thanks
Comment #17
svendecabooterIn core/modules/migrate_drupal/src/Plugin/migrate/process/d6/TimeZone.php:
// Convert the integitger value of the offset (which can be either
// negative or positive to a time zone name.
to
// Convert the integer value of the offset (which can be either
// negative or positive) to a time zone name.
In core/modules/migrate_drupal/src/Tests/d6/MigrateSystemDateTest.php
* Upgrade variables to user.*.yml.
to
* Upgrade variables to system.*.yml.
I think your attachment for the timezones issue is missing?
Comment #18
densolis commentedHere is the spread sheet attachment that should have been with comment 14.
Comment #19
densolis commentedI am guess that the simpletest failed because my machine is on day light saving time and the test machine is not. Below is the timezone_name_from_abbr for the same offset with and without daylight saving times. As you can see, they produce different results.
I change my simple test value from Europe/London to Europe/Paris.
Comment #20
densolis commentedThere is an issue when run the simple test on this patch.
As I said above, the function timezone_name_from_abbr will produce DIFFERENT results for the SAME offset value depending upon your daylight saving time setting. As you can see above, the offset value of 3600 will return either a value of Paris or London depending on the daylight saving time setting on your server.
On my dev machine, daylight saving time is on, so the function returns a value of Europe/London. And the simple test worked. However, when the test bot ran the same simple test - it failed. So the ONLY change I made to the simple test is I changed Europe/London to Europe/Paris and the simple test passed. So I am guessing the simple test is run on a machine with the daylights saving time disabled.
Does anyone have a suggestion on how to get around this so that the simple test will run on any machine regardless of value of daylight savings time?
Comment #21
ultimikeDennis,
Looking pretty good so far.
One other thing you need to do is to add the migration, test, and dump file to core/modules/migrate_drupal/src/Tests/d6/MigrateDrupal6Test.php. This is the "big test" that simulates a full migration. Once you take a look at the file, it's pretty obvious how to do it. Then, go ahead and run this test locally to make sure all is well.
In addition, there's a couple of minor fixes below.
Thanks,
-mike
Need to remove trailing spaces.
Let's keep consistent and make "time zone" one word.
Comment #22
densolis commentedMike,
Here is another version of the patch. This includes the change to the complete D6 migration test.
Comment #24
densolis commentedThe above failure was caused by the MigrateUserTest.php simple test. It checks the System Date, but it never sets the D6 value for the field via the dump table.
So I modified the MigrateUserTest.php to load the Drupal6SystemDate.php dump table. So here is the next version of the patch.
Comment #25
densolis commentedSet to needs review.
Comment #27
ultimikeDennis,
Welcome to the current hell that is our dump files (we're working on fixing this: #2348875: Improving our dump files). Here's what is going on...
When the MigrateUserTest runs by itself (not as part of MigrateDrupal6Test), there is no default timezone set (since your variable migration isn't a dependency of the user migration), so the
$this->assertEqual($user->getTimeZone(), $time_zone);test always passes, as users with "timezone_name = NULL" will be migrated with a NULL timezone.During the Drupal6Test, all migrations run, including your new d6_system_date, so users with "timezone_name = NULL" in D6 will get the default timezone as set in your migration, so the
$this->assertEqual($user->getTimeZone(), $time_zone);tests now fail.So, what is the solution - I _think_ we'll probably have to make the d6_system_date migration a dependency of d6_user. Then, in MigrateUserTest, we'll have to add $id_mappings for d6_system_date. Before you do this though, we should probably confirm this approach with either chx or benjy.
Also - one other thing: while you added the d6_system_date migration and the Drupal6SystemDate dump to MigrateDrupal6Test, you forgot to add MigrateSystemDateTest...
-mike
Comment #28
densolis commentedMike,
hmm. I know what you are saying, but why does the MigrateD6Test results show that MigrateSystemDateTest ran four times and passed each time? The results are about 24 lines below the MigrateUserTest has the failed messages.
Where does the MigrateSystemDateTest.php go in the MigrateD6Test.php? I looked through it and did not see where it went.
The only place I can see it going is in the Namespace section of MigrateD6Test. Is there were it goes?
Also, since it is not in MigrateD6Test, why did it get run within MigrateD6Test?
Dennis
Comment #29
ultimikeDennis,
Because the MigrateSystemDateTest works fine. It is the presence of this new migration that is now causing the MigrateUserTest to now fail. Before the d6_system_date migration was added, there wasn't a default timezone for users to be assigned to, and the MigrateUserTest went happily on its way, knowing that
$user->getTimeZone()will be NULL for NULL input from D6. Now, with the d6_system_date migration working, a NULL input from D6 will be turned into whatever the d6_system_date migration migrates in as the default. Make sense?MigrateSystemDateTest goes in the
getTestClassesList()method of MigrateDrupal6Test.Yeah, this is tricky as well. I assume you're looking at https://qa.drupal.org/pifr/test/894538 - you are correct that is shows MigrateSystemDateTest as having run and passed. But, keep in mind that the test bot runs the MigrateSystemDateTest by itself (which is what we see in the results) and as part of MigrateDrupal6Test. It's sneaky, I know.
Making more sense now?
-mike
Comment #30
densolis commentedMake more sense now? No.
Here are my questions.
I get this. The MigrateSystemDateTest is run by itself and it appears to run after the MigrateDrupal6Test and the associated MigrateUserTest.
So, given this - where is the value of $time_zone = Europe/Paris coming from in MigrateUserTest. Since the D8 timezone has not been set - why is it not null? Remember, my process has not yet test. And when it does run, it runs by itself and does not affect the values in MigrateUserTest. So how is this value being set? If I'm not setting it, what process is?
The comments
and
seem to be contradicting each other. If I forgot to add the MigrateSystemDateTest, how can it be causing the MigrateUserTest to fail?
The default is UTC - not Europe/Paris. If Europe/Paris is not coming from the MigrateSystemDateTest or the d6_system_date.yml - where is it coming from?
I do not know how to make MigrateUserTest dependent on MigrateSystemDateTest. I searched for the string "depend" in the d6/test source directory and almost every test has the string "depend" in it. I could not find one where it was dependent upon another test.
Comment #31
ultimikeDennis,
The following may be of use in MigrateUserTest::setup():
I actually think you probably need this instead of the d6_system_date id_settings...
Then, based on what we discussed during the weekly hangout, you still need to add d6_system_date as a dependency to d6_user, we don't need id_mappings in d6_user, but we do need to mark the migration as complete, using something like this (via MigrateNodeTestBase and MigrateTestBase::prepareMigrations()):
-mike
Comment #32
ultimikeDennis,
This is getting complicated, and I'm not sure it should be.
As we discussed as we screenshared, d6_user.yml uses the UserUpdate7002 process plugin to transform the D6 user "timezone" field as follows:
We need to determine if both the D6 timezone_name and event_timezone fields are null, should we set the D8 timezone to null or to the D8 system.date.timezone.default value.
-mike
Comment #33
benjy commentedIf we can't decide what to set it to, lets just set it to the D8 default unless NULL in D8 has some kind of special meaning like it does for filter formats?
Comment #34
densolis commentedBenjy,
I am going to modify the user test so that if it's logic returns a null, then I will set the time to the system configuration time.
I've been on the road this last week and have not had much time. I will some time in the next couple of days to work on this.
Dennis
Comment #35
densolis commentedAll,
Just to remind everyone, the problem is this simple test errors out because the time zone return a time zone of Null for three test. Here is the dump code for \core\modules\migrate_drupal\src\Tests\Dump\Drupal6User.php. The following value are set for "expected_timezones":
I have been looking at the below code in \core\modules\migrate_drupal\src\Tests\d6\MigrateUserTest.php
The assertEqual fails because GetTimeZone returns a different value than $time_zone. As I read Benjy's comment above, I should change the code to:
But I thought the code I thought that code
$source->expected_timezone ?:is doing the same thing as the propose IF statement. The Dump test code set the value of expected_timezone to Null.What am I missing? I know it is something obvious.
Thanks,
Dennis
Comment #36
benjy commentedhmm, when I said that I wasn't talking about the test, more the actual processing itself?
Comment #37
mgiffordJust unassigning to help see if we can move this issue ahead.
Comment #38
quietone commentedComment #39
quietone commentedReroll and move into appropriate modules.
Comment #41
quietone commentedTrying ultimike's suggestion in #31.
Comment #43
quietone commentedComment #44
quietone commentedComment #46
quietone commentedI didn't expect that one to pass.
I don't fully understand what is happening here. I set the system time to "America/New York" but that is not what is returned in the test. It doen't help that it works locally.
Comment #47
quietone commentedComment #48
quietone commentedI'm not sure the above patch is correct. I made changes based on experimenting with D6 locally, not from vast experience with D6.
One change is that /migrate_drupal/src/Tests/Table/d6/Users.php is modified so that the timezone and timezone_name are always in sync. And the other is that migrate/process/d6/UserUpdate7002.php is modifed so that if the source timezone is NULL is it transformed to the system default timezone.
Comment #49
quietone commentedRemoved 'loadDump' line.
Comment #50
phenaproximaLooks pretty good to me. I have only minor changes to request:
Nit: extra line of whitespace.
The plugin also needs a unit test.OK, maybe not; see #55.For consistency, let's use
$this->config('system.date')here.The config object should be injected in the constructor (using ContainerFactoryPluginInterface), in order to improve testability.
Comment #51
quietone commentedCan do on the two simple edits. But .. I don't know how to write a Unit test for the process plugin and I don't know how to inject the config object into the constructor.
Comment #52
chx commentedYou can look at
./core/modules/block/src/Plugin/migrate/process/d6/BlockTheme.phpas an example of the injection asked for.Comment #53
chx commentedAnd no, the process plugin does not need a unit test, the migrate_drupal process plugins are not reusable waste of effort to unit test them, the integration test is sufficient. Check it: most (all?) don't have a test.
Comment #54
chx commentedAnd no, the process plugin does not need a unit test, the migrate_drupal process plugins are not reusable waste of effort to unit test them, the integration test is sufficient. Check it: most (all?) don't have a test.
Comment #55
phenaproximaBecause the plugin is quite simple in this case, I agree that we don't really need a test. But in general process plugins should be unit tested same as anything else. Migrate Drupal source plugins aren't terribly re-usable either but we have written extensive unit tests for those, so why would we skip testing the process plugins?
Comment #56
chx commentedIt was a gut feeling -- people might actually need the D6/D7 source plugins for other purposes than D8 migration. Also, most (all?) migrate_drupal process plugins are extremely simplistic.
With all that said I am not against (who am I to be against) writing tests for them but so far we don't have any so why block this issue on that? A followup (series of followups) can be done if we want to test them.
Comment #57
quietone commentedFrom #50
1. fixed
2. fixed
3. possibly fixed. On IRC chx pointed out that core/modules/block/src/Plugin/migrate/process/d6/BlockTheme.php did a similar thing. I used that as a guide for #3. Thanks chx.
No unit test is in this patch.
I'm getting 12 file related exceptions on MigrateUserTest locally but I've seen in the past that I get these errors but testbot is fine.
Comment #59
quietone commentedAn attempt at a unit test. Not sure the directory is correct because there is another plugin test at
core/modules/system/tests/src/Unit/Migrate/source/d6/MenuTest.php while others are in core/modules/field/tests/src/Unit/Plugin/migrate.
Comment #60
quietone commentedComment #61
quietone commentedPlease review the patch in #57.
After feedback from chx and phenaproxima, we all agree that the test should be removed. So, just revert to the previous patch #57. When #57 was submitted it failed tests, but on retest it passed and the original test changed to a pass.
Comment #64
quietone commentednamespace error
Comment #66
quietone commentedActually, my error. Left testing crud in the patch.
Comment #67
phenaproximaA few more things, then this looks terrific to me.
The static mapping here shouldn't be necessary -- the config system will cast the value to boolean (or the appropriate type) automatically. Direct mapping should suffice.
The variables go into the system.date config object, let's change the comment to say as much.
This doesn't appear to be used in the plugin.
What is $x for?
Comment #68
quietone commented1-3. Fixed
4. testing crud, a dummy line where I set a breakpoint. the debugger highlights breakpoint in such a way that they are hard to read, so I use a dummy line. Probably better to see if I can configure the highlighting.
Comment #69
phenaproximaKickass.
Comment #70
phenaproximaChanging the test group.
Comment #71
phenaproximaBack to RTBC.
Comment #72
webchickCommitted and pushed to 8.0.x. Thanks!
Comment #75
liam morlandIn Drupal 7, it works to put the following in settings.php
What is the equivalent in Drupal 8?
Comment #76
quietone commented