Problem/Motivation

The date_default_timezone, date_first_day, and configurable_timezones variables do not have a migration to D8 (system.date).

Proposed resolution

Create the migration.

Remaining tasks

Create the migration, write the test.

User interface changes

None.

API changes

None.

Below is a listing of the D6 variable name and the corresponding D8 variable name

D6 Variable Name D8 Variable Name
configurable_timezones system.date.timezone.user.configurable
date_first_day system.date.first_day
date_default_timezone system.date.timezone.default

 
 

#2353681: D6->D8 Migration missing variable: configurable_timezones and #2353675: D6->D8 Migration missing variable: date_first_day have been merged with this issue

Comments

ultimike’s picture

Issue summary: View changes
densolis’s picture

Assigned: Unassigned » densolis

Assigned task to myself.

densolis’s picture

Moved comment to original case entry. (see above)

ultimike’s picture

Issue summary: View changes
ultimike’s picture

Issue summary: View changes
densolis’s picture

StatusFileSize
new11.15 MB

Both the date_default_timezone and configurable_timezones are exposed on the d6 form: /admin/settings/date-time. The prompts exposed on the corresponding form on D8.

I wrote the migration for user-configurable time zone, but the default time zone is going to require plug in so that we can convert from + or - Greenwich time to a time zone.

I have attached a patch of what I have so far.

ultimike’s picture

Dennis - your patch is 11MB - that's probably not right. I'm hiding it for now. Let's touch base on IRC later to figure out what went wrong...

Thanks,
-mike

densolis’s picture

StatusFileSize
new3.13 KB

Mike,

oops - yeah, 11mb might be a big big. I figured out what I did wrong and re-rolled the patch and this time I read the contents of the patch. So here it is again.

I'll be working on the plug in later today.

Dennis

ultimike’s picture

Dennis,

Just a couple of small things to fix while you're still working on this patch:

  1. +++ b/core/modules/migrate_drupal/config/install/migrate.migration.d6_system_date.yml
    @@ -0,0 +1,20 @@
    +  'timezone/user/configurable': ¶
    

    Extra white space at end of line.

  2. +++ b/core/modules/migrate_drupal/src/Tests/d6/MigrateSystemDateTest.php
    @@ -0,0 +1,45 @@
    \ No newline at end of file
    

    Need newline at end of file.

Thanks,
-mike

densolis’s picture

Issue summary: View changes
densolis’s picture

The simple test runs fine. However, when I run the migration using drush, the values from D6 are not correctly migrated to D8.

densolis’s picture

StatusFileSize
new4.6 KB

Mike,

Here is what I have so far. As I said above, the simple test runs fine, but I am having problems getting the migration to migrate the data from D6 to D8. I will try again with a fresh D8 site.

Here is my patch so far.

svendecabooter’s picture

Status: Active » Needs review
StatusFileSize
new4.6 KB

Hi,

Tested your patch, and it seems to work fine with my test D6 & D8 installs.
I tested a few different configurations on the D6 site, and they migrated to D8 as expected.

Attached a new version of your patch with a few typos in comments fixed.

densolis’s picture

the migrate and simple test are working for this patch. There are some issues with the php timezone_name_from_abbr function. It does not return a named time zone for each offset. I have attached a table with the following columns:

  1. Timezone Offset. I got these values by inspecting the element for the D6 times zones page.
  2. In minutes. The offset in minutes.
  3. In hours. The offset in hours.
  4. Timezone_name_from_abbr - I ran each value in the Timezone offset through the function. This column contains the values that were generated for each offset value. NOTE: not all offset values produced a result. Also, the offset of "0" produced "Atlantic/Azores". I was expecting "UTC".
  5. Should be time zone. Using the values from the function system_time_zones(), I found names for all of the offsets that were not found by timezone_name_from_abbr. NOTE: I found time zones in system_time_zones that are not in the Drupal 6 offet list of values. Those are noted in the list with a null offset value.

There have been bug in the php function timezone_name_from_abbr since 2008. It has been fixed, broken, fixed, and broken again.

I forgot to attached the time zone spread sheet. It is in comment 18 below.

densolis’s picture

svendecabooter,

What were the typo? Never mind - I just saw them.

Thanks

The last submitted patch, 13: 2353679-13.patch, failed testing.

svendecabooter’s picture

In core/modules/migrate_drupal/src/Plugin/migrate/process/d6/TimeZone.php:

// Convert the integitger value of the offset (which can be either
// negative or positive to a time zone name.

to

// Convert the integer value of the offset (which can be either
// negative or positive) to a time zone name.

In core/modules/migrate_drupal/src/Tests/d6/MigrateSystemDateTest.php

* Upgrade variables to user.*.yml.
to
* Upgrade variables to system.*.yml.

I think your attachment for the timezones issue is missing?

densolis’s picture

StatusFileSize
new21.5 KB

Here is the spread sheet attachment that should have been with comment 14.

densolis’s picture

StatusFileSize
new4.6 KB

I am guess that the simpletest failed because my machine is on day light saving time and the test machine is not. Below is the timezone_name_from_abbr for the same offset with and without daylight saving times. As you can see, they produce different results.

php -r "print timezone_name_from_abbr('', 3600, 0)";
Europe/Paris

php -r "print timezone_name_from_abbr('', 3600, 1)";
Europe/London

I change my simple test value from Europe/London to Europe/Paris.

densolis’s picture

There is an issue when run the simple test on this patch.

As I said above, the function timezone_name_from_abbr will produce DIFFERENT results for the SAME offset value depending upon your daylight saving time setting. As you can see above, the offset value of 3600 will return either a value of Paris or London depending on the daylight saving time setting on your server.

On my dev machine, daylight saving time is on, so the function returns a value of Europe/London. And the simple test worked. However, when the test bot ran the same simple test - it failed. So the ONLY change I made to the simple test is I changed Europe/London to Europe/Paris and the simple test passed. So I am guessing the simple test is run on a machine with the daylights saving time disabled.

Does anyone have a suggestion on how to get around this so that the simple test will run on any machine regardless of value of daylight savings time?

ultimike’s picture

Status: Needs review » Needs work

Dennis,

Looking pretty good so far.

One other thing you need to do is to add the migration, test, and dump file to core/modules/migrate_drupal/src/Tests/d6/MigrateDrupal6Test.php. This is the "big test" that simulates a full migration. Once you take a look at the file, it's pretty obvious how to do it. Then, go ahead and run this test locally to make sure all is well.

In addition, there's a couple of minor fixes below.

Thanks,
-mike

  1. +++ b/core/modules/migrate_drupal/config/install/migrate.migration.d6_system_date.yml
    @@ -0,0 +1,25 @@
    +  ¶
    

    Need to remove trailing spaces.

  2. +++ b/core/modules/migrate_drupal/src/Plugin/migrate/process/d6/TimeZone.php
    @@ -0,0 +1,39 @@
    + * Process the D6 Timezone offset into a D8 compatible time zone name.
    

    Let's keep consistent and make "time zone" one word.

densolis’s picture

Status: Needs work » Needs review
StatusFileSize
new5.59 KB

Mike,

Here is another version of the patch. This includes the change to the complete D6 migration test.

Status: Needs review » Needs work

The last submitted patch, 22: 2353679-22.patch, failed testing.

densolis’s picture

StatusFileSize
new6.21 KB

The above failure was caused by the MigrateUserTest.php simple test. It checks the System Date, but it never sets the D6 value for the field via the dump table.

So I modified the MigrateUserTest.php to load the Drupal6SystemDate.php dump table. So here is the next version of the patch.

densolis’s picture

Status: Needs work » Needs review

Set to needs review.

Status: Needs review » Needs work

The last submitted patch, 24: 2353679-24.patch, failed testing.

ultimike’s picture

Dennis,

Welcome to the current hell that is our dump files (we're working on fixing this: #2348875: Improving our dump files). Here's what is going on...

When the MigrateUserTest runs by itself (not as part of MigrateDrupal6Test), there is no default timezone set (since your variable migration isn't a dependency of the user migration), so the $this->assertEqual($user->getTimeZone(), $time_zone); test always passes, as users with "timezone_name = NULL" will be migrated with a NULL timezone.

During the Drupal6Test, all migrations run, including your new d6_system_date, so users with "timezone_name = NULL" in D6 will get the default timezone as set in your migration, so the $this->assertEqual($user->getTimeZone(), $time_zone); tests now fail.

So, what is the solution - I _think_ we'll probably have to make the d6_system_date migration a dependency of d6_user. Then, in MigrateUserTest, we'll have to add $id_mappings for d6_system_date. Before you do this though, we should probably confirm this approach with either chx or benjy.

Also - one other thing: while you added the d6_system_date migration and the Drupal6SystemDate dump to MigrateDrupal6Test, you forgot to add MigrateSystemDateTest...

-mike

densolis’s picture

Mike,

hmm. I know what you are saying, but why does the MigrateD6Test results show that MigrateSystemDateTest ran four times and passed each time? The results are about 24 lines below the MigrateUserTest has the failed messages.

Where does the MigrateSystemDateTest.php go in the MigrateD6Test.php? I looked through it and did not see where it went.

The only place I can see it going is in the Namespace section of MigrateD6Test. Is there were it goes?

Also, since it is not in MigrateD6Test, why did it get run within MigrateD6Test?

Dennis

ultimike’s picture

Dennis,

why does the MigrateD6Test results show that MigrateSystemDateTest ran four times and passed each time?

Because the MigrateSystemDateTest works fine. It is the presence of this new migration that is now causing the MigrateUserTest to now fail. Before the d6_system_date migration was added, there wasn't a default timezone for users to be assigned to, and the MigrateUserTest went happily on its way, knowing that $user->getTimeZone() will be NULL for NULL input from D6. Now, with the d6_system_date migration working, a NULL input from D6 will be turned into whatever the d6_system_date migration migrates in as the default. Make sense?

MigrateSystemDateTest goes in the getTestClassesList() method of MigrateDrupal6Test.

Also, since it is not in MigrateD6Test, why did it get run within MigrateD6Test?

Yeah, this is tricky as well. I assume you're looking at https://qa.drupal.org/pifr/test/894538 - you are correct that is shows MigrateSystemDateTest as having run and passed. But, keep in mind that the test bot runs the MigrateSystemDateTest by itself (which is what we see in the results) and as part of MigrateDrupal6Test. It's sneaky, I know.

Making more sense now?

-mike

densolis’s picture

Make more sense now? No.
Here are my questions.

But, keep in mind that the test bot runs the MigrateSystemDateTest by itself

I get this. The MigrateSystemDateTest is run by itself and it appears to run after the MigrateDrupal6Test and the associated MigrateUserTest.

So, given this - where is the value of $time_zone = Europe/Paris coming from in MigrateUserTest. Since the D8 timezone has not been set - why is it not null? Remember, my process has not yet test. And when it does run, it runs by itself and does not affect the values in MigrateUserTest. So how is this value being set? If I'm not setting it, what process is?

The comments

you forgot to add MigrateSystemDateTest...

and

It is the presence of this new migration that is now causing the MigrateUserTest to now fail.

seem to be contradicting each other. If I forgot to add the MigrateSystemDateTest, how can it be causing the MigrateUserTest to fail?

Now, with the d6_system_date migration working, a NULL input from D6 will be turned into whatever the d6_system_date migration migrates in as the default.

The default is UTC - not Europe/Paris. If Europe/Paris is not coming from the MigrateSystemDateTest or the d6_system_date.yml - where is it coming from?

I do not know how to make MigrateUserTest dependent on MigrateSystemDateTest. I searched for the string "depend" in the d6/test source directory and almost every test has the string "depend" in it. I could not find one where it was dependent upon another test.

ultimike’s picture

Dennis,

The following may be of use in MigrateUserTest::setup():

    $config = \Drupal::config('system.date');
    $config->set('timezone.default', 'America/New York');
    $config->save();

I actually think you probably need this instead of the d6_system_date id_settings...

Then, based on what we discussed during the weekly hangout, you still need to add d6_system_date as a dependency to d6_user, we don't need id_mappings in d6_user, but we do need to mark the migration as complete, using something like this (via MigrateNodeTestBase and MigrateTestBase::prepareMigrations()):

$migration = entity_load('migration', 'd6_system_date');
// Mark the dependent migrations as complete.
$migration->setMigrationResult(MigrationInterface::RESULT_COMPLETED);

-mike

ultimike’s picture

Dennis,

This is getting complicated, and I'm not sure it should be.

As we discussed as we screenshared, d6_user.yml uses the UserUpdate7002 process plugin to transform the D6 user "timezone" field as follows:

  1. It ignores whatever the D6 timezone field is set to.
  2. If the D6 timezone_name field is not null, it uses that as the D8 timezone.
  3. Else, if the D6 event_timezone field is not null, it uses that as the D8 timezone.
  4. If both the D6 timezone_name and D6 event_timezone fields are null, then the D8 user timezone is also set to null.

We need to determine if both the D6 timezone_name and event_timezone fields are null, should we set the D8 timezone to null or to the D8 system.date.timezone.default value.

-mike

benjy’s picture

If we can't decide what to set it to, lets just set it to the D8 default unless NULL in D8 has some kind of special meaning like it does for filter formats?

densolis’s picture

Benjy,

I am going to modify the user test so that if it's logic returns a null, then I will set the time to the system configuration time.

I've been on the road this last week and have not had much time. I will some time in the next couple of days to work on this.

Dennis

densolis’s picture

All,

Just to remind everyone, the problem is this simple test errors out because the time zone return a time zone of Null for three test. Here is the dump code for \core\modules\migrate_drupal\src\Tests\Dump\Drupal6User.php. The following value are set for "expected_timezones":

(198  )           'expected_timezone' => array(
(328  )           'expected_timezone' => 'Europe/Berlin',
(353  )           'expected_timezone' => 'Europe/Helsinki',
(378  )           'expected_timezone' => NULL,
(403  )           'expected_timezone' => NULL,
(428  )           'expected_timezone' => NULL,

I have been looking at the below code in \core\modules\migrate_drupal\src\Tests\d6\MigrateUserTest.php

      $time_zone = $source->expected_timezone ?: \Drupal::config('system.date')->get('timezone.default');
      $this->assertEqual($user->getTimeZone(), $time_zone);

The assertEqual fails because GetTimeZone returns a different value than $time_zone. As I read Benjy's comment above, I should change the code to:

      $time_zone = $source->expected_timezone ?: \Drupal::config('system.date')->get('timezone.default');
      <em><strong>if (empty($time_zone)) {
          $time_zone = \Drupal::config('system.date')->get('timezone.default');
      }</strong></em>
      $this->assertEqual($user->getTimeZone(), $time_zone);

But I thought the code I thought that code $source->expected_timezone ?: is doing the same thing as the propose IF statement. The Dump test code set the value of expected_timezone to Null.

What am I missing? I know it is something obvious.

Thanks,

Dennis

benjy’s picture

hmm, when I said that I wasn't talking about the test, more the actual processing itself?

mgifford’s picture

Assigned: densolis » Unassigned

Just unassigning to help see if we can move this issue ahead.

quietone’s picture

Assigned: Unassigned » quietone
quietone’s picture

Status: Needs work » Needs review
StatusFileSize
new4.97 KB

Reroll and move into appropriate modules.

Status: Needs review » Needs work

The last submitted patch, 39: 2353679-38.patch, failed testing.

quietone’s picture

Status: Needs work » Needs review
StatusFileSize
new6.33 KB
new1.12 KB

Trying ultimike's suggestion in #31.

Status: Needs review » Needs work

The last submitted patch, 41: 2353679-41.patch, failed testing.

quietone’s picture

StatusFileSize
new7.07 KB
new983 bytes
quietone’s picture

Status: Needs work » Needs review

Status: Needs review » Needs work

The last submitted patch, 43: 2353679-43.patch, failed testing.

quietone’s picture

I didn't expect that one to pass.

I don't fully understand what is happening here. I set the system time to "America/New York" but that is not what is returned in the test. It doen't help that it works locally.

quietone’s picture

Status: Needs work » Needs review
StatusFileSize
new8.21 KB
new3.82 KB
quietone’s picture

I'm not sure the above patch is correct. I made changes based on experimenting with D6 locally, not from vast experience with D6.

One change is that /migrate_drupal/src/Tests/Table/d6/Users.php is modified so that the timezone and timezone_name are always in sync. And the other is that migrate/process/d6/UserUpdate7002.php is modifed so that if the source timezone is NULL is it transformed to the system default timezone.

quietone’s picture

StatusFileSize
new8.17 KB
new2.14 KB

Removed 'loadDump' line.

phenaproxima’s picture

Status: Needs review » Needs work
Issue tags: +Needs tests

Looks pretty good to me. I have only minor changes to request:

+++ b/core/modules/migrate_drupal/src/Plugin/migrate/process/d6/TimeZone.php
+  public function transform($value, MigrateExecutableInterface $migrate_executable, Row $row, $destination_property) {
+

Nit: extra line of whitespace.

The plugin also needs a unit test. OK, maybe not; see #55.

+++ b/core/modules/system/src/Tests/Migrate/d6/MigrateSystemDateTest.php
+  public function testSystemDate() {
+    $config = \Drupal::config('system.date');

For consistency, let's use $this->config('system.date') here.

+++ b/core/modules/user/src/Plugin/migrate/process/d6/UserUpdate7002.php
+    if ($timezone === NULL) {
+      $timezone = \Drupal::config('system.date')->get('timezone.default');
+    }

The config object should be injected in the constructor (using ContainerFactoryPluginInterface), in order to improve testability.

quietone’s picture

Can do on the two simple edits. But .. I don't know how to write a Unit test for the process plugin and I don't know how to inject the config object into the constructor.

chx’s picture

You can look at ./core/modules/block/src/Plugin/migrate/process/d6/BlockTheme.php as an example of the injection asked for.

chx’s picture

And no, the process plugin does not need a unit test, the migrate_drupal process plugins are not reusable waste of effort to unit test them, the integration test is sufficient. Check it: most (all?) don't have a test.

chx’s picture

And no, the process plugin does not need a unit test, the migrate_drupal process plugins are not reusable waste of effort to unit test them, the integration test is sufficient. Check it: most (all?) don't have a test.

phenaproxima’s picture

the migrate_drupal process plugins are not reusable waste of effort to unit test them

Because the plugin is quite simple in this case, I agree that we don't really need a test. But in general process plugins should be unit tested same as anything else. Migrate Drupal source plugins aren't terribly re-usable either but we have written extensive unit tests for those, so why would we skip testing the process plugins?

chx’s picture

It was a gut feeling -- people might actually need the D6/D7 source plugins for other purposes than D8 migration. Also, most (all?) migrate_drupal process plugins are extremely simplistic.

With all that said I am not against (who am I to be against) writing tests for them but so far we don't have any so why block this issue on that? A followup (series of followups) can be done if we want to test them.

quietone’s picture

Status: Needs work » Needs review
StatusFileSize
new12.03 KB
new7 KB

From #50
1. fixed
2. fixed
3. possibly fixed. On IRC chx pointed out that core/modules/block/src/Plugin/migrate/process/d6/BlockTheme.php did a similar thing. I used that as a guide for #3. Thanks chx.

No unit test is in this patch.

I'm getting 12 file related exceptions on MigrateUserTest locally but I've seen in the past that I get these errors but testbot is fine.

Status: Needs review » Needs work

The last submitted patch, 57: 2353679-57.patch, failed testing.

quietone’s picture

StatusFileSize
new11.58 KB
new2.95 KB

An attempt at a unit test. Not sure the directory is correct because there is another plugin test at
core/modules/system/tests/src/Unit/Migrate/source/d6/MenuTest.php while others are in core/modules/field/tests/src/Unit/Plugin/migrate.

quietone’s picture

Status: Needs work » Needs review
quietone’s picture

Please review the patch in #57.

After feedback from chx and phenaproxima, we all agree that the test should be removed. So, just revert to the previous patch #57. When #57 was submitted it failed tests, but on retest it passed and the original test changed to a pass.

phenaproxima queued 57: 2353679-57.patch for re-testing.

The last submitted patch, 57: 2353679-57.patch, failed testing.

quietone’s picture

StatusFileSize
new12.03 KB
new575 bytes

namespace error

Status: Needs review » Needs work

The last submitted patch, 64: 2353679-64.patch, failed testing.

quietone’s picture

Status: Needs work » Needs review
StatusFileSize
new10.12 KB
new1.72 KB

Actually, my error. Left testing crud in the patch.

phenaproxima’s picture

Status: Needs review » Needs work
Issue tags: -Needs tests

A few more things, then this looks terrific to me.

+++ b/core/modules/system/migration_templates/d6_system_date.yml
+  'timezone/user/configurable':
+    plugin: static_map
+    source: configurable_timezones
+    map:
+      0: FALSE
+      1: TRUE

The static mapping here shouldn't be necessary -- the config system will cast the value to boolean (or the appropriate type) automatically. Direct mapping should suffice.

+++ b/core/modules/system/src/Tests/Migrate/d6/MigrateSystemDateTest.php
+ * Upgrade date time variables to system.*.yml.

The variables go into the system.date config object, let's change the comment to say as much.

+++ b/core/modules/user/src/Plugin/migrate/process/d6/UserUpdate7002.php
   /**
+   * Contains the configuration object factory.
+   *
+   * @var \Drupal\Core\Config\ConfigFactoryInterface
+   */
+  protected $configFactory;

This doesn't appear to be used in the plugin.

+++ b/core/modules/user/src/Plugin/migrate/process/d6/UserUpdate7002.php
+  public static function create(ContainerInterface $container, array $configuration, $plugin_id, $plugin_definition) {
+    $x = 100;

What is $x for?

quietone’s picture

Status: Needs work » Needs review
StatusFileSize
new9.89 KB
new1.88 KB

1-3. Fixed
4. testing crud, a dummy line where I set a breakpoint. the debugger highlights breakpoint in such a way that they are hard to read, so I use a dummy line. Probably better to see if I can configure the highlighting.

phenaproxima’s picture

Status: Needs review » Reviewed & tested by the community

Kickass.

phenaproxima’s picture

Status: Reviewed & tested by the community » Needs review
StatusFileSize
new9.9 KB
new516 bytes

Changing the test group.

phenaproxima’s picture

Status: Needs review » Reviewed & tested by the community

Back to RTBC.

webchick’s picture

Status: Reviewed & tested by the community » Fixed

Committed and pushed to 8.0.x. Thanks!

  • webchick committed b06eee2 on 8.0.x
    Issue #2353679 by quietone, densolis, phenaproxima, ultimike: D6->D8...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

liam morland’s picture

In Drupal 7, it works to put the following in settings.php

$conf['date_first_day'] = 1;

What is the equivalent in Drupal 8?

quietone’s picture

Assigned: quietone » Unassigned