Migration D6 > D8 of CCK date fields

A couple of notes in pursuit of solution. (I'm just coming to D8 so writing a migration plugin myself is at the moment beyond me.)

To recap, date fields are not migrating. However, some aspects of a date field DO migrate; specifically, in the D8 Config table, the (serialized) data for a migrated node-type does include proper information for date fields.

For example: if your D6 site includes a node type "Projects" with a CCK date field "content_field_deadline", the D8 site after migration will include a node type "Projects", but there will be no "field_deadline". However, the CONFIG table entry core.entity_view_display.node.projects.default will include some info on the missing date field:

field_deadline:
label: inline
weight: 13
type: datetime_default
settings:
format_type: fallback
timezone_override: ''
third_party_settings: { }

In other words, some aspects of the cck > D8 migration process are working for date fields, even though the actual fields don't migrate.

Can you dump the data from content_node_field and content_node_field_instance for the relevant fields? We do have tests for date fields so i'm guessing it must be something specific to your field configuration.

Also, how are you migrating, with migrate_upgrade? Are there any errors in the logs after migrating?

Thanks for looking in on this! And to ask one question: so date fields have been known to migrate properly from D6 to D8?

I used migrate_upgrade (via drush, and via UI). Interestingly, there are no errors relating to the date fields. Expected errors relating to Node/User Reference fields and other problems which I anticipated.

Note that right now I'm just trying to find out whether date migrations work in general - i.e., whether I should be troubleshooting my migration, waiting for or helping with a plugin, or devising an ugly workaround.

In the Drupal 6 content_node_field table:

field_request_deadline:
  type: date
  global settings:
Array
(
    [default_value] => blank
    [default_value_code] => 
    [default_value2] => same
    [default_value_code2] => 
    [input_format] => m/d/Y - H:i
    [input_format_custom] => 
    [increment] => 1
    [text_parts] => Array
        (
        )

    [year_range] => -3:+3
    [label_position] => above
)
 db_columns:
Array
(
    [value] => Array
        (
            [type] => varchar
            [length] => 20
            [not null] => 
            [sortable] => 1
            [views] => 1
        )

)

Drupal 6 content_node_field_instance:

field_request_deadline:
  widget_type: date_popup
  widget_settings: 
Array
(
    [default_value] => blank
    [default_value_code] => 
    [default_value2] => same
    [default_value_code2] => 
    [input_format] => m/d/Y - H:i
    [input_format_custom] => 
    [increment] => 1
    [text_parts] => Array
        (
        )

    [year_range] => -3:+3
    [label_position] => above
)

display_settings:
Array
(
    [weight] => 17
    [parent] => group_request_scheduling
    [label] => Array
        (
            [format] => inline
        )

    [teaser] => Array
        (
            [format] => default
            [exclude] => 0
        )

    [full] => Array
        (
            [format] => default
            [exclude] => 0
        )

    [4] => Array
        (
            [format] => default
            [exclude] => 0
        )

    [2] => Array
        (
            [format] => default
            [exclude] => 0
        )

    [3] => Array
        (
            [format] => default
            [exclude] => 0
        )

    [token] => Array
        (
            [format] => default
            [exclude] => 0
        )

)
  widget_module: date

And in Drupal 8 Config table, within core.entity_view_display.node.request.default:

[field_request_deadline] => Array
                (
                    [label] => inline
                    [weight] => 15
                    [type] => datetime_default
                    [settings] => Array
                        (
                            [format_type] => fallback
                            [timezone_override] => 
                        )

                    [third_party_settings] => Array
                        (
                        )

                )

A couple of notes:
- this iteration has the D6 date field as part of a field group; but other migration attempts with field group removed had same result.

Additional note, possibly relevant:

Within the Config table, core.entity_view_display.node.request.default did NOT include date fields in the dependencies->config array.

[dependencies] => Array
        (
            [config] => Array
                (
                    [0] => field.field.node.request.body
                    [1] => field.field.node.request.field_request_additional_links
                    [2] => field.field.node.request.field_request_assets
                    (etc...)
                    [16] => node.type.request
                )

My D6 site uses Datestamp type date fields. The content is migrated but in the wrong format. The node__field_date.field_date_value contains a unix timestamp: '1458086400'. The D6 source table also uses unix timestamp. When I create a D8 node the data is stored as '2016-03-14T22:52:20'.

I tried exporting from a clean 6.38 to a clean 8.0.5 with the same results: no date fields were migrated.

D6
- modules installed: cck, date
- 3 fields created on the page content type: field_date_test, field_datetime_test and field_timestamp_test. Default settings for all three.
- 1 page created with the same value in all three fields: 04/02/2016 - 13:49

D8
- modules installed: migrate, migrate_upgrade, migrate_drupal, migrate_plus, migrate_tools

Migrations were run using drush 8.0.5.
I have attached dumps of the relevant D6 tables. Let me know if I can run some more tests to help debug that issue.

It appears that no migrate processes, templates, tests, etc. were created to migrate Date fields to D8.

At least not in Core nor in the Date module, though a separate issue was created in the date module issue queue here:
#2655308: Convert datestamp fields D6 to D8

If this is going to be solved in Core, then the other should be marked as a duplicate.

While we're at it, let's make this a "Feature request".

Okay, so my bad. There is definitely migrate code in core to support date migrations. Unfortunately, it doesn't appear to be working properly. At least not for the above user, nor for me. So I'm changing this to a "Bug report".

In D6, I have a date field with these global_settings:
a:7:{s:11:"granularity";a:2:{s:4:"year";s:4:"year";s:5:"month";s:5:"month";}s:11:"timezone_db";s:0:"";s:11:"tz_handling";s:4:"none";s:6:"todate";s:0:"";s:6:"repeat";i:0;s:16:"repeat_collapsed";s:0:"";s:14:"default_format";s:6:"medium";}

The field instance uses date_text widget with these widget_settings:
a:10:{s:13:"default_value";s:3:"now";s:18:"default_value_code";s:0:"";s:14:"default_value2";s:4:"same";s:19:"default_value_code2";s:0:"";s:12:"input_format";s:11:"Y-m-d H:i:s";s:19:"input_format_custom";s:3:"F Y";s:9:"increment";i:1;s:10:"text_parts";a:0:{}s:10:"year_range";s:5:"-3:+3";s:14:"label_position";s:4:"none";}

The field is not created during the migration.

I have added migrate plugins for the datetime module It DEFINITELY needs work, because it still doesn't work, but I want to put my work out there and get some help.

Basically, the field type isn't mapping properly, and I don't know why. When it gets to CckFieldPluginBase.php line 75, in getFieldType(), it can't find the mapping to "date time", so it just thinks the field type is "date".

Linking issue: #2631736-51: Cckfield Plugins must distinguish core versions

Also, I was able to import the date fields by adding the following to CckFieldPluginBase.php line 75:

    if ($field_type == 'date') {
      return 'datetime';
    }

The data in given in #5 has a widget of 'date_popup' and the one in #12 has a widget of 'date_text'. I don't see a migration path for them. I modified the d6 dump such that these widgets are being used by two of the existing date fields on the story content type and entered data for all the date fields. Tests have been added to MigrateFieldTest and MIgrateNodeTest.

Unfortunately, there is still a failure in MigrateNodeTest. The date data for all three fields is migrated to D8 but isn't retrieved when loading the node.

Let's start with a reroll first.

Fixed the failure.

I was initially scared of reviewing this patch until I realized that 99% of it is changes to the D6 database fixture. Looks great! Thank you all.

Rerolled. The conflicts were with timestamps in drupal6.php, so hopefully, tests pass.

Taking a second look at the patch, I'm wondering why new assertions haven't been added to the tests for the migrations changed in this patch...

Rerolled #24 for 8.2.x.

Noting there is a conflict between date_migrate-2566779-24.patch and 2447727-182.patch. date_migrate-2566779-24.patch will not apply after 2447727-182.patch due to merge conflicts in core/modules/migrate_drupal/tests/fixtures/drupal6.php and core/modules/field/migration_templates/d6_field_instance_widget_settings.yml.

The test fixture looks straightforward to fix (see drupal6.php_.rej_.txt) but the changes to d6_field_instance_widget_settings.yml look more significant (d6_field_instance_widget_settings.yml.rej_.txt), since 2447727-182.patch changes the 'options/type' to no longer be a static_map.

A couple of small things, then I think this is probably RTBC.

1. +++ b/core/modules/field/tests/src/Kernel/Migrate/d6/MigrateFieldTest.php
   @@ -58,9 +58,13 @@ public function testFields() {
   +    $this->assertIdentical("datetime", $field_storage->getType(), t('Field type is @fieldtype. It should be datetime.', array('@fieldtype' => $field_storage->getType())));
   +    $field_storage = FieldStorageConfig::load('node.field_test_date');
   +    $this->assertIdentical("datetime", $field_storage->getType(), t('Field type is @fieldtype. It should be datetime.', array('@fieldtype' => $field_storage->getType())));
   

   Let's use assertSame(), and we don't need to translate the assertion message.

2. +++ b/core/modules/migrate_drupal/tests/fixtures/drupal6.php
   @@ -45630,10 +45459,6 @@
   -  'name' => 'i18n_lock_node_article',
   -  'value' => 'i:1;',
   -))
   -->values(array(
   

   Why did this get removed?

3. +++ b/core/modules/node/tests/src/Kernel/Migrate/d6/MigrateNodeTest.php
   @@ -65,6 +65,10 @@ public function testNode() {
   +    $this->assertSame('2013-01-02T04:05:00', $node->field_test_date->value,'Date field with text widget is correct');
   +    $this->assertSame('1391357160', $node->field_test_datestamp->value,'Date field with text widget is correct');
   +    $this->assertSame('2015-03-04 06:07:00', $node->field_test_datetime->value,'Date field with text widget is correct');
   

   There needs to be a space after the second argument for each of these calls.

Fails to apply.

The reroll.

Back to NW for the issues in #29.

Rerolled again against 8.2.x after new commits there on Oct 21. As you can see in the interdiff, only line offsets have changed.

Also, I'm bold enought to unassign from @phenaproxima as it seems there is need for more contribution before a new round of review.

Addressing the items in #29.
1 and 3. Fixed
2. i18n_lock_node_article exists, and is the same value, before and after the patch is applied. I can't explain what git diff is doing. However, it does look like the dump has unnecessary changes and needs to be trimmed down to just what is needed. That is something I can't do now, but maybe in a few days.

Assigning to myself for review.

Can we get a no test patch that doesn't have the facets in it? It might make it easier for reviewing.

I'll try to review this in the next week.

I created date, datestamp, and datetime fields on a D6 site and performed a migration. All three fields were created on the node, but only the date field renders properly - as pointed out in #7 above, the actual values are copied as-is rather than converted to the "2014-03-14T07:03:00" form as used in D8.

+++ b/core/modules/node/tests/src/Kernel/Migrate/d6/MigrateNodeTest.php
@@ -65,6 +65,10 @@ public function testNode() {
+    $this->assertSame('1391357160', $node->field_test_datestamp->value, 'Date field with text widget is correct');
+    $this->assertSame('2015-03-04 06:07:00', $node->field_test_datetime->value, 'Date field with text widget is correct');

Well, this does match the actual behavior - unfortunately, it's not the behavior we actually want:P.

On that side note then, does #2820490: FormatDate process plugin need to move to the core issue queue?

@heddn: Yes! That plugin is generally useful enough to consider for core anyway, and it would address the problem here.

Postponing on the FormatDate plugin.

sorry to bother... but what exactly do i have to do now to be able to migrate D6 to D8 date fields?
i wonder if there are any sites out there NOT using datefields in D6?
how come this has not been handled by core?

Because there are many, many, many Migrate-related issues that need doing, and too few of us who are willing and/or able to do them. This is why core development moves very slowly. If you want to unblock this issue, working on #2820490: FormatDate process plugin would be a great way to do that.

This is a blocker to considering the Drupal-to-Drupal migration path stable.

Drupal 8 now has two datetime types: datetime and date. In patch #34 datetime is hardcoded (in FieldSettings.php getSettings(): 'datetime' => array('datetime_type' => 'datetime'),). What can I do if I want to use date only? In Drupal 6 the field settings where set by granularity to year + month + day. How would the solution look like, I I want to set "date only" only for one field (field_date_only) and use datetime all the others ... with less coding possible.

Use the process plugin from the related issue

@heddn: You mean #2820490: FormatDate process plugin? I am using that one too, its for the value formatting, but don't help for the field settings itself.

My problem is related to core/modules/field/src/Plugin/migrate/process/d6/FieldSettings.php: I don't can't see how to to overwrite datetime_type somewhere to set it to date (for date only).

#2820490: FormatDate process plugin is in. So no longer postponed.

Probably needs a reroll too.

The reroll.

Corrected the tests that mikeryan pointed out in #38 were wrong. Added a cck date plugin. The value passed to FormatDate process plugin is array of value and delta and since FormatData expects a string the tests will fail. How do we deal with delta?

So more work to do. I'm on holiday for this long weekend, maybe someone can move this along?

Fixed DateField::processCckFieldValues().
Changed the type map so that datestamp is mapped to timestamp.
Updated the text in MigrateNodeTest

Setting to NR helps...

Added date related tests in MigrateFieldTest. Fixed the failing test by adding the to_format for the datestamp type. And, if the field formatter does not exist an invalid from_format will be passed to FormatDate, which will throw an exception.

1. +++ b/core/modules/datetime/src/Plugin/migrate/cckfield/d6/DateField.php
   @@ -0,0 +1,81 @@
   + *   type_map = {
   ...
   +class DateField extends CckFieldPluginBase {
   

   Should we be extending from CCK or the new Field plugin?

2. +++ b/core/modules/datetime/src/Plugin/migrate/cckfield/d6/DateField.php
   @@ -0,0 +1,81 @@
   +  public function getFieldFormatterMap() {
   

   Do we need to implement this? Perhaps no.

3. +++ b/core/modules/datetime/src/Plugin/migrate/cckfield/d6/DateField.php
   @@ -0,0 +1,81 @@
   +    $process = [
   +      'plugin' => 'iterator',
   +      'source' => $field_name,
   +      'process' => $process,
   +    ];
   

   This is about what I would expect. Good.

Moving back to NW for #56.

@heddn, thanks for the prompt review.

1. The Cck rename patch has not yet been ported to 8.3.x and this is issue is against 8.3.x

2. DateField extends CckFieldPluginBase which is abstract class implementing MigrateCckFieldInterface. getFieldFormatterMap is defined in MigrateCckFieldInterface but not in CckFieldPluginBase,

3. Sweet!

Actually, this needs work to clean up the fixture.

Removed unnecessary changes to the d6 test fixture. That is,kept only changes to the date fields.

Let's port this to 8.4.x so that we can use the Field plugin annotation and base class, rather than Cckfield. Then we can backport to 8.3.x if needed.

OK, here it is. The interdiff is pretty useless/odd, it shows changes for things that were not changed.

1. +++ b/core/modules/datetime/src/Plugin/migrate/field/d6/DateField.php
   @@ -0,0 +1,81 @@
   +      default:
   +        // Set an invalid format which forces a MigrateException in format_date.
   +        $from_format = '';
   

   I don't know how I feel about purposefully setting a value which will throw a MigrateException in another plugin. It seems like unnecessary tight coupling to me. That said, I'm not sure what else we could do. If we stick with this approach, we'll probably need to explicitly test this code path.

2. +++ b/core/modules/field/tests/src/Kernel/Migrate/d6/MigrateFieldInstanceTest.php
   @@ -96,6 +96,41 @@ public function testFieldInstanceMigration() {
   +    $field = FieldConfig::load('node.story.field_test_date');
   

   It doesn't make a whole lot of difference, but I think we should add instanceof checks immediately after these lines, like so:

   $this->assertInstanceOf(FieldConfig::class, $field)
   

   That way, we can avoid fatals if the field doesn't exist.

3. +++ b/core/modules/field/tests/src/Kernel/Migrate/d6/MigrateFieldTest.php
   @@ -62,6 +62,14 @@ public function testFields() {
   +    $this->assertSame("datetime", $field_storage->getType(), t('Field type is @fieldtype. It should be datetime.', array('@fieldtype' => $field_storage->getType())));
   +    $field_storage = FieldStorageConfig::load('node.field_test_datestamp');
   +    $this->assertSame("timestamp", $field_storage->getType(), t('Field type is @fieldtype. It should be timestamp.', array('@fieldtype' => $field_storage->getType())));
   +    $field_storage = FieldStorageConfig::load('node.field_test_date');
   +    $this->assertSame("datetime", $field_storage->getType(), t('Field type is @fieldtype. It should be datetime.', array('@fieldtype' => $field_storage->getType())));
   

   We should be using short array syntax here ([]). I'd also like see some assertInstanceOf() checks added to avoid fatals if the field storages were not migrated properly.

Items 2and 3 fixed.
As for the first one, I wasn't fond of that default value either. Considering that the DateField process is only executed when the type matches it shouldn't even be possible to get there if the type is not one of date, datestamp or datetime. Still, I'd prefer a default value. How about just using any of the three options, say 'date'?

If it's not possible to get to that switch-case unless the incoming type is not one of date, datestamp, or datetime, let's throw either a MigrateException or InvalidArgumentException directly in the default case, rather than expect the FormatDate plugin to do it for us.

Also, small nitpick, but:

+++ b/core/modules/datetime/src/Plugin/migrate/field/d6/DateField.php
@@ -0,0 +1,81 @@
+  public function processFieldValues(MigrationInterface $migration, $field_name, $data) {
+
+    switch($data['type']) {

Extra blank line.

Forgot to mention -- since the committers will likely kick it back without this, let's also add test coverage to ensure that an exception is, in fact, thrown if the incoming field type is invalid.

OK, now throwing a MigrateException and test added.

Test was in wrong directory.

Looks great! Let it roll.

1. +++ b/core/modules/datetime/src/Plugin/migrate/field/d6/DateField.php
   @@ -0,0 +1,79 @@
   +    // See d6_field_formatter_settings.yml and FieldPluginBase
   +    // processFieldFormatter().
   

   Nitpick: If this is in reference to a method on a class, they should be separated by :: and not a newline :)

2. +++ b/core/modules/datetime/src/Plugin/migrate/field/d6/DateField.php
   @@ -0,0 +1,79 @@
   +        throw new MigrateException(sprintf('Field %s is an unknown date field type.', var_export($data['type'], TRUE)));
   

   Is this really the field name or the date type name? If its the date type name, then is the error message helpful identifying which field has the problem? If we also know the field name, should we print the field name AND the date type name, so its easier to figure out where it comes from? Otherwise we are holding back useful information that could be really beneficial in debugging this error.

Working on this.

#71.1 Is fixed.
#71.2 Field name and data type are now labeled and specified.

$ git status
On branch 8.4.x
Your branch is up-to-date with 'origin/8.4.x'.

$ git apply --index 2566779-73.patch
error: patch failed: core/modules/migrate_drupal/tests/fixtures/drupal6.php:3403
error: core/modules/migrate_drupal/tests/fixtures/drupal6.php: patch does not apply

Tagging novice for the re-roll.

Re-rolled

Yay, it passed! Back to RTBC.

Let's get this tested on 8.3.x as well due to our recent problem with backports of migrate stuff. Does this require anything not yet backported to 8.3.x?

@Gabor Hojtsy: Yes, it requires #2683435: CCK does not exist in Drupal 7, rename Migrate's cckfield plugins to field plugins to be backported to 8.3.x.

Postponing on #2683435: CCK does not exist in Drupal 7, rename Migrate's cckfield plugins to field plugins.

No longer blocked. Retesting against 8.3.x. Setting back to RTBC in expectation.

Committed/pushed to 8.4.x and cherry-picked to 8.3.x, thanks!

Tagging for mention in 8.4.0 release notes.