Problem/Motivation

The last time a given migration imported content is useful information. This was provided in the D7 Migrate via the migrate_ui dashboard and the drush migrate-status command.

Proposed resolution

At the completion of an import operation, save the current datetime via State API.

Remaining tasks

Implement the functionality and tests.

User interface changes

N/A for core (Migrate Plus will make use of it in contrib).

API changes

Add getLastImported()/setLastImported() to MigrationInterface and Migration.

Comments

nicoloye’s picture

StatusFileSize
new2.42 KB

Trying to understand the global structure of our new migrate module, so sorry if the logic is wrong, I'm still novice on D8 ;)
Here is a patch, c&c are welcome.
Surely needs more work.

Currently the states are saved with this name : "migrate_last_imported:[migration id]", maybe another naming convention should be used ?

nicoloye’s picture

Status: Active » Needs review

Status: Needs review » Needs work

The last submitted patch, 1: migrate-last-imported-time-2429089-1.patch, failed testing.

mikeryan’s picture

  1. +++ b/core/modules/migrate/src/Entity/Migration.php
    @@ -350,6 +350,10 @@ public function checkRequirements() {
    +        $this->setLastImported();
    

    Here you're setting it where another migration is checking to see if this migration has completed at some time in the past. The place to set it would be at the point the migration actually finished running. The return points in MigrateExecutable::import() would be the place to do that.

  2. +++ b/core/modules/migrate/src/Entity/Migration.php
    @@ -477,6 +481,24 @@ public function setTrackLastImported($track_last_imported) {
    +    $key = 'migrate_last_imported:' . $this->id;
    

    We're saving other similar information using keyvalue rather that state - see https://www.drupal.org/node/2429085#comment-9829053 - I think this should be consistent with the others.

  3. +++ b/core/modules/migrate/src/Entity/Migration.php
    @@ -477,6 +481,24 @@ public function setTrackLastImported($track_last_imported) {
    +    return date('Y-m-d H:i:s', $last_imported/1000);
    

    I don't think the API should do the formatting, that should be left to the caller (just return the timestamp).

  4. +++ b/core/modules/migrate/src/Entity/MigrationInterface.php
    @@ -273,6 +273,21 @@ public function isTrackLastImported();
    +   *   The current system of record of the migration.
    

    A little too much pasted here.

nicoloye’s picture

Oops, sorry for that, I'll check this tomorrow. Thanks for the feedback :)

nicoloye’s picture

Status: Needs work » Needs review
StatusFileSize
new2.12 KB

Here is a fixed patch.

mikeryan’s picture

  1. +++ b/core/modules/migrate/src/Entity/Migration.php
    @@ -477,6 +477,24 @@ public function setTrackLastImported($track_last_imported) {
    +    $last_imported = \Drupal::state()->get($key, FALSE);
    

    Still looking for that change from state to keyvalue - see the previously-linked issue and the current patch using keyvalue: https://www.drupal.org/files/issues/track_current_state_of-2429085-13.patch

  2. +++ b/core/modules/migrate/src/MigrateExecutable.php
    @@ -345,6 +345,7 @@ public function import() {
    +    $this->migration->setLastImported();
    

    There are a couple of error returns in this function, I think we should track the time there as well.

Looking good - I've got a migrate_plus patch using this at #2429107: Display last imported time with drush migrate-status:

$ drush ms
 Group: default  Total  Imported  Unprocessed  Last imported       
 beer_term       3      3         0            2015-04-20 09:45:34 
 beer_user       4      0         4                                
 beer_node       3      0         3                                
 beer_comment    5      0         5
mikeryan’s picture

Status: Needs review » Needs work
nicoloye’s picture

Assigned: Unassigned » nicoloye

Ah, I may have missed it. I'll fix this.

anavarre’s picture

@nicoloye if you could also post an interdiff with new patches, that would be great. Thanks!

nicoloye’s picture

@anavarre, I'll do that, no problem :)

nicoloye’s picture

Status: Needs work » Needs review
StatusFileSize
new3.14 KB
new2.54 KB

Here is a new patch & interdiff.
@mikeryan, for some reason drush ms returns nothing on my local install. I tried importing stuffs through migrate_drupal ui or manifest file but maybe it's not the right way, how can I test this ?

mikeryan’s picture

Did you enable migrate_example? That's what provides the migrations for testing.

Also note that for now you need to add

$databases['migrate'] = $databases['default'];

to your settings.php (pending group support getting committed).

nicoloye’s picture

Status: Needs review » Needs work

OK so the values are wrongly saved.

root@VMDEB7-DEV:/var/www/actency/ddd/2_d8/drupal# drush ms
 Group: default  Total  Imported  Unprocessed  Last imported
 beer_term       3      0         3            1970-01-01 01:00:00
 beer_user       4      0         4            1970-01-01 01:00:00
 beer_node       3      0         3            1970-01-01 01:00:00
 beer_comment    5      0         5            1970-01-01 01:00:00

I can't figure what I've done wrong, I compared these changes with other keyValue entries in the module and I don't see much differences.

mikeryan’s picture

+++ b/core/modules/migrate/src/Entity/Migration.php
@@ -477,6 +477,23 @@ public function setTrackLastImported($track_last_imported) {
+    return $migrate_last_imported_store;

Oops - you're returning the keyvalue store rather than the value returned by get().

nicoloye’s picture

Ouch, how could I miss that ... ^^"

nicoloye’s picture

Status: Needs work » Needs review
StatusFileSize
new3.1 KB
new537 bytes

Here it should be good.

mikeryan’s picture

Status: Needs review » Needs work
+++ b/core/modules/migrate/src/MigrateExecutable.php
@@ -242,6 +242,7 @@ public function import() {
+      $this->migration->setLastImported();

OK, my previous statement was a little broad - since the requirements exception is preventing import from happening at all, I would say we should not set the last imported time here.

With the removal of that line, I'll be ready to declare this rtbc. Thanks!

nicoloye’s picture

Status: Needs work » Needs review
StatusFileSize
new2.75 KB
new537 bytes

Thanks ! I removed the time tracking for requirements exceptions.

Status: Needs review » Needs work

The last submitted patch, 19: migrate-last-imported-time-2429089-19.patch, failed testing.

mikeryan’s picture

Status: Needs work » Reviewed & tested by the community

Status: Reviewed & tested by the community » Needs work

The last submitted patch, 19: migrate-last-imported-time-2429089-19.patch, failed testing.

anavarre’s picture

Status: Needs work » Reviewed & tested by the community

Per #22

xjm’s picture

Priority: Minor » Normal
Status: Reviewed & tested by the community » Needs work
Issue tags: +Needs tests

This seems like a great idea, and the implementation is nice and clean. Needs tests though.

  1. +++ b/core/modules/migrate/src/Entity/Migration.php
    @@ -477,6 +477,22 @@ public function setTrackLastImported($track_last_imported) {
    +    $migrate_last_imported_store = \Drupal::keyValue('migrate_last_imported');
    ...
    +    $migrate_last_imported_store = \Drupal::keyValue('migrate_last_imported');
    

    Minor, but is there a reason that we're not injecting the KV store dependency?

  2. +++ b/core/modules/migrate/src/Entity/Migration.php
    @@ -477,6 +477,22 @@ public function setTrackLastImported($track_last_imported) {
    +  public function setLastImported() {
    ...
    +    $migrate_last_imported_store->set($this->id(), round(microtime(TRUE) * 1000));
    

    Should we add an optional parameter to set a specific time? If we don't want to make that functionality public, we could at least make this protected and add a public wrapper like setLastImportedNow()... or something like that with a better name.

    Also, isn't it normal for most setters to return $this? I could be mistaken.

  3. +++ b/core/modules/migrate/src/Entity/MigrationInterface.php
    @@ -273,6 +273,19 @@ public function isTrackLastImported();
    +   * Get the migration time of last import.
    ...
    +   * Set the migration time of last import.
    

    Very minor: Should be "Gets" and "Sets" per https://www.drupal.org/node/1354#functions.

mikeryan’s picture

Status: Needs work » Postponed

At the moment, the plan is for this to be accomplished by #2535458: Dispatch events at key points during migration.

mikeryan’s picture

Status: Postponed » Closed (duplicate)
Issue tags: -Needs tests

Contrib can do this itself now: https://www.drupal.org/node/2544874