Problem/Motivation

The current status (STATUS_IDLE, STATUS_IMPORTING, STATUS_ROLLING_BACK, STATUS_STOPPING, STATUS_DISABLED) of a migration is not currently being tracked. The status in D7 served two purposes:

  1. As a semaphore to prevent multiple operations from being invoked on a given migration at the same time.
  2. To provide status information to the user (through the migrate_ui dashboard and drush migrate-status command).

Proposed resolution

Maintain the current status of each migration via State API (default in the absence of this state is STATUS_IDLE). STATUS_IMPORTING and STATUS_ROLLING_BACK should be set at the start of operations and reset to STATUS_IDLE upon completion. No import/rollback operation should be permitted to begin when the status is anything other than STATUS_IDLE. A running import/rollback task should periodically check the status - if it is STATUS_STOPPING, it should cleanly exit and set the status back to STATUS_IDLE.

Remaining tasks

Implement the functionality and tests.

User interface changes

N/A for core (Migrate Plus will make use of it in contrib).

API changes

Add getMigrationStatus()/setMigrationStatus() to MigrationInterface and Migration.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

mikeryan’s picture

Looking at this, I see setMigrationResult/getMigrationResult is using the keyvalue store rather than state. Why? It's not clear to me what the tradeoffs would be for using keyvalue rather than state...

mikeryan’s picture

Status: Active » Needs review
FileSize
6.18 KB

Here it is - using state until/unless someone explains why keyvalue would be better. See #2429105: Display migration status (importing, rolling back) with drush migrate-status for a migrate_plus patch using this.

Status: Needs review » Needs work

The last submitted patch, 2: track_current_state_of-2429085-2.patch, failed testing.

Status: Needs work » Needs review
mikeryan’s picture

FileSize
6.18 KB

Oops, a little debug code snuck in...

Status: Needs review » Needs work

The last submitted patch, 5: track_current_state_of-2429085-5.patch, failed testing.

mikeryan’s picture

OK, digging in locally, I see this will fail too, first failure:

1) Drupal\Tests\migrate\Unit\MigrateExecutableTest::testImportWithValidRow
Failed asserting that null is identical to 1.

What's happening is that in the unit case context, this breaks out of the loop:

    $this->migration->setMigrationStatus(MigrationInterface::STATUS_IMPORTING);
    ...
      // Primarily looking for STATUS_STOPPING here, but we should quit on any
      // unexpected status.
      if ($this->migration->getMigrationStatus() != MigrationInterface::STATUS_IMPORTING) {
        break;
      }

This works when run for real in migrate_plus, seems like the state isn't properly being stored/retrieved - is there anything special needed in terms of mocking when dealing with state?

The last submitted patch, 2: track_current_state_of-2429085-2.patch, failed testing.

benjy’s picture

  1. +++ b/core/modules/migrate/src/Entity/Migration.php
    @@ -372,6 +372,24 @@ protected function getEntityManager() {
    +  public function setMigrationStatus($status) {
    +    \Drupal::state()->set('migrate.' . $this->id() . '.status', $status);
    +  }
    +
    +  /**
    +   * {@inheritdoc}
    +   */
    +  public function getMigrationStatus() {
    +    $status = \Drupal::state()->get('migrate.' . $this->id() . '.status');
    +    if (is_null($status)) {
    +      $status = MigrationInterface::STATUS_IDLE;
    +    }
    +    return $status;
    

    State is just a wrapper around the key value store. I think we decided against it so we could a few things:

    * Allows us to retrieve or delete multiple entries at the same time.
    * Not have to build the unique ids for each storage like you have here.
    * Not have to check is_null on that result since keyValue->get() allows you to specify a default.

  2. +++ b/core/modules/migrate/src/MigrateExecutable.php
    @@ -237,6 +237,15 @@ protected function getSource() {
    +    if ($this->migration->getMigrationStatus() != MigrationInterface::STATUS_IDLE) {
    

    Should we have a strict check here since STATUS_IDLE is 0 it seems easy to have false positives.

  3. +++ b/core/modules/migrate/src/MigrateExecutable.php
    @@ -261,11 +271,17 @@ public function import() {
    +      if ($this->migration->getMigrationStatus() != MigrationInterface::STATUS_IMPORTING) {
    

    Same here.

  4. +++ b/core/modules/migrate/src/MigrateExecutable.php
    @@ -237,6 +237,15 @@ protected function getSource() {
    +      $this->message->display(
    +        $this->t('Migration @id is busy with another operation', array(
    

    These should be on the same line, just the array values returned to a newline.

mikeryan’s picture

Not have to build the unique ids for each storage like you have here.

We need to have unique IDs, though, each migration has a distinct status at any given time. And, looking at the results, I believe they also need to use unique IDs - right now it's just a global migrate_result key, which presents the possibility of race conditions if running multiple migrations in parallel. As for checking is_null, only getMigrationStatus() has to deal with that, no biggie.

Should we have a strict check here since STATUS_IDLE is 0 it seems easy to have false positives

Can't hurt!

These should be on the same line, just the array values returned to a newline.

Will do!

Back to the test failures - any insights? I suspect maybe we need to somehow mock the state? (/me feels like PhpUnit is mocking him...)

benjy’s picture

Well using the KeyValue store we also use a unique Id, apart from it's only unique within the migration_result kv store.

I'll take a look at that test later today.

benjy’s picture

Status: Needs work » Needs review
FileSize
1.47 KB
7.64 KB

Not pretty but something like the attached should do it.

mikeryan’s picture

FileSize
7.89 KB
2.8 KB

OK, here I've:

  1. Switched to using the keyvalue store directly
  2. Applied @benjy's other suggestions
  3. Initialized the status in @benjy's test change to STATUS_IDLE

Thanks!

benjy’s picture

+++ b/core/modules/migrate/src/MigrateExecutable.php
@@ -238,9 +238,8 @@
+    if ($this->migration->getMigrationStatus() !== MigrationInterface::STATUS_IDLE) {
+      $this->message->display($this->t('Migration @id is busy with another operation', array(
           '@id' => $this->migration->id(),

Would it be worth outputting the migration status it is busy with?

mikeryan’s picture

FileSize
10.07 KB
3.36 KB

OK - introducing a function to get translated strings for the possible statuses, this will also be useful to contrib tools (UI and drush).

benjy’s picture

+++ b/core/modules/migrate/src/MigrateExecutable.php
@@ -238,9 +239,11 @@
+          '@status' => Migration::getTranslatedStatus($status),

I'm not entirely sure about translations but I don't quite understand why this couldn't be something like $this->migration->getStatusLabel().

Then getStatusLabel() could simply be return $this->t($this->getMigrationStatus());

I think i'm missing why it needs to be a static method and available at any time?

mikeryan’s picture

FileSize
9.71 KB
3.27 KB

I had poked around core looking for best practices for translating integer codes into translated text and borrowed it from RfcLogLevel. This is simpler though...

benjy’s picture

  1. +++ b/core/modules/migrate/src/Entity/Migration.php
    @@ -15,6 +15,7 @@
    +use Drupal\Core\StringTranslation\TranslationWrapper;
    

    No longer needed.

  2. +++ b/core/modules/migrate/src/MigrateExecutable.php
    @@ -237,6 +238,16 @@ protected function getSource() {
    +          '@status' => $this->t($this->migration->getStatusLabel()),
    

    @status is already used within a $this->t() so I don't know we need to run the status through t() on it's own.

    Sorry, I may have implied that in my last comment.

mikeryan’s picture

FileSize
9.46 KB
465 bytes

@status is already used within a $this->t() so I don't know we need to run the status through t() on it's own.

The string passed to $this->t() is 'Migration @id is busy with another operation: @status', which will produce something like 'Migration article_node is busy with another operation: Importing'. Translated to Spanish that would end up as something like 'article_node migración está ocupado con otra operación: Importing' if we don't translate the label going in, would it not?

chx’s picture

While it is correct that nothing in this code translates the status label and so getStatusLabel should be $this->t('Stopping') but why do we have this function on the entity???? I'd expect the entity to have a statusLabel field and this is the getter for it. Weird! Why is this not on MigrateExecutable?

benjy’s picture

+++ b/core/modules/migrate/src/Entity/Migration.php
@@ -214,6 +214,26 @@ class Migration extends ConfigEntityBase implements MigrationInterface, Requirem
+  public function getStatusLabel() {

@@ -372,6 +392,22 @@ protected function getEntityManager() {
+  public function setMigrationStatus($status) {

I think the naming of getStatusLabel() and getMigrationStatus()? should be consistent.

@chx, are you suggesting the getStatusLabel method be on the executable?

mikeryan’s picture

@chx: We're managing the statuses themselves in the entity, I'm not clear on the advantage of moving the labels to the executable?

Some refactoring in this patch:

  1. Removed "Migration" from the APIs.
  2. Moved the label mappings to a class property, making it easier to add more statuses.
  3. Renamed the existing checkStatus() method to avoid confusion.

Status: Needs review » Needs work

The last submitted patch, 22: track_current_state_of-2429085-22.patch, failed testing.

mikeryan’s picture

Status: Needs work » Needs review
FileSize
10.39 KB
8.44 KB

Oops.

mikeryan’s picture

FileSize
827 bytes

I keep running one interdiff behind...

benjy’s picture

  1. +++ b/core/modules/migrate/src/MigrateExecutable.php
    @@ -468,33 +488,15 @@ public function saveQueuedMessages() {
    -    /*
    -     * @TODO uncomment this
    -    if ($this->getStatus() == MigrationInterface::STATUS_STOPPING) {
    -      return MigrationBase::RESULT_STOPPED;
    -    }
    -    */
    -    // If feedback is requested, produce a progress message at the proper time
    -    /*
    -     * @TODO uncomment this
    -    if (isset($this->feedback)) {
    -      if (($this->feedback_unit == 'seconds' && time() - $this->lastfeedback >= $this->feedback) ||
    -          ($this->feedback_unit == 'items' && $this->processed_since_feedback >= $this->feedback)) {
    -        $this->progressMessage(MigrationInterface::RESULT_INCOMPLETE);
    -      }
    -    }
    -    */
    

    So we don't need this anymore? Is there an issue to bring it back somewhere?

  2. +++ b/core/modules/migrate/src/Tests/MigrateStatusTest.php
    @@ -0,0 +1,62 @@
    +    $migration = entity_create('migration', array());
    +    $migration->set('id', 'migration_status_test');
    +    $migration->set('migration_groups', array('Testing'));
    +    $migration->set('source', array('plugin' => 'empty'));
    +    $migration->set('destination', array(
    +      'plugin' => 'config',
    +      'config_name' => 'migrate_test.settings',
    +    ));
    

    Not to worried, but did you know you can pass all these values to the second param of entity_create()

I think this is ready other than my question in point 1. Nice work.

mikeryan’s picture

The first removed @todo comment is being handled directly in the loop in import() ("Primarily looking for STATUS_STOPPING..."). As for the second, I haven't opened a core issue yet for it but the migrate_plus issue is #2432985: Implement --feedback option on migrate-import - I'm not sure exactly how we'll handle feedback going forward but I'm fairly certain generating progress messages in MigrateExecutable is not the way.

Not to worried, but did you know you can pass all these values to the second param of entity_create()

Yes... Just in the mindset of treating the migration as a config entity, it feels natural to me to set configuration values this way, but if using the $values array is a preferred convention I'm fine with that.

benjy’s picture

Status: Needs review » Reviewed & tested by the community

I don't think it's preferred I just wondered if you was aware. Everything else looks good.

alexpott’s picture

Status: Reviewed & tested by the community » Needs work
  1. +++ b/core/modules/migrate/src/MigrateExecutable.php
    @@ -468,33 +488,15 @@ public function saveQueuedMessages() {
       /**
    -   * Checks for exceptional conditions, and display feedback.
    -   *
    -   * Standard top-of-loop stuff, common between rollback and import.
    +   * Checks for exceeded resource thresholds.
        */
    -  protected function checkStatus() {
    +  protected function checkThresholds() {
    

    This should detail its return value.

  2. +++ b/core/modules/migrate/src/MigrateExecutable.php
    @@ -468,33 +488,15 @@ public function saveQueuedMessages() {
         return MigrationInterface::RESULT_COMPLETED;
    

    Does returning this from a method called checkThresholds() make sense?

mikeryan’s picture

This historically (in D7) was checking various possible conditions that might lead to exiting the main loop in import(), kind of a grab-bag of stuff, and returned the RESULT constant that import() would propagate up. Now all it's doing is checking memory and time thresholds - I went in there to remove the comments that have been obsoleted by this work, and went out-of-scope by renaming it. Looking more carefully, regardless of what its name should be, I think it should be returning a boolean and letting import() set the appropriate RESULT - but, that's out of scope for this patch, it should be dealt with separately. Or, we've come this far, I can just toss that boolean return (and its interpretation in import() into this one...

mikeryan’s picture

Currently considering whether to implement some or all of this in contrib using events (#2535458: Dispatch events at key points during migration) - the one bit I really think needs to be in the core the semaphore aspect.

mikeryan’s picture

Rerolled, in light of #2545672: Handle various migration interruption scenarios. Whichever of these two gets in first, the other should add setting of the STOPPING status when setting the interruption.

mikeryan’s picture

Priority: Normal » Major
Issue tags: +Migrate critical, +blocker
phenaproxima’s picture

+++ b/core/modules/migrate/src/Entity/Migration.php
   /**
+   * Labels corresponding to each defined status.
+   *
+   * @var array
+   */
+  protected $statusLabels = [
+    self::STATUS_IDLE => 'Idle',
+    self::STATUS_IMPORTING => 'Importing',
+    self::STATUS_ROLLING_BACK => 'Rolling back',
+    self::STATUS_STOPPING => 'Stopping',
+    self::STATUS_DISABLED => 'Disabled',
+  ];

Ideally these should be translated. Perhaps getStatusLabel() should do that?

+++ b/core/modules/migrate/src/Entity/Migration.php
+  public function setStatus($status) {
+    $migrate_status_store = \Drupal::keyValue('migrate_status');
+    $migrate_status_store->set($this->id(), $status);
+  }

Nit: can this be one line? \Drupal::keyValue()->set()

+++ b/core/modules/migrate/src/Entity/Migration.php
+  /**
+   * {@inheritdoc}
+   */
+  public function getStatus() {
+    $migrate_status_store = \Drupal::keyValue('migrate_status');
+    return $migrate_status_store->get($this->id(), static::STATUS_IDLE);
+  }

Ditto.

+++ b/core/modules/migrate/src/Entity/Migration.php
+  /**
+   * {@inheritdoc}
+   */
+  public function getStatusLabel() {
+    if (isset($this->statusLabels[$this->getStatus()])) {
+      return $this->statusLabels[$this->getStatus()];
+    }
+    else {
+      return '';
+    }
+  }

Micro-optimization -- let's call getStatus() once here to save the overhead.

+++ b/core/modules/migrate/src/Entity/Migration.php
     $migrate_result_store = \Drupal::keyValue('migrate_result');
     $migrate_result_store->set($this->id(), $result);

Can this also be converted to one line?

+++ b/core/modules/migrate/src/Tests/MigrateStatusTest.php
+    $status_list = array(
+      MigrationInterface::STATUS_IDLE,
+      MigrationInterface::STATUS_IMPORTING,
+      MigrationInterface::STATUS_ROLLING_BACK,
+      MigrationInterface::STATUS_STOPPING,
+      MigrationInterface::STATUS_DISABLED,
+    );
+    foreach ($status_list as $status) {
+      $migration->setStatus($status);
+      $this->assertIdentical($migration->getStatus(), $status);
+    }

Why bother testing every individual status? Surely it's enough to just test one and be done with it?

+++ b/core/modules/migrate/tests/src/Unit/MigrateTestCase.php
+    $migration->expects($this->any())
+      ->method('getStatus')
+      ->will($this->returnCallback(function() {
+        return $this->migrationStatus;
+      }));

For readability, can we use $migration->method('getStatus')->willReturnCallback(function() { ... }) instead?

+++ b/core/modules/migrate/tests/src/Unit/MigrateTestCase.php
   /**
+   * @var \Drupal\migrate\Entity\MigrationInterface::STATUS_*
+   */
+  protected $migrationStatus = MigrationInterface::STATUS_IDLE;

It's not clear how this is to be used in the context of the test. Can you add a bit of documentation here explaining it?

Beyond these minor things, this seems good to go.

mikeryan’s picture

Ideally these should be translated. Perhaps getStatusLabel() should do that?

They are translated when used, e.g.

+          '@status' => $this->t($this->migration->getStatusLabel()),

There was previous discussion of this above - when/where should the status label get translated? I think we have it right here, at the point where it is being used in a translatable message, so we're sure it has the same translation context as the message containing it.

The rest of the feedback is addressed in the attached patch.

phenaproxima’s picture

Status: Needs review » Reviewed & tested by the community

Works for me. Thank you for accommodating my OCD-ness.

webchick’s picture

Status: Reviewed & tested by the community » Fixed

Committed and pushed to 8.0.x. Thanks!

  • webchick committed 004d26c on 8.0.x
    Issue #2429085 by mikeryan, benjy: Track current state of migrations
    

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.