While the Migrate module was originally developed for the purpose of performing complete site migrations, where the source data is always the system of record, it can be used for other scenarios. For example, you might define a migration process for the purpose of updating only specific fields on existing content with new data, or after initially performing a full migration of new content you might want to remigrate the legacy data while preserving some changes you've made on the destination side. To control this behavior, the Migration class supports setting the system of record for the migration.

  /**
   * Indicate whether the primary system of record for this migration is the
   * source, or the destination (Drupal). In the source case, migration of
   * an existing object will completely replace the Drupal object with data from
   * the source side. In the destination case, the existing Drupal object will
   * be loaded, then changes from the source applied; also, rollback will not be
   * supported.
   *
   * @var int
   */
  const SOURCE = 1;
  const DESTINATION = 2;
  protected $systemOfRecord = Migration::SOURCE;

So, by default the system of record is the source data - if you remigrate existing data (e.g., with --update on your drush migrate-import command), the Drupal content is completely replaced by the content from the source. However, in the constructor for your migration you can set

  $this->systemOfRecord = Migration::DESTINATION;

If you do this, then when you run drush migrate-import only the fields that you have mapped in the migration will be replaced on the Drupal side - all other fields will retain their existing values.

So, when you are defining a migration whose purpose is updating existing content, just set systemOfRecord and add field mappings for the specific fields you want to update, leaving the rest unmapped. Of course, there is another way to circumvent changing the entire systemOfRecord and still protecting a few fields. Described in a protecting migration fields on update cookbook example.

If you are updating content from a previous migration you must map the NID to the primary key it relates to and tell Migrate what the initial migration was called so it knows what node to update.

$this->addFieldMapping('nid', 'key')
         ->sourceMigration('Beer');

If you'd like to run an initial migration to create content from an external source, and also be able to run subsequent migrations that update existing content but preserve some Drupal-side changes, you need to define separate Migration classes for the latter purpose with systemOfRecord set to Migration::DESTINATION.

Comments

Prodigy’s picture

Important distinction between System of Record and use of the HighWater field.

System of Record (Destination) is typically used when updating specific fields for previously imported content. Example: You've made changes to a CCK Select list and want to update only that changed field for all of your content, but not erase anything you've previously imported.

HighWater You're performing an incremental migration and you've just been given 200 new articles that weren't present when you initially performed the migration. You'd specify a 'highwater' field (typically a date reflecting time changed, i.e node changed) to only update content that is = or > the value in your 'highwater' field.

drupalmind’s picture

I have data coming from two different sources. I need to run one source at node create and other to be import at node update. migrate provide to map nids with some sources. but i have same Id in both sources. I dont want to map nids with source id but I have mapped source id in node create import in title. now I want to map title(instead of nids) with source id on node update import.
Does some one have ideas?

chrisivens’s picture

If you need to update the information you have added from a migration already with new data from a second migration, you will need to set the system of record as above:

$this->systemOfRecord = Migration::DESTINATION;

Adding to this, you need to define the primary key you are using and the migration it relates to. For instance, if you are updating a node item:

$this->addFieldMapping('nid', 'someKey')
         ->sourceMigration('FirstMigration');

Where FirstMigration is obviously the machine name of the migration you ran to get the data in the first time round.

BTMash’s picture

Hmm, interesting. So you can have one migration piggyback off a secondary migration? That is pretty darn cool! How do rollbacks work in such a scenario?

mototribe’s picture

yeah, very cool! Looks like you can't roll it back, see comments above:

In the destination case, the existing Drupal object will
* be loaded, then changes from the source applied; also, rollback will not be
* supported.

Kind of makes sense. In a roll back you are deleting previously added records. If you would want to roll back updated records you would have to keep track of the previous values. Maybe doable with using revisions ...

mototribe’s picture

actually, in a (systemofrecord=DESTINATION) update migration it will still build a map table and keep track of which records have been updated (or processed). When you do a rollback it just removes those records (or the last x records).

For example, do a UserUpdate migration to set the status to 0 (block a user).
Import the first 10 users to set status to 0.
Now change your script to set status to 1 and roll back and import 5 users.
The last 5 users now get changed to status 1 and the first 5 are still 0.

Paul Lomax’s picture

During user migrations using this method the user name gets prefixed with _2 as it is updated. Is there a way of avoiding this?

svilleintel’s picture

I had the exact same problem and it comes down to your dedupe() method if you trace inside it.
It happens only on the first time the Update class is run.

This is the code ive used to solve the problem. It manually pushes in the migrate_map_destid1 value from the original migration.
Add this code to your Updates class.

  function prepareRow($row) {
    parent::prepareRow($row);
    
    if (!isset($row->migrate_map_destid1)) {
      /* The mappings, importantly destid1, from ContactImport table are added
       * to the row to ensure correct running of the dedupe() function.
       * The dedupe() function will not operate properly if 
       * migrate_map_destid1 is not set, and this can be the case for the first 
       * time the update is running.
       */
      $query = db_select('migrate_map_contactimport', 'map', array('fetch' => PDO::FETCH_ORI_FIRST));
      $query->addField('map', 'destid1');
      $query->condition('sourceid1', $row->contact_id);
      $res = $query->execute();
      if (($db_row = $res->fetchAssoc()) != NULL) {
        // Manually push in the value
        $row->migrate_map_destid1 = $db_row['destid1'];
      } 
    }
  }
kulldox’s picture

Hi,

Can please someone tell me what I'm doing wrong?

So, I've done the commerce_product migration from a CSV file (which is as supplier pricelist), lets call it CSVmig1. It goes in successfully.

This pricelist is updated each 3 days. What I want to do is update just the field_supplier_price on the product.

So I'm doing like this:

<?php
/*
* This class is intended to update existing products
*/

// the input file shuld be a .tsv (TAB Serparated Values)

abstract class MyPriceUpdateMigration extends CommerceKickstartMigration {
  public $migModuleName = 'migrate_MyCSV';
  public $migMachineName = 'product';

  public function __construct() {
    parent::__construct(MigrateGroup::getInstance('fox', array('default')));
    $this->description = t('MyCSV Price Update Migration');
  }
}

class MyCSVPriceUpdateMigration extends MyPriceUpdateMigration {
  public function __construct() {
    parent::__construct();
    // $this->dependencies = array('MyCSVPricelist');

    $this->importFile = 'MyCSV_price_06.11.2012.tsv';
    $this->importDir = drupal_get_path('module', $this->migModuleName).'/../../../../import';
    $this->importProductVocabularyID = taxonomy_vocabulary_machine_name_load('product_category')->vid;
    $this->importSupplier = array_values(taxonomy_get_term_by_name('Supplier1'));
    //The Description of the import. This desription is shown on the Migrate GUI
    $this->description = t('Updateing MyCSV Pricelist ');
    // $this->ProductTypeMap = new MyCSVProductTypeMappings();

    //Source and destination relation for rollbacks
    $this->map = new MigrateSQLMap(
      $this->machineName, 
      array(
          'product_id_csv' => array(
          'type' => 'int',
          'unsigned' => TRUE,
          'not null' => TRUE,
          'alias' => 'update',
        ),
     ),
      // MigrateDestinationNode::getKeySchema()
      MigrateDestinationEntityAPI::getKeySchema('commerce_product', $this->migMachineName)
    );


    //The Source of the import
    $this->source = new MigrateSourceCSV(
      $this->importDir.'/'.$this->importFile, 
      $this->csvcolumns(), 
      array(
        'header_rows' => 3,
        'delimiter' => "\t"
      )
    );
    
    //The destination TBU
    $this->destination = new MigrateDestinationEntityAPI('commerce_product', $this->migMachineName);

    $this->systemOfRecord = Migration::DESTINATION;

    //Field mapping
    $this->addFieldMapping('nid', 'product_id_csv')->sourceMigration('MyCSVPricelist');
    $this->addFieldMapping('field_supplier_price', '6000_csv');
  }

  function csvcolumns() {
    //The defintion of the collumns. Keys are integers. values are array(field name, description).
    $columns[0] = array('group_csv', 'Group');
    $columns[1] = array('sub_group_csv', 'Sub Group');
    $columns[2] = array('manufacturer_csv', 'Manufacturer');
    $columns[3] = array('model_csv', 'Model');
    $columns[4] = array('product_id_csv', 'Product ID');
    $columns[5] = array('quantity_csv', 'Qty');
    $columns[6] = array('promotion_csv', 'Promotion Price');
    $columns[7] = array('15000_csv', '15000 $');
    $columns[8] = array('10000_csv', '10000 $');
    $columns[9] = array('8000_csv', '8000 $');
    $columns[10] = array('7000_csv', '7000 $');
    $columns[11] = array('6000_csv', '6000 $');
    $columns[12] = array('order_csv', 'Order');
    $columns[13] = array('description_csv', 'Description');

    return $columns;
  }
}

?>

But I allwas get "System-of-record is DESTINATION, but no destination id provided". I've tried various combinations.

Thanks in advance.

bubule22’s picture

Hi!
I've the same problem with this message "System-of-record is DESTINATION, but no destination id provided"
Have you find the correct mapping to update 'commerce_product' ?

Thanks in advance.

fonant’s picture

The correct mapping, for a previously migrated set of products using commerce_migrate_ubercart, would seem to be like:

    $this->addFieldMapping('product_id', 'nid')->sourceMigration('CommerceMigrateUbercartProductMyproduct');

but this doesn't work. For some reason the destination product_id ends up being an array of 'destid1' (the product_id) and 'destid2' (the product_revision) instead of just the single destination product_id value.

Help requested here: #2201565: Can't create a separate migration using mappings generated by this module

http://www.fonant.com - Fonant Ltd - Quality websites

pounard’s picture

Got the same problem here, migrate_extra entity API implementation will refuse to load the entity if $row->migrate_map_destid1 is not set, but migrate source will always give a NULL value for this field since the new migration class has its own mapping instance: at the very first run, the mapping table is empty, therefore there is no way to have a migrate_map_destid1.

This is actually a bug of the migrate_extras module. Migrate core should probably handle this algorithm by itself and leave a lesser but simpler destination interface to implement: due to the current architecture, every destination class must copy/paste this lookup algorithm, I'm not surprised this happens.

leex’s picture

I have a situation where I'm updating users who were previously migrated. I made a copy of the migration and made it into a destination like so:

Added:

$this->systemOfRecord = Migration::DESTINATION;

Changed mapping field from 'name' to 'uid' (where previously the unique_id was used to create the username):

$this->addFieldMapping('uid', 'unique_id')
         ->sourceMigration('AccountMigration');

Make sure to remove any unwanted mappings!

screon’s picture

Hi, I tried it as well in my migrate class but it throws a warning in drush:

addFieldMapping: custom_id was previously mapped from   [warning]
other_id, overridden

Is this normal? I don't really understand what ID's to use for the mapping...

ohthehugemanatee’s picture

Worth noting that if you set $this->systemOfRecord = Migration::DESTINATION; it will ONLY be able to process records that have an extant node.

If you want to perform a migration where SOME nodes are new, and SOME are just updates, I think you have to create stubs for entities that don't exist yet.

You never REALLY learn to swear until you own a computer.

rferguson’s picture

What I'm trying to do sounds like this. I have an old website that I'm bringing content over from. The problem is I want the client to be able to go into their new drupal site and start mapping nodes to menu links. So far this has been problematic for me as the menu links disappear when I re-run drush mi. I've tried making the systemOfRecord=DESTINATION once imported but that would only handle updates. I've also been meaning to try this solution https://www.drupal.org/node/983290 using the prepare method but I'm wondering if it's the best. My basic requirements are being able to import/update nodes and preserve any links created for those nodes (the idea being that their old website can live on right up until launch with new content continuously being imported while they build out the information architecture on the new site).

milos.kroulik’s picture

Were you able to accomplish your goal? I also have the same need.

jonaskills’s picture

If you are updating content from a previous migration you must map the NID to the primary key it relates to and tell Migrate what the initial migration was called so it knows what node to update

What if I want to update content that was not added from a migration?

e.g. I am trying to update stock values for commerce_product entities. There was no initial migration that imported the products so I can't define a sourceMigration to map keys.

I just want to update fields on the commerce_product using the unique sku.

Just setting

$this->systemOfRecord = Migration::DESTINATION;

in my constructor returns "System-of-record is DESTINATION, but no destination id provided" when trying to run the migration.

Thanks!
-TOm

osopolar’s picture

Did you figured out a way? Maybe prepareRow() could be used instead of sourceMigration() to map the source to existing entities?

Edit: Answering my question: I set $row->migrate_map_destid1 in prepareRow(), that works fine.

lookatthosemoose’s picture

Thanks for the tip osopolar.
In my scenario, I did not have UID in the source file to directly reference like $row->uid.

But, I was able to use EntityFieldQuery to query for the user's UID using another unique field and then set $row->migrate_map_destid1

$query = new EntityFieldQuery();
$user_result = $query->entityCondition('entity_type', 'user')->fieldCondition('field_accountid', 'value', $row->AccountID, '=')->execute();
if (isset($user_result['user']) && count($user_result['user']) == 1) {
      $user_uids = array_keys($user_result['user']);
      $uid = array_pop($user_uids);
      $row->migrate_map_destid1 = $uid;
}
nyariv’s picture

How would you implement this in the Drupal 8 version?

nyariv’s picture

I figured it out. Drupal 8 migrate depends on drush, and when using the --update flag in the command it automatically changes the systemOfRecord.

firfin’s picture

Subject says it all no?
Here's a link https://www.drupal.org/node/2808425

ranavaibhav’s picture

Cant seem to figure out how to go about updating existing content (not created/imported through Migrate). Any sample code to share? ver 8.5

Jackie R

killwaffles’s picture

Is it possible to set the system of record in the UI?

Stephen Ollman’s picture

Confirmed that with Drupal 8, using the command 'drush mi --update {group}' does indeed update the record instead of adding a new one without any reference to the systemofrecord

Certified Drupal Site Builder 7 & 8