I have a migration that requires me to migrate data from one database {legacy} to another {new}. Both are on the same server, both are defined in settings.php as part of the db array. I have this arrangement on my local machine and on a remote server. Strangely, this migration works locally, but not on the remote machine...

The crux of the problem is that it seems to be looking for the migrate_status table in the legacy database, when it should be looking for it in the new database! I can't image why it would look in the wrong place in one environment and not another.

Details below.

The migration class (simplified).

class legacyServiceProviderMigration extends Migration {
  public function __construct() {
    parent::__construct();

    $this->map = new MigrateSQLMap($this->machineName,
      array(
        'pid' => array(
          'type' => 'int',
          'unsigned' => FALSE,
          'not null' => TRUE,
          'description' => 'Persistent Record ID',
          'alias' => 'b',
        )
      ),
      MigrateDestinationNode::getKeySchema()
    );

    // We are getting data from tables in the Drupal default database - first,
    // set up a query for this data.
    $fields = array(
      'pid',
    );
    $query = Database::getConnection('default', 'legacy')
      ->select('baserecords', 'b')
      ->condition('latitude', '', '<>')
      ->fields('b', $fields);
      
    // Create a MigrateSource object, which manages retrieving the input data.
    $source_fields = array(
      'pid' => t('Persistent Record ID'),
    );

    $this->source = new MigrateSourceSQL($query, $source_fields, NULL, array('map_joinable' => FALSE));
    $this->destination = new MigrateDestinationNode('service_provider');

    // Assign mappings TO destination fields FROM source fields.
    
    // Core node fields.
    $this->addFieldMapping('is_new')->defaultValue(TRUE);
    $this->addFieldMapping('title', 'businessname');
}

The db array (usernames and pws removed)

$databases['default']['default'] = array(
  'database' => 'new',
  'host' => 'localhost',
  'port' => '',
  'driver' => 'mysql',
  'prefix' => '',
);

$databases['legacy']['default'] = array(
  'database' => 'legacy',
  'host' => 'localhost',
  'port' => '',
  'driver' => 'mysql',
  'prefix' => '',
);

I also use a function in the custom migration module that serves to map categories:

/**
 * Get the categories associated with a company.
 */
function legacy_migration_get_company_cats($pid) {
  db_set_active('legacy');
  $query = db_select('companyheadings', 'ch');
  $query->fields('ch', array('normalizedid'));
  $query->rightJoin('categories', 'cat', 'ch.normalizedid = cat.categoryid');
  $query->condition('pid', $pid);
  $result = $query->execute()->fetchAll();
  
  $categories = array();
  foreach ($result as $key => $entry) {
    foreach ($entry as $cat_type => $cat) {
      $categories[] = $cat;
    }
  }
  $categories = array_unique($categories);
  db_set_active();
  return count($categories) ? legacy_migration_get_local_tids($categories) : NULL;
}

The error from drush (addresses obfuscated):

Migration failed with source plugin exception: SQLSTATE[42000]: Syntax error or access violation: 1142 SELECT command denied to user           [error]
'new'@'address.obfuscated.com' for table 'categories'
<em class="placeholder">PDOException</em>: SQLSTATE[42S02]: Base table or view not found: 1146 Table                                           [error]
&#039;legacy.migrate_status&#039; doesn&#039;t exist: SELECT 1 AS expression
FROM 
{migrate_status} migrate_status
WHERE ( (machine_name = :db_condition_placeholder_0) ) FOR UPDATE; Array
(
    [:db_condition_placeholder_0] =&gt; legacyServiceProvider
)
 in <em class="placeholder">MigrationBase-&gt;endProcess()</em> (line <em class="placeholder">761</em> of <em
class="placeholder">/mnt/www/html/new/docroot/sites/all/modules/migrate/includes/base.inc</em>).
WD php: PDOException: SQLSTATE[42S02]: Base table or view not found: 1146 Table 'legacy.migrate_status' doesn't exist: SELECT 1 AS     [error]
expression
FROM 
{migrate_status} migrate_status
WHERE ( (machine_name = :db_condition_placeholder_0) ) FOR UPDATE; Array
(
    [:db_condition_placeholder_0] => legacyServiceProvider
)
 in MigrationBase->endProcess() (line 761 of /mnt/www/html/new/docroot/sites/all/modules/migrate/includes/base.inc).
PDOException: SQLSTATE[42S02]: Base table or view not found: 1146 Table &#039;legacy.migrate_status&#039; doesn&#039;t exist: SELECT 1 AS expression
FROM 
{migrate_status} migrate_status
WHERE ( (machine_name = :db_condition_placeholder_0) ) FOR UPDATE; Array
(
    [:db_condition_placeholder_0] => legacyServiceProvider
)
 in MigrationBase->endProcess() (line 761 of /mnt/www/html/new/docroot/sites/all/modules/migrate/includes/base.inc).
Drush command terminated abnormally due to an unrecoverable error.                                                                             [error]
<h1>Uncaught exception thrown in shutdown function.</h1><p>PDOException: SQLSTATE[42S02]: Base table or view not found: 1146 Table &amp;#039;legacy.semaphore&amp;#039; doesn&amp;#039;t exist: DELETE FROM {semaphore} 
WHERE  (value = :db_condition_placeholder_0) ; Array
(
    [:db_condition_placeholder_0] =&amp;gt; 8628237934ffce03557c278.40479574
)
 in lock_release_all() (line 269 of /mnt/www/html/new/docroot/includes/lock.inc).</p><hr />
Fatal error: Exception thrown without a stack frame in Unknown on line 0

Any insight would be appreciated! Thank you.

Comments

grasmash’s picture

Issue summary: View changes

left out the important function!

albertczyk’s picture

I have some related issue, but with map tables. In theory, when you define the MigrateSQLMap, you can specify the database where map tables will be created. When using the 'default' (local / target) database, the map tables are created in local, but when performing the migration, Migrate looks for them in the remote / source DB, thus getting a SQL error:

SQLSTATE[42000]: Syntax error or access violation: 1142 SELECT command denied to user 'user'@'62.42.x.x' for table 'migrate_map_noticia'

If I define the MigrateSQLMap in the remote DB like

    // We instantiate the MigrateMap
    $this->map = new MigrateSQLMap($this->machineName,
        array(
          'nid' => array('type' => 'int',
                          'unsigned' => TRUE,
                          'not null' => TRUE,
                          'description' => 'D6 Unique node ID',
                          'alias' => 'n'
                         )
        ),
        MigrateDestinationNode::getKeySchema(),
        'sourcce_db'
      );

then the tables are created in the source DB, and the error doesn't appear, but performance is very low (well, in fact I wasn't able to run the migration with map tables in local, so I cannot compare, but I figure out that those tables are being queried/updated intensively and that's not a good thing to do with a remote server).

If at least the error message indicated the line where it happens, the problem would be easier to pinpoint...

mikeryan’s picture

Status: Active » Postponed (maintainer needs more info)

@madmatter23 - I would bet that using db_set_active() is your problem. You should not be using db_set_active() to query the legacy database, use Database::getConnection('default', 'legacy')->select() in place of db_select().

@albertczyk - Your case is different, please open distinct issues rather than tack on to slightly-related ones. That being said, setting the option map_joinable => FALSE in the MigrateSourceSQL constructor will ensure that the map table is accessed separately from the source query, from the default database.

grasmash’s picture

Status: Postponed (maintainer needs more info) » Closed (works as designed)

You were right! db_set_active() is no good for this. Thanks!

Pedro Lozano’s picture

Status: Closed (works as designed) » Active

The problem here, at least in my case, is that the source database tables are prefixed.

When migrate tries to execute this source query:

SELECT n.nid AS nid, n.title AS title, n.status AS status, n.created AS created, n.changed AS changed, n.promote AS promote, n.sticky AS sticky, n.uid AS uid, nr.body AS body, nr.teaser AS teaser, ctv.field_video_embed AS field_video_embed, ctv.field_video_info_value AS field_video_info_value, map.sourceid1 AS migrate_map_sourceid1, map.destid1 AS migrate_map_destid1, map.needs_update AS migrate_map_needs_update, GROUP_CONCAT(DISTINCT(td1.name)) AS terms_tags, GROUP_CONCAT(DISTINCT(td2.name)) AS terms_video_categories
FROM 
{node} n
INNER JOIN {node_revisions} nr ON nr.vid = n.vid
LEFT OUTER JOIN {term_node} tn1 ON tn1.nid = n.nid
LEFT OUTER JOIN {term_data} td1 ON td1.tid = tn1.tid AND td1.vid = 2
LEFT OUTER JOIN {term_node} tn2 ON tn2.nid = n.nid
LEFT OUTER JOIN {term_data} td2 ON td2.tid = tn2.tid AND td2.vid = 6
LEFT OUTER JOIN {content_type_video} ctv ON ctv.vid = n.vid
LEFT OUTER JOIN {sourcedb.migrate_map_videonode} map ON n.nid = map.sourceid1
WHERE  (n.type = 'video') AND (n.status = '1') AND (n.nid = '98525') AND( (map.sourceid1 IS NULL ) OR (map.needs_update = '1') )
GROUP BY n.nid
ORDER BY n.nid ASC

Note that the map table (sourcedb.migrate_map_videonode) is also between {} and it already has the database name appended.

When the database layers adds the prefixes we end up with the following query:

SELECT n.nid AS nid, n.title AS title, n.status AS status, n.created AS created, n.changed AS changed, n.promote AS promote, n.sticky AS sticky, n.uid AS uid, nr.body AS body, nr.teaser AS teaser, ctv.field_video_embed AS field_video_embed, ctv.field_video_info_value AS field_video_info_value, map.sourceid1 AS migrate_map_sourceid1, map.destid1 AS migrate_map_destid1, map.needs_update AS migrate_map_needs_update, GROUP_CONCAT(DISTINCT(td1.name)) AS terms_tags, GROUP_CONCAT(DISTINCT(td2.name)) AS terms_video_categories
FROM 
vb_drupal_node n
INNER JOIN vb_drupal_node_revisions nr ON nr.vid = n.vid
LEFT OUTER JOIN vb_drupal_term_node tn1 ON tn1.nid = n.nid
LEFT OUTER JOIN vb_drupal_term_data td1 ON td1.tid = tn1.tid AND td1.vid = 2
LEFT OUTER JOIN vb_drupal_term_node tn2 ON tn2.nid = n.nid
LEFT OUTER JOIN vb_drupal_term_data td2 ON td2.tid = tn2.tid AND td2.vid = 6
LEFT OUTER JOIN vb_drupal_content_type_video ctv ON ctv.vid = n.vid
LEFT OUTER JOIN vb_drupal_sourcedb.migrate_map_videonode map ON n.nid = map.sourceid1
WHERE  (n.type = :db_condition_placeholder_0) AND (n.status = :db_condition_placeholder_1) AND (n.nid = :db_condition_placeholder_2) AND( (map.sourceid1 IS NULL ) OR (map.needs_update = :db_condition_placeholder_3) )
GROUP BY n.nid
ORDER BY n.nid ASC

Which is obviously incorrect since the prefix has been appended to the database name for the map table.

My settings.php connections array looks something like this:

$databases = array (
  'default' => 
  array (
    'default' => 
    array (
      'database' => 'maindb',
      'username' => 'maindb',
      'password' => 'maindb',
      'host' => 'localhost',
      'port' => '',
      'driver' => 'mysql',
      'prefix' => '',
    ),
  ),
  'sourcedb' => 
  array (
    'default' => 
    array (
      'database' => 'sourcedb',
      'username' => 'sourcedb',
      'password' => 'sourcedb',
      'host' => 'localhost',
      'port' => '',
      'driver' => 'mysql',
      'prefix' => 'vb_drupal_',
    ),
  ),
);
Pedro Lozano’s picture

The only solution for me so far has been to add the sourcedb parameter to the MigrateSQLMap constructor, as @albertczyk pointed out.

But I don't want migrate tables to be in the sourcedb. Is there any solution in this case?

Pedro Lozano’s picture

Status: Active » Closed (works as designed)

Sorry @mikeryan, I had tried the map_joinable solution with no luck, but I just realized I was passing it in the wrong array. It is working now. Thanks.

Pedro Lozano’s picture

Issue summary: View changes

obfuscating