The migration process from mysql 5.1 to postgresql 8.4 aborts with the message:

# drush dbtng-replicate default postgres

WD php: PDOException: SQLSTATE[42P01]: Undefined table: 7 ERROR: relation "locales_source" does not exist [error]
LINE 1: SELECT s.lid, t.translation, s.version FROM locales_source s...
^: SELECT s.lid, t.translation, s.version FROM {locales_source} s LEFT JOIN
{locales_target} t ON s.lid = t.lid AND t.language = :language WHERE s.source = :source AND s.context = :context AND s.textgroup =
'default'; Array
(
[:language] => de
[:source] => @table was successfully created.
[:context] =>
)
in locale() (line 676 of /opt/cms/modules/locale/locale.module).
PDOException: SQLSTATE[42P01]: Undefined table: 7 ERROR: relation "locales_source" does not exist
LINE 1: SELECT s.lid, t.translation, s.version FROM locales_source s...
^: SELECT s.lid, t.translation, s.version FROM {locales_source} s LEFT JOIN {locales_target} t ON s.lid = t.lid AND t.language = :language WHERE s.source = :source AND s.context = :context AND s.textgroup = 'default'; Array
(
[:language] => de
[:source] => @table was successfully created.
[:context] =>
)
in locale() (Zeile 676 von /opt/cms/modules/locale/locale.module).
PHP Fatal error: Exception thrown without a stack frame in Unknown on line 0

after two tables (block and block_bid_seq) have been created. locales_source exists in the source db and contains data.

Comments

josh waihi’s picture

Status: Active » Postponed (maintainer needs more info)

Just tried to replicate the issue but couldn't. Could you please list the steps you took? Sounds like DBTNG didn't create locales_source but am unsure how to replicate this.

manitoba22’s picture

1. create postgres db

  createuser --pwprompt --encrypted --no-adduser --no-createdb drupaladm
  createdb --encoding=UNICODE --owner=drupaladm drupal

2. add it to settings.php

$databases['default']['default'] = array (
      'driver' => 'mysql',
      'database' => 'drupal',
      'username' => 'drupaladm',
      'password' => 'secret',
      'host' => 'localhost',
      'port' => '',
      'prefix' => '',
);
$databases['postgres']['default'] = array (
      'driver' => 'pgsql',
      'database' => 'drupal',
      'username' => 'drupaladm',
      'password' => 'secret',
      'host' => 'localhost',
      'port' => '',
      'prefix' => '',
);

(this is different from your blog post, where $databases['default']['default'] and $databases['default']['pgsql'] was used, which I tried first, but with this, the second database entry did not show up in the Migrator database list)

3. several attempts with the gui (resetting the postgres db with dropdb/createdb after each)

4. run the drush command as given above to get a better readable output

(Perhaps I should mention that some days ago I migrated the mysql database from a host with 5.0 to the current with 5.1. With this I had an issue (http://drupal.org/node/998474), which I solved by truncating search_index and search_total tables.)

manitoba22’s picture

Additional info: Drupal version is 7.4. Enabled modules are:

default # drush pm-list|grep Enabled
 Core         Block (block)                  Module  Enabled        7.4
 Core         Book (book)                    Module  Enabled        7.4
 Core         Color (color)                  Module  Enabled        7.4
 Core         Comment (comment)              Module  Enabled        7.4
 Core         Contextual links (contextual)  Module  Enabled        7.4
 Core         Dashboard (dashboard)          Module  Enabled        7.4
 Core         Database logging (dblog)       Module  Enabled        7.4
 Core         Field (field)                  Module  Enabled        7.4
 Core         Field SQL storage              Module  Enabled        7.4
 Core         Field UI (field_ui)            Module  Enabled        7.4
 Core         File (file)                    Module  Enabled        7.4
 Core         Filter (filter)                Module  Enabled        7.4
 Core         Help (help)                    Module  Enabled        7.4
 Core         Image (image)                  Module  Enabled        7.4
 Core         List (list)                    Module  Enabled        7.4
 Core         Locale (locale)                Module  Enabled        7.4
 Core         Menu (menu)                    Module  Enabled        7.4
 Core         Node (node)                    Module  Enabled        7.4
 Core         Number (number)                Module  Enabled        7.4
 Core         Options (options)              Module  Enabled        7.4
 Core         Path (path)                    Module  Enabled        7.4
 Core         RDF (rdf)                      Module  Enabled        7.4
 Core         Search (search)                Module  Enabled        7.4
 Core         Shortcut (shortcut)            Module  Enabled        7.4
 Core         System (system)                Module  Enabled        7.4
 Core         Taxonomy (taxonomy)            Module  Enabled        7.4
 Core         Text (text)                    Module  Enabled        7.4
 Core         Toolbar (toolbar)              Module  Enabled        7.4
 Core         Update manager (update)        Module  Enabled        7.4
 Core         User (user)                    Module  Enabled        7.4
 DBTNG        DBTNG Migrator                 Module  Enabled        7.x-1.x-dev
 Lightweight  LDAP Authentication            Module  Enabled        7.x-1.0-bet
 Lightweight  LDAP Servers (ldap_servers)    Module  Enabled        7.x-1.0-bet
 Multilingua  Localization update            Module  Enabled        7.x-1.x-dev
 Other        Diff (diff)                    Module  Enabled        7.x-2.0-bet
 Other        Node to alternate formats      Module  Enabled        7.x
 Printer,     Printer-friendly pages         Module  Enabled        7.x-1.x-dev
 User         ak_cleanup (ak_cleanup)        Module  Enabled        7.x
 User         ak_editor (ak_editor)          Module  Enabled        7.x
 User         Wysiwyg (wysiwyg)              Module  Enabled        7.x-2.1
 Core         Bartik (bartik)                Theme   Enabled        7.4
 Core         Seven (seven)                  Theme   Enabled        7.4
 Sonstige     Zen (zen)                      Theme   Enabled        7.x-3.1
manitoba22’s picture

Status: Postponed (maintainer needs more info) » Active
josh waihi’s picture

I noticed your encoding for PostgreSQL is unicode. It should be UTF-8 as Drupal only supports PostgreSQL in UTF-8. Something the Drupal installer picks up but DBTNG Migrator doesn't.

Does this error also occur when you use the GUI? I used the batch GUI to do the migration and it worked for me. This and the encoding are the only differences I have other than the different modules enabled. Not that that should really matter as DBTNG just iterates through enabled modules, installs the schema then copies the contents across.

manitoba22’s picture

After recreating the database with "encoding=UTF-8", I executed Migrator in GUI and got the errors below.

Again, only block and block_bid_seq where created. Maybe there occurs a problem after adding the sequence?

The postgres version is postgresql-8.4.7-1.el6_0.1.x86_64 on centos 6.0

An AJAX HTTP error occurred. HTTP Result Code: 500 Debugging information follows. Path: /cms/?q=batch&id=31&op=do StatusText: Service unavailable (with message) ResponseText: PDOException: SQLSTATE[42P01]: Undefined table: 7 ERROR: relation "locales_source" does not exist LINE 1: SELECT s.lid, t.translation, s.version FROM locales_source s... ^: SELECT s.lid, t.translation, s.version FROM {locales_source} s LEFT JOIN {locales_target} t ON s.lid = t.lid AND t.language = :language WHERE s.source = :source AND s.context = :context AND s.textgroup = 'default'; Array ( [:language] => de [:source] => @table was successfully created. [:context] => ) in locale() (Zeile 676 von /opt/cms/modules/locale/locale.module).Uncaught exception thrown in shutdown function.PDOException: SQLSTATE[42P01]: Undefined table: 7 ERROR: relation "semaphore" does not exist LINE 1: DELETE FROM semaphore ^: DELETE FROM {semaphore} WHERE (value = :db_condition_placeholder_0) ; Array ( [:db_condition_placeholder_0] => 2175106754e3078d0638e87.84210880 ) in lock_release_all() (line 266 of /opt/cms/includes/lock.inc).Uncaught exception thrown in session handler.PDOException: SQLSTATE[42P01]: Undefined table: 7 ERROR: relation "sessions" does not exist LINE 3: sessions sessions ^: SELECT 1 AS expression FROM {sessions} sessions WHERE ( (sid = :db_condition_placeholder_0) AND (ssid = :db_condition_placeholder_1) ) FOR UPDATE; Array ( [:db_condition_placeholder_0] => ghssICYUU_FdNR8sc5NKJWsID9NGuIRQIV8DkNdzpGw [:db_condition_placeholder_1] => ) in _drupal_session_write() (line 203 of /opt/cms/includes/session.inc).

josh waihi’s picture

Status: Active » Needs review

Ah, ok. You have the locale module enabled which, when t() is called, calls locale() while connected to the PostgreSQL database.

I've pushed a commit to the 7.x-1.x branch of the git repository, if you can test it for me, I'll make a new release if it works.

manitoba22’s picture

sorry for the stupid question - I don't know where to find git repository and the updated file (quit new to drupal, and to git, too).

rfay’s picture

@manitoba22, you can actually grab the latest at http://drupalcode.org/project/dbtng_migrator.git/snapshot/refs/heads/7.x..., or you can wait for the next dev release which gets rolled every 12 hours.

Or spend some time with git - it's great. http://drupal.org/documentation/git

manitoba22’s picture

ok - figured it out (I really had to learn it).

The GUI migration went to 100%, and a lists of successfully updated modules was displayed, but at the end an error message occurs:

SQLSTATE[42703]: Undefined column: 7 ERROR: column "l10n_status" of relation "locales_target" does not exist LINE 1: ...target (lid, translation, language, plid, plural, l10n_statu... ^

manitoba22’s picture

With drush dbtng-replicate default postgres, the error message cited above comes first, and then the list of successfully created tables/modules.

josh waihi’s picture

l10n_status isn't a column in locales_target. Try clearing your cache before running the command again. Or maybe one of your modules is referencing a column that doesn't exists, in other words, it needs updating.

manitoba22’s picture

locales_target.l10n_status is indeed missing in the newly created postgres database. However, Drupal seems to work already fine with it - I can navigate, add new content, etc.!

I have gone through some database migrations, and it's unbelievable, that this one worked just on a mouse click!!

josh waihi’s picture

Status: Needs review » Fixed

locales_target.l10n_status is missing by design.

So assuming all is working fine. I'll make a 1.0 release. Thanks for reporting the bug.

manitoba22’s picture

Status: Fixed » Needs review

It is in the mysql database:

mysql> describe locales_target;
+-------------+-------------+------+-----+---------+-------+
| Field       | Type        | Null | Key | Default | Extra |
+-------------+-------------+------+-----+---------+-------+
| lid         | int(11)     | NO   | PRI | 0       |       |
| translation | blob        | NO   |     | NULL    |       |
| language    | varchar(12) | NO   | PRI |         |       |
| plid        | int(11)     | NO   | MUL | 0       |       |
| plural      | int(11)     | NO   | PRI | 0       |       |
| l10n_status | int(11)     | NO   |     | 0       |       |
+-------------+-------------+------+-----+---------+-------+
6 rows in set (0.00 sec)

Perhaps it was added by the l10n_update module.

josh waihi’s picture

If it was added in an update hook or install hook, then dbtng wouldn't know how to migrate it. Schema changes need to be provided via hook_schema_alter.

manitoba22’s picture

Ok, I see. Uninstalling and reinstalling l10n_update fixed the missing column.

Thank you so much for your kind and immediate help. It's really amazing to work togesther across so many time zones.

And thanks again you for this uncredible module!

josh waihi’s picture

Status: Needs review » Fixed

Awesome. And your welcome :)

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

manitoba22’s picture

Status: Closed (fixed) » Active

Unfortunately, the error described above shows up again.

As I could verify, the field locales_target.l10n_status is added by the current 7.x-1.0-beta2 version of Localization update module during l10n_update.install.

As a result of the error, the column is not copied.

josh waihi’s picture

DBTNG Migrator migrates a table in two phases. First it migrates the schema, then the data. This is the function for migrating the schema. As you can see it obtains the schema the same way drupal_get_schema does. So if DBTNG Migrator failed to get the full schema, then Drupal core must also be failing, in which case, this can't be a DBTNG Migrator bug.

Try flushing your schema cache as it may contain old cache that doesn't have the altered schema of locales_target.

Potentially, l10n_update_install should be clearing the schema cache so l10n_update_schema_alter gets called.

manitoba22’s picture

Hi Josh,

Thank you for looking into this once again!

I tried it today on a completely different Drupal installation (different purpose, different machines, different modules installed, different database server - so also postgresql, German as second language and l10n_update installed). To clear the schema cache, before "drush dbtng-replicate" I executed

drush php-eval "cache_clear_all('schema', 'cache');"

on the source (and also on the target) machine, but again got

SQLSTATE[42703]: Undefined column: 7 ERROR: column "l10n_status" of relation "locales_target" does not exist at character 71

I have no idea if this was caused by a flaw in l10n_upate (7.x-1.0-beta2), in dbtng_migrator (7.x-1.0) or drupal core (7.10) itself, or most likely by some mistake on my side (given my weak experience with drupal and php). Is there something I could do to make the problem reproduceable?

josh waihi’s picture

Before you run dbtng's replicate command, what does this give you:

drush php-eval "print_r(drupal_get_schema('l10n_update'));"

It should return an array describing the tables for that module and the locales_target table should then have a l10n_status column in it. If it doesn't, try this:

drush php-eval "print_r(drupal_get_schema('l10n_update', TRUE));"

If it show up then, you should be able to migrate with dbtng no problems.

Also, the cache_clear call is:

cache_clear_all('schema:', 'cache', TRUE);

I do notice the schema build seems to have changed in core from when I first wrote this module, so if this doesn't work, I'll relook at how the schema is built in DBTNG Migrator

manitoba22’s picture

Neither
drush php-eval "print_r(drupal_get_schema('l10n_update'));"
nor
drush php-eval "print_r(drupal_get_schema('l10n_update', TRUE));"
give any results (drush php-eval "print_r(drupal_get_schema('node', TRUE));" does).

The database definitions in the source database are:

  # drush php-eval "print_r(drupal_get_schema('l10n_update', TRUE));"
  # drush sqlc
  psql (8.4.9)
  Type "help" for help.

  drupal_labs=> \d l10n*
                          Table "public.l10n_update_file"
      Column    |          Type          |               Modifiers
  --------------+------------------------+----------------------------------------
   project      | character varying(50)  | not null
   language     | character varying(12)  | not null
   type         | character varying(50)  | not null default ''::character varying
   filename     | character varying(255) | not null default ''::character varying
   fileurl      | character varying(255) | not null default ''::character varying
   uri          | character varying(255) | not null default ''::character varying
   timestamp    | integer                | default 0
   version      | character varying(128) | not null default ''::character varying
   status       | integer                | not null default 1
   last_updated | integer                | default 0
   last_checked | integer                | default 0
   import_date  | integer                | default 0
  Indexes:
      "l10n_update_file_pkey" PRIMARY KEY, btree (project, language)

  Index "public.l10n_update_file_pkey"
    Column  |         Type
  ----------+-----------------------
   project  | character varying(50)
   language | character varying(12)
  primary key, btree, for table "public.l10n_update_file"

                         Table "public.l10n_update_project"
      Column    |          Type          |               Modifiers
  --------------+------------------------+----------------------------------------
   name         | character varying(50)  | not null
   project_type | character varying(50)  | not null
   core         | character varying(128) | not null default ''::character varying
   version      | character varying(128) | not null default ''::character varying
   l10n_server  | character varying(255) | not null default ''::character varying
   l10n_path    | character varying(255) | not null default ''::character varying
   status       | integer                | not null default 1
  Indexes:
      "l10n_update_project_pkey" PRIMARY KEY, btree (name)

  Index "public.l10n_update_project_pkey"
   Column |         Type
  --------+-----------------------
   name   | character varying(50)
  primary key, btree, for table "public.l10n_update_project"


  drupal_labs=> \d locales_target
                          Table "public.locales_target"
     Column    |         Type          |               Modifiers
  -------------+-----------------------+----------------------------------------
   lid         | integer               | not null default 0
   translation | text                  | not null
   language    | character varying(12) | not null default ''::character varying
   plid        | integer               | not null default 0
   plural      | integer               | not null default 0
   l10n_status | integer               | not null default 0
   i18n_status | integer               | not null default 0
  Indexes:
      "locales_target_pkey" PRIMARY KEY, btree (language, lid, plural)
      "locales_target_lid_idx" btree (lid)
      "locales_target_plid_idx" btree (plid)
      "locales_target_plural_idx" btree (plural)

  drupal_labs=> \q
  #
manitoba22’s picture

comment deleted (subject was i18n_status)

manitoba22’s picture

Workarround:

drush -y pm-disable l10n_update
drush -y pm-uninstall l10n_update
drush -y en l10n_update

(recreates the required field)

chi’s picture

Can we invoke hook_schema_alter() from DBTNG Migrator module?

josh waihi’s picture

Status: Active » Fixed

Fixed in 7.x-1.4

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

rudam’s picture

Hello, I tried everything you guys mentioned above and I still get using migrator database check gui:
SQLSTATE[42P01]: Undefined table: 7 ERROR: relation "locales_source" does not exist LINE 1: SELECT s.lid, t.translation, s.version FROM locales_source s... ^

The database was created UTF8:
createdb --encoding=UTF8 -O drupal drupal

Tried drush -y pm-disable l10n_update, drush -y pm-uninstall l10n_update, drush -y en l10n_update and no game.

version 7.x-1.4

rudam’s picture

Update:
I went directly to the migration ignoring the check error and everything went smoothly. Thank you for this wonderful module!