Problem/Motivation

This is step 1 for #2447573: [meta] Make sure 8.x - 8.x hook_update_N() testing is possible.

Proposed resolution

Create the command/script.

Remaining tasks

User interface changes

API changes

Suggested commit message

Please ensure to add larowlan for his initial work on the script in #2447573: [meta] Make sure 8.x - 8.x hook_update_N() testing is possible.

CommentFileSizeAuthor
#69 hook-update-n-test-script-2497323-69.patch26.2 KBjhedstrom
#69 interdiff.txt2.82 KBjhedstrom
#59 hook-update-n-test-script-2497323-59.patch25.75 KBjhedstrom
#59 interdiff.txt8.49 KBjhedstrom
#58 diff.txt2.12 MBdawehner
#57 hook-update-n-test-script-2497323-57.patch24.41 KBjhedstrom
#57 interdiff.txt2.11 KBjhedstrom
#53 hook-update-n-test-script-2497323-53.patch24.18 KBjhedstrom
#53 interdiff.txt6.54 KBjhedstrom
#42 hook-update-n-test-script-2497323-42.patch21.86 KBjhedstrom
#42 interdiff.txt468 bytesjhedstrom
#41 hook-update-n-test-script-2497323-41.patch21.9 KBjhedstrom
#41 interdiff.txt4.1 KBjhedstrom
#35 hook-update-n-test-script-2497323-35.patch21.58 KBjhedstrom
#35 interdiff.txt6.46 KBjhedstrom
#33 hook-update-n-test-script-2497323-33.patch18.72 KBjhedstrom
#33 interdiff.txt2.99 KBjhedstrom
#27 hook-update-n-test-script-2497323-27.patch17.83 KBjhedstrom
#27 interdiff.txt4.08 KBjhedstrom
#25 hook-update-n-test-script-2497323-25.patch16.93 KBjhedstrom
#25 interdiff.txt3.09 KBjhedstrom
#18 hook-update-n-test-script-2497323-18.patch16.84 KBjhedstrom
#18 interdiff.txt4.98 KBjhedstrom
#17 hook-update-n-test-script-2497323-17.patch15.31 KBjhedstrom
#17 interdiff.txt641 bytesjhedstrom
#15 hook-update-n-test-script-2497323-15.patch15.14 KBjhedstrom
#15 interdiff.txt1.2 KBjhedstrom
#14 hook-update-n-test-script-2497323-14.patch15.1 KBjhedstrom
#14 interdiff.txt4.84 KBjhedstrom
#9 hook-update-n-test-script-2497323-08.patch14.36 KBjhedstrom
#9 interdiff.txt7.09 KBjhedstrom
#2 hook-update-n-test-script-2497323-02.patch12.4 KBjhedstrom
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

jhedstrom’s picture

Title: Create a php script that can dump a database for later restoration » Create a php script that can dump a database for testing update hooks
Issue summary: View changes
Parent issue: » #2447573: [meta] Make sure 8.x - 8.x hook_update_N() testing is possible
jhedstrom’s picture

Status: Active » Needs review
Issue tags: +Needs tests
FileSize
12.4 KB

Here's a fairly rough start that combines the starting script from #2447573: [meta] Make sure 8.x - 8.x hook_update_N() testing is possible with a lot of the logic in migrate-db.sh. The console application could probably handle a lot of the bootstrapping that's still happening in the script itself.

Unassigning as I won't have time until next week to pick this up again.

jhedstrom’s picture

Note, the console ansi coloring breaks php syntax, so to test generating, use

./dump-database-d8.php --no-ansi > my-test-script.php

jhedstrom’s picture

Assigned: jhedstrom » Unassigned
jhedstrom’s picture

Issue summary: View changes

Adding some remaining tasks.

dawehner’s picture

  1. +++ b/core/lib/Drupal/Core/Command/DbDump.php
    @@ -0,0 +1,63 @@
    +   * {@inheritdoc}
    +   *
    +   * Overridden so the application doesn't expect the command name as the first
    +   * argument.
    +   */
    +  public function getDefinition() {
    +    $definition = parent::getDefinition();
    +    // Clears the normal first argument (the command name).
    +    $definition->setArguments();
    +    return $definition;
    +  }
    +
    

    Cool tricky, so we have commands which can be reused, but we still have a working script.

  2. +++ b/core/lib/Drupal/Core/Command/DbDumpCommand.php
    @@ -0,0 +1,296 @@
    diff --git a/core/modules/rest/rest.install b/core/modules/rest/rest.install
    
    diff --git a/core/modules/rest/rest.install b/core/modules/rest/rest.install
    index 4bca69b..2afe0b5 100644
    
    index 4bca69b..2afe0b5 100644
    --- a/core/modules/rest/rest.install
    
    --- a/core/modules/rest/rest.install
    +++ b/core/modules/rest/rest.install
    
    +++ b/core/modules/rest/rest.install
    +++ b/core/modules/rest/rest.install
    @@ -11,12 +11,12 @@
    
    @@ -11,12 +11,12 @@
     function rest_requirements($phase) {
       $requirements = array();
     
    -  if (version_compare(PHP_VERSION, '5.6.0', '>=') && version_compare(PHP_VERSION, '7', '<') && ini_get('always_populate_raw_post_data') != -1) {
    +  if (version_compare(PHP_VERSION, '5.6.0') >= 0 && ini_get('always_populate_raw_post_data') != -1) {
         $requirements['always_populate_raw_post_data'] = array(
           'title' => t('always_populate_raw_post_data PHP setting'),
           'value' => t('Not set to -1.'),
           'severity' => REQUIREMENT_ERROR,
    -      'description' => t('The always_populate_raw_post_data PHP setting should be set to -1 in PHP version 5.6. Please check the <a href="https://php.net/manual/en/ini.core.php#ini.always-populate-raw-post-data">PHP manual</a> for information on how to correct this.'),
    +      'description' => t('The always_populate_raw_post_data PHP setting should be set to -1 in PHP versions 5.6 and up. Please check the <a href="https://php.net/manual/en/ini.core.php#ini.always-populate-raw-post-data">PHP manual</a> for information on how to correct this.'),
         );
       }
       return $requirements;
    

    This is part of a different issue, isn't it ;)

jhedstrom’s picture

Assigned: Unassigned » jhedstrom

I'm going to write some tests.

This is part of a different issue, isn't it ;)

Indeed, I forgot to rebase the branch :)

webchick’s picture

Priority: Major » Critical

Since this is blocking a critical, escalating to critical.

jhedstrom’s picture

FileSize
7.09 KB
14.36 KB

This patch:

  • Injects module handler
  • Refactors table discovery to work in simpletests (eg, it uses the db prefix)
  • Adds the beginnings of a test
  • Removes the accidental code from #2

Status: Needs review » Needs work

The last submitted patch, 9: hook-update-n-test-script-2497323-08.patch, failed testing.

jhedstrom’s picture

Issue summary: View changes

Updating issue summary. The tests will load the necessary scripts.

webchick’s picture

Testbot seems to have failed a bunch of patches this afternoon but this one:

SQLSTATE[42S02]: Base table or view not found: 1146 Table 'drupaltestbotmysql.config' doesn't exist: SHOW COLUMNS FROM config; Array ( ) in Drupal\Core\Command\DbDumpCommand->getTableSchema() (line 130 of /var/lib/drupaltestbot/sites/default/files/checkout/core/lib/Drupal/Core/Command/DbDumpCommand.php). 

...seems like it could be legit.

dawehner’s picture

Its great to see good readable code here!!

  1. +++ b/core/lib/Drupal/Core/Command/DbDumpCommand.php
    @@ -0,0 +1,311 @@
    +   * @var \Drupal\Core\Extension\ModuleHandler
    ...
    +   * @param \Drupal\Core\Extension\ModuleHandler $module_handler
    ...
    +  function __construct(Connection $connection, ModuleHandler $module_handler) {
    

    Let's typehint the ModuleHandlerInterface

  2. +++ b/core/lib/Drupal/Core/Command/DbDumpCommand.php
    @@ -0,0 +1,311 @@
    +    $script = <<<ENDOFSCRIPT
    ...
    + \$connection = Database::getConnection();
    ...
    +ENDOFSCRIPT;
    

    Do you really need the escaping of the $ here? At least

     doesn't use it
    </li>
    
    <li>
    <code>
    +++ b/core/scripts/dump-database-d8.php
    @@ -0,0 +1,23 @@
    +Settings::initialize(dirname(dirname(__DIR__)), DrupalKernel::findSitePath($request), $autoloader);
    +$kernel = DrupalKernel::createFromRequest($request, $autoloader, 'prod')->boot();
    ...
    +$application = new DbDump(Database::getConnection(), \Drupal::moduleHandler());
    

    Here is a general question, no idea about a proper answer :) \Symfony\Bundle\FrameworkBundle\Console\Application gets the entire kernel injected, I guess this is fine, as its a much more generic code, but should our DbDump application also get the kernel or does the rule of "the less dependencies the better" apply here?

jhedstrom’s picture

Status: Needs work » Needs review
FileSize
4.84 KB
15.1 KB

re: #12
The fail was legit. The {} were not making it through to the query, so the table was not properly prefixed. Also, I had not enabled the config module :)

re: #13

Do you really need the escaping of the $ here?

I was previously using heredoc, switching to nowdoc so we don't have to escape the $.

Here is a general question, no idea about a proper answer :) \Symfony\Bundle\FrameworkBundle\Console\Application gets the entire kernel injected, I guess this is fine, as its a much more generic code, but should our DbDump application also get the kernel or does the rule of "the less dependencies the better" apply here?

I originally limited the dependencies with the plan of using PHPUnit for the tests. Since we're using KernelTestBase, we could inject the kernel...not sure if it buys us anything.

jhedstrom’s picture

FileSize
1.2 KB
15.14 KB

Oops, missed an instance of ModuleHandler.

jhedstrom’s picture

Issue summary: View changes
Issue tags: -Needs tests
jhedstrom’s picture

FileSize
641 bytes
15.31 KB

This disables the ansi output unless explicitly set. So now a script can be generated without needing to remember to set --no-ansi:

./dump-database-d8.php > target.php
jhedstrom’s picture

FileSize
4.98 KB
16.84 KB

This adds tests for actually loading the generated script. Writing these caught several errors in the script from previous patches (tests FTW).

I've also tentatively removed the trailing .php from the script, which is more in keeping with modern scripts (the php bit is in the header of the script). It also avoids having to call the script dump-database-d8.php from within the command classes.

jhedstrom’s picture

Issue summary: View changes

Updating the issue summary to reflect current status of the script.

dawehner’s picture

  1. +++ b/core/modules/system/src/Tests/Update/DbDumpTest.php
    @@ -0,0 +1,115 @@
    +    // Drop tables to avoid errors.
    +    $tables = ['sessions', 'url_alias', 'config'];
    +    foreach ($tables as $table) {
    +      Database::getConnection()->schema()->dropTable($table);
    +    }
    

    OH that sounds quite scary for a test ... can't we maybe add a feature to export them with a different db prefix? I could imagine that this could be useful for actual upgrade path testing as well?

  2. +++ b/core/modules/system/src/Tests/Update/DbDumpTest.php
    --- /dev/null
    +++ b/core/scripts/dump-database-d8
    

    Well, this makes it more complex to look at the file with your editor, right? I'd just go with .php and be fine, honestly.

  3. +++ b/core/scripts/dump-database-d8
    @@ -0,0 +1,23 @@
    +$application = new DbDump(Database::getConnection(), \Drupal::moduleHandler());
    

    +1 for just adding what is actually needed.

jhedstrom’s picture

OH that sounds quite scary for a test ... can't we maybe add a feature to export them with a different db prefix? I could imagine that this could be useful for actual upgrade path testing as well?

This drops the test tables that are prefixed (simpletestNNNNconfig, simpletestNNNNsessions, etc).

jhedstrom’s picture

+++ b/core/lib/Drupal/Core/Command/DbDumpCommand.php
@@ -61,7 +61,7 @@ function __construct(Connection $connection, ModuleHandlerInterface $module_hand
+    $this->setName('dump-database-d8')

Even if we add the .php back to the script name, should we leave this as-is, or change the actual command name internally to dump-database-d8.php? The only place this surfaces is if a user types ./dump-database-d8 --help:

Usage:
  dump-database-d8

Options:
  -h, --help            Display this help message
  -q, --quiet           Do not output any message
  -V, --version         Display this application version
      --ansi            Force ANSI output
      --no-ansi         Disable ANSI output
  -n, --no-interaction  Do not ask any interactive question
  -v|vv|vvv, --verbose  Increase the verbosity of messages: 1 for normal output, 2 for more verbose output and 3 for debug
cilefen’s picture

Drupal/Core/Command/DbDumpCommand and Drupal/Core/Command/DbDump are easy to confuse. Could there be another way to name those?

dawehner’s picture

Yeah, what about using DbDumpApplication ?

jhedstrom’s picture

FileSize
3.09 KB
16.93 KB

This patch:

  • Moves the script back to dump-database-d8.php
  • Moves the DbDump class to DbDumpApplication
dawehner’s picture

Decide what options/arguments are needed (tables/excluded tables, data/no data, etc) No options/arguments aside from default console options will be added to the initial script. These can go in later if/when needed
Where does the output go? Should output be per-table (as migrate-db.sh is)? Output will go to stdout, and can be piped into files as needed. Output will be for all tables.
Does this script handle the loading too, or is that left to the tests? The tests will load the required scripts
Write tests

We have tests, do we need something more?

Here is just some continues review, sorry for the nitpicking!

  1. +++ b/core/lib/Drupal/Core/Command/DbDumpApplication.php
    @@ -0,0 +1,70 @@
    +  /**
    +   * @var \Drupal\Core\Database\Connection
    +   */
    +  protected $connection;
    +
    +  /**
    +   * @var \Drupal\Core\Extension\ModuleHandlerInterface
    +   */
    +  protected $moduleHandler;
    ...
    +  /**
    +   * @param \Drupal\Core\Database\Connection $connection
    +   * @param \Drupal\Core\Extension\ModuleHandlerInterface $module_handler
    +   */
    +  function __construct(Connection $connection, ModuleHandlerInterface $module_handler) {
    
    +++ b/core/lib/Drupal/Core/Command/DbDumpCommand.php
    @@ -0,0 +1,320 @@
    +   * @var \Drupal\Core\Extension\ModuleHandlerInterface
    +   */
    +  protected $moduleHandler;
    +
    +  /**
    +   * An array of table patterns to exclude completely.
    +   *
    +   * This excludes any lingering simpletest tables generated during test runs.
    +   *
    +   * @var array
    +   */
    +  protected $excludeTables = ['simpletest.+'];
    +
    +  /**
    +   * Table patterns for which to only dump the schema, no data.
    +   *
    +   * @var array
    +   */
    +  protected $schemaOnly = ['cache.*', 'sessions', 'watchdog'];
    +
    +  /**
    +   * @param \Drupal\Core\Database\Connection $connection
    +   * @param \Drupal\Core\Extension\ModuleHandlerInterface $module_handler
    +   */
    +  function __construct(Connection $connection, ModuleHandlerInterface $module_handler) {
    +    $this->connection = $connection;
    +    $this->moduleHandler = $module_handler;
    +    parent::__construct();
    +  }
    

    I hate our super strict docs requirements which doesn't really add any value, but meh we need some one line docs here.

  2. +++ b/core/lib/Drupal/Core/Command/DbDumpCommand.php
    @@ -0,0 +1,320 @@
    +/**
    + * Provides a command to dump the current database to a script.
    + *
    + * @see \Drupal\Core\Command\DbDump
    + */
    +class DbDumpCommand extends Command {
    

    What about describing here what will be part of the dump: tables + modules ...

  3. +++ b/core/lib/Drupal/Core/Command/DbDumpCommand.php
    @@ -0,0 +1,320 @@
    +   * {@inheritdoc}
    +   */
    +  protected function execute(InputInterface $input, OutputInterface $output) {
    +    // If not explicitly set, disable ANSI which will break generated php.
    +    if ($input->hasParameterOption(['--ansi']) !== TRUE) {
    +      $output->setDecorated(FALSE);
    +    }
    +
    

    Let's add --ansi to the ::configure() description so its part of the help text.

  4. +++ b/core/lib/Drupal/Core/Command/DbDumpCommand.php
    @@ -0,0 +1,320 @@
    +    // @todo this is MySQL only since there are no Database API functions for
    +    // table column data.
    

    Does that mean sqlite / pgsql tests will fail then?

  5. +++ b/core/lib/Drupal/Core/Command/DbDumpCommand.php
    @@ -0,0 +1,320 @@
    +
    +  /**
    +   * List of modules enabled for insertion into the script docblock.
    +   *
    +   * @return string
    +   */
    +  protected function getModulesScript() {
    +    $output = '';
    +    $modules = $this->moduleHandler->getModuleList();
    +    ksort($modules);
    +    foreach ($modules as $module => $filename) {
    +      $output .= " *  - $module\n";
    +    }
    +    return rtrim($output, "\n");
    +  }
    

    I'm curious whether we also need enabled themes here?

jhedstrom’s picture

Issue summary: View changes
FileSize
4.08 KB
17.83 KB

I've removed the remaining tasks section since they were all completed.

This patch should address #26 1 and 2.

Let's add --ansi to the ::configure() description so its part of the help text.

It's already part of the help text as the option is provided by the default command class. (See output in #22).

Does that mean sqlite / pgsql tests will fail then?

No, it just means a developer running on either of those couldn't generate a new script. The generated script should work on any backend.

I'm curious whether we also need enabled themes here?

The enabled theme would be available in the exported config, and since themes don't create any tables, this is probably not needed?

dawehner’s picture

It's already part of the help text as the option is provided by the default command class. (See output in #22).

Oh, got it. Sorry

No, it just means a developer running on either of those couldn't generate a new script. The generated script should work on any backend.

Well, so we can't test the command itself, as the test for the command itself would try to generate a script, which does not work on other databases?

The enabled theme would be available in the exported config, and since themes don't create any tables, this is probably not needed?

Oh wait, then I probably don't get the point of the modules ... Ah I see, its pure documentation purposes.

jhedstrom’s picture

Well, so we can't test the command itself, as the test for the command itself
would try to generate a script, which does not work on other databases?

Oh, hmm. That is correct, these tests will fail on non-MySQL.

Status: Needs review » Needs work

The last submitted patch, 27: hook-update-n-test-script-2497323-27.patch, failed testing.

Status: Needs work » Needs review
dawehner’s picture

I can't speak for things in general, but I would be fine for such an edge case "feature" to be opt in for mysql

jhedstrom’s picture

FileSize
2.99 KB
18.72 KB

This patch updates the tests to skip if not running on MySQL. I updated the documentation to indicate why this script is currently MySQL only.

jhedstrom’s picture

Status: Needs review » Needs work

I discovered an issue with the exported script while working over in #2498625: Write tests that ensure hook_update_N is properly run.

Some mappings aren't correct. For instance, the cache_discovery table:

MariaDB [d8]> describe cache_discovery;
+------------+---------------+------+-----+---------+-------+
| Field      | Type          | Null | Key | Default | Extra |
+------------+---------------+------+-----+---------+-------+
| cid        | varchar(255)  | NO   | PRI |         |       |
| data       | longblob      | YES  |     | NULL    |       |
| expire     | int(11)       | NO   | MUL | 0       |       |
| created    | decimal(14,3) | NO   |     | 0.000   |       |
| serialized | smallint(6)   | NO   |     | 0       |       |
| tags       | longtext      | YES  |     | NULL    |       |
| checksum   | varchar(255)  | NO   |     | NULL    |       |
+------------+---------------+------+-----+---------+-------+
7 rows in set (0.01 sec)

MariaDB [d8]> describe simpletest516200cache_discovery;
+------------+---------------+------+-----+---------+-------+
| Field      | Type          | Null | Key | Default | Extra |
+------------+---------------+------+-----+---------+-------+
| cid        | varchar(255)  | NO   | PRI |         |       |
| data       | blob          | YES  |     | NULL    |       |
| expire     | int(11)       | NO   |     | 0       |       |
| created    | decimal(14,3) | NO   |     | 0.000   |       |
| serialized | int(11)       | NO   |     | 0       |       |
| tags       | text          | YES  |     | NULL    |       |
| checksum   | varchar(255)  | NO   |     | NULL    |       |
+------------+---------------+------+-----+---------+-------+
7 rows in set (0.01 sec)

Note the data field in the original table is longblob, while the table created from this script is blob.

jhedstrom’s picture

Status: Needs work » Needs review
FileSize
6.46 KB
21.58 KB

This patch adds the 'size' parameter back to the schema where applicable. I also added tests to verify the generated schemas after loading the tables again.

dawehner’s picture

  1. +++ b/core/modules/system/src/Tests/Update/DbDumpTest.php
    @@ -44,11 +46,40 @@ class DbDumpTest extends KernelTestBase {
    +    // Determine what database backend is running, and set the skip flag.
    +    $this->skipTests = !(Database::getConnection() instanceof Connection);
    

    I'd recommend to check Database::getConnection()->getDatabaseType() Its the kind of code we do in other places as well .

  2. +++ b/core/modules/system/src/Tests/Update/DbDumpTest.php
    @@ -125,14 +159,37 @@ public function testScriptLoad() {
    +    // @todo this is MySQL specific.
    

    Do you mind opening up a follow up to make the script/test database agnostic?

+++ b/core/lib/Drupal/Core/Command/DbDumpCommand.php
@@ -0,0 +1,372 @@
+
+  /**
+   * Returns a schema array for a given table.
+   *
+   * @param string $table
+   *   The table name.
+   *
+   * @return array
+   *   A schema array (as defined by hook_schema()).
+   */
+  protected function getTableSchema($table) {
+    // @todo this implementation is hard-coded for MySQL.

Do you mind also opening up a follow up to provide this kind of functionality: From a given table, extract the schema. Its certainly not required as part of this issue, but would be nice in general

dawehner’s picture

We talked about this issue on a European critical meeting.

a) the script should be renamed to ..._mysql to make it clear b) throw an exception and render some helpful message if the wrong database type is in use

There has been an issue already to have an API in core to extract a table schema from an actual table, but its quite an effort, so it should not block the upgrade path,
what we care about at the end.

amateescu’s picture

I agree with @dawehner that we need a followup to add some introspection capabilities to the db schema API, I would even say that it should be required in order to close #2447573: [meta] Make sure 8.x - 8.x hook_update_N() testing is possible. Otherwise, the patch looks like an awesome start for a generic "db dump" command!

amateescu’s picture

Cross-posted with #37; I'm also ok with not blocking the critical on the schema introspection issue and +1 to a) and b) :)

dawehner’s picture

I know that catch had an issue at some point, I guess its #301038: Add a cross-compatible database schema introspection API

@amateescu
I think its not required because the actual dump will be usable in every database, so it doesn't block you from testing the update path, you just need to use mysql in order to generate the dump, which is a fine limitation.

jhedstrom’s picture

FileSize
4.1 KB
21.9 KB

This adds the mysql-specific naming, and throws an exception if run from a non-mysql backend. I also added a link to #301038: Add a cross-compatible database schema introspection API to the docblock.

jhedstrom’s picture

FileSize
468 bytes
21.86 KB

Previous patch had an unused SafeMarkup.

dawehner’s picture

Status: Needs review » Reviewed & tested by the community

I think this is a good step forward in order to resolve the actual important critical part of having an upgrade path! This would encourage way more people to jump on Drupal 8.

catch’s picture

My schema testing issue was #1119094: Add a test to verify the schema matches after a database update.

For that issue I had started to review https://www.drupal.org/project/schema which has the functionality we need, but at the time there was no Drupal 7 version and I stopped long before starting on a patch - looked like too much work. There is a Drupal 7 version now though.

alexpott’s picture

Status: Reviewed & tested by the community » Needs work
  1. I've been trying to manually test the patch - it does not look like watchdog data is dumped? Ah... this is because of...
      /**
       * Table patterns for which to only dump the schema, no data.
       *
       * @var array
       */
      protected $schemaOnly = ['cache.*', 'sessions', 'watchdog'];
    

    I guess that makes sense. What happens if we want to test something that changes watchdog or session table structure?

  2. Were missing column charset info set the type column in watchdog... but there are plenty more instances.

    Watchdog type before dumping:

    CREATE TABLE `watchdog` (
      `wid` int(11) NOT NULL AUTO_INCREMENT COMMENT 'Primary Key: Unique watchdog event ID.',
      `uid` int(10) unsigned NOT NULL DEFAULT '0' COMMENT 'The users.uid of the user who triggered the event.',
      `type` varchar(64) CHARACTER SET ascii NOT NULL DEFAULT '' COMMENT 'Type of log message, for example "user" or "page not found."',
      `message` longtext NOT NULL COMMENT 'Text of log message to be passed into the t() function.',
      `variables` longblob NOT NULL COMMENT 'Serialized array of variables that match the message string and that is passed into the t() function.',
      `severity` tinyint(3) unsigned NOT NULL DEFAULT '0' COMMENT 'The severity level of the event; ranges from 0 (Emergency) to 7 (Debug)',
      `link` text COMMENT 'Link to view the result of the event.',
      `location` text NOT NULL COMMENT 'URL of the origin of the event.',
      `referer` text COMMENT 'URL of referring page.',
      `hostname` varchar(128) CHARACTER SET ascii NOT NULL DEFAULT '' COMMENT 'Hostname of the user who triggered the event.',
      `timestamp` int(11) NOT NULL DEFAULT '0' COMMENT 'Unix timestamp of when event occurred.',
      PRIMARY KEY (`wid`),
      KEY `type` (`type`),
      KEY `uid` (`uid`),
      KEY `severity` (`severity`)
    ) ENGINE=InnoDB AUTO_INCREMENT=32 DEFAULT CHARSET=utf8 COMMENT='Table that contains logs of all system events.';
    

    Watchdog table after dumping:

    CREATE TABLE `watchdog` (
      `wid` int(11) NOT NULL AUTO_INCREMENT,
      `uid` int(10) unsigned NOT NULL DEFAULT '0',
      `type` varchar(64) NOT NULL DEFAULT '',
      `message` longtext NOT NULL,
      `variables` longblob NOT NULL,
      `severity` tinyint(3) unsigned NOT NULL DEFAULT '0',
      `link` text,
      `location` text NOT NULL,
      `referer` text,
      `hostname` varchar(128) NOT NULL DEFAULT '',
      `timestamp` int(11) NOT NULL DEFAULT '0',
      PRIMARY KEY (`wid`)
    ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
    
  3. diff --git a/core/lib/Drupal/Core/Command/DbDumpCommand.php b/core/lib/Drupal/Core/Command/DbDumpCommand.php
    index 924a8fc..7b951db 100644
    --- a/core/lib/Drupal/Core/Command/DbDumpCommand.php
    +++ b/core/lib/Drupal/Core/Command/DbDumpCommand.php
    @@ -319,10 +319,9 @@ protected function getTemplate() {
      *
     {{MODULES}}
      */
    +use Drupal\Core\Database\Database;
     
    - use Drupal\Core\Database\Database;
    -
    - $connection = Database::getConnection();
    +$connection = Database::getConnection();
     
     {{TABLES}}
     
    diff --git a/core/modules/system/src/Tests/Update/DbDumpTest.php b/core/modules/system/src/Tests/Update/DbDumpTest.php
    index d82a992..f568290 100644
    --- a/core/modules/system/src/Tests/Update/DbDumpTest.php
    +++ b/core/modules/system/src/Tests/Update/DbDumpTest.php
    @@ -11,7 +11,6 @@
     use Drupal\Core\Command\DbDumpApplication;
     use Drupal\Core\Config\DatabaseStorage;
     use Drupal\Core\Database\Database;
    -use Drupal\Core\Database\Driver\mysql\Connection;
     use Drupal\Core\DependencyInjection\ContainerBuilder;
     use Drupal\simpletest\KernelTestBase;
     use Symfony\Component\Console\Tester\CommandTester;
    

    Some fixes I'd make: remove an used use statement and fix the indentation in the script produced.

alexpott’s picture

Also note cache cids...

`cid` varchar(255) CHARACTER SET ascii COLLATE ascii_bin NOT NULL DEFAULT ''

before dumping...

`cid` varchar(255) NOT NULL DEFAULT '',

After reloading the dump.

alexpott’s picture

Also cache tables are missing their key... KEY `expire` (`expire`)
and comment...

<   UNIQUE KEY `comment_field__uuid__value` (`uuid`),
<   KEY `comment_field__comment_type__target_id` (`comment_type`)

Yep it seems all non primary keys are missing.

larowlan’s picture

Fairly vacuous code review, manual testing later

  1. +++ b/core/lib/Drupal/Core/Command/DbDumpCommand.php
    @@ -0,0 +1,378 @@
    +   * Returns a list of tables, not included those set to be excluded.
    

    %s/included/including

  2. +++ b/core/lib/Drupal/Core/Command/DbDumpCommand.php
    @@ -0,0 +1,378 @@
    +    // @todo this implementation is hard-coded for MySQL.
    

    extreme nitpick™: %s/this/This

  3. +++ b/core/modules/system/src/Tests/Update/DbDumpTest.php
    @@ -0,0 +1,195 @@
    +    foreach ($this->tables as $table) {
    +      $this->assertTrue(Database::getConnection()->schema()->tableExists($table), SafeMarkup::format('Table @table created by the database script.', ['@table' => $table]));
    +      $this->assertIdentical($this->originalTableSchemas[$table], $this->getTableSchema($table), SafeMarkup::format('The schema for @table was properly restored.', ['@table' => $table]));
    +    }
    +
    +    // Ensure the test config has been replaced.
    +    $config = unserialize(db_query("SELECT data FROM {config} WHERE name = 'test_config'")->fetchField());
    +    $this->assertIdentical($config, $this->data, 'Script has properly restored the config table data.');
    +
    

    swoon

larowlan’s picture

And something less vacuous

+++ b/core/lib/Drupal/Core/Command/DbDumpCommand.php
@@ -0,0 +1,378 @@
+    // @todo this implementation is hard-coded for MySQL.
+    $query = $this->connection->query("SHOW COLUMNS FROM {" . $table . "}");
+    $definition = [];
+    while (($row = $query->fetchAssoc()) !== FALSE) {
+      $name = $row['Field'];
+      // Parse out the field type and meta information.
+      preg_match('@([a-z]+)(?:\((\d+)(?:,(\d+))?\))?\s*(unsigned)?@', $row['Type'], $matches);
+      $type  = $this->fieldTypeMap($matches[1]);
+      if ($row['Extra'] === 'auto_increment') {
+        // If this is an auto increment, then the type is 'serial'.
+        $type = 'serial';
+      }
+      // Add primary key information.
+      if ($row['Key'] === 'PRI') {
+        $definition['primary key'][] = $name;
+      }
+      $definition['fields'][$name] = [
+        'type' => $type,
+        'not null' => $row['Null'] === 'NO',
+      ];
+      if ($size = $this->fieldSizeMap($matches[1])) {
+        $definition['fields'][$name]['size'] = $size;
+      }

So it seems like we're going to a lot of effort to reverse generate a schema definition in Drupal's agnostic hook_schema format from an existing table. As pointed out by @alexpott, it is missing a few things like keys other than primary. Given we've already locked this to a MySQL specific script, I don't think there is any benefit in reverse-generating the schema in an agnostic hook_schema format. I think we'd be better off using MySQL's SHOW CREATE TABLE syntax in the dumper, storing that output as a series of SQL statments which can be executed directly in the loader.
It will mean a lot less code.
Thoughts?

jhedstrom’s picture

What happens if we want to test something that changes watchdog or session table structure?

The D7 method for this sort of thing allowed for multiple db dump files to be loaded, so that pattern could be used to populate these tables as needed, rather than rely on them being properly populated when the base db dump script is created.

I'll try to address the remaining feedback in a patch later today.

jhedstrom’s picture

I think we'd be better off using MySQL's SHOW CREATE TABLE syntax in the dumper, storing that output as a series of SQL statments which can be executed directly in the loader.
It will mean a lot less code.

This would make the actual upgrade tests dependent on MySQL, which is currently not the case with this patch--only testing the script itself is dependent on MySQL, while the output can be used on all supported backends.

jhedstrom’s picture

Checking for indexes and collation is going to add quite a bit of code to the DbDumpCommand class that would need to be replicated into the DbDumpTest class if we want tests.

Would it be out of scope for this issue, and 8.0.x, to add a few new methods (getTableSchema() and getIndexes()) to the mysql/Schema class with the same @todo messages currently in this patch?

jhedstrom’s picture

Status: Needs work » Needs review
FileSize
6.54 KB
24.18 KB

Here's a patch that catches coalition info, and indexes. I didn't add tests for the coalition since setting those properties in the dump change the schema already being tested (eg, type = 'varchar_ascii'), so they are already tested in a way. I added tests for the indexes.

I have not refactored as suggested in #52, since I have no idea if we can do that refactoring at this time.

This should also address the nits from above.

dawehner’s picture

@effulgentsia and @dawehner talked about this issue and we wondered why we can't call out to mysqldump to generate the dump?
It would be fine to require the usage of that script IMHO, but it would make things way more safe, as all the usecases would be used ...

dawehner’s picture

#54 is kinda wrong in the assumption that mysqldump is database independent. http://stackoverflow.com/questions/5417386/import-mysql-dump-to-postgres... for example shows that pgsql has issues with it.

Given that the approach is probably much better, because it actually allows us to do cross database types update testing

dawehner’s picture

  1. +++ b/core/lib/Drupal/Core/Command/DbDumpCommand.php
    @@ -151,10 +151,11 @@ protected function getTables() {
       protected function getTableSchema($table) {
    

    What about documenting why we can't use mysqldump?

  2. +++ b/core/modules/system/src/Tests/Update/DbDumpTest.php
    @@ -81,7 +87,7 @@ public function setUp() {
    -    $this->installSchema('system', ['sessions', 'url_alias']);
    +    $this->installSchema('system', ['sessions', 'url_alias', 'key_value_expire']);
    

    What about trying to checkout watchdog as well?

jhedstrom’s picture

FileSize
2.11 KB
24.41 KB

I added a note about why dump utilities cannot create a backend-agnostic sql file.

dawehner’s picture

Status: Needs review » Needs work
FileSize
2.12 MB

Did some manual testing

Observations

  • The lenght of an index is not taken into account, see
      'indexes' => array(
        'menu_link_content_field__link__uri' => array(
          'link__uri',
        ),
      ),

    vs - KEY `menu_link_content_field__link__uri` (`link__uri`(30))

  • We don't get the table COMMENT, that seems totally fine, we don't need it for testing purposes
  • UNIQUE KEY is not taken into account, see
    -  UNIQUE KEY `block_content__info` (`info`,`langcode`),
    +  KEY `block_content__info` (`info`,`langcode`),
  • The order of the primary key seems to be mixed up:
    -  PRIMARY KEY (`entity_id`,`revision_id`,`deleted`,`delta`,`langcode`),
    +  PRIMARY KEY (`deleted`,`entity_id`,`revision_id`,`langcode`,`delta`),
    
jhedstrom’s picture

Status: Needs work » Needs review
FileSize
8.49 KB
25.75 KB

This should address ordering of all indexes (including primary keys), and it adds support for unique keys. It also adds support for index length. I added a few more modules and entity schemas to the test to ensure this.

dawehner’s picture

Status: Needs review » Reviewed & tested by the community

Thank you!

+++ b/core/modules/system/src/Tests/Update/DbDumpTest.php
@@ -179,7 +206,8 @@ public function testScriptLoad() {
+    $this->assertFalse(\Drupal::cache('discovery')
+      ->get('test'), 'Cache data was not exported to the script.');

Just in general, I think its totally fine to have this in one line

Status: Reviewed & tested by the community » Needs work

The last submitted patch, 59: hook-update-n-test-script-2497323-59.patch, failed testing.

Status: Needs work » Needs review
jhedstrom’s picture

Just in general, I think its totally fine to have this in one line

PHPStorm got a little overzealous about code formatting and I didn't notice before I posted the patch :(

dawehner’s picture

Status: Needs review » Reviewed & tested by the community

PHPStorm got a little overzealous about code formatting and I didn't notice before I posted the patch :(

Yeah its not a problem IMHO.

Seriously testbot you are drunken

Fabianx’s picture

RTBC + 1

alexpott’s picture

Status: Reviewed & tested by the community » Needs work

There are still some unexpected differences.

  1. Float sizing looks wrong

    Before

    CREATE TABLE `semaphore` (
      `name` varchar(255) CHARACTER SET ascii NOT NULL DEFAULT '' COMMENT 'Primary Key: Unique name.',
      `value` varchar(255) CHARACTER SET ascii NOT NULL DEFAULT '' COMMENT 'A value for the semaphore.',
      `expire` double NOT NULL COMMENT 'A Unix timestamp with microseconds indicating when the semaphore should expire.',
      PRIMARY KEY (`name`),
      KEY `value` (`value`),
      KEY `expire` (`expire`)
    ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='Table for holding semaphores, locks, flags, etc. that…';
    

    After

    CREATE TABLE `semaphore` (
      `name` varchar(255) CHARACTER SET ascii NOT NULL DEFAULT '',
      `value` varchar(255) CHARACTER SET ascii NOT NULL DEFAULT '',
      `expire` float NOT NULL,
      PRIMARY KEY (`name`),
      KEY `value` (`value`),
      KEY `expire` (`expire`)
    ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
    
  2. For some reason if the simpletest module exsits simpletest_test_id is not even created.
    +++ b/core/lib/Drupal/Core/Command/DbDumpCommand.php
    @@ -0,0 +1,428 @@
    +  /**
    +   * An array of table patterns to exclude completely.
    +   *
    +   * This excludes any lingering simpletest tables generated during test runs.
    +   *
    +   * @var array
    +   */
    +  protected $excludeTables = ['simpletest.+'];
    

    So this excludes tables created by the Simpletest module as well. I guess this is an edge case. And an upgrade path for Simpletest is a bit ridiculous. But it does mean that simpletest would be broken if this script was used on a real site.

  3. Before

    CREATE TABLE `file_managed` (
      `fid` int(10) unsigned NOT NULL AUTO_INCREMENT,
      `uuid` varchar(128) CHARACTER SET ascii NOT NULL,
      `langcode` varchar(12) NOT NULL,
      `uid` int(10) unsigned DEFAULT NULL COMMENT 'The ID of the target entity.',
      `filename` varchar(255) DEFAULT NULL,
      `uri` varchar(255) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
      `filemime` varchar(255) CHARACTER SET ascii DEFAULT NULL,
      `filesize` bigint(20) unsigned DEFAULT NULL,
      `status` tinyint(4) NOT NULL,
      `created` int(11) DEFAULT NULL,
      `changed` int(11) NOT NULL,
      PRIMARY KEY (`fid`),
      UNIQUE KEY `file_field__uuid__value` (`uuid`),
      UNIQUE KEY `file_field__uri` (`uri`),
      KEY `file_field__uid__target_id` (`uid`),
      KEY `file_field__status` (`status`),
      KEY `file_field__changed` (`changed`)
    ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='The base table for file entities.';
    

    After

    CREATE TABLE `file_managed` (
      `fid` int(10) unsigned NOT NULL AUTO_INCREMENT,
      `uuid` varchar(128) CHARACTER SET ascii NOT NULL,
      `langcode` varchar(12) NOT NULL,
      `uid` int(10) unsigned DEFAULT NULL,
      `filename` varchar(255) DEFAULT NULL,
      `uri` varchar(255) NOT NULL,
      `filemime` varchar(255) CHARACTER SET ascii DEFAULT NULL,
      `filesize` bigint(20) unsigned DEFAULT NULL,
      `status` tinyint(4) NOT NULL,
      `created` int(11) DEFAULT NULL,
      `changed` int(11) NOT NULL,
      PRIMARY KEY (`fid`),
      UNIQUE KEY `file_field__uuid__value` (`uuid`),
      UNIQUE KEY `file_field__uri` (`uri`),
      KEY `file_field__uid__target_id` (`uid`),
      KEY `file_field__status` (`status`),
      KEY `file_field__changed` (`changed`)
    ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
    

    The uri column is not the same for some reason.

webchick’s picture

Issue tags: +D8 upgrade path

Since this is a sub-issue of #2447573: [meta] Make sure 8.x - 8.x hook_update_N() testing is possible, which is a D8 upgrade path blocker, tagging as such.

Keep up the fantastic work! :D

webchick’s picture

Issue tags: +blocker

More metadata adjustment. Almost done, I swear...

jhedstrom’s picture

Status: Needs work » Needs review
FileSize
2.82 KB
26.2 KB

This should address the 2 schema issues from #66. I left the simpletest logic in place for now.

Before

CREATE TABLE `file_managed` (
  `fid` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `uuid` varchar(128) CHARACTER SET ascii NOT NULL,
  `langcode` varchar(12) NOT NULL,
  `uid` int(10) unsigned DEFAULT NULL COMMENT 'The ID of the target entity.',
  `filename` varchar(255) DEFAULT NULL,
  `uri` varchar(255) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
  `filemime` varchar(255) CHARACTER SET ascii DEFAULT NULL,
  `filesize` bigint(20) unsigned DEFAULT NULL,
  `status` tinyint(4) NOT NULL,
  `created` int(11) DEFAULT NULL,
  `changed` int(11) NOT NULL,
  PRIMARY KEY (`fid`),
  UNIQUE KEY `file_field__uuid__value` (`uuid`),
  UNIQUE KEY `file_field__uri` (`uri`),
  KEY `file_field__uid__target_id` (`uid`),
  KEY `file_field__status` (`status`),
  KEY `file_field__changed` (`changed`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='The base table for file entities.' |

After

CREATE TABLE `file_managed` (
  `fid` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `uuid` varchar(128) CHARACTER SET ascii NOT NULL,
  `langcode` varchar(12) NOT NULL,
  `uid` int(10) unsigned DEFAULT NULL,
  `filename` varchar(255) DEFAULT NULL,
  `uri` varchar(255) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL,
  `filemime` varchar(255) CHARACTER SET ascii DEFAULT NULL,
  `filesize` bigint(20) unsigned DEFAULT NULL,
  `status` tinyint(4) NOT NULL,
  `created` int(11) DEFAULT NULL,
  `changed` int(11) NOT NULL,
  PRIMARY KEY (`fid`),
  UNIQUE KEY `file_field__uuid__value` (`uuid`),
  UNIQUE KEY `file_field__uri` (`uri`),
  KEY `file_field__uid__target_id` (`uid`),
  KEY `file_field__status` (`status`),
  KEY `file_field__changed` (`changed`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 |

Before

CREATE TABLE `semaphore` (
  `name` varchar(255) CHARACTER SET ascii NOT NULL DEFAULT '' COMMENT 'Primary Key: Unique name.',
  `value` varchar(255) CHARACTER SET ascii NOT NULL DEFAULT '' COMMENT 'A value for the semaphore.',
  `expire` double NOT NULL COMMENT 'A Unix timestamp with microseconds indicating when the semaphore should expire.',
  PRIMARY KEY (`name`),
  KEY `value` (`value`),
  KEY `expire` (`expire`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COMMENT='Table for holding semaphores, locks, flags, etc. that…' 

After

CREATE TABLE `semaphore` (
  `name` varchar(255) CHARACTER SET ascii NOT NULL DEFAULT '',
  `value` varchar(255) CHARACTER SET ascii NOT NULL DEFAULT '',
  `expire` double NOT NULL,
  PRIMARY KEY (`name`),
  KEY `value` (`value`),
  KEY `expire` (`expire`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8
jhedstrom’s picture

I also added semaphore and file_managed to the tests (which in turn required the user module and schema), so there is more test coverage now.

dawehner’s picture

Status: Needs review » Reviewed & tested by the community

The feedback from alex got adressed.
There might be more edge cases, I mean SQL can do alot of stuff, but at least standard install is now safe.

alexpott’s picture

+++ b/core/modules/system/src/Tests/Update/DbDumpTest.php
diff --git a/core/scripts/dump-database-d8-mysql.php b/core/scripts/dump-database-d8-mysql.php
new file mode 100755

new file mode 100755
index 0000000..b50f4c0

index 0000000..b50f4c0
--- /dev/null

--- /dev/null
+++ b/core/scripts/dump-database-d8-mysql.php

+++ b/core/scripts/dump-database-d8-mysql.php
+++ b/core/scripts/dump-database-d8-mysql.php
@@ -0,0 +1,23 @@

So do we want the script to executable?

jhedstrom’s picture

Current state of the scripts directory is mixed in terms of being executable or not.

-rw-r--r--  1 jhedstrom  staff     70 Dec  1  2014 cron-curl.sh
-rw-r--r--  1 jhedstrom  staff     82 Dec  1  2014 cron-lynx.sh
-rwxr-xr-x  1 jhedstrom  staff   4265 Feb 16 10:06 drupal.sh
-rw-r--r--  1 jhedstrom  staff   2986 Feb 16 10:06 dump-database-d6.sh
-rw-r--r--  1 jhedstrom  staff   2626 Feb 16 10:06 dump-database-d7.sh
-rwxr-xr-x  1 jhedstrom  staff    743 Jun 10 12:16 dump-database-d8-mysql.php
-rw-r--r--  1 jhedstrom  staff   6814 Dec  1  2014 generate-d6-content.sh
-rw-r--r--  1 jhedstrom  staff  10250 Feb 16 10:06 generate-d7-content.sh
-rwxr-xr-x  1 jhedstrom  staff  12430 Jun  9 09:38 migrate-db.sh
-rwxr-xr-x  1 jhedstrom  staff   1683 Jun  9 09:38 password-hash.sh
-rwxr-xr-x  1 jhedstrom  staff    609 Jun  9 09:38 rebuild_token_calculator.sh
-rwxr-xr-x  1 jhedstrom  staff  41090 Jun  9 09:38 run-tests.sh
drwxr-xr-x  3 jhedstrom  staff    102 Dec  1  2014 test
-rw-r--r--  1 jhedstrom  staff  21908 Feb 16 10:06 transliteration_data.php.txt
-rwxr-xr-x  1 jhedstrom  staff   3145 Jun  9 09:38 update-countries.sh
jhedstrom’s picture

That being said, I don't know that it really matters.

alexpott’s picture

Status: Reviewed & tested by the community » Fixed

Yep I saw the mish-mash - lets leave it like it is.

Committed 0b4add2 and pushed to 8.0.x. Thanks!

  • alexpott committed 0b4add2 on 8.0.x
    Issue #2497323 by jhedstrom, dawehner, alexpott, larowlan: Create a php...
jhedstrom’s picture

Created #2504167: Consolidate mysql-specific schema functions into \Drupal\Core\Database\Driver\mysql\Schema as a follow-up to my thoughts in #52. No idea if that is feasible in 8.0.x, or even in the 8.x.x cycle at all.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.