[meta] Make sure 8.x - 8.x hook_update_N() testing is possible

Here's the dump script, works.
Still needs a generate script.

+++ b/core/scripts/dump-database-d8.php
@@ -0,0 +1,95 @@
+  $schema = drupal_get_schema();

Does drupal_get_schema still return everything? It does not look like it does. We need to get entity schema too.

Migrate does something similar direct from the database, not sure if that is any help here: core/scripts/migrate-dump-d6.sh

@catch I'm not sure that we have to deal with YAML config files. Config is now in the DB so we just have to export the config table - no?

@alexpott, @webchick, @effulgentsia, and I discussed this issue awhile back and agreed that it is critical. We need to be able to test a full site upgrade and add regression tests for hook_update_N() in order to support an upgrade path.

Only test one update hook at a time for now. So a kernel test where we create the table(s) with the old schema definition, add some data, manually call the hook_update_N() and verify the result.

If we went this route, the task would be to insure that each issue that added a hook_update_N included a test, so then this issue would be about making the base test class to facilitate those?

Tagging. I feel like if we got a good list of todos here someone could bang it out.

I started looking into this and it seems that the new dump script added for migrate in #2469623: Process for creating migration source DBs for automated tests can be used as a base for a more generic database dump script for upgrade path testing.

However, it uses CLI arguments and options, and it will quickly grow into a big and hard to maintain script (much like run-tests.sh), so I was thinking that we could use symfony's Console component for it.

One advantage is that php code (the Commands) will be in .php files, so the .sh files will contain just a few lines needed to invoke those commands. The DrupalCI project is also using this component.

A good task for 8.1 can be to convert run-tests.sh to it.

Disadvantages: I don't see any :)

Adding the new shiny tag to get the attention of framework managers.

I've used the Console as an extension / configuration mechanism in a couple of Symfony applications, and agree that it could be useful here. Another alternative could be the Symfony Fixtures bundle, but it might be too closely tied to Doctrine and perhaps not flexible enough to be used here.

Discussed a bit in IRC with @alexpott and he pointed me to some previous discussion about it: #2242947: Integrate Symfony Console component to natively support command line operations

So just to be perfectly clear, I'm not proposing anything like that since it's definitely 8.1 material. I just want to write the code for the dump script in a Command so we can take advantage of the clean API for input options and arguments, nicely formatted output, and, why not, testability of that command. Then have core/scripts/database-dump.sh as three lines of code that executes the command, that's it :)

+1 to using the console framework here:

https://github.com/msonnabaum/XHProfCLI/blob/master/src/XHProfCLI/Aggreg...

is a very good example of how clean such a component looks.

The advantage is that then the classes, etc. are unit testable, configurable, etc.

TODO:

1. Create a script to dump the necessary database tables
2. Write a test to use the output of that script
3. Proof that hook_update_N is now testable.

--

Technical direction has been provided, so removing tag. Still needs approval by framework managers to allow using clean commands via Symfony console for dumping the database. (Technically the dumper could even be its own project as only the tests are needed.)

I just want to write the code for the dump script in a Command so we can take advantage of the clean API for input options and arguments, nicely formatted output, and, why not, testability of that command. Then have core/scripts/database-dump.sh as three lines of code that executes the command, that's it

With this approach though, you'd need to install symfony/console to run the command though right? Because the command execute method has type hinting for the Input/Output interfaces etc? Maybe i'm missing something...

@benjy, not sure what you mean by "install symfony/console". If it's about adding it to core's composer.json then yes, that would be the start :)

I also posted a link in #11 that shows how the two scripts, the command and .sh runner, would look like. Maybe it was too hidden in the text so here's the full url: http://symfony.com/doc/current/components/console/introduction.html#crea...

@benjy, not sure what you mean by "install symfony/console". If it's about adding it to core's composer.json then yes, that would be the start :)

That's exactly what I meant, I thought you were saying somehow we could write it as a Command without symfony/console which is why I was utterly confused. Great, big +1 for adding symfony/console to core!

We should clearly communicate that adding symfony/console does NOT add a console to Drupal but rather just add a console for a specific command as dependency.

@dawehner, that's what I tried to communicate in this issue, see #13 :)

There's a lot of discussion about the script to generate a db for testing, but what still isn't clear to me are the other 2 tasks:

2. Write a test to use the output of that script
3. Proof that hook_update_N is now testable.

Would the script be used for testing every new update hook added to core in the future, or is it intended to just be used by a single test that ensures update hooks work? If the later is the case, we may be able to test update hooks in a functional test without a starting database by using the State API in a test module to alter the schema throughout the course of the test.

IMO we'd want parity with Drupal 7's update testing, so I believe that means a separate test for each update hook, as well as a test to make sure the update process itself works.

re #21 that makes sense--I actually didn't realize we had this for all update hooks in 7.

I opened #2493807: Add symfony/console component to core as a first step.

That one got committed, so using Console for this is now unblocked.

Since there's already a start and much discussion of the db-generation script here, it probably makes sense to continue using this issue for that. I'm working on that now.

I've realized the script will need a child issue (as we can't close this until all tasks are done). I've added #2497323: Create a php script that can dump a database for testing update hooks, with a suggested commit message for larowlan since he worked on the script in #1.

#2493807: Add symfony/console component to core has landed.

Updating the issue summary. I added #2498625: Write tests that ensure hook_update_N is properly run after talking with Webchick in IRC since we can start on the really simple tests without the script.

Explicitly noting that this one is postponed on those other two.

Massive issue summary update to try and explain what we're doing here. Might need double-checking for accuracy.

#2497323: Create a php script that can dump a database for testing update hooks has landed, changing title.

Tentatively testing our new "Plan" issue type now that #2504039: Adjust critical issue counter/link at /drupal-8.0/get-involved to account for "Plan" issue category is in.

All the child issues are now resolved.

Congrats!

3. Proof that hook_update_N is now testable.

I guess we should try it out on an actual issue

Excellent!

Hiding the file so that testbot doesn't keep marking this issue NW.

What's left to do before this can be marked fixed?

@effulgentsia
Comment #38 describes it pretty good. There should be a real usecase for hook_update_N. Once we have proven that, we should be save here.