Add new 10.3.x database dump fixtures, without modules deprecated for removal in 11.x

Changing parent to the issue tracking removal of all the extensions.

Just trying to grasp what we're trying to do in here: The IS talks about the 9.4.4 fixture, #3 tlaks about a 10.3.0 fixture.

Are we going to keep the 9.4.4 fixture around _and_ add a 10.3.0 one?
I would imagine we can loose the 9.4.4 one, since we don't support jumping from 9.x to 11.x?

@Spokje: AFAICT the issue summary is just describing the current, painful situation. I do not think the proposal is to keep a D9 DB dump.

Thanks @Wim Leers, I suspect as much, just wanted to make it clear (and yes, I also felt the pain of fixture-fiddling)

Does it mean that 10.3 should ship both 9.4 and 103.alpha fixtures?

So we know there's pain when changing fixtures, so we combining the changes so there's less pain.

Now what if a core module doesn't have its removal issue committed on time for the release of 10.3.0?

We then have to go through the pain again when changing the fixture painfully made in 10.3.0 in 10.4.0?
Could we move the removal to 12.0.0, so there's less pain?

(Not sure about if there's already specific rules/documentation on this and/or ramifications, just curious and pretty much done with the pain after the gazillion changes we did on fixtures during D9 => D10.)

I came to conclusion to replace module with obsolete stub info.yml file working on #3432134: Remove Action UI module
Removal of the stubs (help_topics and sdc ATM) could use separate issue after fuxtures update

Moreover in case of action module we have no hook to disable it to allow end-user to consider
so it's one line removal from extension.list which could be done here to unblock removal issue

I have avoided making these dumps in the past and I would like to do them this time. I've got a basic script working to do the task. I just need to test the resulting database and have it run on all the source fixtures. So I am assigning this to myself.

Thank you @quietone!

I made a separate issue for the removal of hooks etc, #3439769: Remove all the pre-10.3.0 updates hooks. I found it easier to keep that separate from the changes here to actually use the new dumps. There is a comment in that issue about some files that can probably be removed.

So, I merged that issue and updated tests and comments. As we know that will result in a large MR, and it is 124 files + 143980 − 8136.

One thing to mention is that when making the new dumps I found what I believe is an error in the 'filled' one. In that database node/1 is an Article content type and part of a book. However, the book configuration only allows content type of book to be in book. I resolved that by retaining node/1 and deleting the row from the book database table.

Testing 7403 and there was a merge conflict around

deleted by us:   core/modules/views/tests/src/Kernel/ViewsConfigUpdaterTest.php

So resolved that. But still appears to have test failures now, so moving to NW for that as I don't want to step on anyone's toes.

Applying 7403 to #3422600: Remove Tour module to see if that works for a removal though

But the tests that are failing should they be deleted? Since the db dump has correct array keys

@catch can you rerun the javascript tests on 7404? I can't because of gitlab roles.

Okay so opened separate branch of just the DB dumps for 11.x 7426

The review missed that the steps in #14 are not complete. Also, in #22 the catch asked that the script I used was shared here.

On reviewing everything I found that I missed removing some updates over in #3439769: Remove all the pre-10.3.0 updates hooks. Also, I forgot to change the script to not use drush, which currently doesn't install on 11.x. So, the script pauses and I use the UI to uninstall modules and run updates. With all that fixed these two issues are working as expected for 11.x

MR 7403 - This has the new dumps, changes needed for UpdatePathTestBaseTest to work with the new dumps, and the changes from 3439769 which does all the work to remove hooks.

Curious how it affected locale's tables, for tracker removal there's 2 strings

@andypost, I don't know. How do I find out? Is this different from other modules that have translations? What are the strings?

Specifically here all remains
https://git.drupalcode.org/project/drupal/-/blob/4cd72d0c4a9132bf2540e1f...

Hmm, I have not seen any work like that on the locale tables in the previous extension removal issues. AFAIK, the uninstall process does not alter the locale tables. I checked with a fresh install of Umami and that is true. I installed Umami, then install tracker and comment. After the table size for locale_location and locale_source increased. I then uninstalled those modules and the table size remained the same.

Maybe I am misunderstanding, I should be asleep instead of at the keyboard.

Is there a reason to ship extra unused strings? it will slow down every upgrade test

Tests are passing. This adds dumps for 10.3.0 and updates the two tests that use \Drupal\FunctionalTests\Update\UpdatePathTestBase.

And this is the script I've been developing. It requires ddev and relies on my setup and commands.

@quietone would it be useful to commit the script too? Like you did for the sub tree split

It requires ddev and relies on my setup and commands.

No, I would not commit it because it will only work on my setup.

I can verify the following modules are removed from tests.

Actions
Actions UI
Book
Forum
Tour
Tracker

I see that the new database is added, and updated in code. Someone will still need to validate the references to the modules are removed from there also.

I recreated the dumps locally and diff the output (fun) and did not see anything too surprising. There are some new updates for 10.3.x that change things but all that means is that either we need to recreate them or we need to leave the updates in.

Committed 9fdd5ef and pushed to 11.x. Thanks!
Committed 6e44daa and pushed to 10.3.x. Thanks!

I did not use the 10.3.x branch because the dumps there seem to be flawed and contain reference to an update on 10.3.x that has since been removed. I just manually backported the dumps (and only the dumps) to 10.3.x.

I think we should create a follow-up to ensure we have a test on 10.3.x using the new dumps just to make sure everything is working as expected.

Needs followup for last paragraph in #51

it need one more follow-up for missed statistics settings

->values(array(
  'collection' => '',
  'name' => 'statistics.settings',
  'data' => 'a:2:{s:19:"count_content_views";i:0;s:15:"display_max_age";i:3600;}',
))

There are actually lots more remaining elements to remove, beyond statistics: there are still remainders from aggregator, book, color, forum, and statistics. Most of them are config data and translations.

Translations are supposed to stay, to represent sites that been upgraded from earlier days, the only reason to clean-up is to speed-up of upgrade tests

Yes, there is 'left over' data in the dumps. However, as catch explained in #36, these dumps are "supposed to mimic what happens when you update real sites over time in the hope that this will pick up problems." They are not created from a fresh install with data added. They are created by first installing the previous dumps on 9.4. The Issue Summary has more details about the process. The point is that these dumps are to reflect what really happens when an extension is uninstalled, not what we think should happen. So, the fact that data seems to be left behind is not something to be changed here. This is working as intended.

If you think the uninstall process should be removing more date then that should be discussed in a separate issue.

At least both #56 and #58 needs work for follow-up

regarding #58 - maybe statistics was not uninstalled properly? otherwise it's not clear why it's config is not removed

#58, #62. I discussed this again with catch and we both agree the dumps are correct. All the modules were uninstalled correctly. The 'statistics_settings', and another settings for the removed extensions are not in 'config', they are in 'config_snapshot'. That make sense so one can do a diff on configuration settings. Or, should a new config_snapshot should be made when an module is uninstalled. That would be a discussion for a separate issue.

@catch. thanks for adding the followup.