[meta] How to deprecate Simpletest with minimal disruption

There are two broad categories of things in simpletest:

1) The parts of simpletest required to run run-tests.sh. These we should keep, because according to @alexpott at DrupalCon 2019 we're going to keep run-tests.sh.

2) The parts of simpletest required for the UI form. These should be moved to testing_ui, where the discussion about removing the UI is ongoing.

If we take the test runner parts and turn them into non-module code that lives under core/tests/, then we can move the rest to testing_ui and there's no more simpletest module, other than a directory full of deprecated files.

How do you determine which parts are necessary for the test runner? You open up run-tests.sh and search for 'simpletest_'. Any function that starts with 'simpletest_' but doesn't go on to say 'simpletest_script_' is something we have to keep from the simpletest module.

Some work along these lines started as long ago as 2015, with issues such as #2641632: Refactor simpletest's *_phpunit_*() (and junit) functions etc. to a class, deprecate and others linked from #2866082: [Plan] Roadmap for Simpletest. Obviously, you know... needs reroll.

So it's not simply a question of deprecating WebTestBase, it's a question of moving our testing infrastructure into a better namespace. If we have no desire to do that, then the simpletest module itself can't be deprecated because it's, you know... the most vital part of our QA chain and stuff. :-)

Has anyone come up with a plan here?

Moving WebTestBase to contrib isn't a bad idea, but that still leaves us needing to extract the test runner code from the simpletest module before we remove it.

Can we accomplish all this before the 8.8.0 release? (Technically feasible, but will a release manager maybe not agree?)

In light of #3073936: [policy, no patch] Policy for Drupal 10 deprecations, here's a plan:

• Locate all hard dependencies of run-tests.sh within the simpletest module. (Excluding WebTestBase.)
• Move those dependencies to a third location (ie, not within simpletest module, not within run-tests.sh). There are some issues already about this. This is moving the test runner support parts out of simpletest, not the WebTestBase framework.
• The test runner parts in simpletest can then be deprecated in 8.8.0 for removal before 9.0.0. There is still time, this is not rocket science, patches already exist. The elements being deprecated are for the test runner itself and are basically almost @internal, not for the testing framework which is used by users.
• The remainder of simpletest module is split into simpletest and testing_ui. Alternately, if we're going to turn simpletest into contrib, we don't need to do this step because we can just have the UI be in the contrib module.
• Whatever's left of simpletest module after that basically supports WebTestBase only, and that can be moved to contrib during D9 dev time. It'll even still be called 'simpletest' so no one has to change the namespace in their tests.

This is the more detailed version of 'turn simpletest into contrib.' Extracting simpletest from core will necessarily involve all of these steps. The question is what bits are deprecated and what bits aren't, and whether that is in the way of performing the extraction.

Only by deprecating the contents of core/modules/simpletest/simpletest.module for removal before D9 can simpletest be turned into contrib after 8.8.0-alpha1, unless I misread what the release manager seems to be saying.

Some details:

Here are simpletest classes used by run-tests.sh:

• Drupal\simpletest\Form\SimpletestResultsForm - #3074433: Decouple run-tests.sh from the simpletest module Move to core/tests?
• Drupal\simpletest\TestBase - #3074433: Decouple run-tests.sh from the simpletest module Won’t move. We'll have to come up with a way for run-tests. sh to detect if this class is present before trying to run simpletest framework tests.
• Drupal\simpletest\TestDiscovery - #2863055: Move TestDiscovery out of simpletest module, minimize dependencies

Here are top-level functions used by run-tests.sh, with functions they call, organized by issue:

In simpletest.module:
#2800267: Turn simpletest_clean*() functions into a helper class covers:

• simpletest_clean_environment()
• simpletest_clean_database()
• simpletest_clean_temporary_directories()
• simpletest_clean_results_table()

#2641632: Refactor simpletest's *_phpunit_*() (and junit) functions etc. to a class, deprecate covers:

• simpletest_run_phpunit_tests()
• simpletest_phpunit_xml_filepath()
• simpletest_phpunit_run_command()
• simpletest_phpunit_xml_to_rows()
• simpletest_process_phpunit_results()
• simpletest_summarize_phpunit_result()

#3074043: Move simpletest.module DB-related functions to TestDatabase, deprecate covers:

• simpletest_insert_asssert()
• simpletest_last_test_get()
• simpletest_log_read()

In simpletest.install:
simpletest_schema() - Move to core/tests, call core/tests version from simpletest.install.

And here's how we'll handle the other methods: #3074043: Move simpletest.module DB-related functions to TestDatabase, deprecate That issue is postponed on decisions from this one.

Updating the comment above.

If you agree with this plan, please update the issue summary.

I'm plus one on this plan, thanks for taking the time @Mile23

In terms of

Drupal\simpletest\TestBase - Won’t move. We'll have to come up with a way for run-tests. sh to detect if this class is present before trying to run simpletest framework tests.

Could we use class_alias for that?

Thanks.

Re #11: Could probably just make it conditional with class_exists().

You know why people fear run-tests.sh? Because of things like this code fragment: https://git.drupalcode.org/project/drupal/blob/8.8.x/core/scripts/run-te... also appearing here: https://git.drupalcode.org/project/drupal/blob/8.8.x/core/scripts/run-te...

yeah, plus 1 for conditional with class_exists

🤕

Added a child issue: #3074433: Decouple run-tests.sh from the simpletest module

Thanks, @larowlan and @catch. Updated IS with the detailed stuff.

First issue in the queue: #2863055: Move TestDiscovery out of simpletest module, minimize dependencies

+1 on schedule.

I'll unpostpone all the child issues.

Thanks for not giving up on this, @Mile23 — it will make Drupal's testing infrastructure a lot simpler to understand! 🥳❤️

Filed this child issue so we can refer to it from deprecation docblocks: #3075490: Move simpletest module to contrib

Also more simpletest deprecation for the curious: #2234479: Deprecate hook_test_* hooks in simpletest

#2863055: Move TestDiscovery out of simpletest module, minimize dependencies is in, yay!

Next up in the crosshairs: #2641632: Refactor simpletest's *_phpunit_*() (and junit) functions etc. to a class, deprecate

#2641632: Refactor simpletest's *_phpunit_*() (and junit) functions etc. to a class, deprecate also landed, congrats :) What's next?

Glad you asked. :-)

Up next: #2800267: Turn simpletest_clean*() functions into a helper class

The Current Hotness:

• #2800267: Turn simpletest_clean*() functions into a helper class
• #2234479: Deprecate hook_test_* hooks in simpletest
• #3074433: Decouple run-tests.sh from the simpletest module

And the easiest of all:

• #3028708: Deprecate simpletest_generate_file()

Just a note that I made a patch on #3075490-5: Move simpletest module to contrib so we can see how much of the simpletest module is currently optional.

So it looks like we missed simpletest_process_phpunit_results() in #3074043: Move simpletest.module DB-related functions to TestDatabase, deprecate, so therefore we have a new child issue: #3079988: Replace, deprecate simpletest_process_phpunit_results()

simpletest_process_phpunit_results() just inserts a bunch of records into TestDatabase, but we need to formally deprecate it and stuff.

#3075490-11: Move simpletest module to contrib successfully deletes most of the simpletest module. It also illustrates that even though we converted all the tests to PHPUnit, they still use fixtures within the simpletest module.

We'll have to figure out where to place all those fixtures and then move them.

Because nothing is ever easy, a new child issue: #3080163: Decouple TestFileCreationTrait::getTestFiles() from simpletest module

This is based on the test fails in #3075490-11: Move simpletest module to contrib

In #3075490-17: Move simpletest module to contrib we discover that many functional tests fail with a message like this:

1) Drupal\Tests\layout_builder\Functional\Update\Translatability\LayoutFieldTranslateUpdateConfigAndStorage::testDisableTranslationOnLayouts
No schema for simpletest.settings

This is because functional tests depend on the existence of simpletest.settings to pass important information between the framework and the fixture. We see that FunctionalTestSetupTrait::prepareSettings() adds parent_profile as a setting, which is then consumed in many places, such as ExtensionDiscovery::setProfileDirectoriesFromSettings().

This leads us to file a new child issue: #3080482: Decouple FunctionalTestSetupTrait from the simpletest module

#2800267: Turn simpletest_clean*() functions into a helper class is now in! woot!

Up next: #3074433: Decouple run-tests.sh from the simpletest module

#3080163: Decouple TestFileCreationTrait::getTestFiles() from simpletest module fixed

Nice. Now we're left with #3074433: Decouple run-tests.sh from the simpletest module and #3080482: Decouple FunctionalTestSetupTrait from the simpletest module and we can remove the simpletest module in #3075490: Move simpletest module to contrib if that's what we want to do.

Thanks for the steady progress here! 🙏

Something to contemplate, written here mostly as notes.

Let's assume that a) We want to be able to remove the simpletest module from core, and b) We want a contrib simpletest module.

We might well want run-tests.sh to be able to run WebTestBase tests. In that case we'll need some way for run-tests.sh to do discovery, and also run the tests.

As we remove the requirement for simpletest module from core, by necessity we lose some of the ability to integrate with the contrib.

I think one way to do this would be to basically replicate the (undocumented) 'execute-test' option from run-tests.sh into a pattern that could be discovered from within the contrib module. Basically run-tests.sh runs itself in a new process with 'execute-test' in order to operate in parallel. This should be a behavior we can discover in contrib modules, sort of like a plugin (though we shouldn't use Drupal's plugin system).

We might add some keys to *info.yml so core could recognize that a given extension has a way to both discover and run it's test type, and report back with test results.

Some loose ends from #3075490-28: Move simpletest module to contrib:

• #3082414: Remove non-fixture references to simpletest module from tests
• #3082416: Remove fixture references to simpletest module from tests

OK, so let's spec out a stub simpletest module that will live in D9 core:

It exists for these reasons:

• So that your site doesn't break if you had simpletest module enabled.
• So that there is an upgrade path to D9 in the same circumstance.
• So that even if you upgraded with an enabled core simpletest module, your WebTestBase tests/CI will fail because it can't find that class, and this will send you to the change records or other documentation.
• So that if you try to re-enable the core simpletest module, it can say, hey, don't do that, use the contrib.

This is all enabled by the notion that if you have a contrib module with the same name as a core module, the contrib will win the battle. Because this is the case, we can tell users to disable the core simpletest, enable the contrib and these restrictions will go away.

This looks like a child issue scoped for D9.

That said, I think we can still work on fixing the fixtures, particularly #3082414: Remove non-fixture references to simpletest module from tests which is mostly already done.

#47

contrib module with the same name as a core module, the contrib will win the battle

We'll probably either need to special case simpletest for the composer facade then. We know that currently, almost no modules have any constraint on simpletest (meaning they just have a dependency on drupal/simpletest)

But the facade is built such that if there is a core module, it gets the namespace drupal/simpletest. and if another module happens to conflict with that, it gets the drupal/
- pattern, in this case drupal/simpletest-simpletest.

So.. we might need a followup in the packages.drupal.org repo to special case simpletest so that contrib can call itself drupal/simpletest and the core module sorta wont have a namespace.

Follow-up issue #3084354: Deprecate --browser option from run-tests.sh coming off of #3074433: Decouple run-tests.sh from the simpletest module

#3074433: Decouple run-tests.sh from the simpletest module is at a bit of an impasse.

We need the follow-up mentioned in #46, #47, #48 in order to provide clarity on how to move forward on #3074433: Decouple run-tests.sh from the simpletest module

This is an important major plan, because it is allows up to clean up significant technical debt in core and remove legacy code that's cumbersome to maintain. It's especially important right now, since we have just a few weeks left to finalize the last steps before 8.8.0's alpha and beta deadlines. If that doesn't happen, then we'll be stuck with it until Drupal 10.

That said, as per #3074433-89: Decouple run-tests.sh from the simpletest module comments #89 and #91, it doesn't quite meet the requirements for a critical issue. Critical issues are generally those that render a site unusable, cause data loss, race conditions, security issues, broken upgrade paths, etc. In the case of SimpleTest, deprecating or removing it from core is not a blocker to releasing Drupal 9 -- it's just something that would be really valuable to do in terms of maintainability, and something that we only have the opportunity to do in a new major version.

So I think this issue, and all of its children, should be major rather than critical. They're still hugely valuable to complete before Drupal 9 insofar as it's possible, because the less legacy code we have to maintain for the next three years, the better. Nonetheless, the Drupal 9 release date would not slip if these issues were not done in time for the release deadline. We can consider it a "should-have" for the Drupal 9.0.0-alpha1 issues.

Thanks!

#3074433: Decouple run-tests.sh from the simpletest module is in, so we have accomplished the major goals here of rearranging the technical debt so that other decisions can be made about the future of the simpletest module in core.

The remaining tertiary follow-up issues are not essential to accomplishing this goal. Which we just accomplished. :-)

The next course of action is to come up with a plan for creating a contrib simpletest module, and assuming that's possible, also reducing core's simpletest module to a stub.

• #3075490: Move simpletest module to contrib
• #3084493: Fully deprecate and prepare for removal of SimpleTest module

Thanks, folks.

This is done IMO. There’s still cruft here and there to cleanup but I think that can live with individual issues.

I read the comments concerning followups, #47 - #51, and I've concluded that the followups were either made or the work a potential follow-up was needed for has been completed. I am sure someone will correct me if I am wrong. Therefor, I am removing the tag.