Move simpletest module to contrib

Lets verify the composer namespaceing and ensure we dont have unworkable snafu's for requirements on 'drupal/simpletest'

re #4 I did do some verification, and reported my results to slack, which isnt useful due to its ephemeralness.

In any case, the number of projects that declare a dependency on drupal/simpletest is *extremely* minimal, and really, we ought to be able to make contrib own the drupal/simpletest namespace since I couldnt find any examples that had any sort of constraint whatsoever on it.

+1 to move this out, as this wont affect p.d.o, but we'll probably need a postponed followup to do a namespace switcharoo in the packages.drupal.org issue queue.

So if we don't do anything special, 9.x-compatible modules with simpletests should be able to use.

test_dependencies:
 - drupal:simpletest

in info.yml. That seems acceptable but maybe not ideal since you could probably go a while without realising those tests aren't running.

The other way would be for DrupalCI, when it looks for test_dependencies, to always add simpletest if it's not there for you (which would need to stay in place until either a further announcement, or 9.x EOL).

We should probably update the IS with what the required featureset for the contrib module is, e.g.,

• It will include the legacy UI test runner, which should be installable on simplytest.me in order to test patches without setting up a dev stack.
• It will include SimpleTest UTB, KTB, and WTB base classes so that modules that have not yet converted their full test suite to PHPUnit can simply declare a dependency on simpletest.
• Will it include other base classes that are still included in D8 core currently? E.g. MenuWebTestBase or FileFieldTestBase.

Ideally the contrib module should be as close to the core module as possible to minimize disruptions, but there are some things (like the above) that we want to make sure are not broken or changed before D10. And perhaps the D10 version might remove the simpletest API base classes themselves, but retain the web UI test runner, or something.

Just adding some list markup to the IS.

So was playing around with this in a side issue, and this is what we would need to get this green without further other patches.

The interdiff is what would would need to add to the code_changes_only.txt patch from #28

In short, some small test changes that target simpletest, add a minimal simpletest stub module with an info file that sets it to hidden and the settings config that the update tests use.

Just a note I updated #3084493: Fully deprecate and prepare for removal of SimpleTest module after talking to alexpott - it should not be necessary to have a stub module since people who want to keep using simpletest can use the contrib module. We do need stub modules when we're actually removing a core module outright but that's a different case. Production sites should not have simpletest module enabled anyway.

So I've been trying to get the tests to go green when there is no .info.yml file any more, but this all turns out very hacky.

The problem here is that our current update DB dumps have Simpletest enabled, so all the tests expect the module to remain enabled. When we delete the entire module, we can't uninstall it anymore, and this where things get tricky. There is no good way to 'uninstall' a module when it no longer exists on the file system.

So this is a patch with the absolute minimum stub, a info.yml file. In order to get the upgrade tests to pass, we need an update function to uninstall the Simpletest module, to properly remove the simpletest.settings config.

So the whole 'getting update tests to play nice' is also relevant to #3084493: Fully deprecate and prepare for removal of SimpleTest module

Missed one....

Update tests are a good reason to keep a stub module for now.

Once #3087644: Remove Drupal 8 updates up to and including 88** lands we could remove simpletest from any of the remaining update test database dumps, but until then stub seems easiest.

Is someone planning to write an eulogy? To be very clear I am absolutely not against doing this, the time has certainly come, I am just nostalgic as always. Simpletest has helped Drupal grow from zero tests to almost test driven development. I feel a few words somewhere needs to be said (certainly not by me).

Added release note snippet

We now have a dev release of SimpleTest for D8/D9, see #3076422: [contrib] Move simpletest module to contrib for D9

https://www.drupal.org/project/simpletest/releases/8.x-3.x-dev, this obviously only works with this patch applied to core

#48 gave a laugh; thanks @Lendude.

That said, we do need an actual release note snippet. :) And an actual change record that explains what exactly modules that want a UI testrunner or still have SimpleTest tests should do. Maybe the eulogy release note can go in the change record.

Taking the ability to run Simpletests out now would raise the barrier for D8 -> D9 migrations for contrib and from all I've seen and heard, this is something we want to avoid.
I think we need to operate on the assumption that D9 will be able to run simpletest, but somebody correct me if I'm wrong in my assumptions :)

Ah re-read, its 'move the ability to run the tests to contrib' vs 'having the ability in core' you are talking about? No opinion on that, whatever is easiest for the infra team I guess?

@Mile23

So the next question is:

Should Drupal 9 be able to run simpletest-based tests?

Whether or not it should, is probably a decision and patch that should be separated from this issue about removing the module itself into a follow-up. Because otherwise removing the module is going to be dependent on that decision and/or any DrupalCI plugin we might need to write.

I advocate this approach, because I'm not sure who actually needs WebTestBase

There are several dozen, contrib modules with unconverted WebTestBase. Those modules may not need to do much to have a working runtime Drupal 8 contrib module, meaning that converting the tests would outweigh the effort of the actual module port - because test conversion is one of the least straightforward changes to make. While they'll face the same issue when Drupal 10 comes around, Drupal 9 is the first release where we've promised a continual upgrade path, which hopefully includes most contrib modules being ported by 9.0.0, so minimising the amount of work people need to do seems worth the effort here.

This feels blocked on getting a stable contrib release of SimpleTest (see #3093621: [META] Plan to get a stable D8/D9 release of the SimpleTest module) which is currently blocked on #3077428: Modules that have a composer namespace of project-modulename cannot be patched

I was asked to chime in here from a product management POV, but can't seem to find a concise, objective write-up of the pros/cons and concerns that were raised in whatever other thread that was about removing the UI. Could someone passionate about this issue please help provide that?

According to @Gábor Hojtsy this accounts for a quarter of the remaining deprecations in D9 core. We either need to complete the plan (maybe with a stub module and manual testing of the UX of the upgrade) or un-deprecate it.

We also still need a real release note. :)

If possible, can someone summarize the user impacts of the change, pros and cons, in "three-tweet size"? From the perspective of a module owner. That's the information the product managers have requested to sign off. Thanks!

Heh, "three-tweet size" is a bit of a rough guideline, but basically product managers are interested in impacts of changes. I'm scanning for things like:

- How does this impact core developers?
- How does this impact module developers?
- How does this impact the usefulness of GUI tools like simplytest.me?
- How does this impact sprints (both those participating for the first time and those leading)?

I know from reading this like 11,000 years ago that there were a bunch of concerns raised in these general areas back in the day. What were those specific concerns? How does the current plan address them? If it doesn't address them, what's the workaround, and is it ok? etc.

It is going to take me multiple days over multiple weeks if I need to read each comment individually of multiple 100+ comment issues in order to glean this information. I'm hoping one of the folks who's already deep in this can gather that type of information and answer those types of questions more easily.

Here's a stab at answering those questions, extra details and/or corrections welcome.

1. How does this impact core developers?

Core developers will no longer be able to run tests using the Simpletest UI. However, Simpletest UI output for phpunit is not very helpful. phpunit is easier to run via cli than simpletest was (less command line options etc.)

2. How does this impact module developers?

Modules with simpletests will be able to use the simpletest contributed module to run tests. Either transparently (DrupalCI adding simpletest module to every test run (requires DA help), or via dev-dependencies and a line or two in drupalci.yml (should not require DA help). Support for this is not yet complete yet however, and likely depends on a final decision being made here. Also if we remove simpletest support from run-tests.sh there'll need to be an equivalent script added to the contrib module (or we'll need to keep support for the contrib simpletest module in core's run-tests.sh until such a contrib script is available).

3. How does this impact the usefulness of GUI tools like simplytest.me?

Should not impact simplytest.me, it's not reliant on simpletest any more.

4. How does this impact sprints (both those participating for the first time and those leading)?

Would expect sprints to have been recommending phpunit and the cli runner for some time.

http://grep.xnddx.ru/search?text=extends+DrupalWebTestCase&filename= isn't too crazy (5 pages of results, a few known projects in there). OTOH, that's only grepping D8 contrib. D7 contrib is likely to have tons more, and not all of those modules (let alone custom modules on 66% of all Drupal sites that are still on D7) are ported to D8 yet. We really do not want to add more stumbling blocks in getting people off D7 and onto D9, if we can help it.

I'm a little confused about #3093621: [META] Plan to get a stable D8/D9 release of the SimpleTest module ... if the answer to the above concern is "well now you just need to add "dependencies: simpletest" to your info file" then that seems like not a big deal. Is that it? Or is there some dependency on the contrib module doing something first?

Regarding sprinting, I'm not sure what the "official" resources are for sprinting these days are, but a google search lead me to http://drupalladder.org/lesson/f6499034-7520-2db4-6522-c32c8429e2d7 which is pointing to https://www.drupal.org/docs/7/testing which doesn't seem to imply at all that this is not the official recommended way to be writing tests. "During the Drupal 7 development cycle SimpleTest has become an integral part of the workflow, and it has been moved into core as the Testing module. It has significantly enhanced the way core is developed and made it possible to make major API enhancements with confidence."

Hacking the URL to https://www.drupal.org/docs/8/testing instead provides a buffet of options... one of which is converting SimpleTests to PHPUnit, but nothing once again stating outright that SimpleTest isn't the recommended way to be doing things...

Anyway. Point being, in order to do this, we likely need some docs updates in quite a few places first.

I reviewed http://grep.xnddx.ru/search?text=extends+DrupalWebTestCase&filename= and there are extremely few D8 projects using DrupalWebTestCase. First of all, we can ignore .test files, these are leftovers from D7, they are not picked up by D8. Page 1 has ableplayer, baidu_analytics, baidu_tongji and bootstrap_menu_items projects having .php files which look like legit test cases. Page 3 has session_limit, page 4 has typogrify and cloudwords and it seems this is it. I count seven contrib projects. Just a data point.

As someone who maintains a lot of contrib and has been working very actively on getting contrib ready, I'm a clear +1 on removing it from that perspective.

Points against removing:

* Yes, you did look for the wrong class, there are many more usages, around 43 pages on the search, https://dev.acquia.com/drupal9/deprecation_status/errors reports 575 usage.

* There are actually even more, as quite a few modules have base classes that hey extend from. webform for example had only a single deprecation message on that but actually had hundred or so subclasses, but that is an extreme examples.

* Yes, converting tests can be by far the hardest task in regards to deprecations, when they need to be converted to javascript tests for example, as our BC layers there don't really work. I've spent hours on single test classes.

But, We've also already made a lot of progress. According to https://dev.acquia.com/drupal9/deprecation_status/graphs, the number went down from 650 to 550 already, and that doesn't count a lot more that are available in dev releases and patches and "just" need to be committed. I'd say including patches that's surely another 100 and I wouldn't be surprised if it's twice that. And those base classes are often not actually removed just like core didn't remove them in case other modules still use them.

And these kind of stats are always a bit misleading as we tend to forget that contrib is basically wild west with a lot of unmaintained and barely used modules. Quite a few of the deprecations overall are from modules that were never fully ported to D8 in the first place like api.module.

I believe that sooner or later, pretty much any module out there is going to have a D9 patch. A lot of people are actively looking for modules to provide patches for (counting 13 pages of active issues with the "Drupal 9 compatibility" tag) and the only blocker will be for a maintainer to commit and release that.

That said, I agree we do need a way how the simpletest-as-contrib would actually work on DrupalCI or we could just as well drop it entirely. That's IMHO the main concern here, not contrib on its own. I think it would be possible for it to provide its own run-tests.sh that DrupalCI can run. I think it would be useful to have that detected automatically (grep for WebTestBase, if found, install simpletest and try to run those tests too).

Simpletest (contrib) currently does not work, for a few reasons. The main one being that there's isn't (or shouldn't be) a way to run WTB tests from run-tests.sh once the simpletest module is gone from core. So therefore there's no way to know if the contrib module can pass its own tests.

I think we could do this in three steps.

Leave support for simpletest in the core run-tests.sh, while removing simpletest module from core.

Work on the contrib equivalent to run-tests.sh and ensure it works with drupalci

Remove simpletest support from core's run-tests.sh

Theoretically these could all happen prior to 9.x's release.

If somehow we got stuck with simpletest support in run-tests.sh, this is not the worst thing in the world, we can try to remove it again for Drupal 10 (possibly with no replacement if 500+ WebTestCase has got a lot closer to zero).

@webchick it's less a dependency on the contrib module doing something, than supporting the contrib module in DrupalCI. Although until you can actually see contrib SimplesTest results in DrupalCI we can't really prove the contrib module works. But yes once it's up and running there's either nothing, or nearly nothing, for contrib modules to do to get their old SimpleTests running.

Unless things have changed since I was last involved in this, there isn't a lot of momentum on making the contrib simpletest happen. I have no interest in maintaining it. @larowlan mentioned being interested, but I'm not sure whether that's still the case.

There's not much momentum, but I don't think there will be until there's a clear decision here (although that decision would be easier if there was more momentum), it's a bit chicken and egge.

My 2 cents: actually there are still some interdependencies between Simpletest and the new core PHPUnit testing that would make a clear cut a bit difficult. IMHO we should try to land #3075608: Introduce TestRun objects and refactor TestDatabase to be testable (see there how the interaction with the simpletest database schema for test results is spread aroud, and how the currently-in-core Simpletest is affected by the patch), and possibly #2905007: Allow run-tests.sh to report skipped/incomplete PHPUnit tests, before a contrib Simpletest could be on its own.

I stumbled into this question again while reviewing the remaining @deprecated APIs. 26% of all remaining @deprecated uses in core are now in simpletest module. These ones:

core/modules/simpletest/simpletest.api.php: * @deprecated in drupal:8.6.0 and is removed from drupal:9.0.0. Convert
core/modules/simpletest/simpletest.api.php: * @deprecated in drupal:8.8.0 and is removed from drupal:9.0.0. Convert your
core/modules/simpletest/simpletest.api.php: * @deprecated in drupal:8.8.0 and is removed from drupal:9.0.0. Convert your
core/modules/simpletest/simpletest.api.php: * @deprecated in drupal:8.8.0 and is removed from drupal:9.0.0. Convert your
core/modules/simpletest/simpletest.module: * @deprecated in drupal:8.8.0 and is removed from drupal:9.0.0. Use
core/modules/simpletest/simpletest.module: * @deprecated in drupal:8.8.0 and is removed from drupal:9.0.0. Use
core/modules/simpletest/simpletest.module: * @deprecated in drupal:8.8.0 and is removed from drupal:9.0.0. Use
core/modules/simpletest/simpletest.module: * @deprecated in drupal:8.8.0 and is removed from drupal:9.0.0. Use
core/modules/simpletest/simpletest.module: * @deprecated in drupal:8.4.0 and is removed from drupal:9.0.0. PHPUnit test
core/modules/simpletest/simpletest.module: * @deprecated in drupal:8.8.0 and is removed from drupal:9.0.0. Use
core/modules/simpletest/simpletest.module: * @deprecated in drupal:8.8.0 and is removed from drupal:9.0.0. Use
core/modules/simpletest/simpletest.module: * @deprecated in drupal:8.8.0 and is removed from drupal:9.0.0. Use
core/modules/simpletest/simpletest.module: * @deprecated in drupal:8.8.0 and is removed from drupal:9.0.0. Use
core/modules/simpletest/simpletest.module: * @deprecated in drupal:8.8.0 and is removed from drupal:9.0.0. Use
core/modules/simpletest/simpletest.module: * @deprecated in drupal:8.3.0 and is removed from drupal:9.0.0. Use
core/modules/simpletest/simpletest.module: * @deprecated in drupal:8.3.0 and is removed from drupal:9.0.0. Use
core/modules/simpletest/simpletest.module: * @deprecated in drupal:8.8.0 and is removed from drupal:9.0.0. Use
core/modules/simpletest/simpletest.module: * @deprecated in drupal:8.8.0 and is removed from drupal:9.0.0. Access the
core/modules/simpletest/simpletest.module: * @deprecated in drupal:8.8.0 and is removed from drupal:9.0.0. Access the
core/modules/simpletest/simpletest.module: * @deprecated in drupal:8.8.0 and is removed from drupal:9.0.0. Access the
core/modules/simpletest/simpletest.module: * @deprecated in drupal:8.8.0 and is removed from drupal:9.0.0. Access the
core/modules/simpletest/simpletest.module: * @deprecated in drupal:8.8.0 and is removed from drupal:9.0.0. Use
core/modules/simpletest/simpletest.module: * @deprecated in drupal:8.8.0 and is removed from drupal:9.0.0. Use
core/modules/simpletest/simpletest.module: * @deprecated in drupal:8.8.0 and is removed from drupal:9.0.0. Use
core/modules/simpletest/simpletest.module: * @deprecated in drupal:8.8.0 and is removed from drupal:9.0.0. This function
core/modules/simpletest/src/AssertContentTrait.php: * @deprecated in drupal:8.6.0 and is removed from drupal:9.0.0. Use
core/modules/simpletest/src/AssertHelperTrait.php: * @deprecated in drupal:8.4.0 and is removed from drupal:9.0.0. Use
core/modules/simpletest/src/BlockCreationTrait.php: * @deprecated in drupal:8.4.0 and is removed from drupal:9.0.0. Use
core/modules/simpletest/src/BrowserTestBase.php: * @deprecated in drupal:8.1.0 and is removed from drupal:9.0.0.
core/modules/simpletest/src/ContentTypeCreationTrait.php: * @deprecated in drupal:8.4.0 and is removed from drupal:9.0.0. Use
core/modules/simpletest/src/Exception/MissingGroupException.php: * @deprecated in drupal:8.8.0 and is removed from drupal:9.0.0. Use
core/modules/simpletest/src/InstallerTestBase.php: * @deprecated in drupal:8.6.0 and is removed from drupal:9.0.0.
core/modules/simpletest/src/NodeCreationTrait.php: * @deprecated in drupal:8.4.0 and is removed from drupal:9.0.0. Use
core/modules/simpletest/src/RandomGeneratorTrait.php: * @deprecated in drupal:8.1.1 and is removed from drupal:9.0.0. Use
core/modules/simpletest/src/RouteProvider.php: * @deprecated in drupal:8.6.0 and is removed from drupal:9.0.0. Use
core/modules/simpletest/src/SessionTestTrait.php: * @deprecated in drupal:8.1.1 and is removed from drupal:9.0.0. Use
core/modules/simpletest/src/TestBase.php: * @deprecated in drupal:8.8.0 and is removed from drupal:9.0.0. Instead,
core/modules/simpletest/src/TestBase.php:   * @deprecated in drupal:8.8.0 and is removed from drupal:9.0.0. Use
core/modules/simpletest/src/TestDiscovery.php: * @deprecated in drupal:8.8.0 and is removed from drupal:9.0.0. Use
core/modules/simpletest/src/TestServiceProvider.php: * @deprecated in drupal:8.6.0 and is removed from drupal:9.0.0. Use
core/modules/simpletest/src/UserCreationTrait.php: * @deprecated in drupal:8.4.0 and is removed from drupal:9.0.0. Use
core/modules/simpletest/src/WebAssert.php: * @deprecated in drupal:8.1.1 and is removed from drupal:9.0.0.
core/modules/simpletest/src/WebTestBase.php: * @deprecated in drupal:8.8.0 and is removed from drupal:9.0.0. Instead,

If we decide not to remove the module, we have a bunch of undeprecation to do. If there are dependencies to removing the module in the first place, that will postpone removing all of those deprecated APIs as well I guess.

Looking at the steps in the issue summary and discussion above, we still don't know what will happen, and what the impact will be. Only the first item is certain in the proposal and then "either/or, we'll see", etc.

• Remove simpletest module from Drupal core
• Open a beta-blocking tracking issue to ensure that contrib modules with simpletests can have them run via DrupalCI (with some or no changes). If that's not complete by beta, we'd need to revert the removal commit or discuss further
• In a follow-up, decide whether run-tests.sh should support running TestBase-based tests, either as a supported feature, or as a short-term scaffolding while the contrib module is being developed. But leave that support in for now

I agree with @webchick that we need documentation updates too but until we know how will the new setup work, we cannot document it either. To me it feels like if we keep insisting that all impact be defined up front, then nothing will get done, so as long as we can undo what we did if the impact turns out to be bad, I am fine with it.

#3110983: Verify that simpletest tests can be run in contrib, or add simpletest back to core was added as a blocker to beta, so that should ensure that this happens at least before its too late. As a product manager I think that is sufficient since we cannot tell what will the impact be, just have some rough ideas. Also we don't know if the Drupal Association is tracking this as something to do before the beta.

I think with #3110983: Verify that simpletest tests can be run in contrib, or add simpletest back to core being a beta blocker, I'm comfortable with this approach, too.

We now have a contrib module, and I just committed #3110862: Remove simpletest module from core to Drupal 9. Contributed modules that have SimpleTest tests can re-enable them on DrupalCI with a short snippet added to drupalci.yml

So... I think we can call this issue complete too!

We can leave #3057420: [meta] How to deprecate Simpletest with minimal disruption open for tracking remaining follow-up like #3112432: Prevent core, stub simpletest module from being installed on new sites which isn't 9.0.x-blocking.

Fixing credit.

I've updated the release note to have specific information for contributed modules that still want to run Simpletest on Drupal 9.

A follow up was asked for in # and that work was done in #3083638: Ensure that simpletest and block_place do not get special core treatement. I am removing the tag.