Remove Simpletest UI because we don't want to maintain a graphical test runner

I'm +1 for this, there are graphical runners for phpunit, and most are using phpstorm anyway.

Removing simpletest UI is the same thing as removing simpletest module. This issue should be titled: Quit supporting the simpletest UI by refactoring the non-UI parts of simpletest into core and removing the simpletest module.

I'm -1 on removing the UI and thus the module. I'm +1 on the refactoring part.

-1 for this been 8.x, I still find the Simpletest UI better than the verbose output for BTB which doesn't work with PHPStorm.

There is an alternative approach since a while: #2566767: Don't allow running phpunit-based tests via the UI which could be used as a way to educate more people.

I use the UI when Xdebug sometimes doesn't want to play ball, or if doing a webtest and what the vebose browser output, but I'd be fine to install it from contrib for that. No need to ship it with core.

For debugging you can use https://github.com/mglaman/intellij-drupal-run-tests.

Well, that sounds like a problem of your xdebug configuration :)

RE: #8 Had that problem a while ago, for me it was that the command line was using a different php version than my dev env through the browser. I just ran php --ini and had to add the XDebug conf to the php.ini (incl xdebug.so) that the command line was using.

+1 For this, with the UI remaining in contrib :)

+1 Removing Simpletest has always been the goal of my efforts.

Maintaining it takes too much time. Time that is much better spent with creating or contributing a GUI for phpunit.

For the same reasons I don't think the module would live long in contrib, as there are few people capable of maintaining it, but those will spend their time with more pleasing efforts.

At the moment, core needs improvements for any test runner:

• Better ways to discover tests: #2605664: [Needs change record updates] Align TestDiscovery and bootstrap.php's non-permissive loading of PSR-4 namespaces for traits #2499239: Use test suite classes to discover different test types under phpunit, allow contrib harmony with run-tests
• A way to run tests that isn't PHPUnit (because the testbot needs it: #2189317-5: Get a PHPUnit job type running that is separate and independent from the Simpletest Job type, fix the d.o integration so that we can run the PHPUnit tests independently): #2626986: [meta] Improvements to run-tests.sh #2624926: Refactor run-tests.sh for Console component.
• Since we're using run-tests.sh, all of the top-level functions in simpletest.module which are used by run-tests.sh. #2641632: Refactor simpletest's *_phpunit_*() (and junit) functions etc. to a class, deprecate

Once we've done all that for run-tests.sh, we will have also done it for simpletest UI. It's not that we necessarily need simpletest UI, it's that if we fix simpletest UI for-realsies we'll have a better testing system overall.

Basically: Abandoning simpletest UI is the same as being afraid to look under the sofa cushions for fear of what you'll find. This question of doing away with the UI is a bikeshed question which avoids the reality that once we remove the UI, the testing system will still be a mess.

It's not that scary. :-) We can also make it less scary, and thus easier to maintain.

Well one problem we seem to have is that phpunit doesn't deal with fatals. I think some kind of wrapper script would be great for that, but its something we just need for the testbot and not on each other's local machines.

I started to explore as alternative: https://github.com/VisualPHPUnit/VisualPHPUnit
One issue: https://github.com/VisualPHPUnit/VisualPHPUnit/issues/156

Neutral to -1 on this.

As an old-fart developer who learned how to do things before real debuggers became prevalent, I am much quicker with short debug statements going to output rather using xdebug inside phpstorm. Just having the xdebug enabled also slows down php noticeably, which makes a difference with integration tests.

I also find it quicker to incrementally build long test integration test sequences from verbose HTML output (yes, I know you can get this from run-test.sh, but I think the experience is better when run from the UI).

I also find it quicker to incrementally build long test integration test sequences from verbose HTML output (yes, I know you can get this from run-test.sh, but I think the experience is better when run from the UI).

Well, there is no reason against keeping --browser which does exactly the usecase you describe. It shows you the verbose output you need.

Well, there is no reason against keeping --browser which does exactly the usecase you describe. It shows you the verbose output you need.

No, actually, this issue says --browser isn't worth maintaining because it's the same code that's apparently not worth maintaining in the in-site UI.

FWIW, on 8.x-2.x these days, I can't even run tests based on Drupal\KernelTests\KernelTestBase in the UI: they fail before starting with errors like the one below, although they work just fine from the CLI:

Drupal\KernelTests\Core\Entity\RouteProviderTest::testHtmlRoutes PHPUnit_Framework_Exception: sh: 1: : Permission denied

FWIW, on 8.x-2.x these days, I can't even run tests based on Drupal\KernelTests\KernelTestBase in the UI: they fail before starting with errors like the one below, although they work just fine from the CLI:

Drupal\KernelTests\Core\Entity\RouteProviderTest::testHtmlRoutes PHPUnit_Framework_Exception: sh: 1: : Permission denied

Yeah, creating an issue for that is fine. But yeah things like that are IMHO an argument that its not worth to maintain it. It comes with cost and not that high benefit. Of course we would have to figure out how to get some better verbose message into BTB, but this is kind of an independent problem.

But yeah things like that are IMHO an argument that its not worth to maintain it. It comes with cost and not that high benefit

I guess that depends who you're talking about. I've used the UI for the last 5 years building contrib modules and work projects with D7, I used it throughout the D8 release cycle until all Migrate tests were ported to KTB then I moved to storm. I've not done any D8 work yet, but it is disappointing for me that the UI will be removed.

I'm prepared to help out with maintaining the UI, I would have no doubt fixed the issues reported here once I started using D8 in my day job and wanted to use the UI. Alternatively, i'd be open to look at bringing an existing UI into core such as https://github.com/VisualPHPUnit/VisualPHPUnit if that was a more viable option.

. Alternatively, i'd be open to look at bringing an existing UI into core such as https://github.com/VisualPHPUnit/VisualPHPUnit if that was a more viable option.

Right, this would remove our requirement to maintain it. Note: Its not directly usable, see https://github.com/VisualPHPUnit/VisualPHPUnit/pull/157 for example.

I've not done any D8 work yet, but it is disappointing for me that the UI will be removed.

Would it be disappointing if we provide/there are alternatives for everything simpletest provides? Your comment sounds a bit nostalgic to be honest.

I've not tried it, would it be a suitable replacement even with that issue fixed. I think key parts of the current UI is an easy way to find tests and the ability to select and run multiple tests, or a whole modules tests etc.

Would it be disappointing if we provide/there are alternatives for everything simpletest provides? Your comment sounds a bit nostalgic to be honest.

Sure a bit of nostalgia for something I've enjoyed using, what's wrong with that? ;) But seriously, if we have the equivalent functionality then i'd be fine with that, this is why I mentioned helping out. I actually wonder whether a better CLI interface could be enough.

#14: Before you can remove run-tests.sh you will need to create / install / test a parallel version of phpunit instead. The test runner relies on strong parallelism (concurrency = 32).

#12: What I can see is that the main work we do is completely independent from Simpletest anyway:

- Setting up the tests
- Our various test bases, etc.

and that won't change.

I think the only simpletest remnants are the DB table used for storing test results (and the format), which does not feel that hard for me to maintain.

run-tests.sh also is very independent of simpletest. It just creates a test class and runs it. There really is not much magic to that.

The same is true for the Simpletest UI.

#14: Before you can remove run-tests.sh you will need to create / install / test a parallel version of phpunit instead. The test runner relies on strong parallelism (concurrency = 32).

Can we please ensure to not derail the issue for that? This issue is for the simpletest UI :) Btw. there are parallel test runners we could at least evalulate, like https://github.com/brianium/paratest

The same is true for the Simpletest UI.

Well, we had to put for example quite some effort to scale the simpletest UI to actually work with the amount of tests in D8.

I think the only simpletest remnants are the DB table used for storing test results (and the format), which does not feel that hard for me to maintain.

Simpletest module does the following:

• Provide the test runner code that run-tests.sh uses, both for simpletest-based tests and phpunit-based tests.
• Provide the display that run-test.sh --browser uses (It's the same one that shows you results in the UI).
• Map results between phpunit-based tests and the simpletest schema for both UI and run-tests.sh.
• Basically serve as a repository of code for run-tests.sh.
• Allows sites to disable in-site testing, even though they can still run the tests from run-tests.sh when it's not enabled (using --sqlite).

The reason we hate simpletest is because there are two supported test runners. _simpletest_batch_operation() for instance duplicates a great deal of code from run-tests.sh. Stuff like that.

I've worked on trying to refactor it so that it's maintainable and only a small fraction of my effort has been committed or even really reviewed, which is why I really find it hard to be motivated to continue. It seems to be the same for everyone else who takes this on. Everyone seems to think that the testing system is voodoo rocket science, but it's not. It's just unmaintainable.

The point here is not that the simpletest UI itself is unmaintainable, the point is that the testing system is unmaintainable, and the reluctance to keep the UI is a symptom, rather than a cause. That's why I called it a bikeshed in #13.

What should happen:

• simpletest.module should be nothing but a page that lists tests and lets you run them. This would be super easy to make and maintain, because it's just a Drupal form with some internals glue.
• Test runner code should be a library, similar to a Component or an external library. It should not have a dependency on anything else in Drupal. Any function that starts with simpletest_*() should be refactored into this library.
• The test runner code should live under core/tests/. Its Composer dependencies should be require-dev.
• There should only be one API to this library, and it should be designed so that each concurrent test run could also be a batch run.
• run-tests.sh should be renamed so it's not embarassing.
• run-tests.sh should be a proper console app so it can be maintained.

Who's ready to roll up your sleeves? Any takers?

I just happened to notice this issue.

Assuming that you mean in this issue "remove the ability to run tests whose output looks like SimpleTest UI does now", I am about -10000 on removing the Simpletest UI.

When I am developing tests for a contrib module, I use the UI a lot. It is just plain handy when you are writing tests for the UI of a module, and debugging the test itself. When you are trying to figure out if you have a UI bug or a test bug, you need to look at the verbose output, and having it right there in the browser is very handy. You can scroll down the list of assertions, find the red ones, and see what was going on just above it to debug your tests. I really really really do not want to lose that ability.

Also, in the User Guide project, I am using a browser-based test, with special verbose output, and a browser script to automate making screenshots for the different language versions of the User Guide. Without the Simpletest UI, I would not be able to do this automation, or would have to figure out a different way to do it. I really do not want to lose that!

When I am developing tests for a contrib module, I use the UI a lot. It is just plain handy when you are writing tests for the UI of a module, and debugging the test itself. When you are trying to figure out if you have a UI bug or a test bug, you need to look at the verbose output, and having it right there in the browser is very handy. You can scroll down the list of assertions, find the red ones, and see what was going on just above it to debug your tests. I really really really do not want to lose that ability.

Hint, --browser is your friend.

It would really help if the run-tests script had some documentation at the top. At least to say "To learn how to run this script, use the command "php run-tests.sh --help".

I will try --browser --verbose and see if it works for my use cases. Thanks!

You might also put something about --browser --verbose into the issue summary, to avoid alarming random folks who come across this issue and say NO!!!! ;)

I find --browser to be awkward because it's not contained to a single window. Tests often take a long time to run so when I run them I tend to go off and do something else, then come back later to view the results. With the --browser option, whatever else I'm doing gets interrupted by a random browser window popping up when the tests are finished.

That's surely possible to improve (it could be a clickable link rather than automatically opening a browser window), but there are other benefits to having a UI as well: easier to find specific tests to run, generally easier to use, etc.

The comment from @Mile23 in #28 also makes a lot of sense to me.

Oh. Yeah, if --browser puts up random links all the time into my browser, that would be totally lame. That would definitely not work well for me.

Ideally, what would happen is that at the end of the test, it would put up a page just like the test output currently in the Simpletest UI, which would let you see which assertions passed and failed, and have *links* to the verbose output.

Ideally, what would happen is that at the end of the test, it would put up a page just like the test output currently in the Simpletest UI, which would let you see which assertions passed and failed, and have *links* to the verbose output.

Well, I think that is mostly what it does - it's just that even that one popup (at the end of the test run) is annoying to me because it occurs at a random time.

But that is something which is possible to improve.

I probably better try it before commenting any more. ;)

+1 to #28 too.

--browser is NOT working for me.

First I tried a command like this, from my test site's document root:

php ./core/scripts/run-tests.sh  --color --verbose --browser --keep-results \
  --url [URL TO MY SITE] --class '[MY CLASS NAME]

After waiting 23 minutes for my test to finish (this is the test that automates the screenshots for the user guide, and it takes a while to run on this machine), it came back with 5711 exceptions due to not having permissions to create directories for the output (even though I had chmod'd the directory).

So I tried the same, prefixed by sudo. This resulted in errors too, because root does not own my Firefox process:

(firefox:17596): GLib-GObject-WARNING **: Attempt to add property GnomeProgram::sm-connect after class was initialised
...
(firefox:17596): IBUS-WARNING **: The owner of /home/jhodgdon/.config/ibus/bus is not root!

So. I am back to using the Simpletest UI, unless someone has an idea of how to actually make this work.

+1 for #28

Same here, +1 to #28.

If we have a decent, maintainable test running system, then a Simpletest module that's just a form that runs shouldn't be a big deal.

And it's important to keep the barriers low to people writing and running tests.

Looks like a bit of consensus on #28. If someone wants to make a meta for it and get the proverbial ball rolling, we can get going. Like I say: I've tried to go down this road a few times to roaring silence so if someone else could at least file the issue please. And maybe even review those issues linked in #13. Or even this one: #2780093: Have simpletest, run-tests.sh enforce their dependency on PHPUnit

Otherwise, sure. Make simpletest go away without replacing it. :-)

Yeah, if --browser --verbose actually worked, that would be a replacement... but at least for me, it doesn't. Now, I could probably make a workaround, but it's MUCH nicer to use something that actually works, instead of having to do battle with the test system when you're already doing battle with the code you are trying to debug...

Yeah, that's basically the experience I've had with simpletest for 8 years. Its so bad I've got wrapper scripts with half a dozen shortcuts and gotcha's. If your httpd is down its even slower and worse because it sits there timing out waiting for the response and you have to wait for it to finish to get a reason it failed. Like a _single_ test will take 30min. You should try using phpunit's runner. Despite lacking concurrency its a dream in comparison. You mostly need --browser because run-tests's output is useless but phpunit's is readable and provides great feedback.

We'll have run-tests.sh until we finish #2735005: Convert all Simpletest web tests to BrowserTestBase (or UnitTestBase/KernelTestBase), and even then we still need it so we can do concurrency and handle PHPUnit fatals.

y'all should ping webchick on this before doing it

So, again: As far as I can tell, there is not really a viable alternative to running the tests in the browser, if you want verbose output. I could not get it to work from the command line.

This is *very* useful for debugging. I do not think we should just get rid of the ability to run tests and see the output line by line with the verbose HTML output. What is the plan to replace this functionality?

That sounds good, but I still had various permissions problem using --verbose --browser, as suggested in the issue summary as a replacement for the Simpletest UI. My tests *are* using DrupalGet anyway. See #38.

This user problem is also documented in core/tests/README.md

The problem is that to do --verbose, it needs to run as the web user or root user. But to do --browser, it needs to be me, because I am the owner of the browser process.

@jhodgdon
The problem is that to do --verbose, it needs to run as the web user or root user. But to do --browser, it needs to be me, because I am the owner of the browser process.

Doesn't it print out the URL you could open up manually as well? In case it doesn't we should print it out :)

Well, as-is, --browser and --verbose do not work together, and there is no documentation of all of this stuff (or even a link to where the documentation lives) in the script itself. I would not say that we actually have a viable replacement for the Simpletest UI until --browser --verbose Just Works, or at least has a documented path (documented in the script) to making it work.

Anyway... I think I also saw another issue about removing the script too? This all just seems to me to be moving in the wrong direction.

Well, then the issue summary here is outdated and needs clarification.

Well, people mix up different things here. IMHO we have a couple of things at the same time

* Provide a script to run tests (run-tests.sh)
* Leverage the phpunit test script (phpunit)
* Provide a UI to show which tests are available
* Provide a UI to show the result of the test run

This issue is really JUST about point 3)

OK, fair enough. But if we get rid of Simpletest UI, we are removing access to all 4 points from the UI. So are there issues about the other 3?

For myself... I personally don't care much whether I have to know the name of the test class and/or run from the command line, so I don't care as much about points 1-3. But I do care about point 4... and I think just getting rid of it is not necessarily the best idea. My reasoning:

a) I have found it quite useful to see the verbose output of every drupalGet in tests as I am debugging them. It's convenient to be able to run a test, see the output, click on Verbose where it is supposed to be doing something and see what it did instead, fix the test or the code, run it again, etc. I like the UI for that. Getting rid of it is not so great.

b) I'm using the verbose output, in the browser, to generate automated screenshots for the User Guide... my fear is that by eliminating WebTestBase and the Simpletest UI, I will lose the ability to make automated screenshots, or I will have to completely re-engineer it. It was not too easy to set up in the first place.

I really worry about how this will raise the hurdle for running tests. We want people to run tests, and we want people to write tests, which entails running them. If we want them to do this, then we want as few barriers as possible in their way. People will only keep trying past so many failures before they quit something.

If we remove this UI (and I don't think we should -- see #28 above), then the script for running this from the command line needs to be *as simple as or simpler than* going to a webpage and ticking boxes for the tests you want to run.

run-tests.sh is horrible (incorrectly named for starters) -- for example, it falls over when options are in the wrong order:

~/Sites/8-drupal joachim$ php core/scripts/run-tests.sh --verbose  --class 'Drupal\views_ui\Tests\PreviewTest' --browser
  ERROR: No valid tests were specified.
~/Sites/8-drupal joachim$ php core/scripts/run-tests.sh --verbose --browser  --class 'Drupal\views_ui\Tests\PreviewTest'

... and that ran, but showed me a URL that was a 404... :/

EDIT: removed complaint about the output -- was due to absence of --url parameter, same as the 404. Which is still extra faff to have to bother with.

Do we have a sense about how many people use the test UI https://simplytest.me/ when evaluating patches?

@mpdonadio -- ... um... is that relevant to this discussion? simplytest.me allows you to easily spin up a site with your patch attached. This is about Simpletest UI (admin/config/development/testing) within a Drupal site to run the automated tests.

Yup, once a patch is uploaded to d.org, the test runner does a great job! :)

This is about making sure that people feel able to write tests for their own patches. To write tests, you need to run them locally (or spam d.org with each new attempt... which is not ideal).

#64, you can spin up a site with a patch, and then use that to run tests from Simpletest UI with the patch applied. I frequently use simplytest.me for project reviews so I don't have to disturb my own local dev sites, but I don't know how many people use it to manually test modules / patches and also run tests from the UI. I have done this a handful of times when I don't want to disturb a local dev site.

OK, so we have some complaints about run-tests.sh. I'd suggest that if anyone wants to improve run-tests.sh, they should open an issue so we can work on it. Use this as a parent if you'd like, but at least file the issue: #2626986: [meta] Improvements to run-tests.sh

There are already issues for the run-tests script, including one to change the file name to something sane, and one to use 'env' in the shebang line so it runs with Php on any system. I seem to remember they have been sitting at needs review for a while...

Edit: ah, I see that's already on that meta issue.

FWIW we don't have automatic test running on security patches so it's necessary to run them locally.

@fgm
Please read the issue title, this is not about removing the capability to run tests.

@dawehner indeed: but what I mean is without the graphical runner it is much harder to debug tests since the --browser option to run-tests.sh does not open a browser to the debug output nor, more importantly, provide the URL at which it might be available.

This is what I did initially, since I've been following the present issue since it started. However it turned out the steps outline on [#2116263] just don't provide a way to access the output. The results are kept in a table with thousands of rows, each containing a URL, and no visible way to know which of the rows contains the useful information. Of course, --browser just does nothing (macOS). Plus, the phpunit concurrency is not yet there (yes I know about paratest).

So, since the same page suggests further down to go back to run-tests, that's what I did. But same problem: no usable debug output. The files are there, but no way to related them with the actually failing steps information.

So in the end, it had to be back to the UI.

Hopefully, I missed something, because I would very much prefer to be able to run tests from the CLI if only I could get an easily browsable report, even if it had to be in text format. Any idea ?

--browser should work fine, you are supposed to get a page just like right now with the results, only difference is that you don't select and run tests in the browser, you just look at the results there. I know many people who use it on OSX, maybe you need to configure something, have a look at the code in run-test.sh to see what it executes exactly. Make sure you provide --url as well.

RE #72...

Before removing the Simpletest UI we will convert all tests to phpunit.

That is very Core-centric. People have written tests for Contrib modules as well, and they may not all be converted... contrib modules do not necessarily follow all the changes happening in Core or keep up with them. We should encourage contrib module maintainers to write and run tests, not put up barriers to them doing this.

> Did you copy phpunit.xml.dist to phunit.xml and set printerClass and BROWSERTEST_OUTPUT_DIRECTORY?

And that's a good example of why we DO want to keep a graphical test runner: because it's simple.

I think this issue should be closed, and replacements opened. Its premises are flawed.

> Remove Simpletest UI because we don't want to maintain a graphical test runner

It's not that we don't want to maintain a *graphical* test runner, but that we don't want to maintain *more than one* test runner. And ideally, we want to maintain *no* test runner.

We *do want* a UI for running tests because the barrier to writing (and therefore running) tests should be as low as possible. There shouldn't be configuration to set up. There shouldn't be command-line options that you can get in the wrong order, or that cause unclear failures if you get them wrong. A graphical UI allows users to pick the tests they want to run from a list, and click a button and *just run the test*.

So what we should be doing is reworking Simpletest UI so that it's a front-end for the test runner we don't maintain.

SimpleTest is with us until D9 if not longer, because of the way deprecation works in the core process.

For instance, here's an issue which does some cleanup in simpletest: #2801817: Inject services into SimpletestTestForm and functional cleanup for the test_discovery service Note that it deprecates the simpletest_test_get_all() function for removal before D9. No doubt a maintainer will push back on even that. :-)

It's not that we don't want to maintain a *graphical* test runner, but that we don't want to maintain *more than one* test runner. And ideally, we want to maintain *no* test runner.

Right on.

@fgm
What you describe is really simply a bug. Do you mind opening up an issue? --browser works really fine for many people.

I also think and suggest we postpone this until D9 is closer.

It might be that everyone is so happy with the new test system that this becomes a non-issue anyway. :)

The simpletest APIs are with us until Drupal 9.

UIs aren't an API, so moving this back to 8.3.x (but leaving postponed - a command-line-command generator from test groups and similar would be useful).

If we had a viable command-line test runner that (with as little setup as "Install the Simpletest module") would allow people to run tests and get verbose output that was very similar to the Simpletest UI, I doubt anyone would complain too much. People who write tests and want to run tests can probably handle the command line, if it's as simple as "run this command with these options" and it Just Works (TM).

The problem is that, as much as people here advocating for getting rid of Simpletest UI keep saying this exists, in practice the replacement doesn't exist, or at least doesn't work for everyone, and also the hacks you may need to use to get it working aren't documented.

I was not able to get --browser and --verbose to work for me. See #38.

Oh. I was trying the run tests .sh script. Anyway, at the moment I am using WebTestBase still, not phpUnit.

And I realize you are trying to get rid of WebTestBase in Core at the moment. But not everyone is a Core developer, and asking all of Contrib to update all of their tests at the moment is a lot to ask. IMO the effort to get rid of the simpletest UI at this point is not friendly to Contrib, because all we'll be left with to run WebTestBase tests at that point is the terribly flawed run tests .sh script.

We are working on stop using WebTestBase, that's not the same as removing it. That won't happen before 9.x. But it will definitely happen then. So sooner or later, you will have to convert anyway. But I'm waiting with my own modules as well, until a large amount of core has been converted so that the necessary API's and tools are there and the remaining issues are fixed. Nothing wrong with that.

--verbose and --browser works fine for many people with run-tests.sh for old web tests.. As it has been said before, if it doesn't work for you, then *please* create a bug report, provide infos about your system, how exactly you call it and any other information that might be useful (installed browsers and so on).

Trust me, it is worth the effort. It's just so much faster to use, you can easily run single methods, works very well in combation with the Copy Reference method in phpstorm (both methods and classes), you can easily run groups, you can run tests in parallel which makes them 2x, 3x... 8x faster depending on how many CPU's you have. And the output of the results with --browser is *exactly* the same as when using the UI to run tests.

I didn't make a bug report about the runtests script because after I posted #38, I was basically told "you did it wrong" and "besides, we don't want to support runtests either", so I gave up. I could probably get it to work, but since I have simpletest UI and it works, it was not worth the effort for me.

The report is there in #38 if anyone actually cares about this. But I don't get the impression that anyone wants to keep maintaining the run tests script any more than they want to maintain the simpletest UI.

I see. You should not sudo as root, you should sudo -u apache-user. Which is usually www-data. But I guess that won't solve that problem. I turned the whole thing around and I configured apache to run as my user on my local system and then you don't need to mess with sudo. There is only one downside to that, and that that's drupal then always removes write permissions to sites/default but that can actually be disabled now as well.

According to some googling, http://askubuntu.com/questions/325274/why-do-i-get-ibus-warning-on-runni... and https://bugs.launchpad.net/ubuntu/+source/ibus/+bug/1125944 say you should use gksudo. But that's only a warning anyway and shouldn't actually break anything.

While nobody likes maintaining run-tests.sh, it's going to be around until we remove simpletest (not the UI, the whole thing). so not before 9.x. I use it daily.

Oh @berdir, how do you disable Drupal removing write permissions to sites/default ?

See example.settings.local.php, documented there.

Thanks, I hadn't understood this was what this setting was about. ($settings['skip_permissions_hardening'] = TRUE;)

Found a small documentation issue. see #2811065: Fix docs around --printer option in phpunit.xml.dist

Just noticed this other approach to parallel tests, (not paratest): https://github.com/jwage/phpchunkit

> we don't want to maintain a graphical test runner

Here's my use case why we DO:

Tonight, I saw that one of the tests on Flag module is failing.

But it's 10pm, I've just had a dreadful evening putting the toddler to bed and I want to relax and watch TV. I figure, hey, I can run the test on my laptop at the same time, see if it's an easy fix.

Except I haven't run a test on the CLI in over a month. So I've forgotten the syntax for it. I remember I have to specify the class... but do I have to escape the namespace separators? What about the option to get output? What about the URL option? Where are the docs? What's the option to get the help output?

... and there are too many hurdles. If it weren't for the GUI test runner, I would give up and leave the problem for someone else or for another day.

To be fair: SimpleTestBrowserTest is a bad test for exactly this reason. It's unmaintainable. It's not a good test.

The reason we have it is because we made all the testing code untestable back 10 years ago or whatever and for some reason refuse to work on it, so that tests like SimpleTestBrowserTest have to do the test-ception crap.

The resaon it's still like that is because any time anyone tries to maintain it, they're met with total apathy (at best). That's my user story.

See #28, where we can talk about making the testing system less brittle overall for all uses, thus improving the DX for everybody, make components of the testing system more testable, and, basically replace SimpleTestBrowserTest with some tests that make sense.

#2641632: Refactor simpletest's *_phpunit_*() (and junit) functions etc. to a class, deprecate
#2800267: Turn simpletest_clean*() functions into a helper class
#2624926: Refactor run-tests.sh for Console component.
etc.

Except I haven't run a test on the CLI in over a month. So I've forgotten the syntax for it. I remember I have to specify the class... but do I have to escape the namespace separators? What about the option to get output? What about the URL option? Where are the docs? What's the option to get the help output?

You can put a phpunit.xml file in the root of your project that contains your config, and then you can simply run this:

$ ./vendor/bin/phpunit path/to/my/test.php

Doesn't get simpler than that :)

I can recommend to simply never type out the options on the command line, the very first time you want to run a test, just create the phpunit.xml file and you never have to think about it again.

You can even have it automatically generated when you start a new Drupal 8 project based on my drupal-project fork.

@klausi I'm not saying it's not a bit lazy of me. But I was trying to illustrate the point that users are lazy, and that they give up at hurdles, even when they want the end goal. (It's why commerce sites don't make you create an account, they just create it for you in the background, because being asked to make a username and password makes too many users give up, even though they want to buy the thing in their cart.)

I think that one of the main purposes of having a test runner is so that core and contrib contributors can run tests, so that they can write and work on tests.

Given that goal, we need to keep the hurdles to that as low as possible.

Nothing is going to be as simple as selecting a checkbox is a GUI.

PS. The end of my user story is that I ran the GUI tests, found the bug, and fixed it. Took about 30 minutes. But if if it hadn't been for the GUI, I'd have given up and left it for someone else to fix.

That would have been 30 minutes lost to the Drupal community. Now think of those 30 minutes, and multiply them by everyone we'd like to be able to run tests.

We should definitely make it easier to maintain the GUI test runner, but maintaining a GUI test runner is an *investment*.

@joachim if the GUI test runner had given you the phpunit command line snippet to run, would that have helped? It could also create the phpunit.xml file to download or copy and paste. That way it'd save looking up the cli options or phpunit.xml structure even the first time tests are run. It's still slightly more work than running tests in the UI, but a lot less than figuring out running tests from cli the first time (or after a while) from docs.

A GUI that produces the command line snippet would definitely be better than no GUI at all, yes.

I agree with the removal or at least improving it to give you the command you need to run. I spent a decent amount of hours last week tracking down an error that the UI gave to me which had no real bearing on the actual issue. I can only imagine how many other people are led down the same path, which is just as discouraging to writing tests as getting the command line argument order wrong. Not to be dismissive of the time put into creating and maintaining the Simpletest UI, but it isn't the first time it has happened to me, either.

For those using PHPStorm, Matt Glaman created a plugin to ease the creation and running of tests from PHPStorm itself. It currently has some bugs in different versions of PHPStorm (I am a couple versions behind on ver 10 so it might just be me). So that is an option for those folks using the IDE too.

Made an issue which is basically comment #28. If you thought that was a good idea, please poke your head in on #2866082: [Plan] Roadmap for Simpletest

This is up there with requiring composer that will drive beginners and less technical users away. I'm 100% behind refactoring and reducing the custom functionality of the UI so it can be more streamlined, but to help adoption, help beginners and not throw a huge roadblock in front of folks who haven't mastered the intricacies of their OS's terminal system, we should not assume everyone is going to have a $200+ commercial IDE to avoid their daily dose of terminal anxiety.

So we can look forward to reviews of the issues on #2215035: [meta] Simpletest UI cleanups right? :-)

We just did this #2893117: Improve HTML caching of Simpletest UI test form which improves HTML caching for the test runner UI.

Want to improve the Simpletest UI? Review #2296615: Accurately support multiple @groups per test class so we can move on to #2858652: Support multiple @group test annotations in Simpletest UI

There's also #2301481: Mark test groups as belonging to modules in UI which has been sitting for a while.

And then this one, which wants to deprecate the batch-based test runner which is the one that the Simpletest UI uses, because no one wants to maintain three different test runners all of which have different requirements. Should the runner used by Simpletest UI be deprecated? #2748967: Trigger E_USER_DEPRECATED for BC support in simpletest_run_tests()

So in the past year, not much motion on all those UI-related issues in #110. So I think it's fair to say that even if people want to use this UI form, no one is interested in supporting it.

Therefore, in the interest of graceful deprecation, we should have a follow-up to put some text on the UI that warns users that it's going away in Drupal 9, and also deprecates all the functions that only the UI touches. When that's in, we can un-postpone here.

In #3028663: Split up simpletest into simpletest and testing_ui to enable easier decisions for Drupal 9 I made multiple attempts to get the answer to the following question: what is the (existing) testing UI useful for once all tests are phpunit? This is not explained in this issue either (why it becomes bad once 90% of tests are phpunit?). This is what the summary says now:

Remove Simpletest's UI as soon as 90% of our test cases are converted to PHPUnit tests

It does not explain what changes at 90%. I would add the "Needs issue summary update" tag if it would not already be on the issue.

I know that no one wants to support the SimpleTest UI, but the User Guide project is currently using the SimpleTest base classes and UI in its process that makes/updates automated screenshot images for all of our translation languages whenever new versions of Drupal come out. I tried at one point in the past to convert this process to use JavascriptTestBase (which in theory has the capability of making screenshots), but at the time, there were so many bugs that I gave up. That was a while back, and the JavascriptTestBase class and JS testing system has since had its guts ripped out and has been revamped and rewritten, and it's possible I could get it to work now, but I'm not sure. The User Guide's needs are a bit unique, since the purpose is both to test the UI text that is referenced in the User Guide, and to produce cropped-down screenshots, and the tests/images have to be done in multiple languages.

I guess my path forward could just be to grab the Simpletest base and classes and UI code and put them into the User Guide project when they go away from Core. Or it could be I could convert to JavascriptTestBase -- have all the Core tests successfully been converted that use JS, or are there still bugs? Are there any multilingual JS tests in Core? Any advice/thoughts?

@jhodgdon: Are there any issues in the user guide project that people could help out on to get a solution for that that doesn't tie us to the simpletest UI? I found #3029816: Convert screenshot automation to WebDriverTestBase are there any others?

@Mile23 - that is the issue. I just created that today after discussing with a few people in Slack about the new-ish WebDriverTestBase class. The last time I tried to convert (to JavascriptTestBase), the log of what happened and why I abandoned the effort is in the related issue:
#2856747: Convert screenshot automation to JavascriptTestBase

If you or anyone else is interested in helping out... there is quite a bit of documentation about the current screenshot process in a README file in this directory:
https://cgit.drupalcode.org/user_guide/tree/auto_screenshots
The current test classes are in the src/Tests directory under there. There is a base class, and then subclasses for each language.

Moving this back to 8.x. If we want to remove the SimpleTest UI, we can do so in 8.x - there's no bc issue because it's purely a user interface with no data to migrate, the replacement is using the CLI. Whether we want to completely remove it or not is a different question but one that is version agnostic.

imho we should postpone this for now until the code is refactored a bit in #3028663: Split up simpletest into simpletest and testing_ui to enable easier decisions for Drupal 9.

I was able to get rid of the need for anything Simpletest-related in the User Guide tests. I am no longer needing the test running UI or the Simpletest base classes. So... anything I objected to before, you can ignore. :)

I was able to get rid of the need for anything Simpletest-related in the User Guide tests

Awesome! Great work

So:

• We need an IS update here.
• Once we have said IS update, we also need product manager signoff on this change since what is and isn't included in core needs product management signoff (as well as FM and RM signoff).

For me, this is OK so long as (a) the test runner UI exists in contrib and (b) the command-line script in core still presents the links to the verbose output in the handy way it does currently.

We ended up moving SimpleTest to contrib, so marking as outdated.