Convert all Simpletest web tests to BrowserTestBase (or UnitTestBase/KernelTestBase)

Just a note when doing this we should also consider whether the test would be better as a kernel or unit tests - neither of which we had for the first several years we were adding web tests.

Open child issues to convert web tests for each core module.

Not sure whether this is the best idea. Conceptual similar tests, aka. tests with similar scope, is often a better idea, because its easier to figure out what is going on.

Yes that gets tricky as well. There's a fair bit of duplication between the test coverage for modules like node/taxonomy and entity system tests. Similarly between tests in Views module vs. tests in views-data-providing modules. That's no fun to pick apart but would be good if we did.

So issues by theme, then sub-issues for those sounds good.

Yes yes yes! but unfortunately it is not as straight forward as KTBNG conversion. I'm still struggling to convert DER WTB tests to BTB #2469609: Convert all WebTestBase to BrowserTestBase.
We, as Drupal developers, have to change our mentality about browser base testing. We don't have to think page as a giant html string or form as an giant array to pass to a function. We have to write tests in a way we test them on a browser. Like visit a link, click a link, click a form element. Form is not big piece of array we have to set. Input of the fields should be set in a way we set them while browsing.

For a start BTB::submitForm() is super flawed.
Exhibit A from DER #2712545: Field setting form is broken after 2412569 http://cgit.drupalcode.org/dynamic_entity_reference/tree/tests/src/Funct... and http://cgit.drupalcode.org/dynamic_entity_reference/tree/src/Tests/Dynam...
In WTB

    $this->drupalPostForm(NULL, array(
      'cardinality' => FieldStorageDefinitionInterface::CARDINALITY_UNLIMITED,
      'settings[exclude_entity_types]' => FALSE,
      'settings[entity_type_ids][]' => ['user', 'entity_test'],
    ), t('Save field settings'));

BTB equivalent

    $page = $this->getSession()->getPage();
    $entity_type_ids_select = $assert_session->selectExists('settings[entity_type_ids][]', $page);
    $entity_type_ids_select->selectOption('user');
    $entity_type_ids_select->selectOption('entity_test', TRUE);
    $assert_session->selectExists('cardinality', $page)
      ->selectOption(FieldStorageDefinitionInterface::CARDINALITY_UNLIMITED);
    $page->uncheckField('settings[exclude_entity_types]');
    $this->submitForm([], t('Save field settings'), 'field-storage-config-edit-form');

@jibran
Can you elaborate why we have to? There is submitForm() for this kind of work.

And then there is a way we assert stuff on the page in WTB and BTB.
For example, before #2711963: Modernized ToolbarIntegrationTest ToolbarIntegrationTest.php was asserting stuff like WTB, treating page as giant piece of html string, and after #2711963: Modernized ToolbarIntegrationTest ToolbarIntegrationTest.php has a nice flow to it.

I'm still trying to find a perfect balance with this apporoch as well in #2702233: [backport] Add JavaScript tests for Form API #states: required, visible, invisible, expanded, checked, unchecked. How to assert stuff on a page with form? Should we add FormWebAssert like WebAssert or not, as pointed in #2702233-30: [backport] Add JavaScript tests for Form API #states: required, visible, invisible, expanded, checked, unchecked? Or we shouldn't do this at all as suggested in #2702233-25: [backport] Add JavaScript tests for Form API #states: required, visible, invisible, expanded, checked, unchecked.

@jibran
Can you elaborate why we have to?

Short answer BTB is designed to work with different type of drivers and GoutteDriver sets element differently then PhantomJSDriver.

Given that KTB/UTB tests typically live in a different namespace than WTB tests, how stupid of an idea would it be to do this as

1. Write equivalent KTB/UTB for existing WTB classes and similar tests that can run in a simpler space.
2. Make old tests as @deprecated
3. Run with both sets of tests for a while and see if we have regressions that one test catches that the other doesn't
4. Remove the @deprecated test before 9.0.0.

My fear is that that WTB has some implicit side effect coverage going on, and if we simplify (for lack of a better term) we may actually be losing some test coverage.

(this thought is somewhat half baked, but I have click save and head to a meeting)

@mpdonadio
I think this could be an interesting strategy for some subset of the tests. Maybe some which are really side wide.

Sweet, and the second one includes all of phpunit's discovery processes, which are high for a single test, but for a full-testsuite or group are lower

@larowlan
Are you sure this is not triggered by less tests at the moment? What would happen if we have the same amount of tests as we have with simpletest now?

I don't find the debugging experience as good with BTB as WTB, the UI is very helpful for finding the rendered HTML closest to the failure or request you want to inspect. IMO, we should fix that before we replace everything with BTB.

When a test fails phpunit stops execution immediately, so the last printed link to the HTML response from your test is the place where it broke. No fiddling around in the Simpletest UI and desperately searching for the line where it actually broke and what might be the bad response. For me that is a lot better than Simpletest UI.

I agree, this is something we really need to teach people. We fail early, because in simpletest, further failures could be totally misleading once the system was in a broken state at some point.

Sure - there is still room for a lot of developer experience improvements, but I think we should not hold up conversion to BrowserTestBase because of that. Can be parallel efforts.

Oh yeah totally. Please open issues if you have ideas!

Snagging #2737805: Convert web tests to browser tests for forum module

Found in #2737805: Convert web tests to browser tests for forum module that submitting admin/modules page to uninstall/install a module kicks out the logged in user. That one will be fun to debug...

Added a random related issue: #2738567: Add test trait for drupalGetTestFiles and drupalCompareFile

We need to fix #2575725: Run PHPunit tests one at a time via UI to avoid exception when selecting too many before doing too much BTB conversion since the fact they take some time to run means that the approach of trying to run all the PHPUnit tests first and in the form submission is quite flawed.

Added a new subissue for more assertions: #2750941: Additional BC assertions from WebTestBase to BrowserTestBase

I would like to help with the dblog tests conversion, I been working on several issues of this module for the last 2 months and I feel quite comfortable to help with this task. However, reading the comments, and specially #6 it seems this is not easy to implement.

I think it could be a good idea to update the summary issue to reflect what are the main changes need to be made in order to convert a WebTestcase into Unit or Kernel Test base.

So in the section "Proposed resolution" in addition to "create child issues" we should have more details regarding when a Kernel Test applies, or when a Unit test applies. I'm pretty sure this documentation already exists but we need links to this docs, otherwise I don't see how other people can join to help in this plan.

Amazing. Thanks!

Thanks a ton klausi!

I still believe we should go through the individual tests and identify places which could be rather a kernel test.

When adding new, untested, assertions to AssertLegacyTrait, please double check the visibility. If a method is added that is public instead of protected it will cause fatal errors in real life BrowserTestBase implementations in contrib.

See for example: #2744089: Fix visibility of AssertLegacyTrait::assertLink(), #2752315: Fix visibility of AssertLegacyTrait::assertNoEscaped(), #2757139: Fix visibility of AssertLegacyTrait::buildXPathQuery().

I personally think the AssertLegacyTrait should not be included by default in BrowserTestBase at all, instead it should be included in the converted legacy tests.

When a test fails phpunit stops execution immediately, so the last printed link to the HTML response from your test is the place where it broke.

This is great when you have a nice fast unit test, run fail, fix and run again, great. But when you're talking about a web tests that can run for 3-10minutes, if it fails after 5 minutes on an assertion, then you run it again for another 5 minutes and it fails on the next line, that's just annoying. Sometimes it's better to know all the failures so you can attempt to fix more than one at a time. stopOnFailure="false" would be a better default for longer running browser tests IMO.

Sure - there is still room for a lot of developer experience improvements, but I think we should not hold up conversion to BrowserTestBase because of that. Can be parallel efforts.

The problem here is, we seem to favour doing conversions and getting to the new shiny rather than adding the polish it needs. The verbose output only works for drupalGet() and drupalPost(), which is ironic given one of the benefits of BTB is familiar API's (mink) but if you use Mink directly, you don't get any verbose output. Also, other BTB functions like clickLink() do requests without capturing verbose output. IMO, we need a reliable way to capture output whenever a request happens in the test.

The --stop-on-failure configuration only helps you when running multiple tests

Ahh, i see that's a shame.

Verbose output: we already hacked the HTTP client for mink to catch errors

How come we override the client rather than the driver? They look pretty similar looking at them.

Just to point out the failure discussion ... when there is a failure, anything could happen afterwards, much like you can prove anything when you start with "0 = 1". Given that the only way to fix tests properly is to fix the first failure.

Sure anything can happen but most of the time you can infer quite well what went wrong. Take for example the following code, it would be helpful to see all failures, for example if the array was reversed vs being in a random order, it could help you find the issue right away. There are many situations like this imo.

$this->assertEqual(1, $files[0]->id()); // Fail, 1 does not equal 4.
$this->assertEqual(2, $files[1]->id());
$this->assertEqual(3, $files[2]->id());
$this->assertEqual(4, $files[3]->id());

Sure anything can happen but most of the time you can infer quite well what went wrong. Take for example the following code, it would be helpful to see all failures, for example if the array was reversed vs being in a random order, it could help you find the issue right away. There are many situations like this imo.

Mh fair, but its seriously something the design of PHPUnit will never be able to support. The alternative for this usecase though would be to use something like the following:

$file_ids = array_map(function ($file) { return $file->id();}, $files);
$this->assertEquals([1, 2, 3, 4], $file_ids);

I pretty strongly disagree with #49 and #46. It really depends on what you are doing.

Within a test, if doing something sequential, then I can see stopping at the first failure. But, in other cases, seeing all the fails can be beneficial. If feeding a test with a data provider, then I want to see all of the fails at once: I can likely address several of the fails at once. I was working on an integration test last night, where seeing all of the fails at the same time pointed to the true cause of my problem.

When doing a full test run (which outside of just the UnitTestCase is mostly infeasible on a local machine) aborting after the first fail (even within a particular test) will probably end up with more full runs on the bot.

I think evaluating whether we can downgrade tests, which will shorten runs, will be the biggest benefit both for local dev and load on the bots.

@klausi #20: I think the debugging experience is better with PHPUnit and BrowserTestBase. Once you have copied core/phpunit.xml.dist to phpunit.xml and enabled BROWSERTEST_OUTPUT_DIRECTORY + configured printerClass="\Drupal\Tests\Listeners\HtmlOutputPrinter" you get very useful links when running the test.

Can we make sure that's in the docs somewhere if it isn't already?

New issue/bug that blocks some test conversions #2767655: Allow WebAssert to inspect also hidden fields.

I provided a patch that is green in #2767655: Allow WebAssert to inspect also hidden fields. That needs review.

#51 - Hopefully #2760905: The documentation should be more explicit about PHPUnit requesting the webserver user to perform all functional tests is enough?

#2757007: Convert all book web tests to BrowserTestBase is ready for review.

I posted #2770549: [plan] Discuss desired API and test suite following deprecation of SimpleTest to help address some of the architectural and long-term questions without derailing the conversions.

@klausi agreed, that would be very valuable.

Hmm, also, -1 for doing this on a per-module basis. I'd recommend trying #2770921: Feb 21st: Convert chunk of WTB to BTB by just moving classes, changing use statements adding traits and seeing what shakes out there, not only in terms of the API but also in terms of DrupalCI compatibility. E.g. there is a recent random fail in forum tests that may be caused by a difference between SimpleTest and BTB, but when it's on a per-module basis it becomes difficult to debug or track down. Or, see https://www.drupal.org/pift-ci-job/390573

And honestly I'd also kind of feel we need to have the documentation mentioned above before we do conversions, because we've missed the mark on the documentation gate so far for BTB. I wouldn't yet recommend reverting the 3 or 4 per-module conversions that have gone in, but it is something we might need to consider.

I'm all for not backporting actual test changes, this is not adding that much value.

Backporting test framework changes though have a couple of big advantages:

1. People more likely will see those improvements and leverage it as part of their daily workflow
2. Contrib authors potentially have less of an issue to keep their tests working on the two current important branches of core

Found a bug in a previous conversion #2773389: BrowserTestBase::getTextContent() wrongly returns the raw content.

Yet another one that is required by more than one test conversion #2773733: Fix AssertLegacyTrait::assert(No)Text to handle html encoded strings (part 3). So I opened a dedicated issue.

Ran across this today https://elliot.land/7-reasons-to-use-concise-on-top-of-phpunit thought it might be interesting following on from the convo in #47, it provides a verify() option so it doesn't quick fail the test. Also, the tool claims to have a better CLI/runner than PHPUnit. Could be worth an issue to discuss if anyone is interested.

@benjy
Sure open up an issue, but its unrelated with BTB, because concise is on a lower level.

For example when I review those patches I actually look at the converted tests and see whether they could be fundamentally approved.

I suggest beginning by converting per test base class.

In the entire hierarchy of WebTestBase, there are 930 classes. 403 of those are direct subclasses. 31 are *TestBase classes, which means ~500 subclass from test bases. 31 issues for 500 classes.

Here's our first round: Make an issue for each of these test base classes. Convert the base class, pick up the pieces on the subclasses.

Found 31 matches of extends webtestbase in 31 files.	
AggregatorTestBase.php	
abstract class AggregatorTestBase extends WebTestBase {      [position 13:35]	
BlockTestBase.php	
abstract class BlockTestBase extends WebTestBase {      [position 11:30]	
BlockContentTestBase.php	
abstract class BlockContentTestBase extends WebTestBase {      [position 12:37]	
CommentTestBase.php	
abstract class CommentTestBase extends WebTestBase {      [position 16:32]	
ContentTranslationTestBase.php	
abstract class ContentTranslationTestBase extends WebTestBase {      [position 14:43]	
FieldTestBase.php	
abstract class FieldTestBase extends WebTestBase {      [position 12:30]	
FileFieldTestBase.php	
abstract class FileFieldTestBase extends WebTestBase {      [position 14:34]	
FileManagedTestBase.php	
abstract class FileManagedTestBase extends WebTestBase {      [position 13:36]	
ImageFieldTestBase.php	
abstract class ImageFieldTestBase extends WebTestBase {      [position 25:35]	
MenuWebTestBase.php	
abstract class MenuWebTestBase extends WebTestBase {      [position 10:32]	
MigrateUpgradeTestBase.php	
abstract class MigrateUpgradeTestBase extends WebTestBase {      [position 13:39]	
NodeTestBase.php	
abstract class NodeTestBase extends WebTestBase {      [position 12:29]	
PathTestBase.php	
abstract class PathTestBase extends WebTestBase {      [position 10:29]	
RESTTestBase.php	
abstract class RESTTestBase extends WebTestBase {      [position 13:29]	
SearchTestBase.php	
abstract class SearchTestBase extends WebTestBase {      [position 11:31]	
ShortcutTestBase.php	
abstract class ShortcutTestBase extends WebTestBase {      [position 13:33]	
InstallerTestBase.php	
abstract class InstallerTestBase extends WebTestBase {      [position 17:34]	
StatisticsTestBase.php	
abstract class StatisticsTestBase extends WebTestBase {      [position 10:35]	
AjaxTestBase.php	
abstract class AjaxTestBase extends WebTestBase {      [position 10:29]	
CacheTestBase.php	
abstract class CacheTestBase extends WebTestBase {      [position 10:30]	
PageCacheTagsTestBase.php	
abstract class PageCacheTagsTestBase extends WebTestBase {      [position 12:38]	
DatabaseWebTestBase.php	
abstract class DatabaseWebTestBase extends WebTestBase {      [position 11:36]	
ToolkitTestBase.php	
abstract class ToolkitTestBase extends WebTestBase {      [position 11:32]	
MenuTestBase.php	
abstract class MenuTestBase extends WebTestBase {      [position 7:29]	
ModuleTestBase.php	
abstract class ModuleTestBase extends WebTestBase {      [position 14:31]	
SystemConfigFormTestBase.php	
abstract class SystemConfigFormTestBase extends WebTestBase {      [position 14:41]	
UpdatePathTestBase.php	
abstract class UpdatePathTestBase extends WebTestBase {      [position 39:35]	
TaxonomyTestBase.php	
abstract class TaxonomyTestBase extends WebTestBase {      [position 11:33]	
TourTestBase.php	
abstract class TourTestBase extends WebTestBase {      [position 10:29]	
UpdateTestBase.php	
abstract class UpdateTestBase extends WebTestBase {      [position 26:31]	
ViewTestBase.php	
abstract class ViewTestBase extends WebTestBase {      [position 19:29]	

Per base class is something that we also did more or less in kernel tests although I often did more than one there except for the really big ones. I think that does work fairly well.

One thing there is that contrib/custom code might also extend from those (extending from others is also possible but less likely. I sometimes do that for different backends, e.g. redis queue testing). what we did for kernel tests is mostly keeping the old base classes around, so contrib has some time to shift from the old to the new base classes and doesn't break every time we commit a new patch.

I actually would like to get as close as possible to making the conversion of core a single, scheduled change, that we would do during a beta or RC phase, or immediately when a minor branch opens. Edit: The work already going on to add BC layers will work well with that approach.

(Editing to remove comments from the wrong issue, sorry).

(...) making the conversion of core a single, scheduled change...

@xjm, this is not possible for some of the tests, especially for those that are testing JS features in forms by setting hidden fields. This is because of Mink architecture that tests everything from the perspective of a human. An example is #2757007: Convert all book web tests to BrowserTestBase and I'm sure we'll find others.

We should try to get as close to possible to converting the whole test suite in one massive, scheduled patch with a single scope once we have provided as much BC as possible, but it may be a few massive patches (one for the simplest scope to just change which class is extended, another for a specific change throughout core that is impossible to provide a BC layer for, another different change that is impossible to provide a BC layer for, etc). It would be during a beta/RC phase or immediately after a new minor branch is opened. This mitigates the intermittent small disruptions and makes it something we can announce in advance and plan for, once we're sure the documentation and the DrupalCI support and the BC layer and everything else is in really great shape.

There are a number of reasons for this, many outlined in https://www.drupal.org/core/scope. This was also pointed out in #4 on this issue by @dawehner. A few specifics that are of concern for me in this case:

• Documentation. We have comprehensive documentation for SimpleTest in the handbook and API docs, but comprehensive docs for BTB are pretty much missing. @dawehner has a starter patch in #2469713: Step 2: Create a JavaScriptTestBase using PhantomJs Driver/Binary but there is still a long way to go.
• Contrib may extend base classes, as @Berdir points out. If we provide enough BC to make core test conversions as close as possible to a single scope, the contrib pain will be eased a lot. Let's say we want 60%-80% of web tests to work just by changing the class it extends. If we also have a clear guide on how to convert the remaining 20%, then contrib developers can plan well in advance that we will do it on a scheduled date.
• Consistency. Non-core developers are confused, put off, and sometimes actually distressed by the fact that core inconsistently uses different APIs for the same thing. The closer we can get to changing core all at once, the more this DX issue is mitigated.
• As-yet unknown issues and infrastructure support. There are still some things happening on DrupalCI that seem to be exposed by BTB tests and not simpletest. Having one conversion of 5 test classes introduce 75% random failures on secondary environments is a huge problem. If we make the conversion patches more conceptually scoped (and bigger), we can make sure they are tested on all environments and also give enough heads up to ensure d.o infrastructure resources are available to fix blockers. The same is true for bugs we may not yet know about in our BTB test running, etc. within core itself.

@claudiu.cristea, I crossposted, but hopefully the latest comment clarifies. Thanks!

I'd modify @Mile23's suggestion as follows:

1. Add BC layers to make as much as possible pass in #2770921: Feb 21st: Convert chunk of WTB to BTB by just moving classes, changing use statements adding traits until we are out of ideas on how to safely add BC. Test it extensively on all supported environments.
2. Identify specific changes we can't provide BC for, but that could be scripted.
3. Meanwhile, make sure things like documentation etc. get completed in parallel.
4. Then go down the route of changing base classes for remaining things.

We'd stack a series of a few large but individually conceptually simple issues together as a scheduled conversion, and then possibly work out the stragglers, rather than the other way around.

@xjm #71.1: If #2770921: Feb 21st: Convert chunk of WTB to BTB by just moving classes, changing use statements adding traits isn't just exploratory but is supposed to be part of a process it really needs an IS update for how it fits in.

Agreed on stacking large but simple issues, but I'm not sure what the atomic unit is here. We'd want to be able to review each piece without going insane. :-)

If we care that contrib is using our simpletest base classes, then we can make a BTB version of the base classes alongside the WTB ones, with deprecation info. We leave the WTB ones alone and modify the code core uses.

OTOH, there's already a bunch of work done in conversion per-module, so perhaps while the documentation is coming together we could grab those? We could postpone any issue for a module that defines a test base class.

Thanks @Mile23!

The work so far in the per module issues is definitely still useful; it gives us diffs for the comprehensive change record/conversion docs and proves BC layers that immediately work across core and not just in that module. I just don't think we should actually commit them individually at this time.

Per module patches also help with reviews, as maintainers of the particular modules can review their space.

Could we agree on the following approach maybe? @xjm
This gives quick progress now while still adding the BC layers we might want.

1. Take #2770921-10: Feb 21st: Convert chunk of WTB to BTB by just moving classes, changing use statements adding traits and remove every failing test
2. Add BC layers like described in #2770921-11: Feb 21st: Convert chunk of WTB to BTB by just moving classes, changing use statements adding traits
3. Try to move all use statements again
4. Goto Step 2
5. At some point identify tricky tests, Update path etc.

Haven't seen this in any of the related issues. A lot of tests have assertions like this:

    $this->drupalPostForm(NULL, $edit, t('Save'));
    preg_match('|entity_test/manage/(\d+)|', $this->url, $match);
    $id = $match[1];
    $this->assertText(t('entity_test @id has been created.', array('@id' => $id)), 'Entity was created.');

Has a precedence been set yet for how to handle this pattern ($this->url isn't in BrowserTestBase)? Or do we need to add $this->url?

Seems it would be better to assert before the post that no items in the database matches the posted criteria then load the item using the same criteria afterwards. You can then assert both that the entity was created and you have the id you're looking for to assert the message on the page.

Thanks so much klausi++

Thanks @klausi. Made some minor changes. Great work.

Opened #2784159: Remove CURL timeout in BTB for some better debugging experience

Small subissue: #2787923: Add a getDrupalSettings() method to JavascriptTestBase

So glad to see the handbook page! Thanks @klausi!!

The summary could definitely use an update here, for the handbook page, my comment in #69 and the discussions we had after it, etc.

Found one that should have been a kernel test, #2791955: Convert \Drupal\node\Tests\NodeAccessTest to a KernelTest - reviews welcome

If this is an initiative it surely would be nice if "each test runs in a completely new Drupal instance, which is created from scratch for the test" -- would get an option to not do that. Exercising an existing site is helpful and simpletest 7.x-2.x had an option for that while I know that's contrib, well, there's no place to put that now as simpletest is being removed and many contrib was core'd in D8.

We are already using the feature described in #92 with BTB and JTB for testing client projects by copying the whole setUp method, not calling parent::setUp and omitting anything which installs drupal or assumes a new environment. A very handy feature which would be fantastic if it was made more accessible and easier to get started with.

I can't but I am sure @Sam152 can.

Sure.

Also credit to @larowlan for doing the legwork to make this possible for our internal projects. Probably worth getting your thoughts on how we could achieve this.

Small little other issue: #2793451: Provide a debug helper to output the current HTML page

@chx
Please next time just open up a new issue, as this kind of stuff derails other discussions. Thank you!

Another related issue #2296635: Evaluate performance impact of limiting test discovery to *Test.php filename suffix

Before we ditch simpletest (which was running *.test files until very recently) we need to solve the following: When developing a test, you run phpunit path/to/test.php and the test passes but there is no warning the test won't be picked up when running a group of tests because it's not called *Test.php, because it violates PSR0/4 or whatever else it needs.

Alternatively we need to ditch phpunit and use a modern framework. Fantasy.

Note: #2296635: Evaluate performance impact of limiting test discovery to *Test.php filename suffix actually fixes the problem described by chx by failing in case a test class doesn't line up the with the expectations.

So well, can someone please decide the way forward?

IMHO going on with individual issues is still not a bad idea, because, well, we have found a tons of things along the way, while being able to work in parallel. Unlike codestyle issues, where the outcome is sort of predictable, its not on those conversion issues IMHO.

But well, whatever we do, please @xjm, @alexpott decide on something, because this is obviously better than waiting.

@chx: When developing a test, you run phpunit path/to/test.php and the test passes but there is no warning the test won't be picked up when running a group of tests because it's not called *Test.php, because it violates PSR0/4 or whatever else it needs.

phpunit --filter MyTest does full discovery and then filters with a regex. It's a bit slower but you can do sanity check with it when you add the test file.

@chx: When developing a test, you run phpunit path/to/test.php and the test passes but there is no warning the test won't be picked up when running a group of tests because it's not called *Test.php, because it violates PSR0/4 or whatever else it needs.

For my understanding is this a complaint against phpunit or how Drupal run's tests? The described behaviors seem to be consistent with how tests work in simpletest unless I'm missing something.

I've opened #2794285: [meta] Add backwards compatibility layer to browsertestbase to ease conversion from webtestcase to track the bc layer for webtestbase - please add additional child issues to that if you know of them.

Hi there! This is an issue that we'd ideally like to move to the new "Drupal core ideas" queue when/if #2785735: [policy, no patch] Agile process for bigger changes in core (Part A: Ideation) goes through (hoping for the next week or two). If you could read that issue and provide your thoughts on that, that would be great!

Actually, yeah, good call. Sorry I was just quickly running down the list of meta plans looking for things that seemed to fit.

Still, if you have thoughts, feel free to add them in that issue! :)

Another child issue: #2795037: BTB: Add cronRun function

Another one: #2795041: BrowserTestBase: Add drupalPostWithFormat

Another one: #2795043: Standardize setup methods of all views tests

Another one: #2795045: Make SchemaCheckTestTrait available outside of the config module

Another one: #2795049: Convert web tests to browser tests for history module

Another one: #2795051: Move \Drupal\simpletest\WebTestBase::drupalBuildEntityView into a trait and make it available in BTB

Another one: #2795071: [meta] Tests using raw DOMElement

Another one: #2795073: Add titleNotEquals to WebAssert and a assertNoTitle to AssertLegacyTrait

Another one: #2795085: Add assertNoCacheTag to assertLegacyTrait

Another one: #2795111: WTB to BTB, add getRawContent() in BTB

> that it would need buy-in from the community or core maintainers - we already have their blessing

You left out "Some of".

Another novice task: #2795543: Document that only plain string values are allowed in \Drupal\Tests\WebAssert::buildXPathQuery

Another non novice one: #2795567: Use Symfony's VarDumper for easier test debugging with dump()

Another novice one: #2795579: Adapt assertCacheContexts to not rely on the return value of assertIdentical

I have a question about this issue: Can you still make Verbose output in BrowserTestBase?

The reason I am asking is that we are using the Simpletest UI (which I also understand you want to get rid of, with as far as I can tell no currently viable alternative, which I commented on, on the other issue) to make automated screen shots for the User Guide project.

EDIT: What I meant by "other issue" here was: #2750461: Remove Simpletest UI because we don't want to maintain a graphical test runner

What I did was make a method that works like verbose output, that puts out a cleaner HTML file with some injected JavaScript in it, and then I use a combination of a Greasemonkey script and Image Magick to grab screenshots of these HTML output files and trim them down to nice images for the User Guide.

Will this be possible to do in BrowserTestBase, do you think? So far I have not been able to produce verbose output from the command line at all, and I would hate for the User Guide to die in Drupal 9 due to not being able to keep automating the screen shots.

When one would ask me, I would just commit it to 8.3.x, also all the other conversions, but keep adding the BC layers / additional assertions to 8.2.x as well.

We certainly killed the momentum completely which we had.

I'd love to see this proceeding, too, so that I can staging refactoring tests / split out kernel/unit tests at the same time.

#128 seems like a good idea to me, one caveat is whether git handles the renames when cherry-picking patches between 8.3.x and 8.2.x, if it doesn't that'd be a pain.

Added related #2801883: Add an empty $modules property in BrowserTestBase.

This probably needs to be handled after all conversions #2803621: Break BrowserTestBase & children dependency on Simpletest, deprecate stub BC traits

(Linking the issue that apparently fixed the random fails related to the Forum tests, which @dawehner had me find.)

Moving to the ideas queue as an active initative as per https://www.drupal.org/core/roadmap

Ok, this was listed as an initiative issue at https://www.drupal.org/core/roadmap. Also that it is a plan could also put it in the ideas queue, especially if it needs some signoffs, but if you believe this is the best place, let's keep it here.

Note that since that page now links to ideas queue lists, this being in the core queue will decrease the visibility as it will not be listed anymore.

I hope I am wrong about this, but in WTB, assertText() is case sensitive:

\Drupal\simpletest\AssertContentTrait

  protected function assertTextHelper($text, $message = '', $group = 'Other', $not_exists = TRUE) {
    if (!$message) {
      $message = !$not_exists ? SafeMarkup::format('"@text" found', array('@text' => $text)) : SafeMarkup::format('"@text" not found', array('@text' => $text));
    }
    return $this->assert($not_exists == (strpos($this->getTextContent(), (string) $text) === FALSE), $message, $group);
  }

Note the strpos in the actual assert().

In BTB, assertText is case insensitive. The BC later eventually calls \Behat\Mink\WebAssert

   public function responseContains($text)
    {
        $actual = $this->session->getPage()->getContent();
        $regex = '/'.preg_quote($text, '/').'/ui';
        $message = sprintf('The string "%s" was not found anywhere in the HTML response of the current page.', $text);

        $this->assert((bool) preg_match($regex, $actual), $message);
    }

Note the 'ui' modifier on the regex.

If this is true, what does this mean for WTB to BTB conversions or our current BC layer?

EDIT: I traced this out wrong. The deprecated BTB assertText eventually calls the BTB version of assertTextHelper(), which does a strpos. But, the docs say to use the Mink equivalents, which are case insensitive.

My last comment in demo test form. Note which of the BTB assertions is the first fail, which should be the one on line 28. Just setting NR for a bot run.

I couldn't find an existent issue on d.o., but prior to converting all of these, it would be really good to have a solution, or a documented workaround to running xdebug within page requests made from functional tests.

There is a PHPUnit PR from @dawehner to PHPUnit to fix this, but that would need to be pulled into core to utilize.

Interesting, I can debug both test code and web requests in functional tests with PHPStorm. Isn't this problem solved by increasing the number of connections in the PHPStorm XDebug settings @jhedstrom? If I am misunderstanding you, please post an example where the debugger won't kick in.

Yes, I can confirm that xdebug is working for Unit/Kernal/Browser/JavaScript test but only from cli(phpunit) or phpstrom not sure about run-tests.sh. Just to be on the safe side I raised the number of connections to 5.

@juampynr @jibran I've opened #2850891: xdebug breakpoints not working in web requests from BrowserTestBase to discuss further since we are having varying results :)

Here is an issue for system module: #2862508: Convert system functional tests to phpunit

Added #2863267: Convert web tests of views

Added #2863268: Convert web tests to browser tests for tracker module

I think one of the recent edits lost some changes.

Added #2863372: Convert dblog web tests to BrowserTestBase

Added #2863416: Convert web tests to browser tests for filter module

Added #2863597: Convert web tests to browser tests for toolbar module

Added #2863607: Convert WebTestBaseTests of BigPipe to BrowserTestBase.

Added #2863644: Convert web tests of block.

Added #2863996: Convert web tests to browser tests for ckeditor module

Add following
#2864005: Convert web tests to browser tests for block_content module
#2863842: Convert web tests to browser tests for search module
#2863969: Convert web tests to browser tests for options module
#2863984: Convert [1/3] web tests to browser tests for responsive_image module

Add #2864035: Convert web tests to browser tests for rdf module

Added: #2864008: Convert web tests to browser tests for content_moderation module

Add #2864088: Convert web tests to browser tests for shortcut module

Add #2864121: Convert web tests to browser tests for statistics module

I think this only affects the RDF tests, #2864201: XPath in WebTestBase and BrowserTestBase use different roots. Need to see the lang attribute on <html> is tested .

Statistics and many modules require the conversion of \Drupal\simpletest\AssertContentTrait::assertNoPattern(). Follow #2864257: Convert web tests AssertNoPattern to Browser Test

Add #2865407: Convert web tests to browser tests for contextual module but waiting after #2795041: BrowserTestBase: Add drupalPostWithFormat

Add #2865413: Move RegisterSerializationClassesCompilerPassTest.php to correct location

fix wrong markup list

Added: #2865537: Convert web tests to browser tests for minimal profile

If I understand this correctly we have like 890 to go:

$ find core/modules -print | grep "src/Tests" | wc -l
893

Added: #2866513: Convert web tests to browser tests for comment module

Added details for migrate modules.

Added to IS:
system: #2862508: Convert system functional tests to phpunit

Added note on Wizard tests to IS, separate issue created: #2867777: Convert views WizardTestBase tests to phpunit

Added various issues for module conversions in the issue summary

All module conversion issues created!

michielnugter Thanks a lot!

Link fix.

All new issues that are postponed contain the tests in the IS that it is postponed on.

Updated issue summary with the current best approach and some hints/tips on common conversions.

Add sub-issue for language module #2871740: Convert web tests to browser tests LanguageTourTest for language module

We start to make good progress, see the progress report in the issue queue.

Make #2872059: Convert web tests to browser tests for locale module.

Added 'phpunit initiative' tag to this issue and all it's sub-issues. Will add it to every related issue so we can use the tag to track all issues related to the initiative.

Updated IS to state the tag usage.

Impressive work, seriously!

Updating the todo list, good progress!

Updated IS.

//Naveen

This has landed #2747167: Convert first part of web tests of views_ui

These landed #2757023: Convert all aggregator web tests to BrowserTestBase and #2867304: Convert web tests to browser tests for migrate_drupal_ui module and #2863372: Convert dblog web tests to BrowserTestBase

Updating IS

Two more landed today #2870441: Convert web tests to browser tests for content_translation module #2872059: Convert web tests to browser tests for locale module

//Naveen

#2870459: Convert web tests to browser tests for taxonomy module has landed! Added #2887134: Convert web tests to browser tests for taxonomy module Part -2 to the list.

Can I quickly say how impressed I am how many tests we've already converted?

#2864005: Convert web tests to browser tests for block_content module has landed today. Added follow-up #2887311: Convert BlockContentTypeTest web tests to browser tests for block_content module to the IS

#212,
In 8.4.x latest Head 578 conversions are left. As per IS, in 8.2 we have 1152 tests. So from 8.2, we have converted ~570+ tests so far.

$ find core/modules -print | grep "src/Tests/" | wc -l
     578

P.S: I'll PM you the link of the PHPUnit initiative website over IRC which @lendude has shared with me. It's a nice one, it shows the progress in graphs.

//Naveen

@naveenvalecha no need to share, I've just moved it live:

http://simpletest-countdown.org

Want to know how the initiative is progressing? Check the website to see how we are doing! Currently we are taking a fresh datapoint every tagged release of Drupal, so about every month.

The site is a work in progress, so any feedback is appreciated, please just use the contact form on the site or send me a tell on IRC.

Thanks, @lendude!

#2888539: Convert TrackerUserUidTest web tests to browser tests and #2212869: Extend ContactAuthenticatedUserTest for anonymous user landed today. Added them to IS

//Naveen

#2870464: Convert web tests to browser tests for update module, #2889350: Convert BookRelationshipTest web tests to BrowserTestBase, #2889352: Convert ContactFieldsTest & ContactLinkTest web tests to browser tests for contact module and #2889353: Convert ForumIntegrationTest web tests to browser tests for forum module landed today.
Updated IS accordingly.
//Naveen

#2870440: Convert web tests to browser tests for config_translation module landed as well. Updated IS accordingly.

//Naveen

No need to state Views and WizardTestBase test shouldn't be converted as both have landed.

Want to know how we did going from 8.3.3 to 8.3.5? The scores are up on http://simpletest-countdown.org

Really great progress was made, thanks everybody for helping out! Plenty more to do, so lets keep it up :)

#2775553: Convert web tests to browser tests for REST module is no longer blocked. I converted one of the tests — anybody else want to continue? :)

List conversions which are blocked grouped by blockers