PHPUnit initiative

Sign me up!

I assigned @mglaman up manuall.

Thanks for filing this!

Things I see missing from the roadmap are:

• The documentation.
  • Daniel has a rudimentary patch to update some codebase documentation.
  • klausi has a handbook page started for BTB. We need to review these docs.
  • We also need to write comprehensive docs for JSTB, including possibly reaching out to the PhantomJS maintainers. (nod_ said it was difficult to even know how to test a certain interaction.)
• The process for informing contributors about the upcoming change.
• The test runner UI parity that we discussed.

Oh, and also the "after the big bang" part: How we plan to proceed once the conversion is done (including for example a series of issues to convert the test suite away from the legacy methods one at a time, the eventual process to deprecate SimpleTest, etc.).

Moving to the ideas queue.

We also had identified upgrade path tests as an area that needed particular research for how we will update them.

Let's plan/research those hard conversions on its own issue: #2808777: Research: Complex tests to convert

How about per-module conversions already done but not committed yet? Will they be abandoned?

How about per-module conversions already done but not committed yet? Will they be abandoned?

That's a fair question. Could most of them be somehow be reused when we would have some sort of forward compatibility layer in place, so people could use mink style elements in WebTestBase already?

Adde

I think @xjm's points are covered in the IS as of now.

Adding myself.

I don't completely understand why a "big bang" is needed - The new PHPUnit could be added, get miles on them in parallel, then Simpletest versions of the identical tests could be removed later. This could be done system-by-system as well. Sorry I'm late to the conversation.

I don't completely understand why a "big bang" is needed

I think we want to avoid the situation where we are maintaining two versions of the same tests, or test files In an ideal world new additions to a BTB would use the Mink-style assertions and WTB would have to use the ST versions.

I just opened up a research issue about API tests: #2812967: API test base: Provide a test helper to test REST and similar APIs

This plan looks great to me and covers what we discussed at Dublin. I think the plan is RTBC.

We plan to discuss the proposed initiative on the core ideas review call today.

We talked about this today on our bi-weekly Ideas queue call. The plan seems very solid, and aims to minimize disruption to other initiatives as much as possible. There was broad agreement from all core committers present that this is awesome.

The core committers discussed this and we all agree (including me) that it would be great to make this an initiative. We'd like this to be minimally disruptive though.

Noting that this Plan has been approved.

Oops, doing this instead.

We should get a spin-off issue in the core queue to put an entry for MAINTAINERS.txt for this initiative.

That issue is here: #2808121: Add PHPUnit initiative to MAINTAINERS.txt

Edit: wrong link. Fixed now. :)

Thank you all a lot!

Instead of wasting our time on random JS fails why not just fix #2775653: JavascriptTests with webDriver?

A proposal #2834021: Provide error test assertions via a trait.

As part of #2847428: [Meta] Modernize dblog module I'm working on #2458191: Provide a storage backend for dblog module which introduces new Kernel tests for the dblog module.

What is the plan for the existent WebTestBase tests? dblog module test coverage is basically with WebTestBase. Some of those tests could be removed because we now have phpunit tests. Do we leave them untouched?. Is any discussion regarding that type of conversions?

Phpunit 6.0 came out. We cannot switch directly to it, as its just compatible with php 7.0, but we can use some FC layer introduced by sebastian: #2850797: Prepare our phpunit tests to be BC compatible with phpunit 5.x/6.x

#2851541: [policy, no patch] No new WebTestBase tests in Drupal 8 after mass-conversion has been proposed as a followup to #2770921: Feb 21st: Convert chunk of WTB to BTB by just moving classes, changing use statements adding traits, which should help reduce conflicts or the need for rework as we move on to next steps.

I recently updated the testing gate policy for this initiative:
https://www.drupal.org/node/1203498/revisions/view/10353348/10353373

However, it's actually a bit too in-the-weeds there. One thing I noticed while adding references to that page is that it's really hard to tell where to start when reading our testing documentation. (The recent handbook docs upgrade doesn't help with this either.)

I think we need a simple, high-level page summarizing what PHPUnit is, what the four kinds of PHPUnit tests Drupal provides API for are, and when you would use each of them. Then, that page can link to unit, KTB, BTB, and JSTB docs in the handbook as appropriate.

We'd also include a small note under the BTB section that says to still extend existing SimpleTest web tests until they are converted rather than adding duplicate tests, with a link to the old SimpleTest documentation.

Updating the summary for this.

So #2782663: Convert web tests to browser tests for syslog module just percolated up the RTBC queue again, and it is basically a conversion of a single xpath assertion to be BTB-compatible, and then moving the test. Maybe we should have a single issue for converting tests that had to be skipped because of an xpath assertion, rather than converting one test at a time? Thanks!

#56, I would say yes. Those tests should have been blocked by #2784537: Add legacy assertFieldByXPath()/assertNoFieldByXPath() method for browser tests, which has been in for a while and is working well.

Note: These tests might require more changes than just the xpath ones.
We could still provide the forward compabiility layer in #2809181: Provide forward compatibility layer for BrowserTestBase::xpath and thereby really just have to deal with the xpath changes for now. This allows IMHO in total more scoped changes.

@dawehner, yeah, I would say if the test passes with just adjusting the xpath, then put it together in a single patch. If it has another issue also blocking it or requires rewriting, then follow up on the individual test as needed. Exploring #2809181: Provide forward compatibility layer for BrowserTestBase::xpath sounds like a good idea too.

@klausi, can we maybe start with a sub-meta that lists the tests that need to be fixed for xpath and how? Then we can go from there. If it indeed only makes sense to convert per-module, then we'll do that, but it would be helpful to have an issue that helps with the big picture on it rather than one-offs without the full context. And maybe we can spot a pattern or two that will make things more consistent.

I'll create a subissue for that.

So I think there is no easy common ground of conversions for all of core. I think we need to continue on a case by case basis and improve our compatibility with Simpletest as we go (tracked in #2794285: [meta] Add backwards compatibility layer to browsertestbase to ease conversion from webtestcase).

Mh rereading your comment let me think about it again. I agree just making straight conversions don't cut it.

What about trying to keep conversions together on your github again, and then try to find common patterns by doing so?

I don't think there is really a sensible alternative to the per module other than per test. The xpath conversions are going to depend on what is looked at as far as I can see. Given that this is a one of task that we won't regress on like coding standards I think it is appropriate to just get things done.

Thank you alexpott!

Added child: #2863044: [plan] Remove simpletest dependencies from PHPUnit-based tests

Hmm. The components changed. Pretty sure this is approved.

Yes we discussed changing the components, but I don't think Active Initiative was supposed to be removed. I pinged @webchick about it.

Do we have an approach for batch?

Old web-test-base used to process the batch until it was done.

A post that triggers a batch doesn't do that with BTB, after the first page (progress cycle) we are back to the test runner/assertions.

Something like this in base class would be useful?

/**
   * Checks for meta refresh tag and if found call drupalGet() recursively.
   *
   * This function looks for the http-equiv attribute to be set to "Refresh" and
   * is case-sensitive.
   *
   * @param int|NULL $max_count
   *   Maximum number of refreshes. NULL for unlimited.
   * @param int $count
   *   Current refresh count.
   */
  protected function checkForMetaRefresh($max_count = NULL, $count = 0) {
    if ($max_count && $count > $max_count) {
      return;
    }
    if ($refresh = $this->xpath('//meta[@http-equiv="Refresh"]')) {
      // Parse the content attribute of the meta tag for the format:
      // "[delay]: URL=[page_to_redirect_to]".
      if (preg_match('/\d+;\s*URL=(?<url>.*)/i', $refresh[0]->getAttribute('content'), $match)) {
        $this->drupalGet($this->getAbsoluteUrl(Html::decodeEntities($match['url'])));
        $this->checkForMetaRefresh($max_count, $count + 1);
      }
    }
  }

@larowlan Do you mind creating its own issue for that?

Thanks @klausi and @dawehner.

Batch processing works without Javascript, so no need for JTB, it uses the <noscript><meta equiv="refresh"... trick

We should not loose the PHP test coverage of batch processes IMHO. For most batch processes its an implementation detail that its using JS mostly. At least for me JavascriptTestBase is about testing JS.

At this point I think we can also update https://www.drupal.org/contributor-tasks/write-tests to not tell contributors to use SimpleTest. :)

Active initiative component is back. Woot! See #70.

Presented for your bemusement: #2866082: [Plan] Roadmap for Simpletest

Updated IS to reflect 'phpunit initiative' tag usage and added me and LenDude to the regular contributers. Will move our names up once we're the official leads.

Let's make it a bit clearer.

As PHPUnit 4 is no longer supported, any plan on switch to PHPUnit 5?

Is it possible right now to use PHPUnit 5/6?

@el7cosmos:

There is an issue open for the upgrade to a newer PHPUnit here: #2578813: Figure out PHPUnit version support. It seems that for now in Drupal it's not possible to use PHPUnit 5 or 6.

There is an issue open for the upgrade to a newer PHPUnit here: #2578813: Figure out PHPUnit version support. It seems that for now in Drupal it's not possible to use PHPUnit 5 or 6.

Yeah, we can't as the mimimal php version is 5.5

Added this old issue as a child: #2825078: Deprecate UnitTestCase dependency on block module

I never did add this as related.

Latest count

find core/modules -print | grep "src/Tests" | grep "Test.php" | wc -l
302

Hello all,
is there any chance to be part of this initiative?
Thank you for reply.
Hermann

@hazong
A good way would be to jump onto the #phpunit slack channel, I'm happy to point to tasks there for you.

Can we get another update of http://simpletest-countdown.org/? Last update was on November 16, 3 months ago in 10 days.

@Wim Leers yes we can! Updated to 8.5.0-alpha1

No more low hanging fruit, so it's slowing down as expected.

Still, 29 fewer Simpletest tests :) Thanks for updating that, @Lendude!

There are a bunch of test base deprecations from #2770921: Feb 21st: Convert chunk of WTB to BTB by just moving classes, changing use statements adding traits. Ones like this: https://cgit.drupalcode.org/drupal/diff/core/modules/aggregator/src/Test...

This was from before we were working hard on adding change records, so there isn't one for the PHPUnit initiative.

We'd like to @trigger_error() on pre-existing deprecated test base classes, as in #2886547: Add trigger_error to deprecated TestBase base classes/traits that have none, reference CR

But there are no change records to add to the deprecation messages, so we should either figure out a CR for this initiative, or determine a more appropriate existing CR to add.

I'd like to get involved with this. I'll take a look for "phpunit initiative" tagged issues and see if I can find a good one to start with.

@opdavies
Feel free to also join #testing on Drupal slack :)

Thanks @dawehner. I was in #testing already but I've also joined the #phpunit channel.

We are so close now:

find core/modules -print | grep "src/Tests" | grep "Test.php" | grep -v simpletest
core/modules/system/src/Tests/Session/SessionHttpsTest.php
core/modules/file/src/Tests/FileFieldWidgetTest.php
core/modules/field_ui/src/Tests/ManageDisplayTest.php
core/modules/quickedit/src/Tests/QuickEditAutocompleteTermTest.php
core/modules/quickedit/src/Tests/QuickEditLoadingTest.php

I think we should close this after #3027952: [Plan] Remove the usage of deprecated methods in tests.

I think this can be marked as fixed now as #2735005: Convert all Simpletest web tests to BrowserTestBase (or UnitTestBase/KernelTestBase) is done.

Now we have #3038234: The PHPUnit Initiative is complete! so marking this as fixed.

::points to 'needs change record' tag::

Are the change record links still missing from the deprecations? If so, I think that would be a dedicated issue ideally not a holdup of the whole initiative meta issue?

is that it? https://www.drupal.org/node/3129738