Background

We want to get rid of simpletest.module in Drupal 9. We want to prepare the latest Drupal 8 development branch as much as possible before opening the Drupal 9 development branch (see #2608062: [policy, no patch] Pre-requisites for opening Drupal 9). Therefore we need to convert all web tests to BrowserTestBase or UnitTestBase or KernelTestBase and deprecate WebTestBase.

More discussion can be found in the original phpunit conversion issue at #2735005: Convert all Simpletest web tests to BrowserTestBase (or UnitTestBase/KernelTestBase).

Process for the big bang conversion patch (DONE!)

Because changing tests to a new system is a disruptive change we want to schedule a date in February 2017 where we commit one "big bang" patch prepared in #2770921: Feb 21st: Convert chunk of WTB to BTB by just moving classes, changing use statements adding traits. It will contain scripted test case conversions where only files are moved to use BrowsertestBase.

Converting Simpletests per module

After this big bang patch is committed there will be some remaining Simpletests that we need to convert to PHPUnit manually per module. This is coordinated in #2735005: Convert all Simpletest web tests to BrowserTestBase (or UnitTestBase/KernelTestBase).

We will deprecate Simpletest's WebTestBase in 8.4.x, but not for 8.3.x releases. Between the release of 8.3.0 and 8.4.0 we will polish our PHPUnit browser tests and then WebTestBase will be deprecated and forbidden for new tests with the release of 8.4.0 in October 2017. See #2847678: Deprecate WebTestBase in favor of BrowserTestBase.

Process for compatibility and better tests

As a parallel effort we will improve the compatibility layer between the old WebTestBase and BrowserTestBase. Those issues will be committed to Drupal core immediately when they are ready (no need to wait for them until February 2017.

Documentation updates

Not in scope

todo

Team and resources

Initiative coordinators:

Communication and planning:
todo

Regular contributors

  • Claudiu Cristea (claudiu.cristea)
  • Matt Glaman (mglaman)
  • Jibran Ijaz (jibran)
  • larowlan when time permits
  • naveenvalecha
  • Mile23
  • Please add yourself ...

Senior advisors
oh yeah, we need that too. Maybe @alexpott?

IRC: #drupal-phpunit
slack: #phpunit

Issue creation

Please add the tag 'phpunit initiative' to every issue and add the issue to the correct Component (if it's a test for the comment module, add it to comment.module).

Signoffs

We have signoffs from a Drupal 8 release manager (xjm) and a Drupal 8 framework manager (alexpott) as well as Dries.

Comments

klausi created an issue. See original summary.

mglaman’s picture

Sign me up!

claudiu.cristea’s picture

Issue summary: View changes
dawehner’s picture

Issue summary: View changes

I assigned @mglaman up manuall.

klausi’s picture

Issue summary: View changes

Added a new IRC channel: #drupal-phpunit and set it to auto-join for me. Please chat there about this!

jibran’s picture

Issue summary: View changes
larowlan’s picture

Issue summary: View changes
klausi’s picture

Opened #2808121: Add PHPUnit initiative to MAINTAINERS.txt to get this into MAINTAINERS.txt.

Now we should recruit a project manager to help us with communication and marketing.
Example tasks:
* Schedule weekly meetings in IRC or hangouts
* Ping people on IRC before the meeting starts
* Publish update posts about progress every 2-3 weeks
* Assign reviewers to issues that contributors work on
etc.

xjm’s picture

Thanks for filing this!

Things I see missing from the roadmap are:

  • The documentation.
    • Daniel has a rudimentary patch to update some codebase documentation.
    • klausi has a handbook page started for BTB. We need to review these docs.
    • We also need to write comprehensive docs for JSTB, including possibly reaching out to the PhantomJS maintainers. (nod_ said it was difficult to even know how to test a certain interaction.)
  • The process for informing contributors about the upcoming change.
  • The test runner UI parity that we discussed.
klausi’s picture

Assigned: Unassigned » Dries
Issue summary: View changes

Assigning to Dries to take a look at this initiative and sign it off.

xjm’s picture

Oh, and also the "after the big bang" part: How we plan to proceed once the conversion is done (including for example a series of issues to convert the test suite away from the legacy methods one at a time, the eventual process to deprecate SimpleTest, etc.).

klausi’s picture

Assigned: Dries » Unassigned

OK, let's make the issue summary a bit more complete first.

xjm’s picture

Project: Drupal core » Drupal core ideas
Version: 8.3.x-dev »
Component: phpunit » Plan
Status: Active » Needs work

Moving to the ideas queue.

xjm’s picture

We also had identified upgrade path tests as an area that needed particular research for how we will update them.

klausi’s picture

I think we already discussed that this is not really a core idea because we will do the phpunit conversion in any case and we decided that long ago. It is just a matter of coordination how we get there :) See #2735005-109: Convert all Simpletest web tests to BrowserTestBase (or UnitTestBase/KernelTestBase).

Anyway, we can move it back to core once the issue summary is ready.

klausi’s picture

Issue tags: +Dublin2016
dawehner’s picture

Issue summary: View changes

Let's plan/research those hard conversions on its own issue: #2808777: Research: Complex tests to convert

claudiu.cristea’s picture

How about per-module conversions already done but not committed yet? Will they be abandoned?

dawehner’s picture

How about per-module conversions already done but not committed yet? Will they be abandoned?

That's a fair question. Could most of them be somehow be reused when we would have some sort of forward compatibility layer in place, so people could use mink style elements in WebTestBase already?

naveenvalecha’s picture

Issue summary: View changes
klausi’s picture

@claudiu.cristea: yes, we can start conversion issues per module where the tests need manual updates. Then we get more insight into common patterns we need to change and might come up with better legacy compatibility for our browser tests.

The most important thing is fighting random test fails right now that we see in #2808085: Pipe char in locators break Mink and Symfony element search and #2770921: Feb 21st: Convert chunk of WTB to BTB by just moving classes, changing use statements adding traits.

klausi’s picture

Issue summary: View changes
dawehner’s picture

Issue summary: View changes

Adde

klausi’s picture

Issue summary: View changes
Status: Needs work » Needs review

I think I covered xjm's points now in the issue summary. Also openend a couple of sub-issues.

Can somebody review the issue summary now and provide feedback?

Mile23’s picture

I think @xjm's points are covered in the IS as of now.

Mile23’s picture

Issue summary: View changes

Adding myself.

rfay’s picture

I don't completely understand why a "big bang" is needed - The new PHPUnit could be added, get miles on them in parallel, then Simpletest versions of the identical tests could be removed later. This could be done system-by-system as well. Sorry I'm late to the conversation.

mpdonadio’s picture

I don't completely understand why a "big bang" is needed

I think we want to avoid the situation where we are maintaining two versions of the same tests, or test files In an ideal world new additions to a BTB would use the Mink-style assertions and WTB would have to use the ST versions.

klausi’s picture

@rfay: several random test failures have been identified during the first conversion attempts to BrowserTestBase (example: #2737805: Convert web tests to browser tests for forum module). If we go on with converting then core committers will see more random test failures and will have to revert back and forth conversions until the new browser test system becomes more stable. That is another reason why we plan a big bang patch and a separate conversion branch that we will heavily test against all environments first.

Currently #2808085: Pipe char in locators break Mink and Symfony element search is a hard blocker that causes random fails for example.

dawehner’s picture

I just opened up a research issue about API tests: #2812967: API test base: Provide a test helper to test REST and similar APIs

klausi’s picture

Issue summary: View changes

@xjm: are you now happy with the outline in the issue summary?

klausi’s picture

Issue summary: View changes
klausi’s picture

Issue summary: View changes
klausi’s picture

Quick status update:
We are still waiting on xjm to get feedback from Dries about officially launching the phpunit initiative.

It looks like we managed to get the random test fails under control with some help from the fine Mink and Symfony folks. Our big bang conversion issue is currently green, yay! #2770921: Feb 21st: Convert chunk of WTB to BTB by just moving classes, changing use statements adding traits.

We need help with reviews for the following issues:
#2820442: SimpleTestBrowserTest has random fails when BrowserTestBaseTest gets larger
#2808085: Pipe char in locators break Mink and Symfony element search

We need help with contributions, for example if you want to try the new core javascript tests there is #2809161: Convert Javascript/AJAX testing to use JavascriptTestBase to pick from :)

We also have some bad news: #2605664: [Needs change record updates] Align TestDiscovery and bootstrap.php's non-permissive loading of PSR-4 namespaces for traits is stalled out of confusion and not going anywhere after 200 comments. If you have any idea how to proceed there - please help!

xjm’s picture

Status: Needs review » Reviewed & tested by the community

This plan looks great to me and covers what we discussed at Dublin. I think the plan is RTBC.

We plan to discuss the proposed initiative on the core ideas review call today.

webchick’s picture

We talked about this today on our bi-weekly Ideas queue call. The plan seems very solid, and aims to minimize disruption to other initiatives as much as possible. There was broad agreement from all core committers present that this is awesome.

Dries’s picture

The core committers discussed this and we all agree (including me) that it would be great to make this an initiative. We'd like this to be minimally disruptive though.

webchick’s picture

Status: Reviewed & tested by the community » Fixed

Noting that this Plan has been approved.

webchick’s picture

Component: Plan » Active Initiative
Status: Fixed » Active

Oops, doing this instead.

We should get a spin-off issue in the core queue to put an entry for MAINTAINERS.txt for this initiative.

xjm’s picture

That issue is here: #2808121: Add PHPUnit initiative to MAINTAINERS.txt

Edit: wrong link. Fixed now. :)

klausi’s picture

yay! Thanks all!

dawehner’s picture

Thank you all a lot!

klausi’s picture

Status update:
We are still waiting for core committers to commit which is RTBC for quite some time #2784537: Add legacy assertFieldByXPath()/assertNoFieldByXPath() method for browser tests. It is also the last puzzle piece missing for our bing bang conversion patch in #2770921: Feb 21st: Convert chunk of WTB to BTB by just moving classes, changing use statements adding traits.

Unfortunately we are struggling with lots of intermittent test failures in many places in our test framework, see #2829040: [meta] Known intermittent, random, and environment-specific test failures. One part of that are test failures in JavascriptTestBase on PHP 5 which we try to figure out in 2 issues:
#2830485: \Drupal\Tests\outside_in\FunctionalJavascript\OutsideInBlockFormTest fails randomly
#2832672: Upgrade jcalderonzumba/* for better test performance

At this point I cannot recommend the use of JavascriptTestBase because it exposes these instabilities on the testbot. Hopefully we can resolve those problems soon.

Documentation for JavascriptTestBase was started at https://www.drupal.org/node/2832672 . It could use some common tips and tricks we use when writing javascript tests and a step by step example such as on https://www.drupal.org/docs/8/phpunit/phpunit-browser-test-tutorial . Please edit!

For people using the Simpletest UI we have now an improved PHPUnit integration in 8.3.x, so you should get somewhat more meaningful output from PHPUnit (done in #2809117: Integrate PHPUnit verbose output in Simpletest UI).

Please post any other updates that I forgot!

jibran’s picture

Instead of wasting our time on random JS fails why not just fix #2775653: [PP-2] JavascriptTests with webDriver?

klausi’s picture

@jibran: we don't know if that will fix the random test fails - in the end it is just a different interface to PhantomJS. But in any case I would love to see some progress on it, please reroll the patch there!

claudiu.cristea’s picture

claudiu.cristea’s picture

klausi’s picture

Issue summary: View changes

Status update:
Phase 1 of the PHPunit initiative is nearing completion now. We are ready to commit a larger chunk of conversions from the old WebTestBase to BrowserTestbase in #2770921: Feb 21st: Convert chunk of WTB to BTB by just moving classes, changing use statements adding traits. We have some random fails in some environments there, but it does not seem related to the issue. Please review!

As planned we will also announce a date when we commit this conversion to 8.3.x. I propose February 27th 2017, shorty before 8.3.0-rc1 is released. Once the conversion issue is RTBC I will coordinate with core committers to get a date fixed and announced.

On the JavascriptTestBase work I think we identified some paths forward to improve random fails during Ajax requests. One takeaway is that you should not issue wait() calls on jQuery progress for example but rather write JS conditions on changes in the user interface or DOM. Example: when waiting for autocomplete results use a JS condition to wait for a list of suggestions to pop up.

We are still in the process of improving Javascript tests, see
#2837676: Provide a better way to validate all javascript activity is completed
#2831603: Improve assertion for testEntityReferenceAutocompleteWidget
#2832672: Upgrade jcalderonzumba/* for better test performance

dagmar’s picture

As part of #2847428: [Meta] Modernize dblog module I'm working on #2458191: Provide a storage backend for dblog module which introduces new Kernel tests for the dblog module.

What is the plan for the existent WebTestBase tests? dblog module test coverage is basically with WebTestBase. Some of those tests could be removed because we now have phpunit tests. Do we leave them untouched?. Is any discussion regarding that type of conversions?

klausi’s picture

I'm not aware of specific dblog test discussions. If the web tests cover the admin UI then they should stay, because that cannot be covered to kernel tests, right? All web tests will be converted to PHPUnit browser tests eventually, also in dblog module. If the web tests are legacy tests that only call the API and do not test the UI, then they should be converted to kernel/unit tests. Please open issues for that as you see fit!

dawehner’s picture

Issue summary: View changes

Phpunit 6.0 came out. We cannot switch directly to it, as its just compatible with php 7.0, but we can use some FC layer introduced by sebastian: #2850797: Prepare our phpunit tests to be BC compatible with phpunit 5.x/6.x

klausi’s picture

Pushed out the core announcement that we will commit our big bang browser test conversion patch on Tuesday: https://groups.drupal.org/node/516229

xjm’s picture

Issue summary: View changes
xjm’s picture

Issue summary: View changes

I recently updated the testing gate policy for this initiative:
https://www.drupal.org/node/1203498/revisions/view/10353348/10353373

However, it's actually a bit too in-the-weeds there. One thing I noticed while adding references to that page is that it's really hard to tell where to start when reading our testing documentation. (The recent handbook docs upgrade doesn't help with this either.)

I think we need a simple, high-level page summarizing what PHPUnit is, what the four kinds of PHPUnit tests Drupal provides API for are, and when you would use each of them. Then, that page can link to unit, KTB, BTB, and JSTB docs in the handbook as appropriate.

We'd also include a small note under the BTB section that says to still extend existing SimpleTest web tests until they are converted rather than adding duplicate tests, with a link to the old SimpleTest documentation.

Updating the summary for this.

klausi’s picture

Issue summary: View changes

Hooray, we have reached the first milestone of the PHPunit initiative! High fives all around and thanks a lot for all your help!

With the mass conversion patch we have now unblocked lots of remaining conversion issues per module. #2735005: Convert all Simpletest web tests to BrowserTestBase (or UnitTestBase/KernelTestBase) is the parent issue for that where we create child issues for each module to make all remaining Simpletests into PHPUnit browser tests. Feel free to jump in now and work on already existing patches or create new issues for core modules that don't have a conversion issue yet.

xjm’s picture

So #2782663: Convert web tests to browser tests for syslog module just percolated up the RTBC queue again, and it is basically a conversion of a single xpath assertion to be BTB-compatible, and then moving the test. Maybe we should have a single issue for converting tests that had to be skipped because of an xpath assertion, rather than converting one test at a time? Thanks!

mpdonadio’s picture

#56, I would say yes. Those tests should have been blocked by #2784537: Add legacy assertFieldByXPath()/assertNoFieldByXPath() method for browser tests, which has been in for a while and is working well.

dawehner’s picture

Note: These tests might require more changes than just the xpath ones.
We could still provide the forward compabiility layer in #2809181: Provide forward compatibility layer for BrowserTestBase::xpath and thereby really just have to deal with the xpath changes for now. This allows IMHO in total more scoped changes.

xjm’s picture

@dawehner, yeah, I would say if the test passes with just adjusting the xpath, then put it together in a single patch. If it has another issue also blocking it or requires rewriting, then follow up on the individual test as needed. Exploring #2809181: Provide forward compatibility layer for BrowserTestBase::xpath sounds like a good idea too.

klausi’s picture

I left a comment in #2809181: Provide forward compatibility layer for BrowserTestBase::xpath why I think it is too late for a forward compatibility layer.

The syslog issue is a single xpath conversion because there is only one old Simpletest in syslog module.

I don't see an easy way to get a list of all tests that require a change of xpath handling. I also think it is much harder to tackle all tests in one issue fixing all the xpath things at once. It is far easier to distribute the conversion work per module on multiple contributors. As you can see in
#2782663: Convert web tests to browser tests for syslog module
#2737805: Convert web tests to browser tests for forum module
the changes that need to be applied are all a bit different and specific to the test case in question. Sometimes we remove methods that don't exist anymore (->parse()). Sometimes we use ->getText() of the NodeElement returned by ->xpath(). Sometimes ->getAttribute(). Sometimes we use other helpers like ->buttonExists().

So I think there is no easy common ground of conversions for all of core. I think we need to continue on a case by case basis and improve our compatibility with Simpletest as we go (tracked in #2794285: [meta] Add backwards compatibility layer to browsertestbase to ease conversion from webtestcase).

Let me know if you have a better idea and what specifically we should do different, otherwise let's continue on a per module basis.

xjm’s picture

@klausi, can we maybe start with a sub-meta that lists the tests that need to be fixed for xpath and how? Then we can go from there. If it indeed only makes sense to convert per-module, then we'll do that, but it would be helpful to have an issue that helps with the big picture on it rather than one-offs without the full context. And maybe we can spot a pattern or two that will make things more consistent.

dawehner’s picture

I'll create a subissue for that.

dawehner’s picture

So I think there is no easy common ground of conversions for all of core. I think we need to continue on a case by case basis and improve our compatibility with Simpletest as we go (tracked in #2794285: [meta] Add backwards compatibility layer to browsertestbase to ease conversion from webtestcase).

Mh rereading your comment let me think about it again. I agree just making straight conversions don't cut it.

What about trying to keep conversions together on your github again, and then try to find common patterns by doing so?

klausi’s picture

@xjm: created a meta issue: #2856916: Converting XPath handling from Simpletest to PHPUnit browser tests. Will try to collect conversion examples there so that we see if easy patterns emerge. Please amend it as you see fit!

@dawehner: what would be the benefit to keep conversions in some external git branch instead of committing them straight to core at this point?

We are going to put module based conversions back to RTBC now when they are ready, let me know if we should still hold back.

alexpott’s picture

I don't think there is really a sensible alternative to the per module other than per test. The xpath conversions are going to depend on what is looked at as far as I can see. Given that this is a one of task that we won't regress on like coding standards I think it is appropriate to just get things done.

dawehner’s picture

Thank you alexpott!

klausi’s picture

We will have an in person PHPUnit initiative meeting today at Drupal Dev Days Seville at 12:00, left corner in main area. Feel free to join if you are in Seville!

klausi’s picture

Mile23’s picture

Component: Active Initiative » Idea
Mile23’s picture

Component: Idea » Approved Plan

Hmm. The components changed. Pretty sure this is approved.

xjm’s picture

Yes we discussed changing the components, but I don't think Active Initiative was supposed to be removed. I pinged @webchick about it.

larowlan’s picture

Do we have an approach for batch?

Old web-test-base used to process the batch until it was done.

A post that triggers a batch doesn't do that with BTB, after the first page (progress cycle) we are back to the test runner/assertions.

larowlan’s picture

Something like this in base class would be useful?

/**
   * Checks for meta refresh tag and if found call drupalGet() recursively.
   *
   * This function looks for the http-equiv attribute to be set to "Refresh" and
   * is case-sensitive.
   *
   * @param int|NULL $max_count
   *   Maximum number of refreshes. NULL for unlimited.
   * @param int $count
   *   Current refresh count.
   */
  protected function checkForMetaRefresh($max_count = NULL, $count = 0) {
    if ($max_count && $count > $max_count) {
      return;
    }
    if ($refresh = $this->xpath('//meta[@http-equiv="Refresh"]')) {
      // Parse the content attribute of the meta tag for the format:
      // "[delay]: URL=[page_to_redirect_to]".
      if (preg_match('/\d+;\s*URL=(?.*)/i', $refresh[0]->getAttribute('content'), $match)) {
        $this->drupalGet($this->getAbsoluteUrl(Html::decodeEntities($match['url'])));
        $this->checkForMetaRefresh($max_count, $count + 1);
      }
    }
  }
dawehner’s picture

@larowlan Do you mind creating its own issue for that?

klausi’s picture

checkMetaRefresh() is in #2855942: Create ::checkForMetaRefresh() on BrowserTestBase and #2757023: Convert all aggregator web tests to BrowserTestBase.

But Batch processing is done with Javascript, right? So a case for JavascriptTestBase?

larowlan’s picture

Thanks @klausi and @dawehner.

Batch processing works without Javascript, so no need for JTB, it uses the <noscript><meta equiv="refresh"... trick

dawehner’s picture

We should not loose the PHP test coverage of batch processes IMHO. For most batch processes its an implementation detail that its using JS mostly. At least for me JavascriptTestBase is about testing JS.

xjm’s picture

Issue summary: View changes

At this point I think we can also update https://www.drupal.org/contributor-tasks/write-tests to not tell contributors to use SimpleTest. :)

Mile23’s picture

Component: Approved Plan » Active Initiative

Active initiative component is back. Woot! See #70.

Mile23’s picture

Presented for your bemusement: #2866082: [plan] Roadmap for Simpletest

michielnugter’s picture

Issue summary: View changes
Issue tags: +phpunit initiative

Updated IS to reflect 'phpunit initiative' tag usage and added me and LenDude to the regular contributers. Will move our names up once we're the official leads.

dawehner’s picture

Issue summary: View changes

Let's make it a bit clearer.

el7cosmos’s picture

As PHPUnit 4 is no longer supported, any plan on switch to PHPUnit 5?

Is it possible right now to use PHPUnit 5/6?

michielnugter’s picture

@el7cosmos:

There is an issue open for the upgrade to a newer PHPUnit here: #2578813: Figure out PHPUnit version support. It seems that for now in Drupal it's not possible to use PHPUnit 5 or 6.

dawehner’s picture

There is an issue open for the upgrade to a newer PHPUnit here: #2578813: Figure out PHPUnit version support. It seems that for now in Drupal it's not possible to use PHPUnit 5 or 6.

Yeah, we can't as the mimimal php version is 5.5

Mile23’s picture

Mile23’s picture

I never did add this as related.

rakesh.gectcr’s picture

Issue summary: View changes
rakesh.gectcr’s picture

Issue summary: View changes