Advertising sustains the DA. Ads are hidden for members. Join today

On this page

Debugging Functional tests

Last updated on
2 September 2025

The Sitemap module has a few Functional and FunctionalJavascript tests, which are sometimes known as PHPUnit Browser tests.

For the sake of brevity and clarity, this documentation will refer to Functional, FunctionalJavascript and PHPUnit Browser tests as Functional tests.

How Functional tests work

When a functional test runs:

  1. A temporary Drupal site is installed to a temporary database,
  2. Any set of modules that the test says it needs are enabled,
  3. Most tests create a user account with a certain set of permissions and logs in as that user,
  4. Most tests navigate to a page in a headless web browser (e.g.: Guzzle, Chromium, etc.),
  5. Most tests assert HTML elements, text, form fields, etc. on the loaded page,
  6. Most tests click links on the page and/or submit a form,
  7. Most tests then assert HTML elements, text, form fields, etc. on the page that results from the action in the previous step.

Problem: What is causing the assertion to fail?

Because Functional tests often mirror what a manual tester would do, they can be fairly straightforward to write, even if you haven't written many tests!

But, because the tests run in a headless browser, and make assertions about what is being "displayed" in that browser, Functional tests can fail with errors that are hard to debug if you cannot "see" what is being displayed in the browser. For example...

The text "lorem ipsum" was not found anywhere in the text of the current page.

... confusingly, this sort of assertion-failure message could happen in one of several scenarios, for example...

  1. The page that the test expected to see is working, but a regression is causing "dolor sit" to appear instead of the expected "lorem ipsum",
    • In this scenario, the error is usually relevant to the test where the error occurred, and thus is usually easier to debug/fix.
  2. A fatal PHP error occurred, and the test is looking for the text "lorem ipsum" on an empty page (i.e.: the so-called "white screen of death"),
    • In this scenario, the error is often related to something that was changed in the patch/merge request where the error first appeared.
  3. A 404 Not Found ("Page Not Found") or 403 Forbidden ("Access Denied") error occurred, and the test is looking for the text "lorem ipsum" on a generic Drupal error page.
    • In this scenario, the error might be related to the set-up portion of the test, to a change in Drupal core, something that changed in the patch/merge request where the error first appeared, or something else.

... if you were manually testing, it might be easy to see the problem, but the contents of the page isn't visible in a PHPUnit test run under default settings.

Solution: HtmlOutputLogger/Printer

Drupal ships with a way for you to see what the headless web-browser saw during the test. In Drupal 11 / PHPUnit 10, this is done with a PHPUnit extension called HtmlOutputLogger; and in earlier versions of Drupal/PHPUnit, this is done with a PHPUnit printerClass called HtmlOutputPrinter.

In CI

Using the HtmlOutputLogger/Printer requires different configuration for Drupal 10 and 11 (both versions of Drupal that we support at time-of-writing), and — to our knowledge — it is non-trivial to write configuration that would run in both the D11 test environment and the D10 test environment, so we don't currently enable it in GitLab CI. Another consideration is that using it makes tests run slower, and maintainers have to wait for tests to finish when we merge a merge-request.

That being said, in the future, ElasticSearch Connector maintainers are open to the idea of enabling it in CI if that would be helpful.

Setup in Drupal 11

To make this work in Drupal 11 / PHPUnit 10...

  1. Set up a local environment for Developing Sitemap 8.x-2.x (run the One-time setup if applicable, then run the Setup for working on an issue)
    1. At time-of-writing, this installs Drupal 11.2 / PHPUnit 11
  2. Add a file named phpunit.xml.dist to the same folder as the sitemap.info.yml, with the following contents:
    <?xml version="1.0" encoding="UTF-8"?>
<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:noNamespaceSchemaLocation="https://schema.phpunit.de/10.5/phpunit.xsd"
  colors="true"
>
  <extensions>
    <bootstrap class="Drupal\TestTools\Extension\HtmlLogging\HtmlOutputLogger">
      <parameter name="outputDirectory" value="sites/simpletest/browser_output"/>
      <parameter name="verbose" value="true"/>
    </bootstrap>
  </extensions>
</phpunit>

  3. Run ddev phpunit

Interpreting the output

When you run PHPUnit with HtmlOutputLogger/Printer enabled, then after it has run all the tests, it will display a section of text prefixed with HTML output was generated. This section lists a bunch of URLs. These URLs point to HTML files, which contain the HTTP responses that the headless browser was served by the test Drupal site (along with some metadata).

For example...

PHPUnit 11.5.35 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.3.23
Configuration: /var/www/html/phpunit.xml.dist

.........F........................                                34 / 34 (100%)

HTML output was generated.
https://sitemap.ddev.site/sites/simpletest/browser_output/Drupal_Tests_sitemap_book_Functional_SitemapBookTest-66-13364898.html

... (there are many URLs printed here, which I'm omitting for brevity) ...

https://sitemap.ddev.site/sites/simpletest/browser_output/Drupal_Tests_sitemap_Functional_SitemapMenuTest-66-86122898.html

... (there are 59 URLs that contain "SitemapMenuTest", which I'm omitting for brevity)...

https://sitemap.ddev.site/sites/simpletest/browser_output/Drupal_Tests_sitemap_Functional_SitemapMenuTest-125-37174452.html

... (there are many more URLs printed here, which I'm omitting for brevity) ...

https://sitemap.ddev.site/sites/simpletest/browser_output/Drupal_Tests_sitemap_Functional_Update_SitemapUpdateSettingsTest-30-35644986.html


Time: 01:06.472, Memory: 10.00 MB

There was 1 failure:

1) Drupal\Tests\sitemap\Functional\SitemapMenuTest::testMenuDepth
Failed asserting that the text of the element identified by '.form-item-plugins-menutools-settings-menu-depth > label' equals 'Amount of levels to display'.
Failed asserting that two strings are equal.
--- Expected
+++ Actual
@@ @@
-'Amount of levels to display'
+'Number of levels to display'

/var/www/html/web/core/tests/Drupal/Tests/WebAssert.php:813
/var/www/html/tests/src/Functional/SitemapMenuTest.php:144

FAILURES!
Tests: 34, Assertions: 985, Failures: 1, PHPUnit Deprecations: 52.

Understanding the file name

These HTML files are typically named as follows:

  1. The first part is the fully-qualified name of the test class, with backslashes (\) replaced by underscores (_).
  2. The second part is a sequential number: each HTTP response in a test function will get a new sequential number. If there are files left over from a previous run, then the sequential number will start at 1 more than the previous run.
  3. The last part is a random-ish number that represents a specific test function in the test class (if the test has a data provider method, I believe this number will also vary).

... for example, in the URL https://sitemap.ddev.site/sites/simpletest/browser_output/Drupal_Tests_sitemap_Functional_Update_SitemapUpdateSettingsTest-30-35644986.html...

  1. the test-class is \Drupal\Tests\sitemap\Functional\Update\SitemapUpdateSettingsTest, 
  2. the sequential number is 30, and 
  3. the random-ish number that represents the specific test function is 35644986.

Understanding the file contents

The HTML files themselves contain the following information:

  1. An ID # section showing the same sequential number from the file name, along with Previous and Next links to make it easier to navigate within a test;
    • Note it still displays a Next link on the last output from a test, which will result in a 404 error if you follow it.
    • If you're debugging a failing test, the last page in a sequence is usually (but not always) where you'll find the problem. That being said, the previous pages in the sequence can provide useful context about how the system got into a failing state.
  2. A Called from section (which typically is too deep in the call stack to be useful);
  3. Information about the type of HTTP request (i.e.: GET request to URL or POST request to URL);
  4. The HTTP response body (which is usually HTML, and is the part of the page that most PHPUnit Functional test assertions are made on); and;
  5. The HTTP response header, serialized as the output from PHP's \var_export() function.

Understanding the test failure

The error in the example test case above...

There was 1 failure:

1) Drupal\Tests\sitemap\Functional\SitemapMenuTest::testMenuDepth
Failed asserting that the text of the element identified by '.form-item-plugins-menutools-settings-menu-depth > label' equals 'Amount of levels to display'.
Failed asserting that two strings are equal.
--- Expected
+++ Actual
@@ @@
-'Amount of levels to display'
+'Number of levels to display'

/var/www/html/web/core/tests/Drupal/Tests/WebAssert.php:813
/var/www/html/tests/src/Functional/SitemapMenuTest.php:144

... shows that an assertion failed in the class \Drupal\Tests\sitemap\Functional\SitemapMenuTest (i.e.: at tests/src/Functional/SitemapMenuTest.php) in public function testMenuDepth(), on line 144, which, in the test, looks like...

$this->assertSession()->elementTextEquals('css', '.form-item-plugins-menutools-settings-menu-depth > label', 'Amount of levels to display');

Looking at the error in the PHPUnit output, it was expecting to find an element on the page with the (CSS) locator .form-item-plugins-menutools-settings-menu-depth > label with the text Amount of levels to display, and while it found the element with that CSS locator, the text in that element was actually Number of levels to display, not Amount of levels to display.

If the thing that caused the error was harder to understand, and I wanted to see what was going on in the browser when this assertion failed, I would have to start by going to the first URL in the test that contains the SitemapMenuTest, i.e.:  https://sitemap.ddev.site/sites/simpletest/browser_output/Drupal_Tests_sitemap_Functional_SitemapMenuTest-66-86122898.html — and following the Next links until I saw the problem.

Fixing the test failure

Recall in our example from the previous section, the text in that element was actually Number of levels to display, but the test was expecting to see the text Amount of levels to display.

In this very-simple, contrived case, there are two solutions:

  1. Change the text in the production code to say Number of levels to display
  2. Change assertion in the test code to say Amount of levels to display

Exactly which of those solutions you'd choose depends on what you were trying to do in your patch/merge-request:

  1. If you'd created the merge request to change the label Number of levels to display so that it says Amount of levels to display instead, then you should be expecting a test failure like this, and you should change the test code to check for the changed text.
  2. If you'd created the merge request to do something unrelated to this label, then that change is probably a regression, so you'd have to look over your production-code changes to figure out why that happened. Likely, you accidentally changed replaced "Number" with "Amount" in the production code, and you should change it back.

Maintainer code review

Among other things, the module maintainers check to make sure that:

  1. changes to production and test code make sense for the issue the change is being proposed in,
  2. changes to the production code have corresponding changes in the test code, and,
  3. changes to the test code have corresponding changes to the production code.

If a maintainer finds a change that doesn't make sense for the issue (e.g.: the issue was filed to add a new Sitemap section, but changes existing sections too, or changes tests for existing sections), that indicates an accidental change or a change to the issue scope, so the maintainer is going to point that out.

If a maintainer finds a change to production code that doesn't have a corresponding change to test code, that indicates that the proposed change isn't properly tested, and a future change to the module could cause that code to break, i.e.: that future changes could break the functionality that you depend on! In this case, the maintainer is going to ask for additional tests.

If a maintainer finds a change to test code that doesn't have a corresponding change to production code, that indicates that the change introduces a regression, i.e.: that the change breaks functionality that someone else depends on! In this case, the maintainer might ask for the regression to be fixed, or to defer the change to a future major version (where we can introduce breaking changes).

Additional reading

Help improve this page

Page status: No known problems

You can: