Problem/Motivation

The output of the nightwatch job on GitLab CI is not easily readable.

After the first test, the output seems to lack newlines:

┌ ────────────────── ✔  default: Tests/toolbarApiTest.js  ─────────────────────┐│                                                                              ││                                                                              ││   [Tests/Toolbar Api Test] Test Suite                                        ││   ──────────────────────────────────────────────────────────────────────     ││   Using: chrome (106.0.5249.103) on LINUX.                                   ││   - Loading url: http://localhost/subdirectory                               ││   ℹ Loaded url http://localhost/subdirectory in 3603ms                       ││   - Loading url: http://localhost/subdirectory/user/reset/1/1699769393/rAx   ││   XXMRqoilkKkm9F1ftirF9ByGekM_5ilpoeJ7cOds/login                             ││   ℹ Loaded url http://localhost/subdirectory/user/reset/1/1699769393/rAxXX   ││   MRqoilkKkm9F1ftirF9ByGekM_5ilpoeJ7cOds/login                               ││   in 1284ms                                                                  ││   - Loading url: http://localhost/subdirectory/admin/modules                 ││   ℹ Loaded url http://localhost/subdirectory/admin/modules in 1468ms         ││   ✔ Element <form.system-modules [name="modules[toolbar][enable]"]> was      ││   visible after 25 milliseconds.                                             ││   ✔ Element <#system-modules-confirm-form> was present after 1542            ││   milliseconds.                                                              ││   ✔ Element <form.system-modules                                             ││   [name="modules[toolbar][enable]"]:disabled> was present after 45           ││   milliseconds.                                                              ││   - Loading url: http://localhost/subdirectory/user/logout                   ││   ℹ Loaded url http://localhost/subdirectory/user/logout in 245ms            ││   - Loading url: http://localhost/subdirectory/user/reset/1/1699769407/C9F   ││   cHTHML3hD-dyWjgTiXgMSfGlriQJUzj-FSH0GTeE/login                             ││   ℹ Loaded url http://localhost/subdirectory/user/reset/1/1699769407/C9FcH   ││   THML3hD-dyWjgTiXgMSfGlriQJUzj-FSH0GTeE/login                               ││   in 356ms                                                                   ││   - Loading url: http://localhost/subdirectory/admin/people/roles/add        ││   ℹ Loaded url http://localhost/subdirectory/admin/people/roles/add in       ││   421ms                                                                      ││   ✔ Expected element <.user-role-form .machine-name-value> to be visible     ││   in 2000ms (67ms)                                                           ││   - Loading url: http://localhost/subdirectory/admin/people/permissions      ││   ℹ Loaded url http://localhost/subdirectory/admin/people/permissions in     ││   1883ms                                                                     ││   - Loading url: http://localhost/subdirectory/admin/people/permissions      ││   ℹ Loaded url http://localhost/subdirectory/admin/people/permissions in     ││   731ms                                                                      ││   - Loading url: http://localhost/subdirectory/user/logout                   ││   ℹ Loaded url http://localhost/subdirectory/user/logout in 164ms            ││   - Loading url: http://localhost/subdirectory/user/reset/1/1699769414/I2A   ││   V9n1kyOJ9KG70PnB6GJG7qr9sFcw9Ei6gT-6rqNI/login                             ││   ℹ Loaded url http://localhost/subdirectory/user/reset/1/1699769414/I2AV9   ││   n1kyOJ9KG70PnB6GJG7qr9sFcw9Ei6gT-6rqNI/login                               ││   in 269ms                                                                   ││   - Loading url: http://localhost/subdirectory/admin/people/create           ││   ℹ Loaded url http://localhost/subdirectory/admin/people/create in 389ms    ││   ✔ User "user" was created successfully (723ms)                             ││   (node:117) MaxListenersExceededWarning: Possible EventEmitter memory       ││   leak detected. 11 error listeners added to [CommandLoader]. Use            ││   emitter.setMaxListeners() to increase limit                                ││   (Use `node --trace-warnings ...` to show where the warning was created)    ││   - Loading url: http://localhost/subdirectory/user/logout                   ││   ℹ Loaded url http://localhost/subdirectory/user/logout in 145ms            ││   - Loading url: http://localhost/subdirectory/user/login                    ││   ℹ Loaded url http://localhost/subdirectory/user/login in 102ms             ││   (node:117) MaxListenersExceededWarning: Possible EventEmitter memory       ││   leak detected. 11 error listeners added to [CommandLoader]. Use            ││   emitter.setMaxListeners() to increase limit                                ││   ✔ Passed [equal]: The user "user" was logged in.                           ││                                                                              ││   – Drupal.Toolbar.models                                                    ││   - Loading url: http://localhost/subdirectory/                              ││   ℹ Loaded url http://localhost/subdirectory/ in 267ms                       ││   ✔ Element <#toolbar-administration> was present after 18 milliseconds.     ││   ✔ Passed [deepEqual]: Keys to check match                                  ││   ✔ Passed [equal]: has menu model                                           ││   ✔ Passed [equal]: menu model is an object                                  ││   ✔ Passed [equal]: has toolbar model                                        ││   ✔ Passed [equal]: toolbar model is an object                               ││   ✔ Passed [equal]: get("activeTab") has expected result                     ││   ✔ Passed [equal]: get("activeTray") has expected result                    ││   ✔ Passed [equal]: get("isOriented") has expected result                    ││   ✔ Passed [equal]: get("isFixed") has expected result                       ││   ✔ Passed [equal]: get("areSubtreesLoaded") has expected result             ││   ✔ Passed [equal]: get("isViewportOverflowConstrained") has expected        ││   result                                                                     ││   ✔ Passed [equal]: get("orientation") has expected result                   ││   ✔ Passed [equal]: get("locked") has expected result                        ││   ✔ Passed [equal]: get("isTrayToggleVisible") has expected result           ││   ✔ Passed [equal]: get("height") has expected result                        ││   ✔ Passed [equal]: get("offsets") bottom has expected result                ││   ✔ Passed [equal]: get("offsets") left has expected result                  ││   ✔ Passed [equal]: get("offsets") right has expected result                 ││   ✔ Passed [equal]: get("offsets") top has expected result                   ││   ✔ Passed [equal]: get("subtrees") has expected result                      ││   ✔ default: Tests/toolbarApiTest.js [Tests/Toolbar Api Test]                ││   Drupal.Toolbar.models (471ms)                                              ││                                                                              ││   – Change tab                                                               ││   - Loading url: http://localhost/subdirectory/                              ││   ℹ Loaded url http://localhost/subdirectory/ in 185ms                       ││   ✔ Element <#toolbar-administration> was present after 14 milliseconds.     ││   ✔ Passed [deepEqual]: Keys to check match                                  ││   ✔ Passed [equal]: has menu model                                           ││   ✔ Passed [equal]: menu model is an object                                  ││   ✔ Passed [equal]: has toolbar model                                        ││   ✔ Passed [equal]: toolbar model is an object                               ││   ✔ Passed [equal]: get("activeTab") has expected result                     ││   ✔ Passed [equal]: get("activeTray") has expected result                    ││   ✔ default: Tests/toolbarApiTest.js [Tests/Toolbar Api Test] Change tab     ││   (577ms)                                                                    ││                                                                              ││   – Change orientation                                                       ││   - Loading url: http://localhost/subdirectory/                              ││   ℹ Loaded url http://localhost/subdirectory/ in 227ms                       ││   ✔ Element <#toolbar-administration> was present after 42 milliseconds.     ││   ✔ Passed [deepEqual]: Keys to check match                                  ││   ✔ Passed [equal]: get("orientation") has expected result                   ││   ✔ Passed [equal]: changing orientation has expected result                 ││   ✔ default: Tests/toolbarApiTest.js [Tests/Toolbar Api Test] Change         ││   orientation (587ms)                                                        ││                                                                              ││   – Open submenu                                                             ││   - Loading url: http://localhost/subdirectory/                              ││   ℹ Loaded url http://localhost/subdirectory/ in 243ms                       ││   ✔ Element <#toolbar-administration> was present after 21 milliseconds.     ││   ✔ Passed [deepEqual]: Keys to check match                                  ││   ✔ Passed [equal]: get("orientation") has expected result                   ││   ✔ Passed [equal]: opening submenu has expected result                      ││   ✔ default: Tests/toolbarApiTest.js [Tests/Toolbar Api Test] Open submenu   ││   (695ms)                                                                    ││                                                                              │└──────────────────────────────────────────────────────────────────────────────┘

https://git.drupalcode.org/project/drupal/-/jobs/316331#L149 (Yes, this is all just one output line...)
Screenshot here: https://www.drupal.org/files/issues/2023-11-12/before.jpg

Problem comes out of a dependency of nightwatch called boxen (https://www.npmjs.com/package/boxen).

This was fixed in version 7.0.1, however nightwatch is (and also will be in 3.x) using 5.1.2.

Fix PR is here: https://github.com/sindresorhus/boxen/pull/81

Steps to reproduce

Proposed resolution

viewing the source of boxen implies we might be able to use the COLUMNS environment variable to trick it into printing newlines.

@longwave in #14

Also turn off the very verbose output to keep all the tests visible in the GitLab output window-thingy

Remaining tasks

User interface changes

API changes

Data model changes

Release notes snippet

Issue fork drupal-3401047

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

Spokje created an issue. See original summary.

spokje’s picture

Issue summary: View changes
spokje’s picture

Title: Nightwatch output on GitLab not easily readible » Nightwatch output on GitLab not easily readable
Issue summary: View changes
StatusFileSize
new628.01 KB
spokje’s picture

Issue summary: View changes
spokje’s picture

Issue summary: View changes

spokje’s picture

Issue summary: View changes
Status: Active » Needs review
smustgrave’s picture

Use patch-packages all the time! Think this will need submaintainer or framework approval for the new packages.

needs-review-queue-bot’s picture

Status: Needs review » Needs work
StatusFileSize
new90 bytes

The Needs Review Queue Bot tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".

This does not mean that the patch needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

spokje’s picture

Rebased

spokje’s picture

Status: Needs work » Needs review

longwave made their first commit to this issue’s fork.

longwave’s picture

Trying an alternative approach, viewing the source of boxen implies we might be able to use the COLUMNS environment variable to trick it into printing newlines.

needs-review-queue-bot’s picture

Status: Needs review » Needs work
StatusFileSize
new90 bytes

The Needs Review Queue Bot tested this issue. It no longer applies to Drupal core. Therefore, this issue status is now "Needs work".

This does not mean that the patch needs to be re-rolled or the MR rebased. Read the Issue Summary, the issue tags and the latest discussion here to determine what needs to be done.

Consult the Drupal Contributor Guide to find step-by-step guides for working with issues.

kostyashupenko made their first commit to this issue’s fork.

kostyashupenko’s picture

Status: Needs work » Needs review

Rebased, now patch from 5534 mr can be applied

smustgrave’s picture

Status: Needs review » Needs work
Issue tags: -Needs subsystem maintainer review

Don't think needs submaintainer if not adding new package.

✔ Passed [ok]: true ok                                                                                                                                                                                                                 │
│    ✔ Passed [ok]: true ok                                                                                                                                                                                                                 │
│    ✔ Passed [equal]: Expected :tabbable to return 1 for element <input type="tel" tabindex="1" />                                                                                                                                         │
│    ✔ Passed [ok]: true ok                                                                                                                                                                                                                 │
│    ✔ Passed [ok]: true ok                                                                                                                                                                                                                 │
│    ✔ Passed [equal]: Expected :tabbable to return 0 for element <input type="tel" tabindex="-1" />                                                                                                                                        │
│    ✔ Passed [ok]: true ok                                                                                                                                                                                                                 │
│    ✔ Passed [ok]: true ok      

Checked MR5534 and output isn't 2 columns anymore but the tests don't make sense. Just a bunch of "true ok"

longwave’s picture

Status: Needs work » Needs review

That is because the tabbable test doesn't provide helpful assertion messages in two of the three assertions in the loop:

        (result) => {
          browser.assert.ok(typeof result.value.actual === 'number');
          browser.assert.ok(typeof result.value.expected === 'number');
          browser.assert.equal(
            result.value.actual,
            result.value.expected,
            `Expected :tabbable to return ${result.value.expected} for element ${result.value.element}`,
          );
        },
smustgrave’s picture

Status: Needs review » Reviewed & tested by the community

Ah okay, know we don't write a lot of nightwatch tests but may be worth opening a follow up to provide better mesages.

quietone’s picture

Status: Reviewed & tested by the community » Needs work
Issue tags: +Needs issue summary update

There are 2 MRs here, which one has been set to RTBC? I think it is the later one, which means the Issue Summary is out of date. Tagging for that as well.

Spokje changed the visibility of the branch 3401047-nightwatch-output-on to hidden.

spokje’s picture

Issue summary: View changes
Status: Needs work » Needs review
Issue tags: -Needs issue summary update

*grmbl* Bloody newbies, always doing a drive-by MR *grmbl*

Anyway....

- Closed my MR
- Rebased longwave's MR
- Removed the unneeded/not working setting of COLUMNS as environment variable
- Updated IS

Since there's technically a code change since the last RTBC (the removed line), putting this back to NR

spokje’s picture

Hmmm, only just realized that we're now not able to see the approximately first third of the tests...
We can only scroll upto around oliveroStickyHeaderToggleTest

We seem to run out of the max amount of lines allowed on GitLab?
Is there a way to up that number for nightwatch log only?

Personally, ATM I think I rather have (faux) two columns with all the results than correct aligned 2/3 of the testlog

longwave’s picture

Is it really helpful to show every passed assertion anyway? Should we set detailed_output to false? https://nightwatchjs.org/guide/configuration/customising-test-output.html

Or we could set disable_colors to true and then the raw log should be more readable?

spokje’s picture

Assigned: Unassigned » spokje
Status: Needs review » Needs work

Is it really helpful to show every passed assertion anyway?

Ah...
I was somehow under the impression that somebody somewhere explicitly wanted this very verbose output.

Or we could set disable_colors to true and then the raw log should be more readable?

But but, I like the pretty colors....

Let's kill detailed_output and that should be more than enough to keep all the tests in view (and color ;)

spokje’s picture

Assigned: spokje » Unassigned
Issue summary: View changes
Status: Needs work » Needs review
smustgrave’s picture

Status: Needs review » Reviewed & tested by the community

Adding variable seems correct to me.

spokje’s picture

Title: Nightwatch output on GitLab not easily readable » [PP-1] Nightwatch output on GitLab not easily readable
Status: Reviewed & tested by the community » Postponed

Whilst we still have the very verbose output, let's use that to check if all deprecations are replaced in #3413135: Replace deprecated functions in Nightwatch tests.
Postponing on that issue.

spokje’s picture

Title: [PP-1] Nightwatch output on GitLab not easily readable » Nightwatch output on GitLab not easily readable
Status: Postponed » Needs review

#3413135: Replace deprecated functions in Nightwatch tests landed.

As a result, the lack of deprecation notices in the log now makes it short enough to be fully viewable in GitLab again.
Since I don't see a massive increase in Nightwatch tests/added assertions to existing test, I removed the detailed_output: false.

Back to NR.

fjgarlin’s picture

Status: Needs review » Reviewed & tested by the community

The change seems simple enough and the output is defo better (see https://git.drupalcode.org/issue/drupal-3401047/-/jobs/633328).
I don't know how much more we can control it, so I'm setting this RTBC.

  • lauriii committed 869929d8 on 11.x
    Issue #3401047 by Spokje, longwave, smustgrave, fjgarlin: Nightwatch...

  • lauriii committed d785cc21 on 10.2.x
    Issue #3401047 by Spokje, longwave, smustgrave, fjgarlin: Nightwatch...

lauriii’s picture

Version: 11.x-dev » 10.2.x-dev
Status: Reviewed & tested by the community » Fixed

Committed 869929d and pushed to 11.x. Also cherry-picked to 10.2.x. Thanks!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.