Problem/Motivation

Maybe one time out of 10 or 20 or so there is a reoccuring test failure, that can be fixed by rerunning the test.

Its always in PHPCS (previous major) with the following:

---- Drupal\Tests\ai_eca\Kernel\Plugin\Action\ChatTest ----
Status    Group      Filename          Line Function                            
--------------------------------------------------------------------------------
Fail      Other      phpunit-3.xml        0 Drupal\Tests\ai_eca\Kernel\Plugin\A
    PHPUnit Test failed to complete; Error: PHPUnit 9.6.22 by Sebastian
    Bergmann and contributors.
    
    Testing Drupal\Tests\ai_eca\Kernel\Plugin\Action\ChatTest
    ..F.                                                                4 / 4
    (100%)
    
    Time: 00:04.422, Memory: 12.00 MB
    
    There was 1 failure:
    
    1) Drupal\Tests\ai_eca\Kernel\Plugin\Action\ChatTest::testWithoutInput
    Failed asserting that 'Hello world! Input: GtHM>&{" My AI/ECA site
    n6d56i93: 'KO"''>&b('\n
    . Config: [].' contains "KO"'>&b(".
    
    /builds/project/ai/vendor/phpunit/phpunit/src/Framework/Constraint/Constraint.php:122
    /builds/project/ai/vendor/phpunit/phpunit/src/Framework/Constraint/Constraint.php:56
    /builds/project/ai/modules/ai_eca/tests/src/Kernel/Plugin/Action/ChatTest.php:124
    /builds/project/ai/vendor/phpunit/phpunit/src/Framework/TestResult.php:729
    
    FAILURES!
    Tests: 4, Assertions: 31, Failures: 1.
---- Drupal\Tests\ai\Kernel\OperationType\Chat\ChatInterfaceTest ----

Proposed resolution

Figure out why it is failing randomly.
Fix it.

Issue fork ai-3499900

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

About issue forks

Comments

marcus_johansson created an issue. See original summary.

bircher’s picture

bircher
Location 🇨🇿
commented
Assigned: Unassigned » bircher
bircher’s picture

bircher
Location 🇨🇿
commented
Version: 1.0.1 » 1.0.x-dev
Assigned: bircher » Unassigned
Status: Active » Needs review

So I ran the test many times in a row to confirm the issue locally, and looking at the error message I had a suspicion.
But I traced it down to see what is happening. And the reason it sometimes fails but not others is that we use $this->randomString() in the parts where this fails. The failure comes from the fact that we use ECA tokens, or more precisely tokens with a \Drupal\eca\Plugin\DataType\DataTransferObject, this in turn gets turned into a string by using Yaml::encode($values). But some random strings get changed by that other than just getting surrounded by '.
The example in the issue summary is KO"'>&b( which becomes 'KO"''>&b(' and that in turn does not contain the original.
This can happen in all the test methods that use the ECA token replacement, so I fixed it by fixing the randomString method in the base class. I ran tests 250 times and they succeeded every time. But we can use another approach too if this is deemed not to be the right approach.

bircher opened merge request !418

mrdalesmith’s picture

mrdalesmith commented
Status: Needs review » Reviewed & tested by the community

Looks good to me.

marcus_johansson’s picture

marcus_johansson commented
Status: Reviewed & tested by the community » Fixed

Thank you bircher, getting merged.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

kristen pol’s picture

kristen pol
she/her
Primary language English
Location Santa Cruz, CA, USA
commented
Title: Fix the random test failure on ChatTest » AI ECA: Fix the random test failure on ChatTest
Component: Other Submodules » AI ECA

We are doing some issue management housekeeping and adding/removing components.

We are removing the "Other Submodules" component and want people to categorize issues with the best module/submodule component.

Moving this issue to "AI ECA".

See #3533272: Update AI module project components for more details.