Problem/Motivation

In #3131900: Refactor assertions that assign return values to variables we realized we have in test code assertion helpers in the form of assert*() methods that return a value to the caller. This is legacy of Simpletest. In PHPUnit, all assertions methods return types are typehinted to void, and PHPUnit any way stops test execution if an assertion fails, so no purpose to return a value anyway.

Proposed resolution

Typehint to void the custom assertion methods; methods that generate a return value should be converted to helpers called by the asserts if needed.

Remaining tasks

User interface changes

API changes

Data model changes

Release notes snippet

CommentFileSizeAuthor
#50 3138078-50.patch528.81 KBmondrake
#50 interdiff_48-50.txt4.12 KBmondrake
#48 interdiff_47-48.txt1.27 KBmondrake
#48 3138078-48.patch525.07 KBmondrake
#47 3138078-47.patch525.16 KBmondrake
#46 3138078-46.patch526.64 KBmondrake
#45 3138078-45.patch528.91 KBmondrake
#44 3138078-44.patch529.78 KBmondrake
#42 interdiff_40-42.txt539 bytesmondrake
#42 3138078-42.patch527.97 KBmondrake
#40 interdiff_38-40.txt2.24 KBmondrake
#40 3138078-40.patch527.96 KBmondrake
#38 3138078-38.patch527.08 KBmondrake
#38 interdiff_35-38.txt22.23 KBmondrake
#35 interdiff_33-35.txt7.16 KBmondrake
#35 3138078-35.patch511.62 KBmondrake
#33 interdiff_30-33.txt3.27 KBmondrake
#33 3138078-33.patch507.61 KBmondrake
#32 3138078-32.patch507.33 KBmondrake
#32 interdiff_30-32.txt2.27 KBmondrake
#31 interdiff_30-31.txt1.42 KBmondrake
#31 3138078-31.patch507.13 KBmondrake
#30 interdiff_27-30.txt93.05 KBmondrake
#30 3138078-30.patch506.87 KBmondrake
#27 interdiff_25-27.txt192.75 KBmondrake
#27 3138078-27.patch420.54 KBmondrake
#25 3138078-25.patch226.97 KBmondrake
#25 interdiff_21-25.txt11.18 KBmondrake
#21 interdiff_20-21.txt1.89 KBmondrake
#21 3138078-21.patch215.66 KBmondrake
#20 3138078-20.patch215.28 KBmondrake
#19 3138078-19.patch215.26 KBmondrake
#19 interdiff_17-19.txt66.51 KBmondrake
#17 3138078-17.patch148.75 KBmondrake
#17 interdiff_14-17.txt17.7 KBmondrake
#16 3138078-16.patch148.71 KBmondrake
#16 interdiff_14-16.txt17.66 KBmondrake
#14 3138078-14.patch132.04 KBmondrake
#14 interdiff_10-14.txt101.38 KBmondrake
#13 interdiff_10-13.txt101.38 KBmondrake
#13 3138078-13.patch132.04 KBmondrake
#12 3138078-12.patch131.69 KBmondrake
#12 interdiff_10-12.txt101.03 KBmondrake
#10 3138078-10.patch31.77 KBmondrake
#10 interdiff_8-10.txt30.87 KBmondrake
#8 3138078-8.patch4.45 KBmondrake
#8 interdiff_6-8.txt2.51 KBmondrake
#6 3138078-6.patch3.23 KBmondrake
#5 3138078-5.patch3.22 KBmondrake
#3 3138078-3.patch3.17 KBmondrake

Issue fork drupal-3138078

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

mondrake created an issue. See original summary.

mondrake’s picture

Status: Needs review » Active
mondrake’s picture

Status: Active » Needs review
StatusFileSize
new3.17 KB

Let's do some discovery.

Status: Needs review » Needs work

The last submitted patch, 3: 3138078-3.patch, failed testing. View results

mondrake’s picture

Status: Needs work » Needs review
StatusFileSize
new3.22 KB
mondrake’s picture

StatusFileSize
new3.23 KB

Status: Needs review » Needs work

The last submitted patch, 6: 3138078-6.patch, failed testing. View results

mondrake’s picture

Status: Needs work » Needs review
StatusFileSize
new2.51 KB
new4.45 KB

Status: Needs review » Needs work

The last submitted patch, 8: 3138078-8.patch, failed testing. View results

mondrake’s picture

Status: Needs work » Needs review
StatusFileSize
new30.87 KB
new31.77 KB

Status: Needs review » Needs work

The last submitted patch, 10: 3138078-10.patch, failed testing. View results

mondrake’s picture

Status: Needs work » Needs review
StatusFileSize
new101.03 KB
new131.69 KB
mondrake’s picture

StatusFileSize
new132.04 KB
new101.38 KB
mondrake’s picture

StatusFileSize
new101.38 KB
new132.04 KB

Status: Needs review » Needs work

The last submitted patch, 14: 3138078-14.patch, failed testing. View results

mondrake’s picture

StatusFileSize
new17.66 KB
new148.71 KB
mondrake’s picture

StatusFileSize
new17.7 KB
new148.75 KB
hash6’s picture

Assigned: Unassigned » hash6
mondrake’s picture

StatusFileSize
new66.51 KB
new215.26 KB
mondrake’s picture

StatusFileSize
new215.28 KB

Rerolled

mondrake’s picture

StatusFileSize
new215.66 KB
new1.89 KB
mondrake’s picture

@hash6 sorry I realised now you called for working on the issue. I was on a train of thought that produced several patches, but I'll stop for now, feel free to go ahead.

hash6’s picture

Assigned: hash6 » Unassigned
xjm’s picture

So the deprecation here will help child implementations. But what about callers? There are callers in contrib that are doing $this->assertTrue($this->assertCustomThing($foo, $bar));

When we correct return values in this issue, those contrib tests will start failing abruptly, with no deprecation. Any ideas for how we could provide deprecation for the callers as well?

mondrake’s picture

Status: Needs work » Needs review
StatusFileSize
new11.18 KB
new226.97 KB

#24 I can't figure out how we could add a void return typehint and at the meantime keep returning a value for BC... a similar issue forced us to rewrite PHPUnit's TestCase class to allow PHPUnit 8 compatibility. I may suggest to get to a complete patch, then decide the one methods that will have to be reverted to current state of returning a value. Personally I wouldn't like that, it'd just be procrastinating the technical debt.

Status: Needs review » Needs work

The last submitted patch, 25: 3138078-25.patch, failed testing. View results

mondrake’s picture

Status: Needs work » Needs review
StatusFileSize
new420.54 KB
new192.75 KB
mondrake’s picture

Status: Needs review » Needs work
mondrake’s picture

Assigned: Unassigned » mondrake
mondrake’s picture

Status: Needs work » Needs review
StatusFileSize
new506.87 KB
new93.05 KB
mondrake’s picture

StatusFileSize
new507.13 KB
new1.42 KB
mondrake’s picture

StatusFileSize
new2.27 KB
new507.33 KB
mondrake’s picture

StatusFileSize
new507.61 KB
new3.27 KB

Status: Needs review » Needs work

The last submitted patch, 33: 3138078-33.patch, failed testing. View results

mondrake’s picture

Status: Needs work » Needs review
StatusFileSize
new511.62 KB
new7.16 KB

Some fixes. The BigPipe failure seems to me a bug on its own.

mondrake’s picture

Status: Needs review » Needs work

The last submitted patch, 35: 3138078-35.patch, failed testing. View results

mondrake’s picture

Status: Needs work » Needs review
StatusFileSize
new22.23 KB
new527.08 KB

Status: Needs review » Needs work

The last submitted patch, 38: 3138078-38.patch, failed testing. View results

mondrake’s picture

Status: Needs work » Needs review
StatusFileSize
new527.96 KB
new2.24 KB

Status: Needs review » Needs work

The last submitted patch, 40: 3138078-40.patch, failed testing. View results

mondrake’s picture

Status: Needs work » Needs review
StatusFileSize
new527.97 KB
new539 bytes

Maybe?

mondrake’s picture

Assigned: mondrake » Unassigned
mondrake’s picture

StatusFileSize
new529.78 KB

Reroll.

mondrake’s picture

StatusFileSize
new528.91 KB
mondrake’s picture

StatusFileSize
new526.64 KB

reroll

mondrake’s picture

StatusFileSize
new525.16 KB

reroll

mondrake’s picture

StatusFileSize
new525.07 KB
new1.27 KB

fixes

Status: Needs review » Needs work

The last submitted patch, 48: 3138078-48.patch, failed testing. View results

mondrake’s picture

Status: Needs work » Needs review
StatusFileSize
new4.12 KB
new528.81 KB

Status: Needs review » Needs work

The last submitted patch, 50: 3138078-50.patch, failed testing. View results

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

mondrake’s picture

mondrake’s picture

Rerolled, reverted changes to base classes and traits.

mondrake’s picture

Status: Needs work » Needs review

This is now reviewable; we need to agree whether we want to add @internal to all assert* methods in concrete test classes, as well as if it's worth adding to them full arg typehinting (done down till the 'comment' module).

longwave’s picture

I suppose the question is do we want to make them all @internal, or do we just make them private?

mondrake’s picture

Personally, I would prefer to keep them protected. It happened to me in contrib that I extend a concrete test class in my module, to ensure that it is compatible with core. Switching to private means I have to rewrite the assert* method in my extended test, even if I do not override it.

longwave’s picture

Status: Needs review » Needs work

Thought about this one for a while and decided I am +1 to full type declarations here. There are no real BC concerns - tests are considered internal and it is fairly unlikely someone is using most of these let alone extending them downstream. Adding type declarations to both arguments and return values sets a good example for the rest of core, and also might help to catch bugs earlier than we might otherwise spot. Therefore the upsides are much greater than the downsides and I think we should go ahead with this.

As regards @internal I guess it can't hurt, we are warning downstream users, and we don't provide BC for tests. If we really didn't want users to extend tests we would declare them final, and we don't do that - so they are free to at their own risk, and that includes using any custom assertions.

mondrake’s picture

#60 fully agree. Also about @internal yes, we are already saying 'no BC' on concrete tests, but also IMHO it does not hurt to say that again, and as I mentioned in #59, if you really want to extend a test class you're on your own - and if you know what you are doing it can still be very useful (think contrib db drivers that legitimately expect to run core Database tests, still overriding some where needed).

mondrake’s picture

Issue tags: +Novice

Adding Novice tag. What needs to be done here is: for the assert* methods that we are already adding a void returntype in this patch (only those, not those in traits or base classes)

  1. add an @internal annotation in their docblocks
  2. add full typehinting to the arguments (where possible), ensuring the signature is aligned with the type int @param docblocks
mondrake’s picture

Assigned: Unassigned » mondrake

Working on doing a few more.

mondrake’s picture

Assigned: mondrake » Unassigned
catch’s picture

When we correct return values in this issue, those contrib tests will start failing abruptly, with no deprecation. Any ideas for how we could provide deprecation for the callers as well?

For random test helpers I think it's fine to break things, those should be considered internal.

For traits or base classes I think we should do some kind of minimal bc/fc effort, but doesn't seem like there are many options:

1. Only change the traits in Drupal 10 (gives time for a CR and rector rules to inform people).

or:

2. Deprecate the entire method and add a new one with no return value - the problem with this though is that it causes churn for callers that aren't relying on the return value, so arguably worse than not doing so.

However we can probably grep contrib for callers to see if we'll actually break anything first (again only for base classes/traits). Also if a method turns out to only have one or two usages in contrib, just pro-actively opening issues against those modules might be enough (i.e. make the grep return empty by the time we get to 9.3.0).

mondrake’s picture

Rerolled. Yes we need to split traits/base classes in their own issues at this stage. I'm +1 for postponing that to D10, and try to do the concrete classes here for D9.3.

mondrake’s picture

Assigned: Unassigned » mondrake

working on this

mondrake’s picture

Assigned: mondrake » Unassigned
Status: Needs work » Needs review
Issue tags: -Novice

Up for review.

longwave’s picture

Status: Needs review » Needs work

A few minor comments. Ran out of time while reviewing, will continue another time.

mondrake’s picture

Status: Needs work » Needs review

Thanks for reviw @longwave. Some fixes and comments in the latest push to the MR.

mondrake’s picture

Rerolled

mondrake’s picture

Rerolled

longwave’s picture

Status: Needs review » Reviewed & tested by the community

As far as I can see there is nothing left to do here, this is cleaning up some Simpletest cruft and bringing our PHP up to modern standards at the same time.

mondrake’s picture

Thanks @longwave - I appreciate it's an eye-spoiling patch to review, this one

mondrake’s picture

Rerolled

mondrake’s picture

Title: Add a 'void' return typehint to custom assert* methods » [D9.3 beta - w/c Nov 8, 2021] Add a 'void' return typehint to custom assert* methods

Suggest to go with beta disruptive window

mondrake’s picture

Rerolled + fixed Drupal\Tests\ckeditor5\FunctionalJavascript\MediaTest + added draft CR + corrected deprecation message

larowlan’s picture

Status: Reviewed & tested by the community » Needs work
Issue tags: +Needs reroll

This no longer applies after the last three disruptive patches went in.

Tried a 3-way merge, but to no avail

both modified:   core/modules/field/tests/src/Kernel/Migrate/d7/MigrateFieldInstanceTest.php
	both modified:   core/modules/views/tests/src/Functional/Handler/HandlerTest.php
mondrake’s picture

Status: Needs work » Reviewed & tested by the community

Rerolled. View's functional HandlerTest removed an assert method in a previous patch, so it was no longer need to adjust it. Another minor patch context change due to dropped usage of list().

mondrake’s picture

Issue tags: -Needs reroll
larowlan’s picture

Status: Reviewed & tested by the community » Needs review
Issue tags: +Needs reroll
larowlan’s picture

Left a question on the MR

mondrake’s picture

I am not aware of a way to do that, sorry.

larowlan’s picture

Status: Needs review » Reviewed & tested by the community

I'll ask another committer their thoughts

larowlan’s picture

Status: Reviewed & tested by the community » Needs review
Issue tags: -Needs reroll

Discussed this with @catch, @xjm and @alexpott and we agreed that because this isn't really a deprecation and more of a code-style concern that we'd remove the deprecation from the patch and just go with the rest of the patch.

When we get phpstan we can look to add a rule to core to ensure all assert* methods return void.

Can we get the changes in the listener removed - thanks!

mondrake’s picture

Status: Needs review » Reviewed & tested by the community

Yep, PHPStan can do better here. Done #85. I think it's OK for me to mark this back to RTBC.

larowlan’s picture

Status: Reviewed & tested by the community » Needs work
Issue tags: +Needs reroll

This needs a reroll to apply cleanly, tried a 3-way merge too and no joy

mondrake’s picture

Status: Needs work » Reviewed & tested by the community

Rerolled

larowlan’s picture

Custom commands failed on the bot

mondrake’s picture

Issue tags: -Needs reroll

Rerolled

catch’s picture

Status: Reviewed & tested by the community » Fixed

Committed/pushed to 9.4.x and cherry-picked to 9.3.x, thanks!

  • catch committed 92836c4 on 9.4.x
    Issue #3138078 by mondrake, larowlan, longwave, xjm: [D9.3 beta - w/c...

  • catch committed 37a5400 on 9.3.x
    Issue #3138078 by mondrake, larowlan, longwave, xjm: [D9.3 beta - w/c...
catch’s picture

Even though we're not enforcing it, published the change record anyway.

huzooka’s picture

From #60:

tests are considered internal

Was/is this documented anywhere? And does this means that e.g. Drupal\Tests\BrowserTestBase or Drupal\KernelTests\KernelTestBase are internal 🤔?

longwave’s picture

@huzooka see https://www.drupal.org/about/core/policies/core-change-policies/drupal-8... - BTB and KTB are not internal, but mostly everything else is.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.