Problem/Motivation

We test these metrics with a range, but the update now always updates them if run with PERF_TEST_UPDATE.

Most actually seem quite stable, but to avoid constant changes, I think it makes sense to only update if it falls outside of the accepted range. Possibly half the range, so we

maybe here or in another issue, also consider making that range relative to the actual number. quite a few metrics are now at around 1k and we test with a range of 2k. We need to define how much, but something like min($value * .1, 2000), so the lower of 10% variation or at most 2k. And the update would use .05 or 1000 to ensure it roughly stays in the middle. The expected range could be just at the end and then still sometimes fail.

Steps to reproduce

Proposed resolution

Remaining tasks

User interface changes

Introduced terminology

API changes

Data model changes

Release notes snippet

Issue fork drupal-3619240

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

berdir created an issue. See original summary.

berdir’s picture

Issue summary: View changes
longwave’s picture

Is there a way we can make these fully deterministic in performance tests so we don't need the ranges at all?

berdir’s picture

@catch might know more than I do. based on the numbers I'm seeing actually change, it seems possible more stable than I assumed as only one changes sometimes for me, but it is still not entirely stable either and I haven't actually verified it's always the same.

berdir’s picture

The range was actually expanded fairly recently in #3588897: Widen the script/stylesheet bytes assertion ranges in performance tests by @catch, but it looks like it wasn't done because there were random fails, just to reduce the number of times it has to to be updated.

Which is actually a very good point. We don't want to update performance tests on umami/previously olivero every time we change a single byte in a css/js file.

So I think yes, we want to keep that range?

catch’s picture

Yes the range isn't because of indeterminacy it was because people got fed up rerolling. Ironically we've sort of reintroduced that by writing it out automatically, so something like this issue, or rounding the number to the nearest 500 bytes or similar would help.

berdir’s picture

> Ironically we've sort of reintroduced that by writing it out automatically, so something like this issue, or rounding the number to the nearest 500 bytes or similar would help.

I do plan to work on this, I'll also see if the comments can be improved a bit.

It's definitely just "sort" of, because it doesn't need to be updated on every change, it's now just also updated when tests need updating because of actual changes.

berdir’s picture

Status: Active » Needs review

This MR now adds a method to calculate the range, and it uses the lower or 20% of the value or 2000 if that would be above, which is from 10k on.

This results in this behavior now when writing it out with debug statements for StandardPerformanceTest:

ScriptBytes: 0 < 0 > 0
StylesheetBytes: 1022 < 1282 > 1542
ScriptBytes: 0 < 0 > 0
StylesheetBytes: 1026 < 1282 > 1538
ScriptBytes: 0 < 0 > 0
StylesheetBytes: 801 < 1001 > 1201
ScriptBytes: 5216 < 6519 > 7822
StylesheetBytes: 801 < 1001 > 1201
ScriptBytes: 5216 < 6519 > 7822
StylesheetBytes: 801 < 1001 > 1201
ScriptBytes: 139667 < 141667 > 143667
StylesheetBytes: 41668 < 43668 > 45668

That seems somewhat reasonable and it should be pretty easy to adjust if we think another percentage is better (or if we drop that idea). A zero also remains a 0, that can't vary without us noticing (we'd still have the files too, but still seems like a good rule to have).

Not 100% sure about some of the changes here. I went with Bytes here in the API of the new method and also changed the existing docs and went from a list of metrics to checking if they end with Bytes. But I'm not entirely sure about that. Could add another method to check that or even run the logic within that one method and check there for the metric name. Open to suggestions.

To test: change one of the Bytes metrics in the assertion files, once within the allowed range, then it shouldn't change when running the test, once outside, then it should update. Note that range is calculated from the stored metric in the file. If you increase it, the range gets larger too. That confused me for a while. For example, to get "1282" to change, it's not 1542 but 1603 because that's that's when 1282 is no longer in range of that. it might also not be those exact with the rounding and calculation, but I don't think that matters too much.

Limitation: This only works when the existing metrics file exists and can be parsed. If it's deleted, it will write whatever value it gets, but I think that's not such a big deal. It's fine to update it, we just don't want to do it all the time.

longwave’s picture

Status: Needs review » Needs work

Added a couple of suggestions.

berdir’s picture

Status: Needs work » Needs review

Thanks for the reviews, used tolerance now.

I started running into my first rebase conflicts on the yml files, and thought for bit about parsing this automatically so we can conveniently just rerun it and keep the current byte values.

But I struggled with a sensible way to do that automatically, I think that gets too complicated. I'll update the CR with some approaches that work for me as I'm getting used to it.

nicxvan’s picture

Status: Needs review » Reviewed & tested by the community

Nightwatch is the failure, I think it's random.

Code changes look good, tolerance is a much better name.

I tested this locally by doing the following.

Running the test normally tests pass.
Updating a byte value StylesheetBytes: 1001 in standardBlockLogin to StylesheetBytes: 1011 tests pass as expected.
Updating a byte value StylesheetBytes: 1001 in standardBlockLogin to StylesheetBytes: 21011 tests fail as expected.

Setting PERF_TEST_UPDATE=1
Updating a byte value StylesheetBytes: 1001 in standardBlockLogin to StylesheetBytes: 1011 tests pass as expected, and no update to StylesheetBytes.
Updating a byte value StylesheetBytes: 1001 in standardBlockLogin to StylesheetBytes: 21011 tests pass as expected, and it is updated to StylesheetBytes: 1001.

  • catch committed c13ef3fb on 11.x
    task: #3619240 Do not automatically update existing Stylesheet/...
catch’s picture

Version: main » 11.x-dev
Status: Reviewed & tested by the community » Fixed

This looks great and will help reduce merge conflicts in a lot of other issues. Also good to have the relative but capped tolerance levels.

Committed/pushed to main and 11.x, thanks!

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.

  • catch committed 80996832 on main
    task: #3619240 Do not automatically update existing Stylesheet/...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.