Problem/Motivation
We test these metrics with a range, but the update now always updates them if run with PERF_TEST_UPDATE.
Most actually seem quite stable, but to avoid constant changes, I think it makes sense to only update if it falls outside of the accepted range. Possibly half the range, so we
maybe here or in another issue, also consider making that range relative to the actual number. quite a few metrics are now at around 1k and we test with a range of 2k. We need to define how much, but something like min($value * .1, 2000), so the lower of 10% variation or at most 2k. And the update would use .05 or 1000 to ensure it roughly stays in the middle. The expected range could be just at the end and then still sometimes fail.
Steps to reproduce
Proposed resolution
Remaining tasks
User interface changes
Introduced terminology
API changes
Data model changes
Release notes snippet
Issue fork drupal-3619240
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
- 3619240-do-not-automatically
changes, plain diff MR !16886
Comments
Comment #2
berdirComment #3
longwaveIs there a way we can make these fully deterministic in performance tests so we don't need the ranges at all?
Comment #4
berdir@catch might know more than I do. based on the numbers I'm seeing actually change, it seems possible more stable than I assumed as only one changes sometimes for me, but it is still not entirely stable either and I haven't actually verified it's always the same.
Comment #5
berdirThe range was actually expanded fairly recently in #3588897: Widen the script/stylesheet bytes assertion ranges in performance tests by @catch, but it looks like it wasn't done because there were random fails, just to reduce the number of times it has to to be updated.
Which is actually a very good point. We don't want to update performance tests on umami/previously olivero every time we change a single byte in a css/js file.
So I think yes, we want to keep that range?
Comment #6
catchYes the range isn't because of indeterminacy it was because people got fed up rerolling. Ironically we've sort of reintroduced that by writing it out automatically, so something like this issue, or rounding the number to the nearest 500 bytes or similar would help.
Comment #7
berdir> Ironically we've sort of reintroduced that by writing it out automatically, so something like this issue, or rounding the number to the nearest 500 bytes or similar would help.
I do plan to work on this, I'll also see if the comments can be improved a bit.
It's definitely just "sort" of, because it doesn't need to be updated on every change, it's now just also updated when tests need updating because of actual changes.
Comment #9
berdirThis MR now adds a method to calculate the range, and it uses the lower or 20% of the value or 2000 if that would be above, which is from 10k on.
This results in this behavior now when writing it out with debug statements for StandardPerformanceTest:
That seems somewhat reasonable and it should be pretty easy to adjust if we think another percentage is better (or if we drop that idea). A zero also remains a 0, that can't vary without us noticing (we'd still have the files too, but still seems like a good rule to have).
Not 100% sure about some of the changes here. I went with Bytes here in the API of the new method and also changed the existing docs and went from a list of metrics to checking if they end with Bytes. But I'm not entirely sure about that. Could add another method to check that or even run the logic within that one method and check there for the metric name. Open to suggestions.
To test: change one of the Bytes metrics in the assertion files, once within the allowed range, then it shouldn't change when running the test, once outside, then it should update. Note that range is calculated from the stored metric in the file. If you increase it, the range gets larger too. That confused me for a while. For example, to get "1282" to change, it's not 1542 but 1603 because that's that's when 1282 is no longer in range of that. it might also not be those exact with the rounding and calculation, but I don't think that matters too much.
Limitation: This only works when the existing metrics file exists and can be parsed. If it's deleted, it will write whatever value it gets, but I think that's not such a big deal. It's fine to update it, we just don't want to do it all the time.
Comment #10
longwaveAdded a couple of suggestions.
Comment #11
berdirThanks for the reviews, used tolerance now.
I started running into my first rebase conflicts on the yml files, and thought for bit about parsing this automatically so we can conveniently just rerun it and keep the current byte values.
But I struggled with a sensible way to do that automatically, I think that gets too complicated. I'll update the CR with some approaches that work for me as I'm getting used to it.
Comment #12
nicxvan commentedNightwatch is the failure, I think it's random.
Code changes look good, tolerance is a much better name.
I tested this locally by doing the following.
Running the test normally tests pass.
Updating a byte value
StylesheetBytes: 1001in standardBlockLogin toStylesheetBytes: 1011tests pass as expected.Updating a byte value
StylesheetBytes: 1001in standardBlockLogin toStylesheetBytes: 21011tests fail as expected.Setting
PERF_TEST_UPDATE=1Updating a byte value
StylesheetBytes: 1001in standardBlockLogin toStylesheetBytes: 1011tests pass as expected, and no update toStylesheetBytes.Updating a byte value
StylesheetBytes: 1001in standardBlockLogin toStylesheetBytes: 21011tests pass as expected, and it is updated toStylesheetBytes: 1001.Comment #14
catchThis looks great and will help reduce merge conflicts in a lot of other issues. Also good to have the relative but capped tolerance levels.
Committed/pushed to main and 11.x, thanks!