Problem/Motivation
Right now, the new performance test assertions will silently write files if asked for assertions that don't exist yet.
It was/is convenient initially to get started, but also risky, if one is renamed and the tests aren't run, it will silently switch to update mode for that and tests on it will never fail.
#3426303: Move login performance testing to Umami was at least in an intermediate step in that state.
Steps to reproduce
Proposed resolution
If a file doesn't exist and update mode isn't set, fail the test.
Another scenario is that there could be leftover files that aren't deleted. We could also assert that all files within a test folder are being used and fail in a teardown if not. That's less of a concern as that's just a bit of cruft, everything that should be asserted still is.
Comments