#1952058: [META] Retesting stale RTBC core patches got deployed recently which is fantastic.
There's a couple of other things that would help to prune the core queues further.
The test on git apply + php -l is more or less instant compared to running the full test suite. Some 24 hour periods see more than 50 core commits, so if we were able to re-test just for those conditions, it would catch patches very early. What it would also enable is re-testing the CNR queue, which is considerably longer (but could get significantly shorter, at least I hope!).
If we also did #2189317: Get a PHPUnit job type running that is separate and independent from the Simpletest Job type, fix the d.o integration so that we can run the PHPUnit tests independently, it might be possible to add PHPUnit to the re-tests as well - whether that's a
Additionally, if we added PHPUnit as a separate step per those tests take around 50 seconds, so those could also happen in a frequent test bot without a lot of additional infrastructure load. That might lull people into a false sense of security though, not sure.
Comments
Comment #1
MixologicThis is probably still somewhat relevant, and probably falls somewhere into our testing policy framework. [#2696421]
Comment #2
MixologicIm going to call this a policy question, and postpone it on actually being able to run these tests separately.
Comment #3
catchTentatively unpostponing this, and re-titling now that phpunit also does kernel and functional testing.
The need here is more or less the same as it was before - a retesting job which only checks if a patch applies, and marks the patch as not applying and the issue needs work if it fails.
This would be considerably faster and less expensive to run than the full test suite, so ideally we could run it on 'needs review' issues maybe every 72 hours or weekly, and RTBC issues maybe every 12 hours.
I think we could switch off actual patch retesting on RTBC issues if we had this, since the case of a non-conflicting cross-commit that fails tests is considerably rarer (and caught by on-commit testing and manual re-tests usually).
Comment #4
catchDuplicate of #2021489: [Policy] Automatically check whether 'needs review' patches apply and pass linting.