At 8.x the Rules project code only has one automated test job. This is "PHP 7.1 & MySQL 5.5, core D8.4" and runs daily. This is good, but we need more.
The default configuration for 8.x issue testing is "PHP 5.5 & MySQL 5.5, D8.4" which we should have as "tested on commit and issue testing default" so that patches are automatically tested (currently after uploading, you have to click 'test/retest' and select the parameters manually)
We also need to test daily at Core 8.5 so that changes regarding deprecated code, for example, are discovered quickly. I believe you need to choose a different db & php combination when picking a different core version, so that the additional tests remain discrete. So I suggest picking PHP7 with Core 8.5 for running daily.
| Comment | File | Size | Author |
|---|---|---|---|
| #7 | rules tests on d.o. 2018-01-19.png | 129.11 KB | jonathan1055 |
| #6 | test options 2018-01-15.png | 208.61 KB | jonathan1055 |
| #4 | automated testing options on 8.x.png | 193.36 KB | jonathan1055 |
Comments
Comment #2
tr commentedYes, I find this important too. Patches aren't getting tested automatically, and can sometimes sit in an issue for days until someone notices and manually starts the test.
We put a lot of effort into writing tests, and mandating that tests accompany patches - let's put the testbot to work for us and help ensure the codebase stays working.
Comment #3
fagook I tried to configure a few more. Let's check when they are run through whether the config is better now.
Comment #4
jonathan1055 commentedThanks @fago, yes this is good.
Just for info, did you pick from the descriptive options (top four radio buttons) or the explicit fixed core version (bottom three options) as in the attached image? Useful to know what might happen when the next core release is made.
Comment #5
fagosry, I don't know anymore and it does not seem to be visible from the automated testing screen. Let's when core version changes.
Comment #6
jonathan1055 commentedYes that's fine. Just for the record since posting the previous image on 9th the versions have moved on. As at 15th Jan we get:
Comment #7
jonathan1055 commentedActually there are two specific things you could do to make the d.o. testing set-up better. Here is what we currently have:
My suggestions are:
These two changes will help to give us good early warning about how Rules performs in all core versions.
* Note - there seems to be a d.o. testing config bug which does not recognise the core version as part of the key when determining what makes one test distinct from the others. If you use the same combo of php version and database then you are only allowed one core version. Adding a new test will always delete the existing one. For the new test above, pick PHP5.6 as this is not utilised yet in any tests, so will ensure uniqueness.
Comment #8
tr commentedDo you have an issue number for this? I have a similar problem with tests overriding other tests, but it's not exactly what you describe. Specifically, I have a PHP 7.2, MySQL 5.5, Core 8.6 daily test that, when it runs, overrides and deletes my PHP 5.5, MySQL 5.5, Core 8.4 on commit/testing default test.
Comment #9
tr commentedI think this can be marked as fixed now. We have daily tests for D7 and D8 which test against the every-changing core development releases, so these should provide early warning of any core changes. And we also have tests on commit which check against a variety of PHP/DB configurations, using the currently supported version of core. These should ensure that any commits will work on current sites. I also added some PostgreSQL tests the other day. I don't think we've ever tested with PostgreSQL before, because these new tests show some errors in both D7 and D8. So I think these new tests will allow us to better support a wide range of platforms.
Comment #10
jonathan1055 commentedThanks for adding the new test config against PostgreSQL, that is useful.
One thing missing, however, is a daily test against the current stable or supported release (which at the moment is 8.6). If a core change in 8.6 was made we would only discover a test failure when the next Rules commit was made, which could be weeks or months later. Relating back to my final comment in #7 above, we should be able to find an "unused" pairing of DB + PHP which could be used for daily tests against the "supported" release (which is currently 8.6.x-dev).
Comment #11
tr commentedDrupal "semantic versioning" is really not very well implemented yet, but supposedly in order for a patch to be put into 8.6.x (the current "supported" version) it needs to be committed to 8.7.x (the current "development" version) first. So I think that by testing daily against the current "development" version, we get a warning or notification about anything that might be put into "supported" ?
I think the best way to test would be to test against all combinations of PHP/DB/Drupal on a daily basis AND for every commit AND for every proposed patch. But I'm trying to keep our resource consumption to the minimum that we actually need.
If you can look at the current tests and propose a change, I'm open to making that change. I think the big win here is the daily tests against development, but also the fact that we now have some people who are trying to stay ahead of the changes and keeps the tests passing. There will always be core changes that break things - the important thing is that we try to find out about problems early and try to fix them before most of the community even knows about them.