#2532352: [policy, no patch] Simplify, automate parts of the project application & release processes suggests adding certain markers of project health/stability to project pages.
Currently we have a tab at https://www.drupal.org/node/3060/qa which has (usually) lots of lovely green bars for passing branch tests.
Showing every branch/environment on the main project page might be a bit much, but we could do something like:
Testing status: PASS (green) / FAIL (yellow/orange) / No testing (yellow/orange)
Then a collapsed fieldset/details showing the results or just linking to the tab.
Code style #1299710: [meta] Automate the coding-standards part of patch review results from DrupalCI is something that could be run against all projects whether they have test coverage or not, and reported in a similar way.
Code coverage #2622922: [meta] Code coverage could be run against all projects with tests, and default to 0 for those without.
Comment | File | Size | Author |
---|---|---|---|
#32 | tests-difference.png | 75.37 KB | benjy |
#24 | Screen Shot 2017-08-09 at 10.46.43 AM.png | 104.52 KB | drumm |
#22 | Screen Shot 2017-08-09 at 11.20.46.png | 67.16 KB | Wim Leers |
#22 | Screen Shot 2017-08-09 at 11.20.40.png | 51.33 KB | Wim Leers |
#19 | with-box.png | 133.23 KB | benjy |
Comments
Comment #2
catchComment #3
Sam152 CreditAttribution: Sam152 as a volunteer and at PreviousNext commentedYes, I love this idea. I've been keen on a build status badge for Drupal.org for a while now. Code coverage is a good one, but we should be careful of giving it too much prominence as web tests obviously wouldn't contribute to it. Maybe the number of passes/successful tests would also be a good metric? Passing (50) is better than Passing (1).
Comment #4
gisleI think automated tests are a wonderful help for a developer for QA purposes.
However, I would be wary about creating a "Testing Badge" to give a project more "Status".
Given the present Testing Framework, it is trivial to earn the proposed "Testing Badge" by creating 10, 100 or 1000 trivial tests and have code passing those tests counted. If this happens, the badge-adorned project may still have more bugs than a bait shop and more holes than a colander (given that the player is unscrupulous about it).
If you go ahead and gamify something, you need to think about how to prevent people to "game" your system to earn the status without doing the work. Counting number of tests passed when the player is also the person writing those tests is not going to fly.
Comment #5
Sam152 CreditAttribution: Sam152 as a volunteer and at PreviousNext commentedA valid concern, but are git vetted users trusted not to game Drupal.org in destructive ways? Anyone can write a script to make thousands of commits (which show up on user profiles). If I found an instance of this, I'd be reporting it to the webmasters queue and moving on. If this is more commonly a useful indicator of quality, I think it's still valuable on the whole.
Comment #6
catchIt's possible, but not easy, to get code coverage via xdebug, for example http://drupal.org/project/code_coverage
Agreed code coverage isn't perfect though, but do think it's much more useful than not having it.
We show that information on the automated testing tab already, but 50 assertions isn't necessarily better than 1. i.e. if I have a 20-line module with a single test/assertion, that's going to be better tested than Views or Core if they had 50 assertions.
Comment #7
Sam152 CreditAttribution: Sam152 as a volunteer and at PreviousNext commentedOn the existing QA pages, I think passing web tests show up as 1, but individual test cases from @dataProviders count for one each. I don't think it's specifically assertions, but I take your point.
Comment #8
drummThis replaces the Download table with blocks that have more room to grow. More per-branch information, like phpcs results or code coverage, has a place to be added. And there’s also more room to show Composer information.
Comment #16
drummDeploying this in the next hour
Comment #17
MustangGB CreditAttribution: MustangGB commentedAny reason we can't have the sub-categories like before?
i.e. "Recommended releases", "Other releases" and "Development releases"
It's quite difficult to parse with everything mashed together.
Comment #18
drummI moved the dev releases next to their tagged releases so the information about each branch was grouped together. And breaking out of the table format gives the information more space to grow, such as adding Composer information, the short description of the release, and more information to help evaluate project/release stability.
Comment #19
benjy CreditAttribution: benjy at Unearthed commentedDo the test results not appear for 8.x project branches? I can only see them against 7.x.
Also, with the coloured bars and the information below, it does feel like they need a box to show that all the information is associated.
With a box
Without a box
Comment #20
fgmMuch as I like these changes, they come with a problem relevant to a number of modules I more or less maintain, and which require non-"standard" services to run their test suite (MongoDB, Beanstalkd, RabbitMQ, Kafka...).
In all these cases, we have to use external CI services (actually always Travis in these cases) since the d.o. infra won't just run anything outside the supported perimeter, which means the modules appear as not tested although they are (including significant coverage in some cases like Beanstalkd).
We discussed this already rather long ago, and I offered to provide a server instance to run these components so that Drupal CI could run the tests for these modules too, but this went nowhere.
Any suggestions on how we could avoid this ?
Comment #21
MustangGB CreditAttribution: MustangGB commentedIf we can't have sub-categories then benjy's boxes seem like the next best thing.
Comment #22
Wim LeersI agree with two concerns I saw raised so far:
Also note that on the "Automated testing" tab, the same test result that is shown on the project page is also highlighted through a black border — apparently because it has the
pift-ci-issue
class, which the D8 equivalent does not have:(Note that https://www.drupal.org/project/big_pipe_sessionless, which is D8-only, also does not have test results listed on its project page, and it also doesn't have a
pift-ci-issue
class on the "Automated testing" tab for any of its results.)On top of these two concerns, I have one more: one of the most important pieces of information for a Drupal module is whether it's working correctly against the next Drupal 8 minor. This is why it's valuable to have daily testing configured: because that can detect BC breaks in the next minor for a contrib module long ahead of time! This is how I noticed #2884337: Fix Symfony 3.2 deprecation notices — the daily automated testing was consequently failing since May 4, I noticed on June 7. It turned out to be due to a deprecation in Symfony that ended up in the next Drupal minor. So that helps encourage contrib modules to stop using deprecated APIs! It also helps module evaluators assess module stability/maintenance level.
Therefore showing development snapshots' automated testing results for daily runs is very important, and I think it should be displayed on the project page.
Comment #23
drummDrupalCI has been pulling in dependencies via Composer for a few months now, including 3rd-party PHP dependencies. That should get many projects more-complete testability. Although, we currently don’t have a way to pull in 3rd-party Docker containers to run the servers themselves. https://www.drupal.org/project/drupalci_testbot is the best issue queue for that.
Comment #24
drummHere is the version of the boxes I’m trying out while reviewing the rest of the comments here.
Comment #25
fgm@drumm, thanks, opened #2900969: How to run servers on which a module depends during tests ? in that queue for followup.
Comment #26
Wim Leers#24 looks like a nice alternative solution to #19. No strong preference here.
Comment #27
drummUnlike the /qa page, this currently resolves the branch labels and queries for the specific branch. The default core branch recently changed from 8.4.x to 8.5.x. Many projects haven’t run tests recently enough to get the new core branch.
I put in a retest for https://www.drupal.org/project/search_api to double check this, and the 8.5.x test is now showing. This also showed me the download table cache needs to be cleared on branch test completion.
I’ll fix the query to better match the /qa page, and fix the caching issue.
Comment #28
Wim LeersYay, thanks @drumm!
Comment #29
MustangGB CreditAttribution: MustangGB commentedMuch better, thanks!
Comment #31
drummThose followups are all deployed now.
I threw in a link to “all results” to go to the /qa page. Showing all of the results on the project page would be a bit much, but I could see doing a followup to do some sort of simple summary to catch things like a daily test failing.
Comment #32
benjy CreditAttribution: benjy at Unearthed commentedThanks Drumm, one other thing is on my 7.x branch the tests results have black bold text and a dark outline but on 8.x it's all green.
See screenshot.
Comment #33
Wim Leers#32 seems related to what I wrote at #22.2.
P.S.: massive improvement, @drumm, thank you!
Comment #34
drummbenjy - re-test those branches to get tests with Drupal 8.5.x.
The bold highlight shows what the current issue testing default is. Primarily to help scan through multiple tests on an issue’s files. It is strict about matching everything up to the project’s current QA configuration. For example, it can help spot stale test results after the default has changed.
Comment #35
mmjvb CreditAttribution: mmjvb as a volunteer commentedNoticed security shields appearing for releases other than Stable, should they?
Also, those shields appear in different colors, what do they mean? Searched for information but couldn't find any!
Comment #36
drummThe colors all mean the same thing.
Which stable release are you seeing a shield on? For example, 8.x-1.3 in #32 is stable, and supported by the project maintainer, but not recommended.
Comment #37
mmjvb CreditAttribution: mmjvb as a volunteer commentedObviously, those that opted into security advisory.
My remark was about shields for releases OTHER THAN Stable: alpha beta and rc !
In addition the shields are now showing on projects that didn't opt in!
Comment #38
drummAh, that’s a regression from #2863103: Include metadata about the security coverage opt-in in the 'extra' attribute of composer.json files and is now fixed.
Comment #40
thomasmurphy CreditAttribution: thomasmurphy as a volunteer commentedWhat would really useful is if the dates of all releases, including dev, were right-aligned so module evaluators could quickly see where the project activity is. It's quite common that the dev version is the real version and the full release version hasn't had any activity for years, that was one of the most useful features of the old download table.