#2532352: [policy, no patch] Simplify, automate parts of the project application & release processes suggests adding certain markers of project health/stability to project pages.

Currently we have a tab at https://www.drupal.org/node/3060/qa which has (usually) lots of lovely green bars for passing branch tests.

Showing every branch/environment on the main project page might be a bit much, but we could do something like:

Testing status: PASS (green) / FAIL (yellow/orange) / No testing (yellow/orange)

Then a collapsed fieldset/details showing the results or just linking to the tab.

Code style #1299710: [meta] Automate the coding-standards part of patch review results from DrupalCI is something that could be run against all projects whether they have test coverage or not, and reported in a similar way.

Code coverage #2622922: [meta] Code coverage could be run against all projects with tests, and default to 0 for those without.

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

catch created an issue. See original summary.

catch’s picture

Issue summary: View changes
Sam152’s picture

Yes, I love this idea. I've been keen on a build status badge for Drupal.org for a while now. Code coverage is a good one, but we should be careful of giving it too much prominence as web tests obviously wouldn't contribute to it. Maybe the number of passes/successful tests would also be a good metric? Passing (50) is better than Passing (1).

gisle’s picture

I think automated tests are a wonderful help for a developer for QA purposes.

However, I would be wary about creating a "Testing Badge" to give a project more "Status".

Given the present Testing Framework, it is trivial to earn the proposed "Testing Badge" by creating 10, 100 or 1000 trivial tests and have code passing those tests counted. If this happens, the badge-adorned project may still have more bugs than a bait shop and more holes than a colander (given that the player is unscrupulous about it).

If you go ahead and gamify something, you need to think about how to prevent people to "game" your system to earn the status without doing the work. Counting number of tests passed when the player is also the person writing those tests is not going to fly.

Sam152’s picture

A valid concern, but are git vetted users trusted not to game Drupal.org in destructive ways? Anyone can write a script to make thousands of commits (which show up on user profiles). If I found an instance of this, I'd be reporting it to the webmasters queue and moving on. If this is more commonly a useful indicator of quality, I think it's still valuable on the whole.

catch’s picture

Code coverage is a good one, but we should be careful of giving it too much prominence as web tests obviously wouldn't contribute to it.

It's possible, but not easy, to get code coverage via xdebug, for example http://drupal.org/project/code_coverage

Agreed code coverage isn't perfect though, but do think it's much more useful than not having it.

Maybe the number of passes/successful tests would also be a good metric? Passing (50) is better than Passing (1).

We show that information on the automated testing tab already, but 50 assertions isn't necessarily better than 1. i.e. if I have a 20-line module with a single test/assertion, that's going to be better tested than Views or Core if they had 50 assertions.

Sam152’s picture

On the existing QA pages, I think passing web tests show up as 1, but individual test cases from @dataProviders count for one each. I don't think it's specifically assertions, but I take your point.

drumm’s picture

Assigned: Unassigned » drumm
Status: Active » Needs review
Related issues: +#2820230: Display Composer require strings on project release table
FileSize
64.17 KB

Screenshot

This replaces the Download table with blocks that have more room to grow. More per-branch information, like phpcs results or code coverage, has a place to be added. And there’s also more room to show Composer information.

  • drumm committed 34e50c1 on 2688127-downloads-ci
    Issue #2688127: Add Release branch information Views handler
    
  • drumm committed 3c4a38d on 2688127-downloads-ci
    Issue #2688127: Remove old testing display on project page
    
  • drumm committed fb57099 on 2688127-downloads-ci
    Issue #2688127: Export redesigned View
    

  • drumm committed 34e50c1 on 2688127-downloads-ci
    Issue #2688127: Add Release branch information Views handler
    
  • drumm committed 3c4a38d on 2688127-downloads-ci
    Issue #2688127: Remove old testing display on project page
    
  • drumm committed fb57099 on 2688127-downloads-ci
    Issue #2688127: Export redesigned View
    

  • drumm committed 34e50c1 on 2688127-downloads-ci
    Issue #2688127: Add Release branch information Views handler
    
  • drumm committed 3c4a38d on 2688127-downloads-ci
    Issue #2688127: Remove old testing display on project page
    
  • drumm committed fb57099 on 2688127-downloads-ci
    Issue #2688127: Export redesigned View
    

  • drumm committed 34e50c1 on 2688127-downloads-ci
    Issue #2688127: Add Release branch information Views handler
    
  • drumm committed 3c4a38d on 2688127-downloads-ci
    Issue #2688127: Remove old testing display on project page
    
  • drumm committed fb57099 on 2688127-downloads-ci
    Issue #2688127: Export redesigned View
    

  • drumm committed 34e50c1 on 2688127-downloads-ci
    Issue #2688127: Add Release branch information Views handler
    
  • drumm committed 3c4a38d on 2688127-downloads-ci
    Issue #2688127: Remove old testing display on project page
    
  • drumm committed fb57099 on 2688127-downloads-ci
    Issue #2688127: Export redesigned View
    

  • drumm committed 34e50c1 on 2688127-downloads-ci
    Issue #2688127: Add Release branch information Views handler
    
  • drumm committed 3c4a38d on 2688127-downloads-ci
    Issue #2688127: Remove old testing display on project page
    
  • drumm committed fb57099 on 2688127-downloads-ci
    Issue #2688127: Export redesigned View
    

  • drumm committed 34e50c1 on 7.x-3.x, dev, 2688127-downloads-ci
    Issue #2688127: Add Release branch information Views handler
    
  • drumm committed 3c4a38d on 7.x-3.x, dev, 2688127-downloads-ci
    Issue #2688127: Remove old testing display on project page
    
  • drumm committed fb57099 on 7.x-3.x, dev, 2688127-downloads-ci
    Issue #2688127: Export redesigned View
    
drumm’s picture

Status: Needs review » Fixed

Deploying this in the next hour

MustangGB’s picture

Any reason we can't have the sub-categories like before?
i.e. "Recommended releases", "Other releases" and "Development releases"

It's quite difficult to parse with everything mashed together.

drumm’s picture

I moved the dev releases next to their tagged releases so the information about each branch was grouped together. And breaking out of the table format gives the information more space to grow, such as adding Composer information, the short description of the release, and more information to help evaluate project/release stability.

benjy’s picture

FileSize
124.85 KB
133.23 KB

Do the test results not appear for 8.x project branches? I can only see them against 7.x.

Also, with the coloured bars and the information below, it does feel like they need a box to show that all the information is associated.

With a box

Concept with a box

Without a box

Concept without a box

fgm’s picture

Much as I like these changes, they come with a problem relevant to a number of modules I more or less maintain, and which require non-"standard" services to run their test suite (MongoDB, Beanstalkd, RabbitMQ, Kafka...).

In all these cases, we have to use external CI services (actually always Travis in these cases) since the d.o. infra won't just run anything outside the supported perimeter, which means the modules appear as not tested although they are (including significant coverage in some cases like Beanstalkd).

We discussed this already rather long ago, and I offered to provide a server instance to run these components so that Drupal CI could run the tests for these modules too, but this went nowhere.

Any suggestions on how we could avoid this ?

MustangGB’s picture

If we can't have sub-categories then benjy's boxes seem like the next best thing.

Wim Leers’s picture

Status: Fixed » Needs work
FileSize
51.33 KB
67.16 KB

I agree with two concerns I saw raised so far:

  1. Echoing the usability concerns in #17. I like the proposal in #18 with the boxes, makes it visually much clearer, like #21 also is saying.
  2. Confirming the bug described in #18: the CDN module's 7.x release is showing the automated testing results, but the 8.x release is not:

    Also note that on the "Automated testing" tab, the same test result that is shown on the project page is also highlighted through a black border — apparently because it has the pift-ci-issue class, which the D8 equivalent does not have:

    (Note that https://www.drupal.org/project/big_pipe_sessionless, which is D8-only, also does not have test results listed on its project page, and it also doesn't have a pift-ci-issue class on the "Automated testing" tab for any of its results.)

On top of these two concerns, I have one more: one of the most important pieces of information for a Drupal module is whether it's working correctly against the next Drupal 8 minor. This is why it's valuable to have daily testing configured: because that can detect BC breaks in the next minor for a contrib module long ahead of time! This is how I noticed #2884337: Fix Symfony 3.2 deprecation notices — the daily automated testing was consequently failing since May 4, I noticed on June 7. It turned out to be due to a deprecation in Symfony that ended up in the next Drupal minor. So that helps encourage contrib modules to stop using deprecated APIs! It also helps module evaluators assess module stability/maintenance level.
Therefore showing development snapshots' automated testing results for daily runs is very important, and I think it should be displayed on the project page.

drumm’s picture

Much as I like these changes, they come with a problem relevant to a number of modules I more or less maintain, and which require non-"standard" services to run their test suite (MongoDB, Beanstalkd, RabbitMQ, Kafka...).

In all these cases, we have to use external CI services (actually always Travis in these cases) since the d.o. infra won't just run anything outside the supported perimeter, which means the modules appear as not tested although they are (including significant coverage in some cases like Beanstalkd).

DrupalCI has been pulling in dependencies via Composer for a few months now, including 3rd-party PHP dependencies. That should get many projects more-complete testability. Although, we currently don’t have a way to pull in 3rd-party Docker containers to run the servers themselves. https://www.drupal.org/project/drupalci_testbot is the best issue queue for that.

drumm’s picture

Here is the version of the boxes I’m trying out while reviewing the rest of the comments here.

Screenshot

fgm’s picture

@drumm, thanks, opened #2900969: How to run servers on which a module depends during tests ? in that queue for followup.

Wim Leers’s picture

#24 looks like a nice alternative solution to #19. No strong preference here.

drumm’s picture

Do the test results not appear for 8.x project branches? I can only see them against 7.x.

Unlike the /qa page, this currently resolves the branch labels and queries for the specific branch. The default core branch recently changed from 8.4.x to 8.5.x. Many projects haven’t run tests recently enough to get the new core branch.

I put in a retest for https://www.drupal.org/project/search_api to double check this, and the 8.5.x test is now showing. This also showed me the download table cache needs to be cleared on branch test completion.

I’ll fix the query to better match the /qa page, and fix the caching issue.

Wim Leers’s picture

Yay, thanks @drumm!

MustangGB’s picture

Here is the version of the boxes I’m trying out while reviewing the rest of the comments here.

Much better, thanks!

  • drumm committed 293e081 on 7.x-3.x
    Issue #2688127: Better match for most recent test
    
  • drumm committed 53e9913 on 7.x-3.x
    Issue #2688127: Add link to all results
    
drumm’s picture

Status: Needs work » Fixed

Those followups are all deployed now.

I threw in a link to “all results” to go to the /qa page. Showing all of the results on the project page would be a bit much, but I could see doing a followup to do some sort of simple summary to catch things like a daily test failing.

benjy’s picture

FileSize
75.37 KB

Thanks Drumm, one other thing is on my 7.x branch the tests results have black bold text and a dark outline but on 8.x it's all green.
See screenshot.

Indicate differences

Wim Leers’s picture

#32 seems related to what I wrote at #22.2.

P.S.: massive improvement, @drumm, thank you!

drumm’s picture

benjy - re-test those branches to get tests with Drupal 8.5.x.

The bold highlight shows what the current issue testing default is. Primarily to help scan through multiple tests on an issue’s files. It is strict about matching everything up to the project’s current QA configuration. For example, it can help spot stale test results after the default has changed.

mmjvb’s picture

Noticed security shields appearing for releases other than Stable, should they?
Also, those shields appear in different colors, what do they mean? Searched for information but couldn't find any!

drumm’s picture

The colors all mean the same thing.

Which stable release are you seeing a shield on? For example, 8.x-1.3 in #32 is stable, and supported by the project maintainer, but not recommended.

mmjvb’s picture

Which stable release are you seeing a shield on?

Obviously, those that opted into security advisory.
My remark was about shields for releases OTHER THAN Stable: alpha beta and rc !

In addition the shields are now showing on projects that didn't opt in!

drumm’s picture

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

thomasmurphy’s picture

What would really useful is if the dates of all releases, including dev, were right-aligned so module evaluators could quickly see where the project activity is. It's quite common that the dev version is the real version and the full release version hasn't had any activity for years, that was one of the most useful features of the old download table.