Problem/Motivation

There is currently no way to know (except for a failing test) what the code coverage is of any particular piece of code in core. Passing tests don't necessarily mean a change is OK.

There were previous efforts to generate code coverage reports for simpletest back when it was added to 7.x, but they never got centralized:

https://www.drupal.org/project/code_coverage

https://www.acquia.com/gb/blog/calculating-test-coverage

http://blog.boombatower.com/code-coverage-reports

Proposed resolution

Add code coverage, for branches only, and display the results somewhere.

Very eventually, it would be great to integrate coverage with api.drupal.org, but really anything is better than nothing.

For implementation, phpunit and simpletest/xdebug would probably need to be separate issues.

Remaining tasks

User interface changes

API changes

Data model changes

Comments

catch created an issue. See original summary.

catch’s picture

catch
he/him
Primary language English
commented
Issue summary: View changes
legovaer’s picture

legovaer
he/him
Primary language Dutch
Location Duffel
commented

I've put a lot of effort in order to have automated code coverage reports on Simpletest tests. This is what we have today for the Scheduler module. This has been generated by Travis CI by using Drupal CI.

I created several Pull Requests in order to have this functionality available. I'm currently working on getting these reports published to coveralls.io as well. I'll update this issue as soon as the PRs got merged.

jonathan1055’s picture

jonathan1055 commented

Brilliant work, legovaer. This will really help module developers, and indeed core developers too. If you would like help in any way please let us know.

Mixologic’s picture

Mixologic
He/Him/His
Primary language English
Location Portland, OR
commented
Project: DrupalCI: Drupal.org Testing Infrastructure » DrupalCI: Test Runner
Component: Code » Jobs and Job Handling
legovaer’s picture

legovaer
he/him
Primary language Dutch
Location Duffel
commented
Assigned: Unassigned » legovaer

I'm almost ready with a prototype which will allow anyone to build a code coverage report for any module. I will be giving a first demo during DrupalCamp Ghent.

As soon as my alpha version is ready, I'll update this ticket.

jonathan1055’s picture

jonathan1055 commented

Excellent. Good luck with the demo.

mile23’s picture

mile23
Primary language English
Location Seattle, WA
commented

+1 on adding coverage reports to test runs. It could be the default text output, or it could be sophisticated with clover or other output format. From the drupalci perspective, this means working on the artifact system and then having d.o do something with it.

We should be able to generate a coverage report pretty easily for PHPUnit-based tests, and we really should only do that because it's built for it.

That is: 'Coverage' on a SimpleTest test isn't really coverage, and shouldn't count for CRAP numbers or what-have-you. In PHPUnit you can specify @covers and @coversNothing to get as granular as you'd like, and SimpleTest doesn't allow for this.

So for core I'd say it's a much better use of time to convert SimpleTest to PHPUnit-based tests which give us a way to control how coverage is reported per unit. #2735005: Convert all Simpletest web tests to BrowserTestBase (or UnitTestBase/KernelTestBase)

For existing SimpleTest-based functional and integration tests we probably need a better way to explain what behavior is being tested so it's discoverable. We could modify SimpleTest to report some piece of the test docblock or an annotation along with a fail or what-have-you. Maybe start with reporting the @group to give some info to drill down. We might also optionally report a stack trace for fails, based on annotation.

legovaer’s picture

legovaer
he/him
Primary language Dutch
Location Duffel
commented

Wanted to give you guys an update on this. Today we've released the first alpha version of Drupal Coverage. You can already see some coverage reports on our POC.

We're now aiming to support Drupal8 tests. In the future we will aim to use DrupalCI as part of our project.

rivimey’s picture

rivimey
she/her
commented

legovaer, I am wondering if I can use this work to explore D7 test coverage of ctools on my local machine (so I can test my local code)? If so, is there help available on how to set it up?

jonathan1055’s picture

jonathan1055 commented

Hi rivimey,
That's a coincidence, as I have just run a coverage report of D7 ctools a few days ago! The reports are viewable at https://rawgit.com/legovaer/dc-travis/reports/ctools/183/index.html

I am working with legovaer on testing the Drupal Coverage Core project. I am not certain yet whether anyone is allowed publically to submit coverage reports. So it might be that we need to communicate off-board on this. I am happy to share what I have done to run this coverage, but I would like to get Levi's permission before I do so.

Jonathan

rivimey’s picture

rivimey
she/her
commented

Hi Jonathan, I presume the reason for 'allowed' is security? Should I use the d.o contact form?

I'm currently trying to improve the test coverage of D7 ctools and if you are able would love to have help: https://www.drupal.org/node/2827100

I've been able to set up the code_coverage module which is working well enough, but there is an anomaly: if the source text for a function is parsed by php not called, it seems to get ignored in the coverage reports and the code in it is completely ignored in the coverage. This particularly affects the ctools plugins :(. Is this a known problem, and is there a fix?