7.x test coverage

In Reduce memory footprint comment #34 I discovered that the automated testing for default time using popup calendar worked fine even though when doing this operation manually it failed. The error was due to function scheduler_date_popup_pre_validate_alter() not being loaded during manual testing but it seems to have been loaded fine during automated testing. This function was in file .edit.inc which is included from scheduler_form_alter()

Is it the case that during automated testing, a call to drupalPost('node/add/page') will use hook_form_alter, do the edit, validation and the save all in one operation and hence the functions loaded when the form was first called are still known when the data is validated? If this is not the case, then how did the automated tests pass?

Understanding this situation will have a bearing on performing the test coverage analysis, hence continuing this discussion here instead.

This task still needs to be done for 7.x. It will differ for 8.x, hence I have created a separate issue #2712465: [meta] 8.x-1.x test coverage

I've been working on some stuff in order to automate a code coverage report for Drupal modules. You can check my work in progress in this branch on github.

In order to see the code coverage report, click on the coverage report badge.

As soon as it's stable, I'll update this ticket. You can already have a good idea on what we have covered so far!

WOW! That is fantastic! What a brilliant idea. The coloured lines of code make it beautifully easy to view the coverage. This is great work.

I'd like to know more about how you are achieving it, i.e. getting the data from simpletest. This is going to make it so much better, compared to the task of checking manually. Are you intending to share this and make it available to others?

[edit: I have just signed-up on Github, https://github.com/jonathan1055 ]

Nice to see that you appreciate the work. As I stated; this is work in progress. I'm currently developing an easier way to implement this as it would be nice if other module maintainers would adopt this.

This won't be Drupal module as it requires an isolated container to execute the tests. I am using Travis CI in order to analyze our tests.

I will give you an update as soon as I have a clean working version of this "thingy".

After working a few weeks on this one, I finally have a more generalized approach. The latest report can be found here.

Now we need to think about how we're going to integrate this with the drupal.org repository. There's already a DrupalCI issue which I'm working on as well which will eventually cover this #2622922 [meta] Code coverage.

Double Wow - this is even better than the first version.
Can you run it against the current 8.x code repository?

After several months of hard work I can now inform you that we have automated code coverage for Scheduler. The last result can be found here.

To keep track of future analyses, check here.

Good work Levi, thank you. I have just logged in to your test site and will give my feedback via https://www.drupal.org/project/issues/drupal_coverage_core

I have added a test for the 'touch' feature, where the created date is changed to match the publishing date.

The coverage report showed that in scheduler.cron.inc this was not being tested. Here is the summary as at the committed code without this patch:

The line not being tested is:

Too busy with the screen grabs I forgot the patch. In addition to the new test I added REQUEST_TIME to the strtotime() calls to ensure that the values will be consistent when checking for equality.

To show that drupal_coverage_core is doing its stuff, here is the .cron.inc report from a new run I initiated on the Scheduler page of the test site.

The coverage for _scheduler_publish() has increased from 91.3% (42/46) to 95.6% (44/46)

The detail of the code show that $n->created = $n->publish_on is now covered by automated testing. We got two extra code lines covered (58 and the closing brace 59).

The overall score for .cron.inc has increased from 72.5% to 74.1% and the total score for the module has gone up by an amazing 0.32% from 58.04% to 58.40%

Thank you to everyone working on the DCC module! This is going to be a great tool to help evaluate automated testing.

Added a test for the lightweight cron - this has been done in 8.x already. This patch does not test the cron admin form (yet) just the urls and the access key.

I have started to add tests for the integration with Rules module. Here's a patch which tests our four reaction rules, based on the tests we already have in 8.x.

These tests were prompted by changes I was working when looking at a problem with Rules in #2898724: PDOException: Column 'nid' cannot be null and I realised we have no Rules test coverage at 7.x

OK testing on d.o. at 7.x still needs the old array syntax array( ) and not [ ]

The new tests pass locally (via web UI and via run-tests.sh). The simpletest log does not have any useful info on what failed, but it seems that $this->admin_user did not get created, due to the scheduler permissions being invalid (which is odd). This implies that the module did not get enabled, so I tracked back and even though there is no error it may be due to the Rules module not being available. This new patch adds Rules as a test dependency in .info

Aargh! Just got hit by #2692407: Test_dependencies are downloaded before applying patches, rather than after (again). The tests for Rules cannot be tested until the change to .info is committed.

Now with the Rules test dependency committed, here is just the .test additions

Tests added for Rules Conditions

Third part of the Rules testing - six events

Some [ ] got in, from 8.x. Changed back to array( )

Missed one.

and again

Now it's the list( ) which is not valid at php5.3 in the foreach statement.
I do need to find a way to test locally with PHP5.3, not 5.5

You might consider disabling the PHP 5.5 and 5.6 tests, and switching to using PHP 7.1 as the primary test option - it'll help ensure the module is compatible with PHP 7 and PHP 7 tests run faster than PHP 5 ones, so it'll save the DA module too :)

Thanks for the suggestions. Yes I have added a weekly test at 7.x and 8.x running PHP7.1. I do not currently have a local setup where I can test PHP7 and my travis testing on 8.x fails when running PHP7 due to not having PHPunit 6

PHPUnit testing framework version 6 or greater is required when running on PHP 7.0 or greater. Run the command 'composer run-script drupal-phpunit-upgrade' in order to fix this.
The command "./vendor/bin/phpunit -c ./core/phpunit.xml.dist ./modules/$MODULE/tests/" exited with 1.

I would like to be able to test using PHP7 before running patches on d.o., and only then will I change the default test config to PHP7. Or is that irrelevant?

I have removed the 7.x PHP5.3 test though.

I'd still suggest having tests run for PHP 5.3 unless the module's info file specifically requires a newer version, you don't want to accidentally leave in some code that'll break older installs, and there's imho no benefit to running PHP 5.5 or 5.6 given you're also running 7.x, any compatibility bugs with 5.5/5.6 will also show up with 7.x.

Added a new test for Non-Enabled types in #3084867-19: Do not check required (un)publish_on date if scheduled (un)publishing is disabled