Problem/Motivation

We currently have functional tests which are limited by group, such as CoreD8MySqlPassingTest. We do this so the test runs don't take so long. :-)

They check that the tests run, but they don't check that non-group tests don't run.

We also have CoreNoGroupTest which checks the behavior when no group is specified.

Proposed resolution

Modify existing tests which specify DCI_TestGroups to verify that non-group tests did not run.

Remaining tasks

User interface changes

API changes

Data model changes

Original issue

While working with @Mixologic on debugging a failed test, I discovered that no matter how I invoked the drupalci command, the vagrant hosted local always generated an "All tests will run." test run.

Set up: vagrant local-hosted DrupalCI, with drupalci init using default settings (PHP 5.5, Mysql, etc.).

Here's a sample script I've used to invoke a test run:

DCI_Fetch='https://www.drupal.org/files/issues/file-example-2102651-torenware-v6.patch,modules/examples' \
DCI_Patch='file-example-2102651-torenware-v6.patch,modules/examples' \
DCI_TestItem='directory:modules/examples' \
DCI_TestGroups=' --module file_example '
DCI_AdditionalRepositories='git,git://git.drupal.org/project/examples.git,8.x-1.x,modules/examples,1;'  \
DCI_Concurrency=4 DCI_Color=true ./drupalci run simpletest

This sets up a run with the command:

cd /var/www/html && sudo -u www-data php /var/www/html/core/scripts/run-tests.sh  --url http://localhost/checkout --dburl mysql://drupaltestbot:drupaltestbotpw@drupaltestbot-db-mysql-5-5/simpletest_1446092781  --color --keep-results --sqlite /var/www/html/results/simpletest.sqlite --concurrency 4 --php /opt/phpenv/shims/php --all

I've tried a number of different combinations, but all of them run all of the tests.

Even the example from the Running DrupalCI Locally page does this:

Running 

DCI_Concurrency=4 DCI_Testgroups=action DCI_Color=true ./drupalci run simpletest

results in "All tests will run."

cd /var/www/html && sudo -u www-data php /var/www/html/core/scripts/run-tests.sh  --url http://localhost/checkout --dburl mysql://drupaltestbot:drupaltestbotpw@drupaltestbot-db-mysql-5-5/simpletest_1446141728  --color --keep-results --sqlite /var/www/html/results/simpletest.sqlite --concurrency 4 --php /opt/phpenv/shims/php --all

I'm not sure if the documentation needs to be changed, or if the command line tool is broken right now, but I don't see a way to get a run restricted to a single test, class, or module.

Comments

Torenware created an issue. See original summary.

Torenware’s picture

Torenware commented
Issue summary: View changes
Torenware’s picture

Torenware commented
Issue summary: View changes
isntall’s picture

isntall commented
Status: Active » Needs review

by removing
DCI_TestGroups=' --module file_example '

I was able to get the desired outcome

DCI_Fetch='https://www.drupal.org/files/issues/file-example-2102651-torenware-v6.patch,modules/examples' \
DCI_Patch='file-example-2102651-torenware-v6.patch,modules/examples' \
DCI_TestItem='directory:modules/examples' \
DCI_AdditionalRepositories='git,git://git.drupal.org/project/examples.git,8.x-1.x,modules/examples,1;'  \
DCI_Concurrency=4 DCI_Color=true \
./drupalci run simpletest

here's an excerpt of my local output on a vagrant vm using the dev branch of drupalci.

....
cd /var/www/html && sudo -u www-data php /var/www/html/core/scripts/run-tests.sh  --url http://localhost/checkout --dburl mysql://drupaltestbot:drupaltestbotpw@drupaltestbot-db-mysql-5-5/simpletest_1446616218  --color --keep-results --color --concurrency 4 --sqlite /var/www/html/results/simpletest.sqlite --php /opt/phpenv/shims/php --directory modules/examples
Command created as exec id a7132dc6

Drupal test run
---------------

Tests to be run:
  - Drupal\tour_example\Tests\TourExampleTest
  - Drupal\block_example\Tests\BlockExampleTest
  - Drupal\block_example\Tests\BlockExampleMenuTest
  - Drupal\cron_example\Tests\CronExampleTestCase
  - Drupal\Tests\phpunit_example\Unit\AddClassTest
  - Drupal\Tests\phpunit_example\Unit\DisplayManagerTest
  - Drupal\Tests\phpunit_example\Unit\ProtectedPrivatesTest
  - Drupal\phpunit_example\Tests\PHPUnitExampleMenuTest
  - Drupal\email_example\Tests\EmailExampleTest
  - Drupal\config_entity_example\Tests\ConfigEntityExampleTest
  - Drupal\cache_example\Tests\CacheExampleTestCase
  - Drupal\simpletest_example\Tests\SimpleTestExampleMockModuleTest
  - Drupal\simpletest_example\Tests\SimpleTestExampleTest
  - Drupal\field_example\Tests\FieldExampleMenuTest
  - Drupal\field_example\Tests\TextWidgetTest
  - Drupal\field_example\Tests\Text3WidgetTest
  - Drupal\Tests\file_example\Kernel\StreamWrapperTest
  - Drupal\Tests\file_example\Unit\SessionWrapperTest
  - Drupal\file_example\Tests\FileExampleTest
  - Drupal\node_type_example\Tests\NodeTypeExampleTest
  - Drupal\examples\Tests\ExamplesTest
  - Drupal\plugin_type_example\Tests\PluginTypeExampleTest
  - Drupal\tablesort_example\Tests\TableSortExampleTest
  - Drupal\page_example\Tests\PageExampleTest
  - Drupal\content_entity_example\Tests\ContentEntityExampleTest
  - Drupal\js_example\Tests\JsExampleTest
  - Drupal\dbtng_example\Tests\DBTNGExampleTest

Test run started:
  Wednesday, November 4, 2015 - 05:50

Test summary
------------

Drupal\block_example\Tests\BlockExampleTest                   27 passes
Drupal\Tests\phpunit_example\Unit\AddClassTest                27 passes
Drupal\cron_example\Tests\CronExampleTestCase                 35 passes
Drupal\Tests\phpunit_example\Unit\DisplayManagerTest           1 passes
Drupal\Tests\phpunit_example\Unit\ProtectedPrivatesTest        4 passes
Drupal\block_example\Tests\BlockExampleMenuTest                8 passes             1 exceptions
Drupal\email_example\Tests\EmailExampleTest                   21 passes
Drupal\tour_example\Tests\TourExampleTest                     30 passes
Drupal\phpunit_example\Tests\PHPUnitExampleMenuTest            8 passes
Drupal\config_entity_example\Tests\ConfigEntityExampleTest    82 passes
Drupal\simpletest_example\Tests\SimpleTestExampleMockModuleT  12 passes
Drupal\cache_example\Tests\CacheExampleTestCase               59 passes
Drupal\field_example\Tests\FieldExampleMenuTest                8 passes
Drupal\Tests\file_example\Kernel\StreamWrapperTest             3 passes
Drupal\Tests\file_example\Unit\SessionWrapperTest              1 passes
Drupal\field_example\Tests\TextWidgetTest                    117 passes                            2 messages
Drupal\simpletest_example\Tests\SimpleTestExampleTest         35 passes
Drupal\field_example\Tests\Text3WidgetTest                   117 passes                            2 messages
Drupal\examples\Tests\ExamplesTest                            20 passes
Drupal\node_type_example\Tests\NodeTypeExampleTest            43 passes             1 exceptions
Drupal\file_example\Tests\FileExampleTest                    672 passes   3 fails  24 exceptions  36 messages
Drupal\plugin_type_example\Tests\PluginTypeExampleTest        11 passes             6 exceptions
Drupal\js_example\Tests\JsExampleTest                         10 passes
Drupal\tablesort_example\Tests\TableSortExampleTest           27 passes
Drupal\page_example\Tests\PageExampleTest                     90 passes             2 exceptions
Drupal\content_entity_example\Tests\ContentEntityExampleTest 225 passes
Drupal\dbtng_example\Tests\DBTNGExampleTest                   59 passes

Test run duration: 3 min 57 sec

HTTP/1.1 200 OK
Content-Type: application/vnd.docker.raw-stream


Completed execute:command
....

and on the 20151029_production tag

....
cd /var/www/html && sudo -u www-data php /var/www/html/core/scripts/run-tests.sh  --url http://localhost/checkout --dburl mysql://drupaltestbot:drupaltestbotpw@drupaltestbot-db-mysql-5-5/simpletest_1446616717  --color --keep-results --color --concurrency 4 --sqlite /var/www/html/results/simpletest.sqlite --php /opt/phpenv/shims/php --directory modules/examples
Command created as exec id 21ab1bc3

Drupal test run
---------------

Tests to be run:
  - Drupal\tour_example\Tests\TourExampleTest
  - Drupal\block_example\Tests\BlockExampleTest
  - Drupal\block_example\Tests\BlockExampleMenuTest
  - Drupal\cron_example\Tests\CronExampleTestCase
  - Drupal\Tests\phpunit_example\Unit\AddClassTest
  - Drupal\Tests\phpunit_example\Unit\DisplayManagerTest
  - Drupal\Tests\phpunit_example\Unit\ProtectedPrivatesTest
  - Drupal\phpunit_example\Tests\PHPUnitExampleMenuTest
  - Drupal\email_example\Tests\EmailExampleTest
  - Drupal\config_entity_example\Tests\ConfigEntityExampleTest
  - Drupal\cache_example\Tests\CacheExampleTestCase
  - Drupal\simpletest_example\Tests\SimpleTestExampleMockModuleTest
  - Drupal\simpletest_example\Tests\SimpleTestExampleTest
  - Drupal\field_example\Tests\FieldExampleMenuTest
  - Drupal\field_example\Tests\TextWidgetTest
  - Drupal\field_example\Tests\Text3WidgetTest
  - Drupal\Tests\file_example\Kernel\StreamWrapperTest
  - Drupal\Tests\file_example\Unit\SessionWrapperTest
  - Drupal\file_example\Tests\FileExampleTest
  - Drupal\node_type_example\Tests\NodeTypeExampleTest
....
Torenware’s picture

Torenware commented

@isntall -- I'm not getting your results, not on dev, and not on production. The script you put in #4 gives me "All tests will run" in both branches. Maybe there's a stray setting somewhere I'm missing, but I'm definitely not getting just the tests for examples.module.

What exact configuration (i.e., via ./drupalci init) are you using? I especially need these for dev, since I can't make heads or tails what the first group of choices mean, much less which is appropriate:

Please select the numbers corresponding to which DrupalCI web environments to support. Separate multiple entries with commas. (Default: [8])
  [0] drupalci/base
  [1] drupalci/base-dev
  [2] drupalci/db-base
  [3] drupalci/db-base-dev
  [4] drupalci/php-base
  [5] drupalci/php-base-dev
  [6] drupalci/web-base
  [7] drupalci/web-base-dev
  [8] all

The other two groups are obvious enough; the defaults are the same as production, and I took the defaults.

I blew away my /home/vagrant/.drupalci/ directory, so I'm running defaults. Are you sure there isn't anything in your configuration that got in from someplace other than the script you posted? Also, I pulled dev a few minutes ago.

Something is still significantly different from my set up from yours. Let's see if we can figure out what that is.

isntall’s picture

isntall commented

I don't use drupalci init.

Here are my steps (the drupalci command is slightly different than before, i usually specify the php version, db type/version, etc.)

$ cd drupalci_testbot # where ever that is
$ vagrant up #build a fresh vagrant box from the dev branch
$ vagrant ssh #ssh into the vagrant box
$ docker pull drupalci/web-5.5 && docker pull drupalci/mysql-5.5
$ cd /opt/drupalci_testbot
$ DCI_RunScript=/var/www/html/core/scripts/run-tests.sh \
DCI_PHPVersion=5.5 \
DCI_DBVersion=mysql-5.5 \
DCI_RunScript=/var/www/html/core/scripts/run-tests.sh \
DCI_JobType=simpletest \
DCI_ComposerInstall=False \
DCI_CoreBranch=8.0.x \
DCI_Color=true \
DCI_Concurrency=4 \
DCI_Fetch='https://www.drupal.org/files/issues/file-example-2102651-torenware-v6.patch,modules/examples' \
DCI_Patch='file-example-2102651-torenware-v6.patch,modules/examples' \
DCI_TestItem='directory:modules/examples' \
DCI_AdditionalRepositories='git,git://git.drupal.org/project/examples.git,8.x-1.x,modules/examples,1;'  \
./drupalci run simpletest

and excerpt from this run
cd /var/www/html && sudo -u www-data php /var/www/html/core/scripts/run-tests.sh --url http://localhost/checkout --dburl mysql://drupaltestbot:drupaltestbotpw@drupaltestbot-db-mysql-5-5/simpletest_1446663422 --color --keep-results --concurrency 4 --color --sqlite /var/www/html/results/simpletest.sqlite --php /opt/phpenv/shims/php --directory modules/examples

What i usually do, when using a vagrant box is use open a second bash shell and use export

$ vagrant ssh #ssh into the vagrant box
$ docker pull drupalci/web-5.5 && docker pull drupalci/mysql-5.5
$ bash
$ cd /opt/drupalci_testbot
$ export DCI_RunScript=/var/www/html/core/scripts/run-tests.sh
$ export DCI_PHPVersion=5.5
$ export DCI_DBVersion=mysql-5.5
$ export DCI_RunScript=/var/www/html/core/scripts/run-tests.sh
$ export DCI_JobType=simpletest
$ export DCI_ComposerInstall=False
$ export DCI_CoreBranch=8.0.x
$ export DCI_Color=true
$ export DCI_Concurrency=4
$ export DCI_Fetch='https://www.drupal.org/files/issues/file-example-2102651-torenware-v6.patch,modules/examples'
$ export DCI_Patch='file-example-2102651-torenware-v6.patch,modules/examples'
$ export DCI_TestItem='directory:modules/examples'
$ export DCI_AdditionalRepositories='git,git://git.drupal.org/project/examples.git,8.x-1.x,modules/examples,1;' 
$ ./drupalci run simpletest

and an excerpt from this run
cd /var/www/html && sudo -u www-data php /var/www/html/core/scripts/run-tests.sh --url http://localhost/checkout --dburl mysql://drupaltestbot:drupaltestbotpw@drupaltestbot-db-mysql-5-5/simpletest_1446663760 --color --keep-results --color --concurrency 4 --sqlite /var/www/html/results/simpletest.sqlite --php /opt/phpenv/shims/php --directory modules/examples

work does need to be done on drupalci init, our main focus has been drupalci run simpletest and making that work.

Torenware’s picture

Torenware commented

@isntall: thanks. That procedure works. The only thing I haven't figured out is where the test assets end up.

At some point, somebody needs to update the docs, since that procedure doesn't work anymore. I'm not adverse to doing it, although I don't have enough experience with the system yet to generalize the instructions.

EDIT: turns out the environment maps its mounts into /var/lib/drupalci/. To find exactly where, see "Accessing Results" on Running DrupalCI Locally.

mile23’s picture

mile23
Primary language English
Location Seattle, WA
commented
Project: DrupalCI: Drupal.org Testing Infrastructure » DrupalCI: Test Runner
Status: Needs review » Active

OK, so in the past you'd use DCI_TestItem, which would parse module:whatever out to --module whatever, which seems much better to me.

Now, you have to use DCI_TestGroups which is expected to have --module whatever (complete with end spacing) since it's a direct string replacement, which seems inferior to me. Though it does allow you to specify multiple modules, I suppose.

Moving to the specific test runner project.

Mixologic’s picture

Mixologic
He/Him/His
Primary language English
Location Portland, OR
commented
Component: Code » Vagrant/Local Testing
mile23’s picture

mile23
Primary language English
Location Seattle, WA
commented
Title: drupalci command does not restrict to test group » Write a test to ensure that drupalci restricts tests by group, module name, etc.
Issue summary: View changes
Issue tags: -Vagrant
Parent issue: » #2680565: [meta] Create functional tests of run command output to test production essentials

Pretty sure this has been solved, so I'm re-scoping to write a test case.

mile23’s picture

mile23
Primary language English
Location Seattle, WA
commented
Title: Write a test to ensure that drupalci restricts tests by group, module name, etc. » Modify tests to ensure that drupalci restricts tests by group, module name, etc.
Mixologic’s picture

Mixologic
He/Him/His
Primary language English
Location Portland, OR
commented

Both DCI_TestItem and DCI_TestGroups , and now even DCI_RTTestGroups work in the following ways:

"all" => --all
"module:projectname" => "--module projectname"
"class:\fully\qualified\classname" => "--class \fully\qualified\classname"
"file:src/module/tests/filename" => "--file src/module/tests/filename"
"directory:modules/projectname" => "--directory modules/projectname"

Anything else gets passed directly through, so an actual groupname like Url works but so does "--directory modules/projectname"

Mixologic’s picture

Mixologic
He/Him/His
Primary language English
Location Portland, OR
commented
Status: Active » Fixed
Mixologic’s picture

Mixologic
He/Him/His
Primary language English
Location Portland, OR
commented

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.