While Unit tests are great for testing the specific classes and their functionality, we also need some "end-to-end" testing to ensure Features actually works as expected.

This issue will cover the creation of Kernel SimpleTests to handle the basic drush command functionality.

  • Add configuration(s) to a Feature
  • List the feature to ensure the added config is present
  • Export the feature
  • Check that the proper files were generated by the export
  • Make a config change and ensure the feature list shows the change
  • Import the feature to ensure the change is removed
  • Re-make the change and export the feature and check that the export has been updated
CommentFileSizeAuthor
#8 2616404-7.patch2.04 KBmpotter
#6 2616404-5.patch2.02 KBdawehner

Comments

mpotter created an issue. See original summary.

dawehner’s picture

dawehner
Primary language German
commented

Do you plan to actually test the drush commands? I have never seen anyone doing that, but this would be great if this is possible.

This issue will cover the creation of Kernel SimpleTests to handle the basic drush command functionality.

You can also use the phpunit based kernel tests, which work basically exactly the same as simpletest based kernel tests, but with some of the niceties of PHPUnit.
Just imagine that phpunit should have been named "PHP testing runner" instead.

mpotter’s picture

mpotter commented

Yeah, still learning about PHPUnit vs SimpleTest.

I'd love to test the actual drush commands, but not sure how the test framework handles (or doesn't) the printed output. Ideally you'd capture the output from drush print so that you could test the results, but I don't see anything that does that.

My guess is that I'll need to just replicate the functionality of the drush commands in a test rather than calling the drush commands directly.

mpotter’s picture

mpotter commented

@dawehner can you post some links to info on using "phpunit based kernel tests"? All I can find are docs that imply anything that needs the Drupal db needs to use SimpleTest. Since testing the drush commands involve testing config that actually gets stored into the active config in the db, I'm not sure I see how to use phpunit for this. I see some of the code you wrote and the use of "mock" classes, but I'm not sure that can be used for real functional testing.

mpotter’s picture

mpotter commented
StatusFileSize
new features-kerneltest_for_drush-2616404-5.patch1.57 KB

OK, I need some help here.

Here is a starting patch that creates a trivial kernel test starting point. When I add this patch I can see the test in the Tests UI, but when I run it, I get "No test results to display." So somehow it's still not running the testFeatureList() function.

Can somebody who has more experience with PHPUnit in Drupal 8 tell me what I'm doing wrong here?

dawehner’s picture

dawehner
Primary language German
commented
StatusFileSize
new 2616404-5.patch2.02 KB

I'd love to test the actual drush commands, but not sure how the test framework handles (or doesn't) the printed output. Ideally you'd capture the output from drush print so that you could test the results, but I don't see anything that does that.

I remember that drush has its own test suite, not sure though how usable it is.

\Drupal\KernelTests\KernelTestBase was developed as a 1to1 replacement of \Drupal\simpletest\KernelTestBase so most of the things
work exactly the same way, you can pretty much access your module's API, but you don't have yet any table/config installed.

We crossposted each other ...
To give you a rough idea how this could look like, here is an example

dawehner’s picture

dawehner
Primary language German
commented

To run the tests I always use the following bit on my shell:

  export SIMPLETEST_DB='mysql://root@localhost/dev_d8'
  export SIMPLETEST_BASE_URL='http://d8.dev'
  ./vendor/bin/phpunit -c core modules/features/tests # This runs for example all tests inside that particular folder
mpotter’s picture

mpotter commented
StatusFileSize
new 2616404-7.patch2.04 KB

OK, thanks for the example. Looks like I was on the right track. However, it doesn't show your test in the Testing UI. I had to modify the Namespace and add the @group option (just the namespace might have been sufficient).

Still not really sure why my test didn't work though.

mpotter’s picture

mpotter commented

OK, this is interesting. I found the line that you had in your example that was missing in mine that enabled my test to start working:

protected $strictConfigSchema = FALSE;

That seems rather obscure and not something I have seen in documentation.

dawehner’s picture

dawehner
Primary language German
commented

OK, thanks for the example. Looks like I was on the right track. However, it doesn't show your test in the Testing UI. I had to modify the Namespace and add the @group option (just the namespace might have been sufficient).

Oh yeah, I just can't be bothered to add those, if running them via phpunit gives me the same result :)

Here is a starting patch that creates a trivial kernel test starting point. When I add this patch I can see the test in the Tests UI, but when I run it, I get "No test results to display." So somehow it's still not running the testFeatureList() function.

One key fundamental thought difference between simpletest and phpunit is that phpunit thinks that a broken test is the exception aka.
it doesn't give you any output, unless it fails,
so no results to display means there is no fail.

OK, this is interesting. I found the line that you had in your example that was missing in mine that enabled my test to start working:

protected $strictConfigSchema = FALSE;

That seems rather obscure and not something I have seen in documentation.

The problem here is #2392561: Add configuration schema, so by default Drupal checks the schema in tests more strictly than in normal life.

mpotter’s picture

mpotter commented

Interesting that without the $strictConfigSchema I get the "No test results to display". When I added that (and when using your test), I *do* get a proper:

1 pass, 0 fails, 0 exceptions
Drupal\Tests\features\Kernel\FeaturesDrushTest

kind of result in the UI. So there is a difference between "no results" and "passing".

Anyway, now I think I can make good progress on this tomorrow. Too bad I haven't figured out how to run my xdebug for PHPStorm within the PHPUnit stuff within my local Docker container. I can debug the normal Drupal code on the site just fine, but the PHPUnit doesn't seem to connect to the debugger for some reason.

dawehner’s picture

dawehner
Primary language German
commented

kind of result in the UI. So there is a difference between "no results" and "passing".

Oh well, the integration between the simpletest UI and phpunit is not ideal, which is why I always prefer running via phpunit directly ...

Anyway, now I think I can make good progress on this tomorrow. Too bad I haven't figured out how to run my xdebug for PHPStorm within the PHPUnit stuff within my local Docker container. I can debug the normal Drupal code on the site just fine, but the PHPUnit doesn't seem to connect to the debugger for some reason.

The only thing I could imagine is that you have a limited amount of connection allowed at the same time? Usually I think you need like 4 in order to run the main drupal, then the batch process in there and then the test or something like that, see phpstorm => settings => languages & frameworks => PHP => Debug "Max. simult. connections"

mpotter’s picture

mpotter commented

So when I try following the instructions for running PHPUnit tests for Drupal. "cd core" and then run "../vendor/bin/phpunit ../modules/contrib/features/tests/src/Kernel/" gives me:

PHPUnit 4.8.11 by Sebastian Bergmann and contributors.

SS

Time: 9.33 seconds, Memory: 5.50Mb

OK, but incomplete, skipped, or risky tests!
Tests: 2, Assertions: 0, Skipped: 2.

So it sees both of our tests, but skips them because they are "risky"?? So wondering what command you are using to run these from the command line?

dawehner’s picture

dawehner
Primary language German
commented

@mpotter
With --verbose you should see some more information.
I could imagine that you forgot the two environment variables:

 export SIMPLETEST_DB='mysql://root@localhost/dev_d8'
  export SIMPLETEST_BASE_URL='http://d8.dev'
mpotter’s picture

mpotter commented

Yep, that was it. But also found the core/scripts/run-tests script that handles this.

dawehner’s picture

dawehner
Primary language German
commented

Just a tip, if you copy the core/phpunit.xml.dist to phpunit.xml you can add additional values and also add environment variables, see https://phpunit.de/manual/current/en/appendixes.configuration.html#appen...

  • mpotter committed eb5c526 on 8.x-3.x 
    Issue #2616404 by mpotter, dawehner: Create Kernel test for Generation...
mpotter’s picture

mpotter commented

Added a new Kernel test for the Generation plugins to commit eb5c526 based on the templates posted in patches above.

Leaving this issue open for adding additional integration tests.

nedjo’s picture

nedjo
he/him/his
Primary language English
commented
Related issues: +#2618390: Failing test for archive write

Related bug: #2618390: Failing test for archive write.

nedjo’s picture

nedjo
he/him/his
Primary language English
commented

I'm working on Kernel tests for the assignment methods.

nedjo’s picture

nedjo
he/him/his
Primary language English
commented
Status: Active » Needs review

Created child issue: #2619650: Kernel test coverage for assignment plugins.

nedjo’s picture

nedjo
he/him/his
Primary language English
commented
Status: Needs review » Active

The last submitted patch, 6: 2616404-5.patch, failed testing.

Status: Active » Needs work

The last submitted patch, 8: 2616404-7.patch, failed testing.

nedjo’s picture

nedjo
he/him/his
Primary language English
commented
Status: Needs work » Active