On #2148255: [meta] Make better D8 api.d.o landing page, linked to high-level overview topics, and put it in Core api.php files, we made a patch that included a stub Topic page for api.drupal.org (i.e., a @defgroup) titled:
Automated Tests
This can be found in file core/modules/system/core.api.php where it says
@defgroup testing
The documentation to go on this page needs to be written. The idea is:
a) Write a few paragraphs about the topic.
b) Link to more detailed documentation on
https://drupal.org/developing/api/8
(or nearby)
c) If the more detailed documentation does not yet exist, create stub page(s), link to the stub pages, and add a note to this issue stating that the stub pages need to be filled out.
d) If the topic has related classes, interfaces, and functions -- appropriate for an overview -- add
@ingroup testing
to their documentation headers. That will make these classes etc. show up on the Topic page on api.drupal.org. Only include classes/functions that are appropriate for an overview page please!
For more info -- documentation standards for @defgroup/@ingroup:
https://drupal.org/coding-standards/docs#defgroup
Comment | File | Size | Author |
---|---|---|---|
#12 | interdiff.txt | 3.03 KB | jhodgdon |
#12 | 2216557-testing-docs-12.patch | 7.29 KB | jhodgdon |
#10 | 2216557-testing-docs-10.patch | 6.82 KB | jhodgdon |
Comments
Comment #1
jhodgdonTaking this on...
Comment #2
jhodgdonHere's a first pass.
Comment #3
BerdirYay :) Here's some feedback...
See below, you should also tag DrupalUnitTestBase.
I don't think all tests are regression tests?
While that is the theory, it's not always that easy to write a test first, as you might not know how exactly it should behave until you fixed it :) Especially for non-unit tests.
I'd leave out the test-first philosophy and instead just say that bugfixes need to come with a test to proof that the bug is fixed and stays fixed.
A lot here is about drupal.org and processes we have here. That's quite different to most other API documentations that we have. Not sure how much of that should really be in here.
run-tests.sh needs the testing module to be installed too (at least unless you use it in the non-parent-installation mode).
Also, we should probably also mention that you can run (phpunit) unit tess with phpunit, which is way easier to use and faster than run-tests.sh and I always recommend to use that directly when dealing with unit tests.
We do not test functions with phpunit, as we do not load them, anything that relies on functions needs ugly workarounds right now...
According to PSR-4, they need to be in tests/src I think.
I would leave out the arguments part because phpunit does support arguments in combination with data providers. See \Drupal\Tests\Core\Access\AccessManagerTest for example. But I'd avoid explaining that here, just leave that part out.
I *think* we're discussing to switch to 4.0 somewhere, so we should make sure this link is updated when we switch. Probably find and cross-post to the relevant issue.
You should only use WebTestBase if you need to do web requests, otherwise use DrupalUnitTestBase, which is *much* faster and recommended to use whenever possible. WebTestBase does a full installation with the requested modules while DrupalUnitTestBase only created a in-memory pseudo-installation where you need manually create the tables you need and include all modules you need, including dependencies and required modules.
I'd try to generalize this a bit and explain that setUp() can be used to set up the necessary environment that is common to all test methods in a given test class, for example content types (maybe use node types as we're in API documentation? content types is so confusing with nodes, content, entities, and * types)
This might also be a good place to explain that each test method runs in a new, isolated test environment and can not rely on anything that it does not set up itself.
Additional in this case is anything that is not a required module, so defining the necessary modules is almost required (you need to enable yourself too). Maybe rewrite a bit to "You must define all required modules to run the test..." or so?
Comment #4
sunA very nice summary on the 3 main concepts for tests by @mongolito404 in #2232271-36: [Meta] Use Behat for validation testing:
We can simply replace/remove the JavaScript parts, state PHPUnit as tool for unit testing, DUTB for integration testing, and WebTest for validation/acceptance testing, and the resulting summary will give a quick + concise introduction to the main concepts and tools we're using. :-)
Comment #5
jhodgdonThanks for the detailed review! I think I've addressed all the comments... Some notes:
item 2 - I do think that the main purpose of our testing suite is regression testing, but I agree this is not the only purpose... reworded slightly.
Item 4 - OK. Moved this section after the explanation of Writing, because it depends on knowing what PHPUnit is now.
Item 9 - Decided just linking to phpunit.de was better.
Comment #6
jhodgdonNote: The patch in #5 was submitted cross-post with #4, so it doesn't include the suggested summary. I don't actually think we want to go into so much detail...
Comment #7
jhodgdonRegarding the different types of testing...
Unit testing is a *type* of testing (a set of things to test), distinct from Functional testing.
Integration testing is a *phase* of testing, distinct from Regression testing.
Verification and Validation are two distinct testing *processes*, as defined in that wikipedia link.
The summary in #5 mixes up types vs. phases vs. processes... Some of our tests are originally written as verification tests, some are validation tests. On another axis, some of our tests are functional and some are unit tests. And although we often write patches with tests included (integration phase), once we've committed all of these tests to Drupal Core, they become a regression test suite, which helps ensure we don't break stuff we've already done.
So... I'm not sure all of that belongs in the patch, but I don't actually agree that the summary in #5 clarifies the differences between different testing types, phases, or processes.
Comment #8
BerdirDrupalUnitTestBase, not UnitTestBase (also the @ingroup), the second should really be deprecated in favor of PHPUnit/UnitTestCase. (There is an issue to convert all test that directly extend from it to phpunit, but it's... complicated (for example due to the not-testing-functions "rule", so things need to be converted to methods/classes first).
See https://drupal.org/node/1829160 for a pretty good change record about DrupalUnitTestBase and the difference to other tests.
Maybe we should have something like developing/api/testing that wraps both phpunit and simpletest documentations which would give us a place to explain the different base classes in more depth and link to detailed instructions for each, for example DrupalUnitTestBase could be based on that change record...
Comment #9
jhodgdon#8 - Good catch, thanks! New patch attached.
Regarding the idea of having something in developing/api that covers Testing as a whole, we already have separate sections for PHPUnit and Simpletest at
https://drupal.org/simpletest
https://drupal.org/phpunit
I would think both of those should have sections explaining the classes and getting into more depth. Probably developing/api (the 8.x page) should link to both of those pages? They're outside of developing/api I think...
Comment #10
jhodgdonReroll for PSR-4 and clean apply.
Can someone please review this so we can get it in? It has to be better than what is there now (which is nothing)... Please?
Comment #11
chx CreditAttribution: chx commented> "mocking" class
Instead: "mock" object.
> You can define additional modules to be enabled by overriding the $modules member variable
I would say by specifying or something like that -- while technically you are overriding the $modules the test framework uses reflection and gathers all the $modules so it's not like a test class extending say a base class will override which modules exist.
> DrupalUnitTestBase
Has been renamed to KernelTestBase
> The class name needs to end in the word Test.
Nope. It needs to be in the right directory/namespace and have a getInfo method. The "ends in the word Test" is just PHPunit making lives harder. Simpletest doesn't create minefields like that.
> * PHPUnit tests can also be run from the PHPUnit framework.
This needs a link somewhere because the only way to do this AFAIK is to change into the core directory and run phpunit from there (or perhaps there's some magic argument to show phpunit where the xml is)
Comment #12
jhodgdonThanks very much for the review! I believe I have addressed all of your comments, which all seemed to be good ideas to me.
Comment #13
chx CreditAttribution: chx commentedLooks good to me.
Comment #14
jhodgdonUnassigning to avoid confusion between "I patched this" and "I'll commit this".
Comment #15
webchickAwesome.
Committed and pushed to 8.x. Thanks!