Since the general cleanup of SimpleTest is just about finished I figured we could discuss the new task of unit testing. We still need the functional tests completed and go through and clean-up old tests, but the API changes and coding standards are just about done.

I will layout what my current understand of the plan from conversations on IRC so that we can formal document the discussion and ensure that everyone is on the same page.

Class Scope
There will be one class (test case) per file and one file per core file. This will provide a simple convention that makes sense for unit testing.

Example: ->
block.module    -> block.module.test

The classes will then be named in a similar fashion. -> BlockAdminIncTestCase
block.module    -> BlockModuleTestCase

With the large number of unit tests a better way of grouping tests needs to be created.

Function Names
Each test method should be related to the function it tests in a 1-to-1 relationship.


db_query  -> test_db_query
_db_query -> test__db_query    <- double underscore

To perform unit testing each function needs to be isolated from all other functions. To do this all "sub calls" to other drupal functions need be located and each drupal function needs to be overridden to return a static value. The current plan is to use runkit.

There are several things that need to be figured in out for this to work.

  • At what scope should functions be overridden? - Globally, per file, per function, or use a default and deviate where necessary?
  • If functions are not overridden globally should they be reset after the current scope exits?
  • How should functions be overridden? Using runkit_function_[add|redefine]? Or define the function in the file and use runkit_function_rename?
  • If functions are overridden per test function how can we use runkit_function_rename since they will conflict when defined?

I have created a module that automates the majority of the above mentioned that is possible to automate. I will update it once a convention has been decided on. I will then generate all the test cases and commit them. This will provide a consistent base for tests to be filled in.



boombatower’s picture

chx had a few ideas.

Common test functions
Create several test functions that can be copied in place of other functions.



Then use runkit_function_copy to copy them to override the necessary functions.

Restore state
After a test is completed all overridden functions should be restored.


also, it's called "unit testing" -- we can be more "elastic" -- not _necessarily_ one test per function, if there is a function which has a helper solely used by this function that's also a unit

congthai’s picture

When I test module simple test on ubuntu, The module is not active. Who know why?

boombatower’s picture

@congthai: Please create a new issue for your problem with more details. This issue is for discussion of unit testing, not issues with the SimpleTest module in general. Thanks

boombatower’s picture

The DrupalTestCase will need to be branched and the current one renamed.

Webchick on IRC proposed: DrupalWebTestCase and DrupalUnitTestCase.

dmitrig01’s picture

Here's a better idea for the simpletest return thing -

function simpletest_change_return($function, $value) {
  runkit_function_redefine($function, '', 'return '. var_export($value, 1) .';');

Very simple, eh?
Just call simpletest_change_return('drupal_get_form', array('#type' => 'form')); and drupal_get_form(); will return array('#type'=>'form').

boombatower’s picture

As a note this will still need to restore to the previous state as described above.

dmitrig01’s picture

Not exactly.
If we figure out how to run one test per page load, then it would work (the changes are per-page-load).

boombatower’s picture

That would make it all run slower, be harder to compile the results, and has no benefit.

webchick’s picture

We had a long talk (we being boombatower, Crell, catch, myself, and probably other people I forget) about how adding a runkit requirement is going to make it so that only 0.000001% of people can run unit tests, which would basically defeat the purpose. And a better approach is refactoring of these monster functions that call functions that call functions as Crell did for the registry patch. See before/after here for a guide:

pwolanin’s picture

I agree with that sentiment - runkit may have a place for advanced tests for critical and complicated functions, but it shouldn't be required for the majority of tests.

Rok Žlender’s picture

+1 for refactoring our code to make it possible to do unit testing rather than using runkit to go around the problem

Wim Leers’s picture


dlangevin’s picture

One thing that hasn't been mentioned much here is "private" vs. public functions. Typically in test-driven development, only public methods are tested since the internal working of a class (private methods) do not need to be tested on their own (given that they are not called ouside of the class itself), and are subject to refactoring. NOT creating tests for private methods (indicated by a leading _) would cut down on the number of tests that need to be created as well as make it simpler to refactor the public methods without rewriting tons of tests.

In addition, I've seen that some people have suggested using runkit or some other methodology to create mocks for what would be considered the "private" functions. I think that this would be a mistake because it would require anyone modifying a private method's parameters or returns to find all of the mocks and modify them as well. This is prone to errors and it would be easier if the errors were pointed out by test failures when a private method is changed.

What does everyone think of this? Since we are using procedural code, there is no enforcement of private vs. public methods, so that could be an issue

boombatower’s picture

I think that only testing public functions is fine, but that won't eliminate the sub-function calls. In other words we will still need to either refactor or use something like runkit. Public functions still call other public functions so we will still need to deal with that.

I think refactoring the code will work in a large number of places, but not for everywhere so I think we still need a contingency plan that uses something like runkit.

dlangevin’s picture

I agree that public functions will still call other public functions, but by using mocks like runkit, you can inadvertently hide the dependencies. For example, if foo() calls bar() and I refactor bar() to return a different value or accept different parameters, the tests for foo() will pass, but in actuality, foo() will be broken.

I do agree that there will be some need for mocks. A good example is db_query(), since its return value and parameters are very unlikely to change and it requires that the database be set up, have the correct data, etc. I am just saying that using it to set up an environment for tests can be dangerous in my opinion.

Also, I am very interested in helping with the unit testing project in general. Boombatower, you seem to be one of the main contributors, especially recently, so if you know of some specific things that you know need to be done, I'd be happy to help.

boombatower’s picture

@dlangevin: That is great that you are interested in helping. We definitely need the help. Hopefully we can come to a consensus that we can use as a base for generating test stubs as mentioned in the original proposal and get them committed.

If you see any unanswered questions that you could comment on that would be great.

We essentially need a proposal on this that we can all agree on so if you want to review all the comments and modify my originally proposal and post it in this thread that would be great.

dlangevin’s picture

OK, here is a modified proposal (I've read through a lot, but please let me know if I missed anything that has been discussed...I'm new here)

SimpleTest Unit Testing Plan Proposal
(Revised from boombatower's original proposal)

File Structure

One class per file, once class per core file. ->

File Naming

Each file is named after the file it tests

Example: ->
block.module -> block.module.test

TestCase Class Naming

The classes will then be named in a similar fashion. -> BlockAdminIncTestCase
block.module -> BlockModuleTestCase


Tests should be grouped by module for unit tests. All tests relating to a single module must pass before it is ready to be committed.

Example: ->
block.module -> block.module.test

These would be in a group called Block. Each test class could be run individually or the group could be run as a whole.

Test Structure

Each public function should have a single test assigned to it. Private functions (denoted by a leading _) do NOT have tests assigned to them. We should make the note to module developers that they are strongly discouraged from using private core functions in their modules because they are not tested and thus subject to refactoring and change.

Test Naming

Test naming would be camelCased to comply with most OOP conventions and SimpleTest's own naming convetion.

db_query -> testDbQuery()

Mock Functions

This one is a little hairy because of the runkit problem mentioned above. I don't see a way around using runkit to generate the mocks that are required to get tests working consistently without a consistent database/file system. However, I propose that no mocks be generated for the module/file that is currently being tested and that it be limited to the current test case (method).

If there is a case where a mock or several mocks might be useful across several methods, a private method can be created and called in each test case that it is needed.

I don't know of a way to redefine functions besides using runkit, so we might have to require that people have it installed to run certain tests. This is especially the case for tests that deal with the database and file system.

Backwards Compatibility

An implication of standardized unit testing is that it forces much more emphasis on a stable public API. If we start changing the parameters and return of public functions, tests using those functions in other modules will start breaking. I have seen other frameworks put fixes (sometimes temporary) when the public API does have to change. This is not really an issue for this proposal, but it is something to think about because having these tests will make it twice as much work to change the public API.


Boombatower has created a module that automates the majority of the above mentioned that is possible to automate. He will update it once a convention has been decided on. He will then generate all the test cases and commit them. This will provide a consistent base for tests to be filled in.

That's it for now. Any thoughts on these issues or others?

floretan’s picture

I'm definitely in favor of refactoring code where possible, and limit mock objects to data sources external to php (filesystem, db, remote data, etc). Besides the fact that most testers won't have the runkit extension installed, it's also a great occasion to improve the general readability of Drupal's code.

chx’s picture

I found a way to redefine functions without runkit. This will require big but simple changes to Drupal core. Let's call the current module and include files "bundles" just to make it simpler to describe what I want to do here. In CVS, Drupal will have a lot of includes -- one function per include. We supply a simple script that builds the aformentioned bundles. So the shipping tarball will have the same modules and includes -- . We run this script before packing Drupal. Also, anyone checking out CVS can run this sript. Once we have the registry in it's easy to patch it to support loading of a bundle of includes instead of just one file. So when you want to unit test function X then you change the filepath of functions called by this function in the registry table to the testing version of these functions and then boot Drupal -- so we will need to run unit tests through the browser as well.

pwolanin’s picture

@chx - this sounds like a very interesting idea, but I think the practical aspects need a little more fleshing out.

For example, every core install should be able to run the unit tests, right? So we'd want to ship in the tarball all the individual function files as well as the "compiled" files?

In this case, I guess development actually happens with diffs against these 1000's of individual function files? Is that practical?

How about the opposite path - a script breaks down the current module and include files into individual function files that get saved within the testing directory somewhere. This allows us to autogenerate predictable file names, and to not worry if a function is moved between files. Then the registry is tweaked to point to this collection of individual files during the tests, and as you propose above, you can substitute individual functions as needed.

chx’s picture

Opposite path -- very good idea, thanks. We might want to ask the registry to do the split, actually -- it has parsing already built in anyways, the necessary code to write them out is a few lines of code. So much nicer than doing it in CVS. Thanks , thanks!

About "every core install should be able to run the unit tests" -- no, not at all. This is not a goal. As many as possible, that is a goal. For eg. if they don't have curl or simplexml extensions, they won't be able to run browser based tests (though unit tests might be able to run from a command line Drupal but then you need php-cli).

pwolanin’s picture

@chx - Yes, you're correct- I should rather have said that my understanding is that Drupal core should ship with all the code and files necessary to do all the tests, though not all may be runable depending on the PHP version and extensions.

dlhubler’s picture

Mocking functions
Uses runkit of course.

Concept of mocking is incredibly handy in and of itself, but bonus feature, it manages the restoring of functions.

I investigated package some, all of it's own unit tests passed for me except something about wildcard. There is not a lot
to the library, and it's not a real release. I'd be happy to contact original owner and inquire about it's status if people
think this would be useful.

beeradb’s picture


jpetso’s picture

Hm... would it be an option to run the function under a different name? Like,

$replacements = array(
  'replaced_function' => $replacement_function,
  ...more of those...
$result = BlockAdminIncTestCase->invoke($function, $args, $replacements);

invoke() then gets the string of the function in question (by parsing the associated source file), tokenizes the function with token_get_all(), replaces the function names in $replacements and also replaces the own function name with something else (say, "simpletest_generated_$function").

Then the parsed tokens are reassembled, written to a temporary include file, that file is included, the renamed/rewritten function is called, and the include file is deleted again.

Did I miss something that would prevent this from working?

andreiashu’s picture


pwolanin’s picture

@jpetso - I think you are missing the point. The idea is to be able to substitute the functions call by whatever function is being tested.

jpetso’s picture

Ok, I'm not surprised that this is the case. Just thought that if the use case is narrow enough, the above approach might work too, but if it doesn't, that's cool with me. Thanks for the clarification.

boombatower’s picture

Status: Active » Closed (won't fix)

Unit Testing in this form will no longer occur.

Gary Feldman’s picture

What exactly is the state of unit testing now? The tutorial page mentions DrupalUnitTestCase, and there appears to be a file implementing it in CVS, but it's not in the current Drupal 6 tar files. Is it simply something that hasn't been put into the D6 branch? I haven't studied it in detail, but it doesn't seem like the sort of thing that would have significant D6/D7 dependencies, and the tutorial page does seem to focus on D6.

Or is the best approach to use SimpleTest directly, without the SimpleTest module?