separate functional from unit tests

Here's a first pass at it. The unit tests fail here after the patch, but I believe they failed before too, mostly because i don't have a mysql user sitting around to create drupals, but also some weird local implementation bugs. I also believe it's testing my system install of drush 4.4 instead of the local 5.x one. :)

Are you saying I need a unit test showing you process isolation for unit tests works? :P I'll at least try out the process isolation...

I like your thinking there greg.

I think there is space for both approaches: test Drush commands and test drush functions. Command-level testing is useful because it allows for broader tests to be developed much more quickly than regular function-based unit tests, which need significant coverage to be effective for proper testing.

Can we just merge in the patch, which supports both approaches? Moshe?

I'd like to push this in now - it's getting hard to maintain those patches separately as unit tests continue to be developped.

I understand. I will not push this without your go. :)

One thing I hope to use this for is to provide an easy way to provide function-level unit tests within provision and other drush extensions.

I don't disagree with the current approach at all, and i also feel it's a great way to easily have people jump in and write tests. But I don't think we should keep people from writing function-level tests, which are also very useful.

Chasing head.

I've reworked this patch on the dev-real-unit branch. It's now working and we have a performance improvement for the test I ported:

anarcat@angela:tests$ phpunit --repeat 10 commandUnitTest.php
PHPUnit 3.4.14 by Sebastian Bergmann.

..........

Time: 2 seconds, Memory: 11.25Mb

OK (10 tests, 90 assertions)
anarcat@angela:tests$ phpunit --repeat 10 commandTest.php
PHPUnit 3.4.14 by Sebastian Bergmann.


Executing: /usr/bin/drush php-eval '$commands = drush_get_commands();print json_encode($commands['\''dl'\''])' --nocolor
.
Executing: /usr/bin/drush php-eval '$commands = drush_get_commands();print json_encode($commands['\''dl'\''])' --nocolor
.
Executing: /usr/bin/drush php-eval '$commands = drush_get_commands();print json_encode($commands['\''dl'\''])' --nocolor
.
Executing: /usr/bin/drush php-eval '$commands = drush_get_commands();print json_encode($commands['\''dl'\''])' --nocolor
.
Executing: /usr/bin/drush php-eval '$commands = drush_get_commands();print json_encode($commands['\''dl'\''])' --nocolor
.
Executing: /usr/bin/drush php-eval '$commands = drush_get_commands();print json_encode($commands['\''dl'\''])' --nocolor
.
Executing: /usr/bin/drush php-eval '$commands = drush_get_commands();print json_encode($commands['\''dl'\''])' --nocolor
.
Executing: /usr/bin/drush php-eval '$commands = drush_get_commands();print json_encode($commands['\''dl'\''])' --nocolor
.
Executing: /usr/bin/drush php-eval '$commands = drush_get_commands();print json_encode($commands['\''dl'\''])' --nocolor
.
Executing: /usr/bin/drush php-eval '$commands = drush_get_commands();print json_encode($commands['\''dl'\''])' --nocolor
.

Time: 6 seconds, Memory: 5.00Mb

OK (10 tests, 100 assertions)

3x performance improvement.

I am in the process of refactoring the bootstrap code to avoid duplication between the bootstrap process in the unit tests and the regular drush process.

er. i can't seem to find any of the code from the dev-real-unit in drush 5.x, nor was I able to find a relevant commit at the said time. it seems that this was simply completely dropped, was it forgotten or refused?

i am trying to implement a test to show that drush_save_data_to_temp_file() fails to create two tmp files during a script (#1603744: can't create two temporary files), and this gets a bit silly as I need to do a lot of the test logic in an eval. With the patches that I sent here, this would have been trivial... while making tests faster.

Crap, sorry I missed this - don't know how. Maybe it's because I was looking only at the 4.x branch. :(