After having a crazy idea of utilizing PSR-0 detection (following reading simpletest.module). I went ahead and created a test patch to try and allow themes to natively provide tests. However, there seems to be just one drawback:
It appears run-test.sh requires full paths for the --file argument.

https://qa.drupal.org/pifr/test/673478

[16:33:24] Scanning sites/default/files/checkout/sites/default/modules/bootstrap for .test files.
[16:33:24] Scanning sites/default/files/checkout/sites/default/modules/bootstrap for /Tests/*.php files.
[16:33:24] Cleaning run-tests.sh environment.
[16:33:24] Command [/usr/bin/php ./scripts/run-tests.sh --concurrency 8 --php /usr/bin/php --url 'http://drupaltestbot983-mysql/checkout' --file sites/default/modules/bootstrap/lib/Drupal/bootstrap/Tests/BootstrapUnitTestCase.php --clean 2>&1] succeeded.
[16:33:24] Initiating run-tests.sh.
[16:33:25] Command [/usr/bin/php ./scripts/run-tests.sh --concurrency 8 --php /usr/bin/php --url 'http://drupaltestbot983-mysql/checkout' --file sites/default/modules/bootstrap/lib/Drupal/bootstrap/Tests/BootstrapUnitTestCase.php 2>&1] succeeded
  Duration: 1 seconds
  Directory: [/var/lib/drupaltestbot/sites/default/files/checkout]
  Completion status: [0]
  Output: [ERROR: No valid tests were specified.].
[16:33:25] Encountered error on [review], details:
array (
  '@reason' => 'run-tests.sh reported no tests were found.  See review log for details.',
)
[16:33:25] Review complete. test_id=673478 result code=8 details=Array
(
    [@reason] => run-tests.sh reported no tests were found.  See review log for details.
)

Notice the difference between where the file was detected:
sites/default/files/checkout/sites/default/modules/bootstrap
and what is sent to run-tests.sh as a --file argument:
sites/default/modules/bootstrap/lib/Drupal/bootstrap/Tests/BootstrapUnitTestCase.php

After talking with jthorson on IRC, he suspects that the code responsible for this is located at:
http://drupalcode.org/project/project_issue_file_review.git/blob/refs/he...

Comments

markhalliwell’s picture

Issue summary: View changes
markhalliwell’s picture

friendly ping :)

jthorson’s picture

Hey Mark,

Can you comment on whether #2273481: Contrib PSR-4 PHPUnit tests are not picked up by PIFR may have helped address this? (I'm guessing not, but just wanted to confirm before diving into this later this week.)

markhalliwell’s picture

Assigned: Unassigned » markhalliwell
Status: Active » Postponed

Sure. I'll double check sometime next week while at DC.

markhalliwell’s picture

Assigned: markhalliwell » Unassigned
Status: Postponed » Active

Sorry, this got buried.

I just re-tested #2138693: Provide UnitTests for theme, but it again failed same problem as the issue summary.

markhalliwell’s picture

Status: Active » Closed (duplicate)
Related issues: +#2769599: [PP-1] Allow themes to be testable