Modernizing Testbot: Testing Wish-List

This issue is intended as a collection point for various operational features, functionality, or capabilities that folks would like to see on the new testing infrastructure. While it is a bit of a 'dumping ground' at the moment, I do intend to go in and clean it up, eventually.

Please feel free to add your own requirements to the list or in the comments below ... but if an item requires any discussion or debate, lets handle that in a separate issue and add the reference link here.

Testbot Wishlist

New patch submissions

When a new patch is submitted to an issue, what would we like to happen?

Code Tests

- PHPUnit tests
- Performance Testing
- Static Analysis tests - analyze cyclomatic complexity etc.
- Code Coverage analysis - show whats tested

Front end/System tests

- Core Simpletests
- Contrib Simpletests
- qunit? (javascript unit tests) - qunit test runner page
- testswarm? (javascript unit tests on ALL browsers)
- Visual regression testing
- Mink/Selenium/Casper/Wraith/Browserstack like tools
- Behat Testing: #2597802: Add the ability to run Behat on DrupalCI

Change Management and Review tools

- Patch application to HEAD
- Detect conflicting patches
- Automatically generate Interdiffs
- Coding Style Review - Coder Module/Codesniffer (support - #1791872: [Policy, no patch] Add special tag to identify issues ready for a high level only review and #1299710: [meta] Automate the coding-standards part of patch review)

Core Commit reactions/background jobs

When a core commit happens, what could happen in the background that can be automated?

- Automatic patch re-rolling
- Patch validity tests (patch still applies?)
- See if 'needs review' still applies, and tag/nw
- Need reroll on RTBC patches
- Retesting of RTBC patches nightly
- Automatically update generated documentation

Automated Project Analysis

How can we leverage the automated q.a. infrastructure to facilitate new module developers?

Excellent background info - http://jthorson.doesdrupal.com/node/28
- PAReview
- Git Repo Health Check
- Git repository health check jobs

Testbot behavioral control

How can we put power in the hands of the developers to test what they need tested?

Selectable Environment (Assign test to specific bot)

- Developers able to create new environments
- Environment specific tests possible.

Configurable control of behavior driven by developers

- Ability to influence the behavior of the testbots from issue queue (.travis.yml/settings.testing.php/etc)
- Selectable environment that the tests run in.
- Configure which tests get run (simpletest/phpunit/performance/qunit etc)
- Configurable Fast-failure option
- Per-file properties: Test/Don't test
- Execute only selected test subset/blacklist tests
- Prioritize specific tests, then full run
- Cancel tests in progress
- Test Dependencies/chaining (test patch after patch x)

Systemic configuration of priorities

- Ability to prioritize Core vs. Contrib
- Ability to prioritize Environments (default first, and others if idle).
- Ability to prioritize by test category
- Ability to prioritize by issue priority
- Run set of tests in parallel
- Ability to 'blacklist' tests system wide
- Batch 'cancel' and batch 'retest'
- Try x times, and then give up and mark the test as bad (?)

Reporting, data, and output

What kinds of data and reports would we like to have available?

Test Metrics

- Timestamp individual tests within a run (test timing metrics)
- Test durations (per test)
- Commit id that patch applied to

Communications

- Notifications for test originator upon test completion (email)
- Estimated Time to begin testing
- Estimated Time to complete testing
- Compressed Summary test results inline in issues
- Enhanced details and reports on qa.drupal.org

Reports

- Real-time view of test in progress (console)
- Historical test results/details
- Code Coverage Reports
- Performance Reports /Current/Historical
- Views: Last x failed, last x environment fails, etc.

Data Formats

- Standardized Response Formats
- Compressed detailed results
- Separate status for 'environment' failure versus 'test' failure
- log test failures, time, patch

Meta Infrastructure

Resource management/monitoring

- Worker heartbeat with full status communications
- dynamic spinup/down of resources as needed
- Separate environment setup from testing phase
- Pre-test Environment checks (sufficient disk, etc)
- View queue
- Reorder queue
- Cancel queue

- Support Multiple Environments

Environment Examples

• Databases
  • Mysql
    • MyISAM testing
    • InnoDB testing
  • Postgres
  • Sqlite
• Caching Backends
• Database
• APC
• PHP versions
  • PHP 5.3
  • PHP 5.4
  • PHP 5.5
  • PHP 5.6
• PHP extensions