We plan on enabling all sorts of new functionality with the Modernizing Testbot initiative, which will require the design and development of a new user interface on drupal.org.

I anticipate a re-design of the automated testing tab on project pages (which was put there in anticipation of this initiative), as well as needing some way to display test results within the issue queues.

For the issue queues, I believe that placing test results in the comment stream is not scalable ... as we start to have multiple test types (and non-test functionality), then this will start to create too much noise in the comment thread. We'll need to rethink where/how tests are triggered and results are displayed. Perhaps a block, or a local tab per issue ... ideas?

Comments

mile23’s picture

The UX for specifying environment requirements should be a .yml file at the top level of the project.

Let's call it .drupalci.yml.

Make it as code-compatible as possible with .travis.yml. If no file is given, test using sane defaults.

Specify PHP versions and extensions, database types, etc.

php:
  - 5.4
  - 5.5
database:
  - mysql
  - sqlite

Note that we just specified 4 test runs.

The bot will run through all the permutations and then report pass/fail.

We could also say things like this:

fixture:
  - defaultFixture
  - fixtures/
  - none

defaultFixture allows us to specify a default testbot fixture, which is whatever the testbot is being told is the best way to preload a database for tests.

fixtures/ would be a directory in the project repo, which might have a naming convention such as fixture.mysql for mysql, and so forth.

none allows us to specify that the full Drupal installation happen per test.

If we combine this with the yaml above, we end up with 12 test runs. This makes it a bit of a necessity to quit on any fail. So: If you know a specific environment is failing, you alter .drupalci.yml in the patch.

Furthermore:

strict:
  - phpcs
  - phpunit
  - none

none will turn off strict for whatever definition that is.

Importantly, adopting this strategy means that the test runner can take on the responsibility for reconfiguring itself for each run. The dispatcher doesn't need to concern itself with provisioning the testbot. The bot will also be easier to use locally, since you can just change the file and run the test.

mile23’s picture

The other big deal here is that I just designed a replacement for drush.make. :-)

jthorson’s picture

Title: Design UX for Drupal.org integration » Design UI for Drupal.org integration

Ooops ... I mislabeled that. Sorry! I meant that we need a User Interface. :)

nick_schuch’s picture

Here are some diagrams based on the sprinting we did in Amsterdam.

My goal was to take the content type and generate a minim viable project. Here is what I've got:

Wireframe - Latest

Wireframe - Page

nick_schuch’s picture

StatusFileSize
new50.59 KB

Here is an early build of the front page of the results site, still WIP but thought I would share.

DrupalCI results front page

andypost’s picture

#4 looks nice, there's no reason to make it more complicated

cilefen’s picture

@nick_schuch good work

Bojhan’s picture

Sorry, but could this get a requirements list? I honestly have no idea what we are designing, what user goals we need to support what the artefacts are etc.

nick_schuch’s picture

Thanks @cilefen and @andypost

To quote the project page.

This sub-project is responsible for the long-term archival, storage, and exposure of build artifacts and job/test results; similar to the (non-dispatch related) functions performed by qa.d.o today. It is anticipated that this project could end up being a full drupal site on it's own accord, and the repository will include any custom code related to that build.

This is not meant to be the final product, but a talking point so we can build a list of requirements.

I would really like to have your involvement in the requirements gathering @Bojhan

nick_schuch’s picture

Any update on this one....

We are at the point where we will take this prototype and make it look more like a Drupal.org site. We can then publish it as an early build and gauge the community.

Im going to knock something up that we can iterate on until we are all happy.

shyamala’s picture

I have a wireframe from the current qa site updated at https://docs.google.com/spreadsheets/d/1h8FzQGNdeczkN0HAu6TDZRlyqLdQiv6b....