With #2268449: Run contrib module branch tests against both dev and latest stable core branches we now have custom tests for testing against a non-default core branch, like 8.0.x instead of 8.1.x, and for future one-off tests.

There should be a UI to add these recurring, while limiting each project to a reasonable number of tests. Testing spins up AWS boxes as-needed, each one has a small cost. We need to strike the right balance between flexibility and controlled costs.

Members fund testing for the Drupal project. Drupal Association Learn more

Comments

drumm created an issue. See original summary.

drumm’s picture

Assigned: Unassigned » drumm

This issue will replace the table on projects’ automated testing pages with a larger list UI with more room for options to spread out and be explained. For each branch, we will limit “Issues & on commit” testing to a single environment.

catch’s picture

One point from #2680037: [policy, no patch] Figure out core branch testing needs for contrib modules. We should enforce a minimum version - so that when 8.0.0 support is dropped, any project still configured with it moves to 8.1.0 automatically.

Not sure we'll actually want to mark the 8.0.0 branch unsupported in the project UI though, or not quickly anyway, which would be the easiest way to automate something like that.

  • drumm committed e037ca7 on 2651208-project-ui
    Issue #2651208: Remove overloaded project automated testing UI
    
drumm’s picture

So far I've gutted the edit UI from the projects’ automated testing tab. An example of the new UI is:

Screenshot

Now I need to build out the individual add/edit UI.

drumm’s picture

In going over this with mixologic, we saw some potential in making use of core release labels somehow. One or two words for the targets at https://www.drupal.org/core/release-cycle-overview.

Something like “current development branch” could mean 8.1.x today, and 8.2.x when that opens up. Good labels for the others seem harder to come up with, especially when the beta/rc phase has one more possibility.

(There was some talk of doing something like this for core issues, so open issues don’t have to chase 8.x.x.)

catch’s picture

I can't find the other discussion at the moment, but the proposal there was something like 8.x-PATCH and 8.x-MINOR.

That works exactly until you hit the current point where there are three branches. 8.2.x is already open and it'd be 8.x-NEXT-MINOR or something at the moment.

Running on the next minor before the current stable branch is done, at least for contrib tests, is the least important part of this though - there's hardly any divergence between 8.1.x and 8.2.x at the moment.

Don't think there's an actual issue for the labels, so opened one at #2691883: [policy, no patch] Branch labels for core.

  • drumm committed d61c476 on 2651208-project-ui
    Issue #2651208: Organize environments for custom branch testing
    
  • drumm committed eb06155 on 2651208-project-ui
    Issue #2651208: More-flexible project configuration, add mock options...
drumm’s picture

I expanded the Add [custom] branch test UI to have a Schedule option and default core:

Screenshot

Until #2691883: [policy, no patch] Branch labels for core is further along, I think I'll stick to only putting in a “default” for core, and no additional labels. The schedule options now have more screen real estate to be explained.

And I've organized the environment options, similar to recent changes to the issue test UI:

Screenshot

  • drumm committed 2df9b4e on 2651208-project-ui
    Issue #2651208: Fill out add test logic
    
  • drumm committed 3394bb0 on 2651208-project-ui
    Issue #2651208: Make adding tests possible again, stub out the actual...
  • drumm committed 76e88ad on 2651208-project-ui
    Issue #2651208: Fill out existing issue test environment
    
  • drumm committed 98ffb8d on 2651208-project-ui
    Issue #2651208: Adjust for core_branch being in primary key
    
  • drumm committed 9a5b2b5 on 2651208-project-ui
    Issue #2651208: Fill out default core branch label
    
  • drumm committed de1a910 on 2651208-project-ui
    Issue #2651208: Expand primary key to include core_branch
    
  • drumm committed e4117c5 on 2651208-project-ui
    Issue #2651208: Unify project add test links
    
  • drumm committed e56e94e on 2651208-project-ui
    Issue #2651208: Add core_branch storage for project configuration
    

  • drumm committed 019c477 on 2651208-project-ui
    Issue #2651208: Only show one queued test for each configuration
    
  • drumm committed 4a18e1a on 2651208-project-ui
    Issue #2651208: Fix notice
    
  • drumm committed 4c384f7 on 2651208-project-ui
    Issue #2651208: Add core branch awareness to issue testing
    
  • drumm committed ec4ba7b on 2651208-project-ui
    Issue #2651208: Editing and removing configurations
    

  • drumm committed 019c477 on 7.x-3.x
    Issue #2651208: Only show one queued test for each configuration
    
  • drumm committed 2df9b4e on 7.x-3.x
    Issue #2651208: Fill out add test logic
    
  • drumm committed 3394bb0 on 7.x-3.x
    Issue #2651208: Make adding tests possible again, stub out the actual...
  • drumm committed 4a18e1a on 7.x-3.x
    Issue #2651208: Fix notice
    
  • drumm committed 4c384f7 on 7.x-3.x
    Issue #2651208: Add core branch awareness to issue testing
    
  • drumm committed 76e88ad on 7.x-3.x
    Issue #2651208: Fill out existing issue test environment
    
  • drumm committed 98ffb8d on 7.x-3.x
    Issue #2651208: Adjust for core_branch being in primary key
    
  • drumm committed 9a5b2b5 on 7.x-3.x
    Issue #2651208: Fill out default core branch label
    
  • drumm committed d61c476 on 7.x-3.x
    Issue #2651208: Organize environments for custom branch testing
    
  • drumm committed de1a910 on 7.x-3.x
    Issue #2651208: Expand primary key to include core_branch
    
  • drumm committed e037ca7 on 7.x-3.x
    Issue #2651208: Remove overloaded project automated testing UI
    
  • drumm committed e4117c5 on 7.x-3.x
    Issue #2651208: Unify project add test links
    
  • drumm committed e56e94e on 7.x-3.x
    Issue #2651208: Add core_branch storage for project configuration
    
  • drumm committed eb06155 on 7.x-3.x
    Issue #2651208: More-flexible project configuration, add mock options...
  • drumm committed ec4ba7b on 7.x-3.x
    Issue #2651208: Editing and removing configurations
    

  • drumm committed 5aa951c on 7.x-3.x, 2651208-project-ui
    Issue #2651208: Add core branch awareness to commit testing
    
  • drumm committed 5aa951c on 7.x-3.x, 2651208-project-ui
    Issue #2651208: Add core branch awareness to commit testing
    
drumm’s picture

Status: Active » Fixed

I'll be deploying this shortly.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.