Problem/Motivation

This proposal is based on a core conversation from @webchick and @Gábor Hojtsy, discussed at DrupalCon New Orleans: https://events.drupal.org/neworleans2016/sessions/potential-drupal-8x-an...

Drupal 8 took 4.5 years to complete. We already decided not to do that again and focus on shorter cycles of incremental improvements with semantic versioning instead. That allows predictable timelines where a contribution is released within 6 months, so there is more incentive to contribute. It is also better for users because they see Drupal improving in positive ways twice a year. The release process adjustments did not yet address how we make decisions about what to do and how to do it though, and based on our experience with Drupal 8.0.0 (and prior versions), that needs to be fixed too.

Problems from Drupal 7 and Drupal 8.0.x that we still need to address:

  • Bikeshedding, especially of user-facing changes makes it hard to tell who is in charge
  • You may work hard on something and it may still get rejected at the end because there is no good way to get the right feedback at the right time
  • People don't give directional feedback when needed, standards nitpicks are common even in early stage review
  • We don't validate ideas until after shipping; when it is too late to fix

As for where are we working on making those big changes:

  • Giant core patches are hard to maintain and review, it is hard to collaborate on them
  • Sandboxes are tucked away and not looked at as part of the core process, even though they are easier to collaborate on
  • Contributed modules allow experimentation but often have little visibility
  • Core reviews are too rigorous, not experimentation friendly

Proposed resolution

  1. Iterate quickly and cheaply on ideas without needing to implement anything
  2. Add clear sign-off points to avoid wasting time
  3. Involve the right stakeholders at the right time
  4. Gain visibility for proposals from committers
  5. Reduce barriers to entry into core for new ideas
  6. Clear visibility of priorities for the community
  7. Find a middle ground between rigorous core/sandboxes/contrib/huge patches

Introduce steps where a proposal is only developed to the level required to provide the right kind of feedback. At the idea stage, a few sentences explain what is proposed and stakeholders can decide whether the idea is a no-go, or it should move forward to a planning stage. The plan is fleshed out more with requirements, etc. but no implementation is done yet. When goals are agreed on, prototypes are made, which allows user testing and feedback on the interaction / integration level:

Diagram of core ideation and experimental module process
(larger version)

Drupal 8.x also introduced the concept of experimental modules which fit with the goals of this proposal very well. Experimental modules allow us to ship with new features and improvements without needing to go through the whole rigorous process of stable core inclusion first. An experimental module is not bound to the same stability requirements that a core modules is and can have its own versions. At the same time, it gives visibility to users and is included in the general core process, therefore serving as a middle ground between the previous options for working on bigger changes.

Quicker iteration is enabled thanks to the refinement circle, while meeting all the rigorous requirements for stable core inclusion can be left to the stage where a module matures from experimental to stable. Some time limits should be set on a case by case basis for when we remove an experimental module if it never matures into a stable module. The module in this case could move to being a contributed project.

Applicability and limitations

As for what constitutes an "idea", and how big or small it needs to be to go through this proposed process, practice will tell. Documentation improvements for existing things, bug fixes for functionality, etc. are not ideas while "Make contact module more like a simple form builder" is.

The use of experimental modules is only applicable to self-contained changes. While services can be swapped out in modules (and most things are services in Drupal 8), larger systemic changes are not possible with experimental modules. We may still need to explore feature branches at one point.

Remaining tasks

Discuss!

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Gábor Hojtsy created an issue. See original summary.

Gábor Hojtsy’s picture

Issue summary: View changes
FileSize
121.63 KB
xjm’s picture

This was discussed both in the core conversation and in a meeting that was requested later during the conference sprints. Full discussion notes:
https://docs.google.com/document/d/17LIgHEXBbRm_IJeJRznlLzO5zQO21ImhvyPZ...

In discussions at DrupalCon New Orleans, the group suggested refining this into two phases: The planning phase, in which we iterate with ideas, planning, and prototypes; and the building phase, in which we iterate with the experimental implementation until it becomes stable. In that proposed revision, a completed roadmap issue with full signoffs was the boundary between the two phases. I think several people have a photo of a diagram for this but it looks like it has not been added to the notes yet.

prestonso’s picture

FileSize
356.55 KB

Initiative roadmap from DrupalCon New Orleans

Here's a picture of the initiative roadmap as sketched out during the discussion. I'll also update the NOLA Hard Problems notes with this image as well. Note that nos. 1, 2, and 3 are in a loop titled "Idea" and nos. 4, 5, and 6 are in a line titled "Implementation." Prior to nos. 4, 5, and 6 there are "gates" that must be satisfied before the next step can be initiated.

xjm’s picture

Several people wanted to know what the minimal requirements should be for adding an experimental module, since the proposal indicates the full core gates don't need to be met. I added https://www.drupal.org/core/experimental#requirements based on the consensus from the discussion and previous agreements among the core committers.

xjm’s picture

The photo in #4 was hard to interpret for people who missed the discussion (and also hard to read when flipped from upside down at an angle), so I tried to represent it with a diagram that hopefully someone with more skill can turn into something usable:
https://docs.google.com/drawings/d/1LK1dNWb51Pt-uoKPPy_O9BaehiAluCIjYTDP...

xjm’s picture

@Gábor Hojtsy created an updated version of the diagram:
Diagram of core ideation and experimental module process

xjm’s picture

Issue summary: View changes
webchick’s picture

So, despite the lack of discussion on this issue itself, we've been essentially using this process already for the past several months:

These are all experimental features targeted for 8.2. The things these issues have in common are:

  1. The original "implementation" issue tried to stay as laser-focused as possible on an MVP, without growing scope. This often involved cat-herding to stop people from reverting to old habits like nit-picking whitespace.
  2. Any actionable feedback given as reviews in the implementation issues that are out of scope for MVP are spun off as follow-ups.
  3. When parts of the patch were found to affect areas outside of the experimental module itself, those were spun off into sub-issues that were required to be committed prior to the experimental module making it into core.
  4. A "parent" plan issue based on a template is created in each instance to capture these follow-ups and arrange them into must-have/should-have/could-have for a module to make it into core "properly." This is required prior to the experimental module being committed

This and more is captured in the documentation about experimental modules.

So it feels like the "Part B" in the diagram is pretty well covered. We still have some kinks to work out regarding various "goldilocks" problems (how much feedback is too much, how many follow-ups are too much, etc.) but I think those will mostly come from experience.

"Part A" on the other hand, hasn't been discussed much at all, beyond a general +1 that we need that separate space. But what it is, where it is, how it works, etc. and most importantly what the "hand-off" is like between part A and part B is still largely unknown.

We had a UX team meeting today and discussed some stuff about this, so going to write up a comment shortly.

webchick’s picture

[Edit: Moved to separate issue instead since that was getting way too long: #2785735: [policy, no patch] Agile process for bigger changes in core (Part A: Ideation)]

webchick’s picture

Title: [policy, no patch] Agile process for bigger changes in core » [policy, no patch] Agile process for bigger changes in core (Part B: Implementation)
Status: Needs review » Reviewed & tested by the community
Issue tags: +Needs documentation

Ok, I decided to spin off the stuff about ideation to its own issue, since that hasn't really been properly discussed yet: #2785735: [policy, no patch] Agile process for bigger changes in core (Part A: Ideation)

If we segment off "Part A" from the rest of this, I think the remainder of the diagram is probably RTBC. We can mark it "Fixed" once it's documented in the handbook.

webchick’s picture

Priority: Normal » Major

Also, since this impacts how we add features to core for the duration of Drupal's lifetime (until we think of something better ;)), seems pretty major. :)

webchick’s picture

Status: Reviewed & tested by the community » Needs work

We discussed this in UX team meeting today. Right now, the diagram makes it appear as though all ideas of a certain size must go through an experimental module process in order to get into core. Versus the page at https://www.drupal.org/core/experimental doesn't indicate this; rather, that experimental is just one possible implementation route that has its own pros/cons, alongside others like "directly patch the stable feature" and "do it in contrib first, then propose it."

So it seems like we need the diagram and accompanying documentation to make this more clear, especially with the decision tree of how to decide how an approved plan should be implemented.

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.9 was released on September 7 and is the final bugfix release for the Drupal 8.1.x series. Drupal 8.1.x will not receive any further development aside from security fixes. Drupal 8.2.0-rc1 is now available and sites should prepare to upgrade to 8.2.0.

Bug reports should be targeted against the 8.2.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.6 was released on February 1, 2017 and is the final full bugfix release for the Drupal 8.2.x series. Drupal 8.2.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.3.0 on April 5, 2017. (Drupal 8.3.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.3.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.6 was released on August 2, 2017 and is the final full bugfix release for the Drupal 8.3.x series. Drupal 8.3.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.4.0 on October 4, 2017. (Drupal 8.4.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.4.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.4 was released on January 3, 2018 and is the final full bugfix release for the Drupal 8.4.x series. Drupal 8.4.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.5.0 on March 7, 2018. (Drupal 8.5.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.5.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.6 was released on August 1, 2018 and is the final bugfix release for the Drupal 8.5.x series. Drupal 8.5.x will not receive any further development aside from security fixes. Sites should prepare to update to 8.6.0 on September 5, 2018. (Drupal 8.6.0-rc1 is available for testing.)

Bug reports should be targeted against the 8.6.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.6.x-dev » 8.8.x-dev

Drupal 8.6.x will not receive any further development aside from security fixes. Bug reports should be targeted against the 8.8.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.9.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.7 was released on June 3, 2020 and is the final full bugfix release for the Drupal 8.8.x series. Drupal 8.8.x will not receive any further development aside from security fixes. Sites should prepare to update to Drupal 8.9.0 or Drupal 9.0.0 for ongoing support.

Bug reports should be targeted against the 8.9.x-dev branch from now on, and new development or disruptive changes should be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.9.x-dev » 9.2.x-dev

Drupal 8 is end-of-life as of November 17, 2021. There will not be further changes made to Drupal 8. Bugfixes are now made to the 9.3.x and higher branches only. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.15 was released on June 1st, 2022 and is the final full bugfix release for the Drupal 9.3.x series. Drupal 9.3.x will not receive any further development aside from security fixes. Drupal 9 bug reports should be targeted for the 9.4.x-dev branch from now on, and new development or disruptive changes should be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.9 was released on December 7, 2022 and is the final full bugfix release for the Drupal 9.4.x series. Drupal 9.4.x will not receive any further development aside from security fixes. Drupal 9 bug reports should be targeted for the 9.5.x-dev branch from now on, and new development or disruptive changes should be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

quietone’s picture

Status: Needs work » Fixed

I asked about this in committer slack and @Gábor Hojtsy and lauriii responded. They both agreed that this issue is fixed, that the work here is complete. There was an understanding that the process would be improved, or iterated on, but that could be done in a separate issue. I will leave it to those move familiar with the history to open a new issue when it is needed.

Therefor, I am closing this as fixed.

Thanks!

quietone’s picture

I forgot to add the parent and to add credit.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

quietone’s picture

Issue tags: -Needs documentation

Removing tag