Problem/Motivation
This proposal is based on a core conversation from @webchick and @Gábor Hojtsy, discussed at DrupalCon New Orleans: https://events.drupal.org/neworleans2016/sessions/potential-drupal-8x-an...
Drupal 8 took 4.5 years to complete. We already decided not to do that again and focus on shorter cycles of incremental improvements with semantic versioning instead. That allows predictable timelines where a contribution is released within 6 months, so there is more incentive to contribute. It is also better for users because they see Drupal improving in positive ways twice a year. The release process adjustments did not yet address how we make decisions about what to do and how to do it though, and based on our experience with Drupal 8.0.0 (and prior versions), that needs to be fixed too.
Problems from Drupal 7 and Drupal 8.0.x that we still need to address:
- Bikeshedding, especially of user-facing changes makes it hard to tell who is in charge
- You may work hard on something and it may still get rejected at the end because there is no good way to get the right feedback at the right time
- People don't give directional feedback when needed, standards nitpicks are common even in early stage review
- We don't validate ideas until after shipping; when it is too late to fix
As for where are we working on making those big changes:
- Giant core patches are hard to maintain and review, it is hard to collaborate on them
- Sandboxes are tucked away and not looked at as part of the core process, even though they are easier to collaborate on
- Contributed modules allow experimentation but often have little visibility
- Core reviews are too rigorous, not experimentation friendly
Proposed resolution
- Iterate quickly and cheaply on ideas without needing to implement anything
- Add clear sign-off points to avoid wasting time
- Involve the right stakeholders at the right time
- Gain visibility for proposals from committers
- Reduce barriers to entry into core for new ideas
- Clear visibility of priorities for the community
- Find a middle ground between rigorous core/sandboxes/contrib/huge patches
Introduce steps where a proposal is only developed to the level required to provide the right kind of feedback. At the idea stage, a few sentences explain what is proposed and stakeholders can decide whether the idea is a no-go, or it should move forward to a planning stage. The plan is fleshed out more with requirements, etc. but no implementation is done yet. When goals are agreed on, prototypes are made, which allows user testing and feedback on the interaction / integration level:
Drupal 8.x also introduced the concept of experimental modules which fit with the goals of this proposal very well. Experimental modules allow us to ship with new features and improvements without needing to go through the whole rigorous process of stable core inclusion first. An experimental module is not bound to the same stability requirements that a core modules is and can have its own versions. At the same time, it gives visibility to users and is included in the general core process, therefore serving as a middle ground between the previous options for working on bigger changes.
Quicker iteration is enabled thanks to the refinement circle, while meeting all the rigorous requirements for stable core inclusion can be left to the stage where a module matures from experimental to stable. Some time limits should be set on a case by case basis for when we remove an experimental module if it never matures into a stable module. The module in this case could move to being a contributed project.
Applicability and limitations
As for what constitutes an "idea", and how big or small it needs to be to go through this proposed process, practice will tell. Documentation improvements for existing things, bug fixes for functionality, etc. are not ideas while "Make contact module more like a simple form builder" is.
The use of experimental modules is only applicable to self-contained changes. While services can be swapped out in modules (and most things are services in Drupal 8), larger systemic changes are not possible with experimental modules. We may still need to explore feature branches at one point.
Remaining tasks
Discuss!
Comment | File | Size | Author |
---|---|---|---|
#4 | nola_initiative_roadmap.jpg | 356.55 KB | prestonso |
#2 | PossiblePlanningProcess.png | 121.63 KB | Gábor Hojtsy |
Comments
Comment #2
Gábor HojtsyComment #3
xjmThis was discussed both in the core conversation and in a meeting that was requested later during the conference sprints. Full discussion notes:
https://docs.google.com/document/d/17LIgHEXBbRm_IJeJRznlLzO5zQO21ImhvyPZ...
In discussions at DrupalCon New Orleans, the group suggested refining this into two phases: The planning phase, in which we iterate with ideas, planning, and prototypes; and the building phase, in which we iterate with the experimental implementation until it becomes stable. In that proposed revision, a completed roadmap issue with full signoffs was the boundary between the two phases. I think several people have a photo of a diagram for this but it looks like it has not been added to the notes yet.
Comment #4
prestonso CreditAttribution: prestonso as a volunteer commentedHere's a picture of the initiative roadmap as sketched out during the discussion. I'll also update the NOLA Hard Problems notes with this image as well. Note that nos. 1, 2, and 3 are in a loop titled "Idea" and nos. 4, 5, and 6 are in a line titled "Implementation." Prior to nos. 4, 5, and 6 there are "gates" that must be satisfied before the next step can be initiated.
Comment #5
xjmSeveral people wanted to know what the minimal requirements should be for adding an experimental module, since the proposal indicates the full core gates don't need to be met. I added https://www.drupal.org/core/experimental#requirements based on the consensus from the discussion and previous agreements among the core committers.
Comment #6
xjmThe photo in #4 was hard to interpret for people who missed the discussion (and also hard to read when flipped from upside down at an angle), so I tried to represent it with a diagram that hopefully someone with more skill can turn into something usable:
https://docs.google.com/drawings/d/1LK1dNWb51Pt-uoKPPy_O9BaehiAluCIjYTDP...
Comment #7
xjm@Gábor Hojtsy created an updated version of the diagram:
Comment #8
xjmComment #9
webchickSo, despite the lack of discussion on this issue itself, we've been essentially using this process already for the past several months:
These are all experimental features targeted for 8.2. The things these issues have in common are:
This and more is captured in the documentation about experimental modules.
So it feels like the "Part B" in the diagram is pretty well covered. We still have some kinks to work out regarding various "goldilocks" problems (how much feedback is too much, how many follow-ups are too much, etc.) but I think those will mostly come from experience.
"Part A" on the other hand, hasn't been discussed much at all, beyond a general +1 that we need that separate space. But what it is, where it is, how it works, etc. and most importantly what the "hand-off" is like between part A and part B is still largely unknown.
We had a UX team meeting today and discussed some stuff about this, so going to write up a comment shortly.
Comment #10
webchick[Edit: Moved to separate issue instead since that was getting way too long: #2785735: [policy, no patch] Agile process for bigger changes in core (Part A: Ideation)]
Comment #11
webchickOk, I decided to spin off the stuff about ideation to its own issue, since that hasn't really been properly discussed yet: #2785735: [policy, no patch] Agile process for bigger changes in core (Part A: Ideation)
If we segment off "Part A" from the rest of this, I think the remainder of the diagram is probably RTBC. We can mark it "Fixed" once it's documented in the handbook.
Comment #12
webchickAlso, since this impacts how we add features to core for the duration of Drupal's lifetime (until we think of something better ;)), seems pretty major. :)
Comment #13
webchickWe discussed this in UX team meeting today. Right now, the diagram makes it appear as though all ideas of a certain size must go through an experimental module process in order to get into core. Versus the page at https://www.drupal.org/core/experimental doesn't indicate this; rather, that experimental is just one possible implementation route that has its own pros/cons, alongside others like "directly patch the stable feature" and "do it in contrib first, then propose it."
So it seems like we need the diagram and accompanying documentation to make this more clear, especially with the decision tree of how to decide how an approved plan should be implemented.
Comment #25
quietone CreditAttribution: quietone at PreviousNext commentedI asked about this in committer slack and @Gábor Hojtsy and lauriii responded. They both agreed that this issue is fixed, that the work here is complete. There was an understanding that the process would be improved, or iterated on, but that could be done in a separate issue. I will leave it to those move familiar with the history to open a new issue when it is needed.
Therefor, I am closing this as fixed.
Thanks!
Comment #26
quietone CreditAttribution: quietone at PreviousNext commentedI forgot to add the parent and to add credit.
Comment #28
quietone CreditAttribution: quietone at PreviousNext commentedRemoving tag