(Split off from #2729061: [policy, no patch] Agile process for bigger changes in core (Part B: Implementation). See that for the whole backstory.)

This issue is to discuss "Part A" of the following diagram:

Diagram of core contribution process.

The problem

Here are a few "case studies" on why something for Part A (ideation/prototyping/planning) is a good thing to have, and why it should probably not be the core queue:

Exhibit A: #2759849: Create a new user-facing core theme

This issue starts out simply enough. Some folks want to band together to create a new theme for core with a visually updated design and ideally based on components, so they can use it as a proof of concept for #2702061: Unify & simplify render & theme system: component-based rendering (enables pattern library, style guides, interface previews, client-side re-rendering). So far, so good. However, it quickly became a discussion focused on implementation details, even down to what framework we would use and why using a framework or not using a framework was a stupid idea. :\

Exhibit B: #2762505: Introduce "outside in" quick edit pattern to core

This initiative had been talked about by Dries on his blog since early this year. The implementation team tried to do the right thing and showcase designs early and often to the Drupal Usability Team, and respond quickly to feedback in refined designs. But because it wasn't clear if "sign-off" was achieved, or conversely that sign-off was needed, implementation here didn't start until far, far late in the game.

Proposed solution

It seems like we could solve these and many other process issues by making a hard, enforced separation between "stuff we're thinking about" and "stuff we're working on."

There are many cool tools for doing ideation (such as https://ideascale.com/) and also prototyping (such as http://notableapp.com/) and we could use those or even build our own in Drupal. But that's either introducing friction (yet another login/lack of integrated notifications) or massive engineering effort into the process. (It may well be worth pursuing one of those options in the longer-term though; issue queues are scary places for non-developers to congregate.)

However, in the spirit of making an MVP for making MVPs ;) seems like for now we could use a separate issue queue on Drupal.org, something like "Drupal core ideas." This could have the following components (categories):

- Idea
- Prototype
- Plan

Each issue in this queue would go through its own Active > Needs review > Needs work > Reviewed & Tested > Fixed cycles, for each component (as appropriate), starting with Idea. And the goal of this queue is to create the signed-off-on plans that become the "finalize foo module" plan issue for each of the initiatives mentioned above, which then gets transferred to the Drupal core queue.

An idea consists of the first three sections (Background/Problem/Proposed Resolution) of the initiative proposal template. (Obviously, if more details are known, those could be filled out as well, but the idea will be evaluated based on these.) The goal is for an Idea to be fairly quick and easy to both write up and understand. The Idea bounces from Needs review to Needs work until eventually folks feel that the idea has merit enough to raise it to a Product Manager for sign-off, and mark it RTBC.

For user-facing issues, during the weekly UX meetings, product managers would review RTBC Idea issues with the Drupal UX team, pulling in others or escalating to the BDFL as necessary.

For developer-facing issues, @tbd on how those are reviewed in a timely fashion.

Once an idea was approved, the issue moves to the "Plan" component.. The "Plan" issue goes through the same Active > Needs review > Needs work > Reviewed & Tested > Fixed cycles, this time pulling on Subsystem maintainer review, Topic maintainer review, $foo Manager review, etc. as needed.

In the event a protoyping phase is needed (e.g. for Outside-in, or for a new API such as Theme Component Library (pseudocode)), we could mark the Plan issue postponed on a "Prototype" issue, and the designer could use whatever tools they want to do protoyping, posting a comment with the place for people to test whenever there was something to look at.

Finally, when the Plan issue gets marked "Fixed" in the "Ideas" queue (meaning all of its sign-offs are done), it moves to the Drupal core queue, so we capture the "Why" as well as the "What/how". The issue would move to "Fixed" as these features became stable parts of Drupal core.

(I don't like how "waterfally" this is, mind you. I think some of these things could happen in parallel; for example, the Workflow Initiative was working on existing known revision system problems while waiting for their plan to be approved; prototyping can be happening while the idea is still being pitched, etc.).

Desired outcomes

Creating this clear separation of "Ideas" vs. "Implementation" results in the following benefits:

1) Because there's nothing to write a patch against, it at least attempts to enforce "we're not talking code here yet."
2) Because of that, ideally, it's easier for people such as end-users, project managers, designers, content authors, and other folks who are traditionally not represented well in the core queue to participate.
3) It creates one central place for all the things that are being thought about, vs. having to know the magic string to search for in the Drupal core queue amongst 30,000 issues.
4) It is very clear when conceptual things are blocked on review/feedback, or things that still need work to achieve sign-off status.

Whew! Thoughts?

Comments

webchick created an issue. See original summary.

yoroy’s picture

Issue summary: View changes
webchick’s picture

Priority: Normal » Major
Issue summary: View changes

Hopefully clarifying that this proposed queue/process is not just for user-facing changes, but also for API-related ones.

Also, since this impacts how we add features to core for the duration of Drupal's lifetime (until we think of something better ;)), seems pretty major. :)

davidhernandez’s picture

I like very much the idea of this being an actual queue somewhere. That makes discoverability much easier, so people who might be interested in helping have a place to look for what is being worked on. It also lets initiative proposers see what is already proposed. We have too much of a problem of things off in different corners and no one knows whats going on at a high level.

pwolanin’s picture

Having a process of deciding on features and even the general implementation approach in issues separate from the actual code/patch issue is something we need in general for core. So, yeah, deciding what features we won't and roughly how they should be built and getting all that signed off before 10's of hours are spent on a patch would be a big improvement.

However, what's the scope - all features need to go through this process (vs bug fixes)? Features of a certain size?

I think there will be some friction as we put the process in place and socialize the new flow.

davidhernandez’s picture

So I think this still doesn't address the "consensus" problems we're always dealing with.

Gábor Hojtsy’s picture

jhodgdon’s picture

This looks like a great approach.

One thing that is missing, I think, is to make a distinction between "minor" features (like adding one setting to an existing thing -- an example could be making an option to open a link in another window on a Link field, if that doesn't already exist), vs. "major" features (like totally revamping how Block placement works, or adding a completely new thing to Drupal that it doesn't currently do at all).

I'm guessing that "minor" features wouldn't go through any of this process, and it's just for "major" features? But the issue summary here doesn't really make that clear, and I'm not sure where the dividing line would be between minor and major.

Other than that, this looks like a great idea for vetting ideas, and probably only trying it out for a while will let the Ideas team know whether it will work well or needs tweaking. Don't fall into the trap of bikeshedding here and trying to envision a perfect process for ideation before you've at least prototyped and tested it... ;)

jhodgdon’s picture

Oh, one other question. Who makes an idea proposal -- does it have to be someone who plans to actually lead the implementation initiative, or can it be anyone with an idea?

Gábor Hojtsy’s picture

@jhodgdon: re major/minor, yes, this process should not be used for "fix typos in code comments" or even "make this link open in a new window" or "replace throbber with a better designed one aligning with the Seven style guide". I don't think we had any good ideas to put a line in the sand on how can one differentiate between the minor and majors though. Any ideas?

Re who can propose plans, since discussing plans will take some time for the respective teams/people, IMHO it would be important that plans where there is already an intention of implementation (whether by submitter or others) are posted. Instead of "I have this idea that would be nice, please someone do it". That said, if core is only constrained by lack of ideas, we should revisit that :)

jhodgdon’s picture

I totally agree that proposing plans in this system, without some idea of how they'd get implemented (feasibility and people to do it) would be a waste of time for reviewers.... But maybe there could be a Component for "Preliminary idea" that could be a way for non-technical people/teams to propose improvements without a developer on board, and developers to look for "wild ideas" that they could join/form initiatives around? These ideas could then have a place to get discussed, without being in formal review, and it would be understood that the Core maintainers and Usability teams etc. would not even look at these until they've moved into the "Idea" phase (when they would need to have some kind of team in place including a developer). It just seems like it might be good for there to be a way for people who are the users of Drupal to propose these things -- most likely, they would have better ideas about the pain points and maybe perspective from other CMS systems they've used, than developers who are caught up in the day-to-day of Drupal Core development.

Regarding major/minor, it seems like this process is for ideas that are "big enough to need their own module or approximately that amount of code" or "major revisions to user or developer experiences". I don't think you can probably draw a formal line, but maybe a few examples would help people decide between filing a Feature Request issue in the Core queue, vs. and Idea issue in the Ideation queue. Anyway, issues can easily be moved between projects, so if a "too big" issue gets filed in Core, it can be moved to Ideation, and vice versa for "too small".

webchick’s picture

I think a loose criteria could be a "significant" change (in reference to the core governance doc) that needs sign-off from 3 or more stakeholders maybe. Like if you're making changes to an admin form a module somewhere that needs the subsystem maintainer's sign-off only? Go nuts. Fixing a bug elsewhere in the system? No problem.

But, if the change you're affecting the node form quite a bit, suddenly that affects Usability, Accessibility, Product, and possibly Framework as well. That'd be a good candidate for this queue.

Similarly, a brand new feature will require at least Product, Framework, and Release manager review, and probably a few subsystem maintainer reviews as well.

Would that line (or something like it) make sense?

jhodgdon’s picture

That makes some sense... although some "small" changes might still require Usability and subsystem maintainer sign-off... I guess that is only 2 though. :)

I think putting those examples from #12 in your eventual documentation on this would be helpful. Plus the ones from #10.

davidhernandez’s picture

Isn't it easier to look at it as anything that needs product manager ownerage/sign off? These kinds of major changes would be things that affect "Drupal The Product". Place block is a major product change, a new theme is a major product change, adding workflow is a major product change, outside in is a major product change. Things that significantly affect those who use or develop with Drupal (which would include big time api changes, too.)

jhodgdon’s picture

Well, it's not just Product Manager -- some of the Big Changes (TM) could be pure Developer Experience, and I think that is not the Product Manager's sphere. But possibly we could make a short list of people who would need to sign off (Product Manager, Architecture person or whatever they are called, etc.).

webchick’s picture

Right. Like other things that would go in here would be Configuration Management, Plugin System, etc. Those are all developer-facing changes.

But maybe "needs sign off from Product Manager or Framework Manager" would be a good guideline indeed.

jhodgdon’s picture

Simple and it seems, accurate too -- a good combination. ;)

karolus’s picture

It's a good thing that Drupal is becoming more responsive to change with this approach.

I've been part of large Web projects on various platforms, and have seen the good and the bad of the process.

Some initiatives I've seen that work well center around user-centered design, and Design Thinking (DT). In DT, it takes a more Agile-like approach--key takeaways for the process are rapid iteration of disposable prototypes. This allows the process to move forward in a rapid manner, allowing changes to be made based on feedback from user testing and validation without a major investment of time and resources. I've employed in on projects (with modifications), and it's served pretty well.

Tools such as InVision are excellent for accomplishing tasks such as what we're looking at, and require relatively minimal investment of time and technology, allowing participants of various skill levels to contribute. In addition, interactions can be added, without creating a single line of code. I use it regularly on my Drupal projects (and I can provide examples). Another benefit is that these prototypes can be shared with developers while ideating, which can help anticipate and deal with potentially major coding issues.

One of the best aspects of having visuals to share is that we're naturally visually-focused, and this gives a tangible quality to the process, drawing more people in to participate. Being active in the local design community, I hear sentiments that Drupal is more developer/enterprise-focused, and not design/UX-friendly. Here's a good way to counter that.

webchick’s picture

We discussed this in UX meeting today, and folks pointed out that it'd be good to have more feedback here esp. from subsystem maintainers and core committers and others who would need to be more actively monitoring the stuff happening here. So if that describes you, and you're one of the 23 followers of this issue who haven't already commented, please do! :) Else, I'll try and reach out to a few folks and ask them to chime in here.

joelpittet’s picture

It would be nice to have somewhere in the proposal of the idea a clear reason why this isn't going into contrib and why it needs to be flushed out in core. That 'discussion' could avoid time upfront if thought through early in the idea.

And the question comes to mind. Should things go into contrib by default if they can go into contrib first?

webchick’s picture

It would be nice to have somewhere in the proposal of the idea a clear reason why this isn't going into contrib and why it needs to be flushed out in core. That 'discussion' could avoid time upfront if thought through early in the idea.

Agreed, the "how" is a good idea to discuss as part of the planning. Some core features, such as Big Pipe, did really well as contrib-first.

And the question comes to mind. Should things go into contrib by default if they can go into contrib first?

No, definitely not. Particularly if it's strategically important to the project. A shining example of this is Media. It has been in contrib for 7+ years, and still isn't done yet. As a result, Drupal core's out of the box experience is painfully behind our competition. While there used to be good arguments against "core first" in the perfectionist tendencies, inability to change things after they're committed, only innovating every 3+ years, etc... Experimental modules and semver largely render those concerns moot.

davidhernandez’s picture

+1 to #20

webchick’s picture

We talked about this on UX meeting last week and felt it was important for there to be more feedback on this plan, esp. from decision-makers who'd need to look at it at least once in awhile. ;)

I've started going around to various issues that seem like they fit the bill for this new queue and added the Drupal core ideas tag to them, asking them (in boilerplate comment) to provide feedback. We'll let that sit for a week or so and see if that generates more feedback. I would love to set this up before DrupalCon, if possible, since Gábor is giving a talk about the new core development process and would love this part not to be "TBD" again.

Mile23’s picture

+1 on this move. The problem is the implementation, eh? :-)

I'd much rather see a new category in the project ('Ideation'?), versus having a separate project like 'Core ideas' or whatever.

It could be that someone proposes a broad Feature Request with a tag like 'Possible Ideation.' Then only a maintainer can promote it to Ideation category, which then has its own rules, and perhaps even an associated feature branch.

This kind of process has been totally missing unless you happened to be at a sprint somewhere where two people reached consensus and therefore the whole community is aligned. :-) So I'm glad it's happening.

Also, I'm coming at this issue from #2735005: Convert all Simpletest web tests to BrowserTestBase (or UnitTestBase/KernelTestBase) which @webchick tagged per #23. That issue is *not* an ideation-type issue. It's an implementation issue, because we've pretty-much already decided to move away from SimpleTest. To me it seems this kind of process would be for a large-scope feature or a re-architecting issue. Questions of *should* we commit to a process that's a lot of work, based on needs, rather than *how* to accomplish that work.

With that in mind, we'd want to be able to decide to close these Ideation issues. What metric will allow us to close on an Ideation issue without proceeding?

Fabianx’s picture

+1 to the proposal

joachim’s picture

> I'd much rather see a new category in the project ('Ideation'?), versus having a separate project like 'Core ideas' or whatever.

I'm also not clear on why this needs to be a separate project.

Also, would the recently added 'Plan' category not be suitable for this?

(But in general, yes, +1!)

lauriii’s picture

+1 to the proposal.

David_Rothstein’s picture

I'd much rather see a new category in the project ('Ideation'?), versus having a separate project like 'Core ideas' or whatever.

Agreed. I don't see what the advantages of a separate project are, and there are several disadvantages:

  1. Harder for maintainers (more places to keep track of).
  2. Less intuitive for inexperienced contributors. (Who would ever guess that the Drupal core issue queue is the wrong place to discuss ideas for Drupal core?)
  3. Not easily extendable to contrib projects who also may want this capability.

But if the proposal is just to add new issue categories like "Idea" and "Prototype" to the existing ones, it sounds great to me.

--

I would also suggest that you consider renaming "Plan" to something else as part of this ("To-Do List", perhaps, given that it's supposed to be for meta issues). This would prevent people from using "Plan" when "Idea" would be more appropriate. And the current "Plan" category encourages open-ended discussions which issue queues aren't good for. #2759849: Create a new user-facing core theme is an example of that - I don't think it's accurate to say that that issue was overrun by a discussion of implemention details, rather it looks to me like some people tried to do design planning (which makes sense) and others started to do implementation/long-term-maintainability planning (which also makes sense) but the problem is that it all happened in the same issue. An initial "Idea" issue (with no planning whatsoever) followed by two related and more-focused "Task" issues for specific decisions about design and implementation would probably have gone better.

webchick’s picture

We could eventually probably dork around with the category lists, but AFAIK that requires DA time, which my understanding is now next to impossible to get, since they have to be focused on keeping the lights on.

The reason I'd like a separate queue as well is this is really a separate audience, or at least a separate context for the same audience. This is end users, designers, UX people, architects, and others who are engaging in high-level planning and validation. A physical separation in terms of the place where it happens vastly reduces the number of overall issues in that space (making it much easier to approach for non-developers, much easier to check for duplicates, etc.),, and also creates a dedicated space for those high-level conversations to happen, and a different place to have conversations around implementation details.

David_Rothstein’s picture

Adding issue categories is technically a code change, but as far as I can tell it's a very simple and zero-risk one (this appears to be the entirety of the change needed to add "Plan") so I assume that option, if chosen, would not be significantly blocked in practice.

webchick’s picture

Well that's good to know! Ok, spun off #2796017: Create new issue categories to account for different planning stages. I'd really rather not block this process on that discussion, though. We could always migrate things back over (manually, if needed) when/if that gets accomplished if we feel like that's a good idea, given the benefits of a separate project outlined in #29.

ptocco’s picture

I would like to add 2 cents to some of the previous comments, for what it's worth. This particular thread seems focused on the need to separate ideas from implementation, and ensuring those ideas are properly vetted before adding them to core. There is of course a need to be cautious so as to not clutter core with things that belong in contrib. I would suggest that the user-facing core theme should first and foremost be a kind of base theme that showcases best practices, with plenty of internal documentation for adding additional subthemes in the contrib community. +1 to all comments suggesting an iterative feedback process on the usability of this theme while it is in development. No doubt, Drupal users deserve and expect a "flagship" core theme. (I found out too late that Bartik was not the polished and perfected theme I thought it was.) D8 should no doubt offer a very simply core theme that invites the contrib community to build upon. I for one would like a core theme that facilitates standardized components and libraries, perhaps even embedding Bootstrap tags into page.tpl files to facilitate customization, rather than placing too much of the load on blocks, and complex variables etc.

My simply takeway is:

1) the need for this theme to be UX tested from a developer point of view

2) the need to incorporate a simple set of best practices, components and libraries that can be readily expanded by the contrib community

Gábor Hojtsy’s picture

@ptocco: Re

There is of course a need to be cautious so as to not clutter core with things that belong in contrib.

The problem is more that people would work on something for core for a long time, investing lots of sweat and tears and then see it rejected because it was not a good fit. We want to reject things that are not a good fit early to save people all that time and effort. (People could still do whatever they want in contrib as always). So its not really about avoiding adding things to core that do not belong there, because those would not be added just because someone worked on them for months, that is not the only deciding criteria. We are trying to set up a system to provide feedback as early as possible on the high level idea.

Re: your comments on the theme, this is not the issue to discuss specific proposals for core features, so your feedback would be at best lost here. Please post on the specific issue.

frob’s picture

I came here from #2724845: Run Drupal as a service to allow non-http based API Initiative which seems like a prime example of a significant change that would benefit greatly from this process.

I would say that this should defiantly be a separate queue from the core queue. My primary thought on it needing to be its own queue is based on usability. The core issue queue is packed and intimidating, especially to new contributors. There are two aspects that make it difficult to use.

  1. The component drop down is unwieldy and intimidating. I recently ran a Drupal Contribute meetup where I was attempting to teach people how to use the Drupal Issue Queue, most specifically the core issue queue. The workflow is logical and easy to understand, telling them what component to assign things to was by far the hardest part. It is a seemingly random list of (sometimes) similar words.
  2. Issue tags are magic to new users. While great in concept, this is a feature of the issue queue that is very opaque to users. It requires secret knowledge of what keys do what. This is a great example of, "Its easy when you know how."

I a very much a +1 for this idea. It gives a single place for bad ideas to die and for good ideas to be put through their paces. Sometimes when dealing with core issues or ideas an issue can sit idle for a long time with little no feedback or worse yet, an idea can be spearheaded by a single person and a bunch of work can be done that never goes anywhere. Making this a clear workflow will really help set expectations and keep people from working on ideas that have fundamental issues but can also limit the bike-shedding that can stymie ideas early on.

Another reason I like this as a separate queue is, I would expect it to be the least amount of work for DA to pull this off and still have something usable. It would be good to get their feedback on this as well.

Fabianx’s picture

#34: That is an excellent and very insightful comment! Thank you!

David_Rothstein’s picture

@frob, on your two points:

  1. I completely agree, but that's an existing usability problem - better to fix it rather than trying to avoid it forever :) I created #2797887: Add a default component 'Unknown' to the Drupal core issue queue now with a very simple improvement for that. (There's also a larger issue at #2660144: [Plan] Update core components.)
  2. As far as I can see, no one discussed using issue tags above... so I'm not sure how that is relevant here? (But yes, in general issue tags aren't great.)
frob’s picture

@David_Rothstein, thank you for opening that issue. It looks like progress is being made to clean up the core components select list.

My points, 1 and 2, are general points on why I don't think it would be a good idea to put this into the core queue. I still think it would be better in its own queue, that way things can be hashed out at a higher level.

frob’s picture

edit, duplicate of #37

cilefen’s picture

It is better as a separate queue because we could change our minds later without having added categories or even more components to the core queue.

Fabianx’s picture

#39: Categories are the same across all projects, but components would be special.

OT, but components should be using a multi group to be more usable.

I don't mind that much if its a different queue or the core queue as I never saw the big difference. So I am neutral on that and can see arguments for both.

yoroy’s picture

Status: Needs review » Reviewed & tested by the community

I'm sensitive to the arguments against a separate queue. An earlier version of this was hashing out research, design and plan in groups.drupal.org/usability and the transition from there over to the core queue was far from smooth.

But this queue is closer to the core queue. You can follow issues, following them in the same tracker on your dashboard. I really think we should at least start with this separate queue so that:

- We are explicit in separating idea from execution, and get used to that idea and mode of working (for large/impactful changes at least)
- For as much as possible in a d.o. queue, it creates a safe space to work through ideas and designs, and those only.
- I hope this queue will as a result also make our product strategy discussions more transparent to the community

Lets try this out and see how it works. Looking forward to discuss some Ideas in Dublin!

Fabianx’s picture

RTBC + 1, lets try it with a separate queue

frob’s picture

+1, now I wish I was going to Dublin.

webchick’s picture

Project: Drupal core » Drupal core ideas
Version: 8.3.x-dev »
Component: other » Idea
Status: Reviewed & tested by the community » Fixed

Ok, this has been RTBC for a few days now, here goes nothing. :)

joachim’s picture

How do we get to the project for this queue? When I look at an issue in the queue, the breadcrumb is to the normal Drupal project.

serg2’s picture

jhodgdon’s picture

Yeah, this project is lacking in breadcrumbs...

webchick’s picture

Hm. I made it a "Drupal core" project, which I guess is what screws with the breadcrumbs. :( Not sure what to do now... if I should delete the project and remake it as a ... module? Or if we should traipse through drupalorg_customizations trying to find the bit of code that's turning the breadcrumbs off.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.