[policy, no patch] Agile process for bigger changes in core (Part A: Ideation)

I like very much the idea of this being an actual queue somewhere. That makes discoverability much easier, so people who might be interested in helping have a place to look for what is being worked on. It also lets initiative proposers see what is already proposed. We have too much of a problem of things off in different corners and no one knows whats going on at a high level.

Having a process of deciding on features and even the general implementation approach in issues separate from the actual code/patch issue is something we need in general for core. So, yeah, deciding what features we won't and roughly how they should be built and getting all that signed off before 10's of hours are spent on a patch would be a big improvement.

However, what's the scope - all features need to go through this process (vs bug fixes)? Features of a certain size?

I think there will be some friction as we put the process in place and socialize the new flow.

So I think this still doesn't address the "consensus" problems we're always dealing with.

Posted https://groups.drupal.org/node/513151 in the core group and @yoroy posted http://www.yoroy.com/pieces/vetting-drupal-product-ideas :)

This looks like a great approach.

One thing that is missing, I think, is to make a distinction between "minor" features (like adding one setting to an existing thing -- an example could be making an option to open a link in another window on a Link field, if that doesn't already exist), vs. "major" features (like totally revamping how Block placement works, or adding a completely new thing to Drupal that it doesn't currently do at all).

I'm guessing that "minor" features wouldn't go through any of this process, and it's just for "major" features? But the issue summary here doesn't really make that clear, and I'm not sure where the dividing line would be between minor and major.

Other than that, this looks like a great idea for vetting ideas, and probably only trying it out for a while will let the Ideas team know whether it will work well or needs tweaking. Don't fall into the trap of bikeshedding here and trying to envision a perfect process for ideation before you've at least prototyped and tested it... ;)

Oh, one other question. Who makes an idea proposal -- does it have to be someone who plans to actually lead the implementation initiative, or can it be anyone with an idea?

@jhodgdon: re major/minor, yes, this process should not be used for "fix typos in code comments" or even "make this link open in a new window" or "replace throbber with a better designed one aligning with the Seven style guide". I don't think we had any good ideas to put a line in the sand on how can one differentiate between the minor and majors though. Any ideas?

Re who can propose plans, since discussing plans will take some time for the respective teams/people, IMHO it would be important that plans where there is already an intention of implementation (whether by submitter or others) are posted. Instead of "I have this idea that would be nice, please someone do it". That said, if core is only constrained by lack of ideas, we should revisit that :)

I totally agree that proposing plans in this system, without some idea of how they'd get implemented (feasibility and people to do it) would be a waste of time for reviewers.... But maybe there could be a Component for "Preliminary idea" that could be a way for non-technical people/teams to propose improvements without a developer on board, and developers to look for "wild ideas" that they could join/form initiatives around? These ideas could then have a place to get discussed, without being in formal review, and it would be understood that the Core maintainers and Usability teams etc. would not even look at these until they've moved into the "Idea" phase (when they would need to have some kind of team in place including a developer). It just seems like it might be good for there to be a way for people who are the users of Drupal to propose these things -- most likely, they would have better ideas about the pain points and maybe perspective from other CMS systems they've used, than developers who are caught up in the day-to-day of Drupal Core development.

Regarding major/minor, it seems like this process is for ideas that are "big enough to need their own module or approximately that amount of code" or "major revisions to user or developer experiences". I don't think you can probably draw a formal line, but maybe a few examples would help people decide between filing a Feature Request issue in the Core queue, vs. and Idea issue in the Ideation queue. Anyway, issues can easily be moved between projects, so if a "too big" issue gets filed in Core, it can be moved to Ideation, and vice versa for "too small".

That makes some sense... although some "small" changes might still require Usability and subsystem maintainer sign-off... I guess that is only 2 though. :)

I think putting those examples from #12 in your eventual documentation on this would be helpful. Plus the ones from #10.

Isn't it easier to look at it as anything that needs product manager ownerage/sign off? These kinds of major changes would be things that affect "Drupal The Product". Place block is a major product change, a new theme is a major product change, adding workflow is a major product change, outside in is a major product change. Things that significantly affect those who use or develop with Drupal (which would include big time api changes, too.)

Well, it's not just Product Manager -- some of the Big Changes (TM) could be pure Developer Experience, and I think that is not the Product Manager's sphere. But possibly we could make a short list of people who would need to sign off (Product Manager, Architecture person or whatever they are called, etc.).

Simple and it seems, accurate too -- a good combination. ;)

It's a good thing that Drupal is becoming more responsive to change with this approach.

I've been part of large Web projects on various platforms, and have seen the good and the bad of the process.

Some initiatives I've seen that work well center around user-centered design, and Design Thinking (DT). In DT, it takes a more Agile-like approach--key takeaways for the process are rapid iteration of disposable prototypes. This allows the process to move forward in a rapid manner, allowing changes to be made based on feedback from user testing and validation without a major investment of time and resources. I've employed in on projects (with modifications), and it's served pretty well.

Tools such as InVision are excellent for accomplishing tasks such as what we're looking at, and require relatively minimal investment of time and technology, allowing participants of various skill levels to contribute. In addition, interactions can be added, without creating a single line of code. I use it regularly on my Drupal projects (and I can provide examples). Another benefit is that these prototypes can be shared with developers while ideating, which can help anticipate and deal with potentially major coding issues.

One of the best aspects of having visuals to share is that we're naturally visually-focused, and this gives a tangible quality to the process, drawing more people in to participate. Being active in the local design community, I hear sentiments that Drupal is more developer/enterprise-focused, and not design/UX-friendly. Here's a good way to counter that.

It would be nice to have somewhere in the proposal of the idea a clear reason why this isn't going into contrib and why it needs to be flushed out in core. That 'discussion' could avoid time upfront if thought through early in the idea.

And the question comes to mind. Should things go into contrib by default if they can go into contrib first?

+1 to #20

+1 on this move. The problem is the implementation, eh? :-)

I'd much rather see a new category in the project ('Ideation'?), versus having a separate project like 'Core ideas' or whatever.

It could be that someone proposes a broad Feature Request with a tag like 'Possible Ideation.' Then only a maintainer can promote it to Ideation category, which then has its own rules, and perhaps even an associated feature branch.

This kind of process has been totally missing unless you happened to be at a sprint somewhere where two people reached consensus and therefore the whole community is aligned. :-) So I'm glad it's happening.

Also, I'm coming at this issue from #2735005: Convert all Simpletest web tests to BrowserTestBase (or UnitTestBase/KernelTestBase) which @webchick tagged per #23. That issue is *not* an ideation-type issue. It's an implementation issue, because we've pretty-much already decided to move away from SimpleTest. To me it seems this kind of process would be for a large-scope feature or a re-architecting issue. Questions of *should* we commit to a process that's a lot of work, based on needs, rather than *how* to accomplish that work.

With that in mind, we'd want to be able to decide to close these Ideation issues. What metric will allow us to close on an Ideation issue without proceeding?

+1 to the proposal

> I'd much rather see a new category in the project ('Ideation'?), versus having a separate project like 'Core ideas' or whatever.

I'm also not clear on why this needs to be a separate project.

Also, would the recently added 'Plan' category not be suitable for this?

(But in general, yes, +1!)

+1 to the proposal.

I'd much rather see a new category in the project ('Ideation'?), versus having a separate project like 'Core ideas' or whatever.

Agreed. I don't see what the advantages of a separate project are, and there are several disadvantages:

1. Harder for maintainers (more places to keep track of).
2. Less intuitive for inexperienced contributors. (Who would ever guess that the Drupal core issue queue is the wrong place to discuss ideas for Drupal core?)
3. Not easily extendable to contrib projects who also may want this capability.

But if the proposal is just to add new issue categories like "Idea" and "Prototype" to the existing ones, it sounds great to me.

--

I would also suggest that you consider renaming "Plan" to something else as part of this ("To-Do List", perhaps, given that it's supposed to be for meta issues). This would prevent people from using "Plan" when "Idea" would be more appropriate. And the current "Plan" category encourages open-ended discussions which issue queues aren't good for. #2759849: Create a new user-facing core theme as part of a Drupal demo is an example of that - I don't think it's accurate to say that that issue was overrun by a discussion of implemention details, rather it looks to me like some people tried to do design planning (which makes sense) and others started to do implementation/long-term-maintainability planning (which also makes sense) but the problem is that it all happened in the same issue. An initial "Idea" issue (with no planning whatsoever) followed by two related and more-focused "Task" issues for specific decisions about design and implementation would probably have gone better.

Adding issue categories is technically a code change, but as far as I can tell it's a very simple and zero-risk one (this appears to be the entirety of the change needed to add "Plan") so I assume that option, if chosen, would not be significantly blocked in practice.

I would like to add 2 cents to some of the previous comments, for what it's worth. This particular thread seems focused on the need to separate ideas from implementation, and ensuring those ideas are properly vetted before adding them to core. There is of course a need to be cautious so as to not clutter core with things that belong in contrib. I would suggest that the user-facing core theme should first and foremost be a kind of base theme that showcases best practices, with plenty of internal documentation for adding additional subthemes in the contrib community. +1 to all comments suggesting an iterative feedback process on the usability of this theme while it is in development. No doubt, Drupal users deserve and expect a "flagship" core theme. (I found out too late that Bartik was not the polished and perfected theme I thought it was.) D8 should no doubt offer a very simply core theme that invites the contrib community to build upon. I for one would like a core theme that facilitates standardized components and libraries, perhaps even embedding Bootstrap tags into page.tpl files to facilitate customization, rather than placing too much of the load on blocks, and complex variables etc.

My simply takeway is:

1) the need for this theme to be UX tested from a developer point of view

2) the need to incorporate a simple set of best practices, components and libraries that can be readily expanded by the contrib community

@ptocco: Re

There is of course a need to be cautious so as to not clutter core with things that belong in contrib.

The problem is more that people would work on something for core for a long time, investing lots of sweat and tears and then see it rejected because it was not a good fit. We want to reject things that are not a good fit early to save people all that time and effort. (People could still do whatever they want in contrib as always). So its not really about avoiding adding things to core that do not belong there, because those would not be added just because someone worked on them for months, that is not the only deciding criteria. We are trying to set up a system to provide feedback as early as possible on the high level idea.

Re: your comments on the theme, this is not the issue to discuss specific proposals for core features, so your feedback would be at best lost here. Please post on the specific issue.

I came here from #2724845: Run Drupal as a service to allow non-http based API Initiative which seems like a prime example of a significant change that would benefit greatly from this process.

I would say that this should defiantly be a separate queue from the core queue. My primary thought on it needing to be its own queue is based on usability. The core issue queue is packed and intimidating, especially to new contributors. There are two aspects that make it difficult to use.

1. The component drop down is unwieldy and intimidating. I recently ran a Drupal Contribute meetup where I was attempting to teach people how to use the Drupal Issue Queue, most specifically the core issue queue. The workflow is logical and easy to understand, telling them what component to assign things to was by far the hardest part. It is a seemingly random list of (sometimes) similar words.
2. Issue tags are magic to new users. While great in concept, this is a feature of the issue queue that is very opaque to users. It requires secret knowledge of what keys do what. This is a great example of, "Its easy when you know how."

I a very much a +1 for this idea. It gives a single place for bad ideas to die and for good ideas to be put through their paces. Sometimes when dealing with core issues or ideas an issue can sit idle for a long time with little no feedback or worse yet, an idea can be spearheaded by a single person and a bunch of work can be done that never goes anywhere. Making this a clear workflow will really help set expectations and keep people from working on ideas that have fundamental issues but can also limit the bike-shedding that can stymie ideas early on.

Another reason I like this as a separate queue is, I would expect it to be the least amount of work for DA to pull this off and still have something usable. It would be good to get their feedback on this as well.

#34: That is an excellent and very insightful comment! Thank you!

@frob, on your two points:

1. I completely agree, but that's an existing usability problem - better to fix it rather than trying to avoid it forever :) I created #2797887: Add a default component 'Unknown' to the Drupal core issue queue now with a very simple improvement for that. (There's also a larger issue at #2660144: [Plan] Update core components.)
2. As far as I can see, no one discussed using issue tags above... so I'm not sure how that is relevant here? (But yes, in general issue tags aren't great.)

@David_Rothstein, thank you for opening that issue. It looks like progress is being made to clean up the core components select list.

My points, 1 and 2, are general points on why I don't think it would be a good idea to put this into the core queue. I still think it would be better in its own queue, that way things can be hashed out at a higher level.

edit, duplicate of #37

It is better as a separate queue because we could change our minds later without having added categories or even more components to the core queue.

#39: Categories are the same across all projects, but components would be special.

OT, but components should be using a multi group to be more usable.

I don't mind that much if its a different queue or the core queue as I never saw the big difference. So I am neutral on that and can see arguments for both.

I'm sensitive to the arguments against a separate queue. An earlier version of this was hashing out research, design and plan in groups.drupal.org/usability and the transition from there over to the core queue was far from smooth.

But this queue is closer to the core queue. You can follow issues, following them in the same tracker on your dashboard. I really think we should at least start with this separate queue so that:

- We are explicit in separating idea from execution, and get used to that idea and mode of working (for large/impactful changes at least)
- For as much as possible in a d.o. queue, it creates a safe space to work through ideas and designs, and those only.
- I hope this queue will as a result also make our product strategy discussions more transparent to the community

Lets try this out and see how it works. Looking forward to discuss some Ideas in Dublin!

RTBC + 1, lets try it with a separate queue

+1, now I wish I was going to Dublin.

How do we get to the project for this queue? When I look at an issue in the queue, the breadcrumb is to the normal Drupal project.

I found the ideas queue at https://www.drupal.org/project/issues/ideas.

Yeah, this project is lacking in breadcrumbs...