Greetings, folks!

I spent the past week and a bit talking with various initiative teams + the other core committers (some of which discussions are still ongoing). Coming out of that, here's a DRAFT of proposed product goals for 8.5/8.6(+) (see the previous 8.4/8.5 product goals draft for comparison) for your review.

Disclaimers

  • This is a proposal for areas of focus, in relative priority order. (Meaning, things closer to the top will generally get more time/focus/attention from committers than things closer to the bottom, where both are competing for such at the same time.) There is general consensus from committers that these are all good, important things to work on.
  • There is no guarantee of the proposed timelines, nor even that any of this actually happens. Drupal remains a "do-ocracy," and things only get done when someone actually does them. :)
  • There is also no guarantee that prioritized items will automatically get committed; the core gates and normal review process still apply.
  • This is still a draft; feedback is still coming in from implementation teams for feasibility, priorities may be reordered, other priorities added, etc.

General goals for 8.5+ (Feature freeze: Jan. 29, 2018)

  1. Complete previous goals that got close, but ultimately missed 8.4.0's cut-off.
  2. Focus on providing more user-facing functionality, since 8.4 was more of an API-focused release.
  3. Try to take on fewer priorities at the same time, in order to better ensure success.

Priorities

At a high-level, here are the priorities for 8.5/8.6(+), in descending order of importance (even though they're all important):

  1. Migrate
  2. Media
  3. API-First
  4. Layouts
  5. Workflow
  6. Outside-In
  7. Out-of-the-Box
  8. Community Initiatives
  9. Community Challenges

#1: Migrate #

Overall roadmap: #2735059: [META] Stabilize the older Drupal to newer Drupal migration system

Targeted for 8.5* (March 2018)

  1. Stabilize the Migrate module (stable core)
    #2859225: [policy, no patch] Mark Migrate (not Migrate Drupal) as Stable
    "As a developer, I want a stable Migration API in core from which to base my custom site migrations."
  2. Stabilize the Migrate Drupal module (stable core)
    #2905736: Mark Migrate Drupal as stable
    "As a developer and/or a site builder, I want a stable and functionally complete Drupal 6 and Drupal 7 to Drupal 8 migration path."
  3. Stabilize the Migrate Drupal UI (stable core)
    #2905491: Mark Migrate Drupal UI as stable
    "As a site builder, I want a simple UI—equivalent to update.php in Drupal 7 and below—that I can use to perform major Drupal version upgrades."

* Note that due to their importance in fostering Drupal 8 adoption, Migrate modules will be marked stable as soon as it is feasible to do so, rather than waiting for 8.5.0's release.

Targeted for 8.6+ (Date TBD; fall of 2018+)

  1. Add Migrate Tools to Drush (contrib)
    [#???]
    "As a developer, I want to be able to use Drush to perform migrations without the need of a separate module."
  2. Enhance Migrate Drupal UI with rollbacks and incremental migrations (stable core)
    #2687843: Add back incremental migrations through the UI / #2687849: Add back rollbacks on migrate_drupal_ui
    "As a site builder, I want to be able to re-run migrations multiple times to incrementally check progress, and roll back if I run into problems."

#2: Media #

Overall roadmap: #2825215: Media initiative: Roadmap

Targeted for 8.5 (March 2018)

  1. Upgrade contrib ecosystem to use the new Media API (contrib)
    #2860796: Plan for contributed modules with Media Entity API in core
    "As a site builder, I want all of my site's media functionality to be compatible with core's media functionality."
  2. Parity with File/Image modules (stable core)
    #2831940: Create file field widget on top of media entity / #2831941: [PP-1] Re-create Image field widget on top of media entity / #2831943: Use "rendered media" (not links) as default media field formatter; add modal to configure the used media view mode
    "As a site builder, I want my site users' file and image uploads to be compatible with other media.."
  3. oEmbed support (experimental core)
    #2831944: Implement media source plugin for remote video via oEmbed
    "As a content author, I want to be able to add remote media (e.g. YouTube videos) to my posts."

Targeted for 8.6+ (Date TBD; fall of 2018+)

  1. Media library (experimental core)
    #2834729: [META] Roadmap to stabilize Media Library
    ("As a content author, I want to be able to select from media I've already uploaded.")
  2. WYSIWYG integration (stable core)
    #2801307: [META] Support WYSIWYG embedding of media entities
    ("As a content author, I want to be able to insert media into my posts via the WYSIWYG editor.")
  3. Metadata improvements (stable core)
    #2836153: Improve metadata mapping UI on Media type form / #2862467: Add complex field mapping to media module
    ("As a site builder, I want a simple and intuitive tool to configure the metadata-to-drupal-fields mappings on my media entities.")

#3: API-First #

Overall roadmap: #2757967: API-first initiative
8.5.x roadmap: #2905563: REST: top priorities for Drupal 8.5.x

Targeted for 8.5 (March 2018)

  1. Improved documentation (Drupal.org)
    Improve https://www.drupal.org/docs/8/core/modules/rest/1-getting-started-rest-c... + https://www.drupal.org/docs/8/core/modules/rest/overview
    "As a developer, I want excellent REST + Serialization documentation on d.o that makes it easy for me to get started."
  2. Improved test coverage (stable core)
    #2824572: Write EntityResourceTestBase subclasses for every other entity type. / #2800873: Add XML GET REST test coverage, work around XML encoder quirks
    "As a developer, I want reliable JSON, HAL+JSON, and read-only XML serialization support for all entities in Drupal."
  3. File upload support (stable core)
    #1927648: Allow creation of file entities from binary data via REST requests
    "As a developer, I want to be able to upload files via REST, JSON API and GraphQL."
  4. Config entity support (stable core)
    #2300677: JSON:API POST/PATCH support for fully validatable config entities + #2869792: [meta] Add constraints to all config entity types
    "As a developer, I want to securely modify config entities—such as content types and taxonomy vocabularies—via REST, JSON API and GraphQL."
  5. Translation support (stable core)
    #2135829: EntityResource: translations support
    "As a developer, I want to be able to read and modify entity translations via REST."
  6. More granular permissions (stable core)
    #2824408: To be able to create/update/delete Term entities via REST, one should not have to grant the 'administer taxonomy' permission / #2808217: To be able to view Vocabulary config entities via REST, one should not have to grant the 'administer taxonomy' permission
    "As a developer, I don’t want to grant administrative permissions to read vocabularies or create terms."
  7. Better validation (stable core)
    #2821077: PATCHing entities validates the entire entity, also unmodified fields, so unmodified fields can throw validation errors / #2820364: Entity + Field + Property validation constraints are processed in the incorrect order
    "As a developer, I don’t want unmodified fields to throw validation errors when PATCHing entities via REST."
    "As a developer, I want understandable validation errors when PATCHING or POSTing entities via REST."
  8. Improved performance (stable core)
    #2765959: Make 4xx REST responses cacheable by (Dynamic) Page Cache + comprehensive cacheability test coverage
    "As a developer, I want faster REST responses for authenticated users."
  9. JSON API (stable core)
    #2836165: New experimental module: JSON API
    "As a developer, I want JSON API to be part of Drupal core"

Targeted for 8.6+ (Date TBD; fall of 2018+)

  1. See #2757967: API-first initiative

#4: Layouts #

Targeted for 8.5 (March 2018)

  1. Custom Landing Pages (experimental core)
    #2884601: Add a Layout Builder to core
    "As a content author, I want to create custom landing pages, by creating one or more page sections and choosing from pre-defined layouts within them."
  2. Sections-based UI for Field Layout (experimental core)
    #2874074: Introduce field type that renders arbitrary elements in a selectable layout
    "As a site builder, I want to be able to configure a default layout for my content types the same way I can configure an individual landing page's layout."

Targeted for 8.6+ (Date TBD; fall of 2018+)

  1. Stabilize Layouts (stable core)
    [#???]
    "As a site builder, I want core's layout functionality to be stable and usable in production websites."
  2. Form Layouts (experimental core)
    [#???]
    "As a site builder, I want the UI for laying out forms to match that of laying out content."

#5: Workflow #

Overall roadmap: #2721129: Workflow Initiative

Targeted for 8.5 (March 2018)

  1. Stable Content Moderation module (stable core)
    #2755073: WI: Content Moderation module roadmap / #2843494: WI: Workflows module roadmap
    "As a content author, I want to be able to create and moderate drafts of my content."
    "As a site builder, I want stable content moderation functionality that I can deploy in production."
  2. More revisionable things (stable core)
    #2725433: WI: Phase A: Use the revision API in more places
    "As a site builder, I want my content authors to be able to create drafts of lots of different types of content: taxonomy terms, menu links, and path aliases."
  3. Workspaces (experimental core)
    #2732071: WI: Workspace module roadmap
    "As a content author, I want the ability to stage collections of content (e.g. article, sidebar, and navigation) from one environment to another."

Targeted for 8.6+ (Date TBD; fall of 2018+)

  1. Full-Site Previews (experimental core)
    #2732081: WI: Phase G2: Full-site preview with Workspace UI
    "As a content author, I want the ability to see what proposed edits to collections of content (e.g. article, sidebar, and navigation) all look like so they can be reviewed as a whole before publishing."
  2. Undo functionality (experimental core)
    #2786135: WI: Phase E: Introduce Trash module
    "As a content author, I want the ability to easily undo deletion of content."
  3. Parent revision support (stable core)
    #2786133: WI: Phase B: Extend the revision API with support for parents
    "As a developer, I want one single way of handling hierarchies in Drupal."
  4. Conflict resolution (experimental core)
    #2867707: WI: Phase H: Conflict management and local workspace merging support
    "As a content author, I want my CMS to alert me if changes I'm about to make are going to overwrite others' changes, and give me the opportunity to fix them."

#6: Outside-In #

Overall roadmap: #2762505: Introduce "outside in" quick edit pattern to core

Targeted for 8.5 (March 2018)

  1. Place blocks from front-end (stable core)
    #2739075: [plan] Make Place Blocks module functionality part of the Block module (etc.)
    As a site builder and content author, I want to be able to place and reorder blocks from the front-end of my site."
  2. Generic off-canvas tray (stable core)
    #2784443: Move off-canvas functionality from Settings tray module into drupal.dialog.ajax library so that other modules can use it
    As a developer, I want the tray to be a generic component that I can integrate with without requiring the Settings Tray module."
  3. Stabilize Settings Tray module (stable core)
    [#???]
    As a content author, I want to be able to make configuration changes to my website from the front-end."
    As a site builder, I want the settings tray module to be stable so that I may rely on it in my production websites."

Targeted for 8.6+ (Date TBD; fall of 2018+)

  1. Unified authoring experience (stable core)
    #2732443: Finalize the behavior for triggering view/edit/build modes from the toolbar (and fix the "disappearing toolbar")
    "As a content author, I want a unified authoring experience between Quick Edit, Settings Tray, Contextual Links, and other UI-based editing functionality."

#7: Out-of-the-Box Experience (new!) #

Overall roadmap: #2847582: Out of The Box initiative

Targeted for 8.5 (March 2018)

  1. New demo profile (experimental core)
    #2809635: Create experimental installation profile #79582: Add example content to the experimental out-of-the-box demo install profile
    "As an evaluator, I want Drupal to demonstrate to me the basics of what it does out of the box with preconfigured functionality and sample content."
  2. New core theme for demo profile (experimental core)
    #2900720: Designs for the Out of the Box experience intiative / #2904243: Implement the Out of the Box designs as a theme for demo_umami
    "As an evaluator, I want Drupal to look stunning out of the box."

Targeted for 8.6+ (Date TBD; fall of 2018+)

  1. New core theme for demo profile (stable core)
    [#???]
    "As an evaluator, I want Drupal's superb out-of-the-box experience to be stable and recommended to me during installation."

#8: Other active, community-driven initiatives #

These are initiatives driven from community activity/interest, which have committer buy-in, and a team actively working on them.

Challenges raised by the community#

These are challenges that need an agreed-upon implementation strategy, buy-in from committers, and a team behind them to be elevated to the previous section.

Next steps

  1. Verify with initiative teams that this all looks more or less like what they told me. :D
  2. Get input into goals/priorities from other contributors/general public
  3. Run past other committers
  4. Finalize text and publish to https://www.drupal.org/core/roadmap.

This is open for comment until September 6, 2017, so that it can hopefully be adjusted/ratified at a core committer meeting the next day, just in time for DrupalCon Vienna!

CommentFileSizeAuthor
#52 migrate.txt2.99 KBrivimey

Comments

webchick created an issue. See original summary.

webchick’s picture

Issue summary: View changes
webchick’s picture

Issue summary: View changes
webchick’s picture

Issue summary: View changes
webchick’s picture

Issue summary: View changes
webchick’s picture

Issue summary: View changes
webchick’s picture

Issue summary: View changes
webchick’s picture

Issue summary: View changes
webchick’s picture

Issue summary: View changes
webchick’s picture

Issue summary: View changes
webchick’s picture

Issue summary: View changes
webchick’s picture

webchick’s picture

Title: *DRAFT* Proposed product goals for Drupal 8.4/8.5(+) core » *DRAFT* Proposed product goals for Drupal 8.5/8.6(+) core

Derp. :P

webchick’s picture

Issue summary: View changes

Jumpy links.

webchick’s picture

Status: Active » Needs review

Ok I think I'm finally done screwing with this now. ;)

star-szr’s picture

Issue summary: View changes

Minor link fix.

wim leers’s picture

Issue summary: View changes

Clarified API-First's "improved docs" target.

wim leers’s picture

Issue summary: View changes

Removed API-First's "GraphQL" target.

IMHO we can't do both JSON API and GraphQL at the same time. The JSON API contrib module is only now maturing (and now has thousands of sites using it). The GraphQL module is nowhere near that — it only has a few dozen sites using it. It needs to mature first before it can be considered for core inclusion. Putting it in core would also significantly lower iteration rate.

The entire API-First Initiative (or rather, the entire API-First ecosystem) is accelerating though. We solved much technical debt in 8.2, 8.3 and 8.4. Which means we can go faster and faster. More and more improvements for core's Serialization+REST modules are also helping the JSON API and GraphQL modules. Which means they can again iterate faster.

So: fingers crossed that we can get most of the 8.5 roadmap done. If we achieve that, then the future for both JSON API and GraphQL look very bright :)

marcoscano’s picture

Issue summary: View changes

Thanks for putting this together!

Replaced "(@todo: "As a foo, I want to be able to boo bar baz my metadata.")" with the real thing.

webchick’s picture

Hahaha, thank you for that. :D

@Wim Leers: Awesome, thanks for the clarification. Another thing that was in the 8.4/8.5 roadmap was OAuth2. Is that still something to add to 8.6+ or..?

webchick’s picture

Issue summary: View changes

Adding a handy-dandy table of contents.

webchick’s picture

Issue summary: View changes
webchick’s picture

Issue summary: View changes

Borked link.

webchick’s picture

Issue summary: View changes

Given the significant security benefits that would offer, I think that's a great one to explicitly call out in the community initiatives section. Good call! Thanks! :D

ifrik’s picture

For "Other community driven":

What about the issues about the admin menu structure? Mainly changing the Content menu item?
We talked about that in a UX meeting a month ago, and its pending "Needs product management review"
https://www.drupal.org/node/2862859

kristiaanvandeneynde’s picture

8-year old ticket with a working patch reporting in. Given how massive this change would be, it might deserve a spot on the list: #540008: Add a container parameter that can remove the special behavior of UID#1

jhodgdon’s picture

Issue summary: View changes

Adding a community initiative to the list in the summary... Hopefully it is OK for me to edit the summary directly; if you don't want this in there, just remove it. :)

Here's what I added as a community initiative:

#2592487: [meta] Help system overhaul - This is an Ideas issue about goals for an updated help system. There is a specific proposal, starting at comment #26 on this issue: taking the (working) sandbox Configurable Help module and getting it into Core. Main goals:
As a developer, I want to be able to bundle multiple easily-translatable (on localize.drupal.org) help topics, in a hierarchical structure including cross-links, with my module.
As a site builder, I want to be able to create and organize the help topics specific to my site.

wim leers’s picture

Issue summary: View changes

Updating the "config entity REST support" target for API-First per #2905563-5: REST: top priorities for Drupal 8.5.x.

ressa’s picture

Improve Composer experience / update Drupal core via GUI

With the move towards Composer, Drupal risks alienating a lot of non-developer users (60%? Number taken from #2874827-31: Drush 8.x doesn't install Drupal 8.4.x and Drush master doesn't install Drupal 8.3.x). See these two issues for more discussion on the topic:

Goal

As a site builder (non-developer), I want to be able to create and maintain Drupal sites easily via GUI-updates or Drush, as opposed to be forced to use the more complex Composer.

Maintaining Drush for Drupal 8

Also, there is the issue about Drush being replaced by Composer, which ties in with the above issue. Not sure if it belongs here, since it's not part of Drupal core or how exactly it should be approached... for more see #2874827: Drush 8.x doesn't install Drupal 8.4.x and Drush master doesn't install Drupal 8.3.x

Goal

Keep Drush working as it currently does for Drupal 8, without requiring Composer.

Proposed solution

Assist the maintenance of Drush for Drupal 8 hosted at Github, fx by sponsoring the maintainers or allocating resources towards developers submitting code to the project.

Question: Why not include Drush in Drupal 8 core, since it is such an essential tool for most site builders?

alison’s picture

+1 (well, plus way more than one) the concerns expressed in #30!! (Clarification: I think @ressa is referring to the pm-* commands in Drush?)

Furthermore, I'd argue that people who struggle with composer aren't even "non-developers," a whole ton of them are very skilled Drupal 7 (and Drupal-8-pre-8.4.x) developers/site builders. I cringe at and am saddened by the idea of pushing away these folks. I know I'm repeating myself, but seriously, this group of people includes a lot of very skilled Drupal site builders, what a disappointment and waste it would be to lose them. And, I really don't care for the attitude I've seen from some people that "developers" are fine in a composer-dependent lifestyle -- as if people who aren't comfortable with composer aren't developers, as if you're either a "developer" or a "I need push-button everything aka WordPress." I'm very confident in telling you, this perception is very out of touch. (In another thread, it was suggested that Drush users would obviously be fine using composer -- I know it was an honest lack of understanding from someone who's deep in core/module development efforts, but I think it's important to let y'all know, it's reeeeally not like that.)

To be honest, in the last few months I've gotten to a place where I'm using composer pretty damn well, all things considered. Overall I'm finally enjoying using Drupal 8, ish, but it's been and still is painful and frustrating, and I know I'm not alone. And I also don't expect pain-free, I don't expect zero learning curve. But, I feel very strongly (can you tell) that forcing site builders to adopt (traditional/command-line) composer or be squeezed out will be a huge bummer, for those Drupal developers and for the community/platform as a whole.

rivimey’s picture

A lot of worthy items in the list. Thank you webchick for assembling it!

I am particularly interested in the Migrate tasks - while it is hard to say the items selected are not worthy of being worked on, my current experience of Migrate is that it works very well for the content in Core but is very hard once you move outside that realm. I would therefore propose:

As a developer exploring moving a D7 site to D8.x, getting my content into my D8 site is a precondition to using D8.

Migrate: support migration of content for more of the commonly used Drupal 7 modules with easily-accessible contrib or core code.

... but you say "we only support core in migrate, and you're asking for contrib modules! no way!

Well, development on many D7 modules is grinding to a halt and I it seems that we probably have all the module-specific migrates we're going to get, unless we do something specific. I have seen up close that the cost of moving sites from D7 to D8 is a big drag on the ecosystem.

Is there also an opportunity for Migrate to suggest, and if needed install, the modules needed enable to support the old site's functionality based on what it finds in the old db? I know that there could be multiple ways: at this stage the important thing is getting the site working at all.

mile23’s picture

But, I feel very strongly (can you tell) that forcing site builders to adopt (traditional/command-line) composer or be squeezed out will be a huge bummer, for those Drupal developers and for the community/platform as a whole.

There are a lot of Composer-related issues, some of which have been sitting dormant for quite some time. Many of them accomplish the goals of allowing things like a UI for Composer stuff. For instance #2494073: Prevent modules which have unmet Composer dependencies from being installed and #2830880: Warn site admins when composer dev dependencies are installed inside of docroot

As a core dev who has poured probably too much energy into trying to make core work well with Composer, it really frustrates me when people say 'don't squeeze out us non-CLI users,' and then no work happens on the issues that give exactly the features they're asking for. Even just a +/-1 on the proposed fix counts as a vote that can keep the momentum going.

If you want these things, at least review issues. There's a composer tag, and a meta: #2002304: [META] Improve Drupal's use of Composer, part 1

And of course, if someone wants to make Drupal 8.5 into the I-never-have-to-think-about-Composer version, then work towards a community initiative to accomplish that goal. Please.

daffie’s picture

@Mile23: Let me start with that I think that it is great that Drupal uses composer.
There are a lot of benefits for Drupal by using composer. I am also very grateful for all the effort you have put into this. Really thank you.
The problem is for people that are unable to run the command line "composer install" is that when you talk about integrating composer usage in Drupal, the only thing that they read/hear is that they can do LESS with Drupal 8.
What for you is easy and was difficult for me and very difficult/impossible for others is installing/upgrading/using a command line interface and the command "composer install".
When those people try to do that and they do not succeed they get angry and frustrated. Especially when they hear/read from you about using composer more in Drupal.
What those people need is a tool with a user interface where they only have to push a button and all the composer install stuff get done.
If you want to read more about their frustration goto: #2845379: Provide optional composer integration but don't force users to understand how to use composer.
In that issue @Mixologic suggest using the open source desktop app https://electron.atom.io/ for creating the run composer tool via a UI.
If the non-developer people have such a tool they will stop complaining about composer and later they will find out what the benefits are of composer support in Drupal 8.
Do not get frustrated with the non-developer people, but try to understand their frustration.
And again let me thank you the effort you have put into expanding composer usage in Drupal (and PHPUnit testing).

webchick’s picture

Issue summary: View changes

Adding a new section to the roadmap of "Community challenges" (better name TBD) and adding the Composer stuff there. (Also the Config management stuff from last time.) I think this is how best to handle contentious issues like this where clearly there's an issue, but no clear agreed-upon path towards resolution, and/or no buy-in from committers on the best path forward, and/or no team to get it done.

Help system might better belong under there, as well. AFAIK this initiative doesn't have a team actively working on it (or am I wrong? either way, it's nice to see you @jhodgdon! :D). Actually, moving everything that's been suggested (except PHPUnit, which I know has all of these things) to that section for now, as a stop-gap. I'm pretty sure the entity access thing does have committer buy-in, but it's not really an "initiative" per se, so... best to wait on those for the committer meeting where they can be funneled into the right place.

In the meantime, though, please keep making suggestions. But please don't get into implementation details in this issue; leave that for the upstream issues.

groovedork’s picture

+1 for number 30

For me a core promise of Drupal has always been that it allows you to build amazing things via a GUI on generic shared hosting environments.

Currently Drupal 8 has been broken that promise. For example, I cannot install the Address module without using Composer, which implies the need to use a command line.

I feel this promise should be unbroken as soon as possible, maybe by incorporating composer into Drupal, and building a GUI on top of composer.

mile23’s picture

@groovedork, @daffie: Please start up a community initiative about Composer. Get organized, set up some requirements, make your case in one place within the proper scope, so the conversation can be meaningful.

@webchick: Thanks.

alison’s picture

@Mile23 re: #33 -- Those are some really good points, it hadn't even occurred to me that there might already be issues like the ones you mentioned, I will check them out, I may be able to contribute to those discussions, as you suggest. I echo @daffie's appreciation for your and many others' extensive work to integrate composer with Drupal and vice versa.

Thank you, @webchick, for noting/documenting the concerns, and also explaining how we can better contribute to the conversation in this issue. Speaking for myself, I feel course-corrected but respected and heard. (Not that I won't post a soliloquy in the wrong place again sometime, but, I appreciate your non-shutting-down guidance.)

(and everybody please make sure #32 isn't lost in the weeds!)

webchick’s picture

@alisonjo2786: Rock! :) Really hope to see you in those issues! Thanks for the awesome response.

Re #32. Thanks for the reminder of that. I've been pondering the past couple of days, but I'm not really sure what to do about it. :\ @rivimey, do you have specific suggestions on modules that are making the transition from D7 => D8 (more) difficult? What you'd like from the UI/Drush to help ease the problems you're running into..?

larowlan’s picture

I'm plus one on the general goals for the initiatives, although feel that the out of the box initiative having something ready for 8.5 is ambitious. So just want to make sure that there is scope for that to be delayed to 8.6 without and punitive measures.

alison’s picture

The two modules I'm the most bummed not to have in D8 are Feeds and Rules -- but I don't know if either of those are bc of core obstacles, or issues within themselves.

(Although, are ppl thinking they aren't dying to have feeds anymore bc they have migrate, even for continuous feed importing stuff?)

(Should I be looking at Feeds/Rules and then if they have core obstacles, come back here and say "i'd love for 8.5.x to have this thing addressed so that we can be closer to having these modules" -- is that an appropriate process, or does it not belong here at all, or is it fine how I've said it already, or?)

wim leers’s picture

Issue summary: View changes

After discussing with @e0ipso and @dawehner.

webchick’s picture

So just want to make sure that there is scope for that to be delayed to 8.6 without and punitive measures.

I've heard this general sentiment expressed a couple of times, so just want to address this head-on.

As stated in the disclaimers section above, the list of priorities/goals isn't a commitment to anything, so there's no fear of punishment/reprisal/what have you. Drupal remains a do-ocracy; only the stuff that people do will actually get done. :) What this document is attempting to do is write down a) what priorities the committer team thinks are important, b) what various teams working on those priorities think (right now, without the benefit of crystal balls :D) that they can accomplish in the next 5 months.

The conversations I've had with teams who set ambitious goals for 8.4 and then missed one or more of them haven't been "damn you, you are all horrible people, never show your face around here again, etc." ;) They've been a lot more along the lines of "that thing you folks are working on is STILL awesome! how can we as a committer team/community better help you achieve your goals this time?"

heddn’s picture

re #41 / #32: perhaps add and idea issue with your requirements for migrate in contrib: https://www.drupal.org/node/add/project-issue/ideas

I'm not sure if that is the right thing, but we can move the issue around to another queue once we get the requirements populated. This isn't a core thing, as rightly pointed out. But as migrate subsystem maintainers, we also are pretty active in the contrib space as well. We maintain drush tools to execute migrations(migrate_tools), sources from xml, json, csv (migrate_plus and migrate_source_csv, respectively), commerce_migrate, etc. We've also contributed code to paragraphs last January to build a destination plugin for it, but need to make that upgrade from d7 paragraphs/field collections more seamless. However, once we get outside of core, the work is scattered across multiple contrib projects with no central direction.

As far as feeds goes, there was a BoF at DC Baltimore organized by feeds folks and migrate subsystem people attended. The general plan that came out of it was to leverage migrate for the lower levels but add the "recuring" feed like functionality on top. But after that discussion, I'm not sure where things landed as the feeds folks haven't approached us with a lot of questions. Nor have I seen any plan issues opened in their issue queue.

adamps’s picture

@webchick, @ressa and others regarding composer I think there is a risk of a loss of vital function.

Here is a summary of the discussion at the end of #2874827: Drush 8.x doesn't install Drupal 8.4.x and Drush master doesn't install Drupal 8.3.x

  • #31 @xjm: Per the DA, 60% of Drupal core installations are from tarball, and therefore not managed by Composer
  • #39 @greg.1.anderson: Drush 8 has never officially supported Symfony 3. The current status of Drush 8 and Drupal 8.4.x is "happens to work"
  • Drush 9 does has removed "drush pm-upgrade" or some other pm commands

So the risk is that 60% of D8 users will lose the mechanism they currently use to apply security updates - any may only notice at the point that they need to apply one. It seems likely that many of these users have no experience of composer and would find it difficult.

On that basis I would like to see this issue elevated out of the community initiatives section. In order to keep D8 usable by the current community I think we need to

  1. #2906637: [META] Drush and core compatibility is fragile
  2. UNTIL WE CAN provide an alternative by simplifying the use of composer: #2845379: Provide optional composer integration but don't force users to understand how to use composer

Please can the leaders of the Drupal community who are planning releases take issue into account?

ressa’s picture

Thanks for creating #2906637: [META] Drush and core compatibility is fragile @AdamPS. I my opinion, not losing the upgrade mechanism which up to 60% of Drupal 8 administrators use, and securing Drush for Drupal 8 into the future (8.5, 8.6, 8.7, etc.) should be product goal #1 for Drupal 8.5/8.6(+) core.

webchick’s picture

In order to declare this as an official priority, we need a) a 'plan' issue in the Drupal core ideas queue that lays out what needs to happen and what the goals are (see others in the queue that are "active initiatives" for examples), b) sign-off from committers, and c) a team to get it done. I think we're lacking all three of these for the problems being outlaid about Composer for site builders. :\

adamps’s picture

@webchick Thanks very much for a quick reply.

I fear we are reaching something of an impasse.

I am a part-time site-builder and minor developer. I could possibly help with a) - certainly I could help define the goals. I doubt I'm the best person to lay out what needs to happen. Sorry I am not in a position to help with c). Even the time I have recently spent to raise carefully researched issues is a challenge in my schedule. I am lucky - I hopefully do have the technical skills to successfully switch to composer as it is. I will shortly cut my losses and switch to that instead of commenting on issues.

@moshe weitzman has corrected me: D8.4 does not even "happen to work" well for most users (the pre-built drush phar still contains Symfony 2 which could clash with Drush 8.4). I predict that the D8.4 release is going to cause substantial frustration and negativity in the less technical part of the Drupal community. There are already many issues such as #2863882: How do I migrate existing D8 site into a Composer managed structure?.

I feel that the leaders of the Drupal community, and those who are paid to develop core (I'm currently commenting in my free time!) have a choice

  • Acknowledge the issues, step forward and take on steps a/b/c on behalf of the site-builders and less technical community
  • Hope for someone else to do it, which might not occur, read the increasing media articles about how Drupal has abandoned the less-technical user, and watch people gradually switch to WordPress which I strongly doubt will have the same issues.

Please, please if "#7: Out-of-the-Box Experience" is important, then surely this is the most important issue in that category. Drupal 8.4 currently does not ship with a satisfactory mechanism to upgrade core.

adamps’s picture

I have raised #2906738: Drupal has lost any satisfactory upgrade and install mechanism to try to clarify and track the problem and raise awareness. This is not asking for a new feature, but rather pointing out that a crucial function that used to work has just stopped working for many users. Please, please can we try to find a way to prioritise this for future releases and identify some people who could work on it?

yoroy’s picture

Thanks for opening that issue @AdamPS. I think it's good to keep in mind that we are always *both* fixing broken things *and* building new things. One doesn't preclude the other, these are not either/or, but always and/and. This particular issue provides the outline of new things we want to achieve.

My please to you would be to not confuse leadership with being paid to work on core. Some people are indeed paid to work on new features because their sponsor would like to see *those features* happen. This does in no way mean that we can somehow reassign those people to work on other things. The composer challenge is real and we're not denying that. It still comes down to getting people and a plan together and start tackling it. And this is not the issue for that :)

adamps’s picture

@yoroy thanks for a polite response making valid points

@webchick It turns out there is already a "Drupal core ideas" issue #2477789: Use composer to build sites, but it is a Feature request. Should it be changed to a plan? Perhaps then we could update the summary of this issue to refer to #2477789: Use composer to build sites instead of the composer issues currently linked?

I take the point that the discussion should not be continued here. It would be great if people could join the conversation on the other issue, in particular to help determine if we can get "committer buy-in".

rivimey’s picture

StatusFileSize
new2.99 KB

webchick, sorry for delayed response. I decided to find out what happened for the upgrade of a curent D7 site I have, as an example, using the current 8.3.7 stable release. The source site is a modestly complex D7 site with 165 enabled modules, of which 128 are contrib. I downloaded into the D8 source tree all the D8 versions of the D7 modules before the migrate, so my D8 site has 111 enabled modules of which 56 were contrib.

A short text file derived from the migrate messages is attached, and there are over 140 modules that are described as 'missing' by the upgrader. What is worse, a lot of actual content, not just config, was thereby omitted from the migrated data with no obvious way to rectify this. I know this is not the place to get into details, but the upgrade process was somewhat alarming...

To pick out a few modules from this list is hard, but for this site I would select the following because their absence results in un-migrated content, not (just) config, behaviour, or presentation:

- field_collection
- addressfield
- phone
- name
- entity
- book
- entityreference
- node_reference ( (=> entityreference)
- nodequeue (=> entityqueue)
- statistics
- uuid
- xbbcode
- xmlsitemap

- and pathauto too!

My point : we need this to be a community effort, spanning as it does multiple modules from many sources, so it seems appropriate to raise it in the context of community initiatives.

heddn’s picture

Re #52, please open a new issue to start tracking these migrate issues. #2859304: Show field type migrations correctly in Migrate Drupal UI is what you mention in your post here and is a release blocker for migrate moving out of stable. There are a lot of false positives as many of these modules you mention have upgrade paths.

rivimey’s picture

heddn, I have done so, as #2906878: [Meta] Support for D7 -> D9 contrib migrate. Hope it's useful.

andypost’s picture

That proposal is mostly about "features"/improvements but what about legacy that nobody care?

We have core modules that nobody maintain and there's no mentions of them in such actionable issues!

For example
- #1255674: [meta] Make core maintainable
- #1803948: [META] Adopt the symfony mailer component
- #1813760: [meta] Clean up @todos and deprecated code
- #2319859: [META] Deprecate contents of database.inc
- #366511: Decouple Comment module from Tracker
- #1261120: Deprecate Tracker module in D10 and move to contrib in D11

I'm trying to think in what we need to deprecate or move to "legacy" core module

mile23’s picture

andrewmacpherson’s picture

Issue summary: View changes
Issue tags: +wcag21

WCAG 2.1 is currently at the W3C working draft stage, and it's a significant expansion to WCAG 2.0.

We're exploring Drupal's response at #2864791: Implement new Success Criteria from WCAG 2.1.

Accessibility is a QA gate in our project governance, rather than a "bold new features" strategic initiative. However accessibility has become a key product differentiator for Drupal, so I feel the large scope of the WCAG 2.1 expansion means we should give it a place in the product-goals plan for minor D8 releases.

I've added it to the "Challenges raised by the community" section here.

I expect we'll be addressing WCAG 2.1 issues over the next few D8 minor releases, say 8.6.0 to 8.8.0. I've outlined a detailed timeline in comments 18 and 21 in #2864791: Implement new Success Criteria from WCAG 2.1.

webchick’s picture

Okay, so 57 comments in, here's where we seem to be at:

  • There's not been much discussion, pro or con, on the "top 7" from this list. I'm going to assume that this means that the things under these headings are not very contentious, because people seem to otherwise have no problem speaking up when they see something awry. :) And if we can indeed achieve all of these things (which, it should be pointed out, would be quite an ambitious feat), 8.5 would be a pretty killer release: a stable migration system (huzzah!), the new core Media API being used out of the box with embedded remote third-party video as a first use case, Drupal's REST API spitting out JSON API by default (and having all of the fixes required to support that), a new drag and drop, section-based landing page and per-content type layout builder, stable content moderation functionality and experimental workspaces functionality (and the revisionable support required to support both), stable versions of the Settings Tray and Place Block functionality (moved to be part of Block module "proper"), and a new experimental "demo" profile/theme showing off what Drupal can do. :)
  • Last time around, the burning community challenge that bubbled up during the roadmap discussion seemed to be the incompleteness of the configuration management system in core, and the competing and incompatible solutions for these gaps in contrib. This time around, it's switched to Composer, and the challenges it presents to the site builder audience. Discussion is taking place at #2845379: Provide optional composer integration but don't force users to understand how to use composer and #2906637: [META] Drush and core compatibility is fragile.
  • Another community challenge raised is the lack of migrations once you get outside of core. While some of these are false-negatives caused by a bug in the UI—#2859304: Show field type migrations correctly in Migrate Drupal UI—others are genuine holes due to modules that have gone largely unmaintained in contrib. See #2906878: [Meta] Support for D7 -> D9 contrib migrate for discussion. Getting some kind of handle on this seems pretty crucial for ensuring sites can move from D6/D7 to D8.
  • There are also a number of community initiatives that folks want to raise the visibility of, some with active external time pressure—such as #2864791: Implement new Success Criteria from WCAG 2.1—and others that have been looming for months/years—such as #2592487: [meta] Help system overhaul and #1255674: [meta] Make core maintainable. For now, added these under a new section, Community Challenges. We need a solid, documented process for either getting these into the "official" priority list, or some other way to escalate them once they have the main requirements met (plan issue, community reviewed/RTBCed, approved by appropriate subsystem & topic maintainers/committers, with a team behind them to do the work). I've noted that as a @todo for the product management + committer team.
  • In the meantime, folks should understand that just because things don't yet appear in the "top 7" (which is always going to be a relatively small number, due to the bandwidth constraints on the committer team) does not mean they aren't important and does not mean you can't work on them! It just is a way of surfacing what committers will be spending the bulk of their time on, so expectations can be aligned accordingly.

It does feel like it's a bit premature to cut off discussion here, so leaving at needs review. On the committer meeting tomorrow I'll try and get sign-off on the "top 7" items, as well as some directional feedback on what they'd like done with the community challenges that have been raised. Please keep the feedback coming!

wim leers’s picture

It's interesting that this one has seen so many more community feedback than #2858592: *DRAFT* Proposed product goals for Drupal 8.4/8.5(+) core. Perhaps this one was publicized better? Or perhaps it's because Drupal 8's adoption is increasing, and therefore more people want to provide feedback based on their experience?

In any case: great to see all this feedback!

xjm’s picture

I like identifying the "Community challenges. I also like raising "Make core maintainable" as a "Community challenge" but it looks like it didn't make it into the summary with the others.

"Make core maintainable" has initiative leads, sort of: the release managers. :P

jhodgdon’s picture

RE #59 -- I don't remember hearing about the previous one, but this one I heard about. Sample of one, but I think "better publicity" is why this one has more comments/feedback.

xjm’s picture

It just is a way of surfacing what committers will be spending the bulk of their time on

Ah. Small quibble with this. I don't think committers necessarily spend the bulk of their time on product priorities. I'd just say that many committers prioritize product priorities first and in roughly that order when possible. :)

I'm not totally sure about the order -- does JSON API have a wider impact than a layout builder? But what's in the list makes sense and seems to reflect what I've seen from the current initiatives.

mile23’s picture

@xjm:

In terms of 'make core maintainable' I found a few ancient perma-postponed issues: #1255674: [meta] Make core maintainable #1273344: Establish heuristics for core feature evaluation

Are there any newer versions of these?

websiteworkspace’s picture

How about an initiative to repair the various bugs in, and restore features omitted from Drupal 8 that used to exist in Drupal 7, before adding additional/new features to Drupal 8?

For example, even the core statistics module does not work correctly and is currently receiving repair work:

https://www.drupal.org/node/2881999

A feature like the foregoing worked for many years in Drupal 7 but doesn't seem to work at all in Drupal 8 core.

Along with such bug fixes/repairs an initiative for a more thorough regression of all existing Drupal 8 core features might increase user base confidence in Drupal 8 stability.

websiteworkspace’s picture

The composer utility ...

How about an initiative to work with the developer(s) of the composer utility to create a "headless" (API only) composer utility (1) than can be integrated into Drupal core directly, so that a - headless composer - driven Admin GUI based, "single click code update", can finally be implemented for Drupal. Such a feature could even have an admin GUI form to configure update directives on a per contributed module basis too, using a "user friendly" version of the underlying "~1.0" and "^1.0" composer update type directives, so that an update from the admin GUI could adhere to fine grained updating directives per module.

If Drupal 8 wants an audience beyond coders and professional site consultancies, an admin GUI based core and contrib module update user-interface seems like a feature that needs to rise to the very top of every other feature under consideration for addition to Drupal core. Other CMS tools have provided GUI based code update functionality for years. The Drupal universe is totally capable of making this happen.

-

(1) if "headless drupal" is possible, using a headless API only composer utility is easily within reach. Other basic technologies like zip gzip have long been available as API components for integration elsewhere for years. Drupal needs headless integration of composer into core as quickly as possible.

mile23’s picture

@DrupalWebsiteBuilder: Please file an issue about this in Drupal core ideas (this project, not core). Thanks.

cilefen’s picture

websiteworkspace’s picture

@Mile23

Drupal 8 suffers deeply while the highest top priorities like one click admin GUI core / module update (with embedded composer logic) get tossed around like hot potatoes, without getting any actual coding attention (yes discussed infinitely on threads like #2845379) (1).

... priorities, priorities, as the buck passing continues ...

(1) personally, I use composer no problem, but the current state of Drupal 8 limits its potential user base and adoption rate ...

mile23’s picture

@DrupalWebsiteBuilder, @cilefen: That's exactly the problem. There's a bunch of 'we don't want to learn Composer' issues and comment hijacks, and too few 'here are specific behaviors what we want instead' issues. We need the latter rather than the former. That's why I suggested in #37 that people start up a more formal community initiative about Composer so that this can happen.

the buck passing continues

Not helpful. Please file an issue with specifics if you want action to occur. Otherwise, it won't.

alison’s picture

@Mile23 I believe @cilefen was just trying to point @DrupalWebsiteBuilder to an issue that could be a more appropriate place for their concerns.

@DrupalWebsiteBuilder: For the composer/dependency management GUI concerns, I strongly urge you to read through and join in on #2845379: Do not force users to understand how to use Composer and #2906637: Drush and core compatibility is fragile -- you can find clarification about the purpose of this thread in @webchick's comments #35 and #58, they were very helpful to me!

@webchick For #64, though, does that need its own issue, or does it fit in here as a "would like to see this in the goals list" thing, or?

yoroy’s picture

All things in this list here are more concrete and specific than wishes. They have a plan and a team working on things. Fixing regressions, stabilizing existing features are all good ideas but need a plan and a team to become a candidate for a list like this (See #47 above).

And to quote #58:

In the meantime, folks should understand that just because things don't yet appear in the "top 7" (which is always going to be a relatively small number, due to the bandwidth constraints on the committer team) does not mean they aren't important and does not mean you can't work on them! It just is a way of surfacing what committers will be spending the bulk of their time on, so expectations can be aligned accordingly.

I understand and appreciate that presenting a list like this also invites discussion on what's not on there. The best way towards momentum on those topics is to start an issue in https://www.drupal.org/project/issues/ideas?categories=All, attract likeminded people and form a plan.

webchick’s picture

Yep, perhaps this image would be helpful, for context: https://www.drupal.org/files/issues/NOLA%20ideation%20process%20discussi...

Ideas for Drupal go through a process of of Idea -> Prototype -> Planning -> Approved initiative roadmap (we are here) -> Implementation.

What this post is attempting to do is:

a) Take all the approved initiative roadmaps and put them in a prioritized list, along with detail as far as what each of the teams think they can accomplish in 8.5 vs. 8.6.

b) Ferret out from the community other things that we need to be mindful of, and start those things on the process of becoming approved initiative roadmaps, for later prioritization. (Hence the urging to move specific discussion on these into the Ideas queue, to get started along this process... because this particular issue will be closed down within a week or so.)

There also seems to be some misunderstanding in at least a few comments here about how core development works... Drupal is a "do-ocracy:" the things that get done happen because someone did them. There is no pool of core developers sitting around and eagerly waiting around for new suggestions of things to work on. Everyone who works on core works on it because either they, personally, are motivated to solve a problem, or because their employer has told them to be motivated to solve a problem. ;) Formulating a really solid plan with clear benefits is one way of attracting developers (or their employers) to become personally motivated to work on your problems. Taking it on yourself, and inviting others who share your concerns is another (and a lot more common). But issuing demands for others to take on something you care about, insulting others for not caring enough about things you care about, etc... this kind of thing will definitely not get it done, and in fact will often have the opposite effect.

Ok, with that out of the way. :) Let's keep discussing.

jhodgdon’s picture

I just want to say thanks for allowing "we have an idea but not a team" items to get into the Community Challenges section on this issue -- great solution to call attention to concerns that are not at the "we have a team and a concrete plan" stage yet, to hopefully get some attention from interested developers.

alison’s picture

Thank you again for all the explaining, everybody!

+1 @jhodgdon #73, well-put.

dqd’s picture

Wow... again. What an effort. Very much appreciated and deserving much respect. Many hard work and many well put observations are stated here. Thanks to webchick at first and Wim Leers, andypost, jhodgdon (I always forget the 2nd d :) ... ), heddn, xjm, all the others.

The only part I felt bogged down a moment was: "As a site builder, I do not want to have to choose between multiple options for uploading media." ... While having missed being part of the discussions before (my fault) these type of list items always make me worry. Because flexibility always strikes usability/simplification. But maybe I have to reread the linked issues above.

+1 @jhodgdon #73, well-put.

@Wim Leers (#59): Well, interest into such issues can have different reasons. It's like the often asked question regarding election statistics: "Which one is the happy or more reflective society? The one which goes to elections, or the one which does not?" ;-) (just joking)

jbitdrop’s picture

I would also like to thank @ all for the hard work on this. It's a very well put report and very useful to get a picture from the birds view. Thank you all!

xjm’s picture

Thanks @diqidoq. Regarding:

The only part I felt bogged down a moment was: "As a site builder, I do not want to have to choose between multiple options for uploading media."

This might be something being lost with the user story. The implementation detail of this is that site builders should be able to always use the new media file and image fields rather than the core file and image fields, in order to have all the added benefits of using the media system in the future. It's not a loss of flexibility; it's adding flexibility. So maybe we should reformulate that user story as:

As a site builder, I want my site users' file and image uploads to be compatible with other media.

Or something along those lines.

xjm’s picture

@webchick, I wonder if we can add a message to this and future roadmap proposals that says something like: "These are the new features we'd like to have. Your contributions will help us get there." Or something. Like on every single slide.

webchick’s picture

Issue summary: View changes

Um. No. :) No other open source projects add those kinds of disclaimers in every single place on their roadmaps lists of product goals, so let's not invent (more :P) Drupalisms here. I'm happy to add whatever other disclaimers to the top, though, in order to help better align user expectations with how core development actually works.

Adjusted the Media user story per #77, though.

webchick’s picture

Component: Idea » Proposed Plan

Also setting the proper component.

xjm’s picture

@webchick, I wasn't aware that this document followed any pattern found elsewhere; are there some examples of other projects'?

webchick’s picture

I don't think there's much uniformity across projects in terms of how this information is presented to non-technical end users, but "a set of issues that are targeted to be closed within a given release" is pretty universal.

GitHub milestones serve this purpose for most other projects; for example:

https://github.com/symfony/symfony/milestones
https://github.com/jquery/jquery/milestones

WordPress uses the same type of system in Trac:

https://core.trac.wordpress.org/roadmap

They, like us, re-adjust these targets periodically to move things out that aren't going to make it, move new things in that came up in between releases, etc.

webchick’s picture

Status: Needs review » Reviewed & tested by the community

There hasn't been much disagreement on at least the "top 7" list, so tentatively moving to RTBC.

Also spun off #2908405: Document the ideas process to try and help with some of the process questions here, so we can better help the "not top 7" get prioritized in some other fashion.

mile23’s picture

Issue summary: View changes

Just adding the one thing: #2866082: [Plan] Roadmap for Simpletest

dqd’s picture

@xjm, thanks for your help to get the picture. That was a good potrayal of the meaning, helping to get it the right way around. xjm++

@webchick, interesting read on https://www.drupal.org/node/2908405 and the linked google doc!

@Mile23, yeah, thanks for the reminder. :) I wanted to chime in there already and lost the trails in the woods.

ressa’s picture

@Mile23 has just created #2908394: Use Composer to build sites without forcing users to learn Composer, which has a goal and suggests a solution to the Composer challenges mentioned here, thank you @Mile23!

webchick’s picture

Status: Reviewed & tested by the community » Fixed

Ok!

Thanks a lot for all the great discussion here, folks! Onward to 8.5.0 awesomeness! :D

ressa’s picture

For everyone interested in the make-Composer-easier issue, amateescu is experimenting with an approach, where a service on drupal.org handles all the Composer dependency resolution, and it looks very promising: #2910136: Experiment: package PHP libraries in a single Phar file

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.