Problem/Motivation

We had several goals in adopting the experimental module process:

  1. To speed up the pace of innovation in core and ship user-facing goodness sooner rather than later.
  2. To get testing and feedback on proposed features from a larger audience than just core developers.
  3. To reduce the development process overhead for adding significant features (i.e., fewer difficult rerolls, 400K patches, 400-comment issues, etc.) while keeping core shippable 100% of the time.
  4. To avoid the need for feature branches (because feature branches lack infrastructural support, would require additional committer overhead and processes, and could have N-dimensional integration conflicts with each other).
  5. To provide expected timeframes and success criteria for completing features, so that we could choose remove the features if they did not stabilize in time or if they were not successful as core features.
  6. To provide earlier core visibility, review, and feedback for initiatives through the core issue queue, so that core requirements, standards, and best practices are an ongoing part of the development process rather than a huge added burden when the feature seems to be complete.

What is working well

Through this process, we have shipped performance enhancements (Big Pipe), stable core Layout and Media APIs, and important accessibility and usability improvements (Inline Form Errors, Settings Tray), among others. Each of these required multiple-months' development timelines, which were achieved while keeping core shippable 100% of the time. The release deadlines associated with experimental modules have been a major factor in this achievement. And because the work takes place in the core queue directly, core committers look at these new proposed features earlier and more often, leading to better expectation management and education on core processes to initiative contributors.

What is not working well

However, one area where we seem to not be achieving our goals is in getting feedback from site builders and developers outside the core development community. While part of this can be attributed to the core issue queue's higher barriers to entry (a problem that is out of scope for this discussion), a major part is because of confusion for many—even core developers—on what expectations they should have with experimental modules.

Core committer Gábor Hojtsy recently did an ad-hoc Twitter poll to get feedback on the experimental modules process. While this poll mainly reached Drupal contributors and enthusiasts, the feedback gathered there mirrors much of what we've heard in the issue queue, at events, within Drupal shops, etc.

Namely:

  1. Users expect features that ship with Drupal core, experimental or not, to offer a higher level of stability than experimental modules currently provide.
  2. No amount of dire warnings (in the UI, in release notes, in keynote presentations, etc.) has changed this expectation.
  3. The "alpha" and "beta" modifiers are not effective, because site builders are frequently forced to use modules in contrib that are labeled "alpha" and "beta" in production. They therefore expect much more stability than core experimental modules have when using these labels (for example, an "alpha"-level experimental API is almost expected to break BC; that means we're responding to developer feedback about it, and this is a good thing).
  4. This expectation mismatch can have negative consequences for the product of Drupal 8 itself, as experimental modules are seen as part of core, and therefore "Drupal 8" is seen as breaking API backwards-compatibility or not providing an upgrade path, versus a particular experimental feature. This undercuts trust in Drupal 8's stability and in our plans to deliver a continuous upgrade path to all future releases.
  5. The warnings themselves create negative perceptions and some Drupal distributions or projects have considered hiding them from users.
  6. Experimental modules sometimes "compete" with contrib solutions, resulting in a perceived year-plus-long uncertainty and slowdown of production fixes for the contrib module.
  7. In practice, trying to remove an experimental module is still very disruptive for sites and for core development, and furthermore causes high stress for community members, so much so that it's almost not doable.
  8. It's been difficult to stabilize experimental modules in a timely fashion, and they contribute to ballooning core technical debt and overhead (since adding an experimental module to core is still a fairly big commitment).

Proposed resolution

In order to respond to both site builder and developer feedback about core experimental modules in Drupal 8, the committer team is planning to make the following changes starting with the Drupal 8.5.x branch (which is open for development now):

  1. Experimental modules that have alpha-level stability will only be committed to development branches of Drupal 8.
  2. If an experimental module has not achieved at least beta-level stability by the alpha1 release of the branch itself, its development will move to the next development branch and the module will not be included in the alpha release. (Or, alternately, the module may be removed from core if there's no clear path to stability.)
  3. Once an experimental module reaches beta stability, we now require (a) upgrade paths, and (b) backwards compatibility (or a deprecated BC layer) for any API improvements.

For example, if an initiative team wanted to add a new experimental module to core for their desired feature, they could introduce a patch that met the requirements for an experimental module and it could be committed to 8.5.x as an alpha-stability experimental module. However, by 8.5.0-alpha1 (the week of January 17, 2018), either the module would need to be in "beta" level stability (which means its API and data model would be considered stable, with upgrade paths and API BC layers provided when needed), or it would be left in the 8.6.x branch, but removed from the 8.5.x branch before tagging the alpha. 8.5.0 would ship without this new functionality, but (if completed in time) it could be available in the 8.6.0 release.

These policy changes are intended to address a number of frustrations with the existing experimental module process (especially points 1-4 in the first section) and to better meet expectations for non-core site builders and developers. By only shipping with beta-level stability or higher experimental modules in stable releases of core, and by requiring core developers to provide BC layers and upgrade paths for their beta-level experimental modules, we are hoping to reach the best compromise possible.

These improvements do not address all the concerns we've heard about experimental modules (for example, slow stabilization, competition with contrib solutions, etc.), so we will continue to gather feedback and refine the process as we switch to this new practice.

Remaining tasks

  • Get community feedback (open until September 6, 2017).
  • Discuss what approach we would take if a module does reach beta stability, but eventually turns out to not be a good fit for core. (E.g.: Mark the module hidden, deprecate its code, add hook_requirements() messages about there being no further development and any available alternatives.)
  • Tweak policy, based on feedback (if required).
  • Update existing documentation.
  • Announce policy change at http://groups.drupal.org/core.

Comments

webchick created an issue. See original summary.

webchick credited catch.

webchick credited xjm.

webchick credited yoroy.

webchick’s picture

This was mostly @xjm and @catch, @yoroy also reviewed, so adding them to the credit thinger.

xjm’s picture

Issue summary: View changes

(Just fixing markup.)

xjm’s picture

Issue summary: View changes

Clarified in the IS that this would apply to 8.5.x-dev, i.e. now (instead of after 8.5.0 which is six months away). The first time there would be any obvious difference would be shortly before 8.5.0-alpha1 in January, but we should set the new expectation with any experimental module we add before then (to make sure we're adding it at the time that works best for them).

xjm’s picture

Issue summary: View changes

Also correcting which difficulties the policy is meant to address; somehow it got changed from the draft we edited.

xjm’s picture

Issue summary: View changes

Adding references to the current definitions for alphas and betas of experimental modules.

xjm’s picture

Issue summary: View changes

And correcting the year on the window for feedback (we are not waiting until Sept. of 2018). ;)

webchick’s picture

Also correcting which difficulties the policy is meant to address; somehow it got changed from the draft we edited.

That's because there was a new item added at the top of the "goals" list courtesy of @yoroy. :)

webchick’s picture

While I'm not seeing any comments here yet, the general feedback at https://twitter.com/webchick/status/901152551734661120 has been overwhelmingly positive so far. Yay!

dnotes’s picture

Well, this is a fantastic summary of all the fuzzy uneasiness I have had about experimental modules. As a site builder and developer outside the core community, these policy changes seem like they would eliminate the uncertainty involved in potentially installing an experimental module, which I have never done to this point because I didn't want the technical debt. I can't be certain that I'll give feedback after installing the modules, but using the module is generally a prerequisite for feedback.

8thom’s picture

linking https://www.drupal.org/node/2873800 as a potential solution.

Rather than removing experimental modules before tagging an alpha release we could just remove the reference in core's composer.json. This still allows you to add the reference back if you understand the risks and need/want to use the experimental module in an alpha state.

It seems counter productive to allow experimental modules to "compete" with contrib solutions, we should rather have a process in which we promote contrib to core.

timmillwood’s picture

I really like the proposed plan.

To me the best way to "remove" experimental modules which are < Beta from core would be to mark them hidden, because it takes more than just reverting a commit or removing a directory from /core/modules to truly remove a module. Marking them has hidden also allows people to still use them within a stable release if they truly want to. As some people have used Content Moderation in production whilst still in Alpha and contributed awesome patches and an unofficial upgrade path.

Example:
In the Workflow Initiative we have been discussing the Workspace module, how we'd like to add this in 8.5.x, but we don't think we'd get it to Beta until 8.6.x, therefore the stable releases of 8.5.0 would include Workspace module as hidden, then unhide in 8.6.0 once it reaches Beta.

rachel_norfolk’s picture

I like it. A lot.

Is there something we can add to the policy that makes it more clear how an experimental module might migrate to contrib, should it eventually not make it as a a full core module?

budda’s picture

Great summary of the current issues with D8 core releases.

I've never really seen the point with including half baked modules in Core over just letting them mature in the land of contrib. Just seems a bit confusing about having half-way modules.

Why not just ditch experimental modules altogether and maybe have something that recommends new modules that may be of interest to download and add in? I know if you don't keep an eye on new modules being published it's easy to miss useful nuggets you may never know your site could have benefited from.

xjm’s picture

Thanks everyone for the feedback so far. For @timmillwood:

To me the best way to "remove" experimental modules which are < Beta from core would be to mark them hidden, because it takes more than just reverting a commit or removing a directory from /core/modules to truly remove a module.

Well, we are very strict on experimental module code being isolated (nothing outside an experimental module's directory should ever reference any of its code; we even push back patches for containing changes outside the module directory and require them to be removed.

In the Workflow Initiative we have been discussing the Workspace module, how we'd like to add this in 8.5.x, but we don't think we'd get it to Beta until 8.6.x, therefore the stable releases of 8.5.0 would include Workspace module as hidden, then unhide in 8.6.0 once it reaches Beta.

We did discuss this approach as well, but the fact is that removing even a hidden module is still an API break and quite disruptive in practice. People can still declare dependencies on them, write code extending their code, etc. no matter how much we plead for them not to. Right now, for example, we've actually made the Media module hidden even though it is stable because its UI isn't ready to be part of the core product. Similarly, BC breaks in Migrate a year ago frustrated a lot of people even though it had no UI at the time and it could only be used by "experts" with Drush.

An answer for @budda's question:

Why not just ditch experimental modules altogether and maybe have something that recommends new modules that may be of interest to download and add in?

That would be a great feature for Drupal in general (and it's often been requested), but it would be a lot of work to build. There are other issues proposing such features on their own merit, for sure. However, it doesn't really help with most of the other goals for experimental modules (such as providing committer review, not having to maintain 400K patches nor manage merges between 14 different feature branches, etc.). So, they do still serve additional purposes beyond end user testing.

xjm’s picture

Is there something we can add to the policy that makes it more clear how an experimental module might migrate to contrib, should it eventually not make it as a a full core module?

Well, this has always been our practice for things that get removed from core. I guess that doesn't really answer your question though. :) I added a sentence https://www.drupal.org/node/2657056/revisions/view/10612521/10616320 since it's just documenting something that people who were around for D7 and/or D8 core development kinda know already.

Also, regarding:

It seems counter productive to allow experimental modules to "compete" with contrib solutions, we should rather have a process in which we promote contrib to core.

Moving stuff to/from contrib isn't entirely a cakewalk since Drupal core "owns" any of the namespaces within a major release once module code ships in a core release. In fact, the machine name of the core experimental module that everyone calls "the Workflow module" is technically workflows, precisely because of such namespacing problems. It also caused pain for BigPipe when it made the jump from contrib to core. The only reason Media is avoiding some nightmarish upgrade path issues is that we renamed the module to media instead of media_entity alongside the move to core.

While namespacing core modules, using composer for core, etc. are longer-term solutions to that problem, they're not entirely practical for the shorter term because we still have to provide BC throughout D8's life for what we already have. Meanwhile, the proposal in the summary is something we can implement with very little overhead and no extra development.

jonathanshaw’s picture

Good plan.

The "alpha" and "beta" modifiers are not effective, because site builders are frequently forced to use modules in contrib that are labeled "alpha" and "beta" in production. They therefore expect much more stability than core experimental modules have when using these labels (for example, an "alpha"-level experimental API is almost expected to break BC;

Yeah. Guilty as charged. The truth is I would happily use a core alpha module, and then be frustrated if it blew up on upgrade, precisely because of the high expectations of core.

The plan helps protect us from our own folly.

effulgentsia’s picture

site builders are frequently forced to use modules in contrib that are labeled "alpha" and "beta" in production

For those who don't know, #2870295: Concerns about beta and RC contrib without security coverage is an issue to try to change this practice. That's unrelated to this issue, but just cross-linking here, so that people reading that sentence don't mistakenly think that that's how it must always be.

DamienMcKenna’s picture

A huge +1 from me.

Sam152’s picture

The only downside from this proposal is not getting feedback from people using these experimental modules in a real-world environment until beta. To test these on an existing site, you would have to upgrade to HEAD first. @timmillwood is correct in saying a lot of the feedback from content_moderation came from the users of the module when it was in alpha.

On the other hand, feedback in beta is still going to be valuable. I do think there should be some provision to make reasonable BC breaks during this time (do we already have this anyway?), after the likely swell of user feedback during beta. Upgrade paths will make this process dramatically less user-hostile but I think it'd be somewhat frustrating to ship an API from a beta experimental module that could have been improved during the 6/12 months of beta.

DamienMcKenna’s picture

I would expect that people who are comfortable running experimental modules and able to deal with the problems of an unsupported upgrade path would be able to juggle a patch.

Sam152’s picture

In most cases that's probably true. In the case of content_moderation there have been blockers that are improving other core subsystems or APIs, which could potentially be a pain to backport. At that point, you'd have equal chance of bugs being because of version incompatibilities vs experimental-ness.

In any case, I'm still +1 but just pointing out there is a small cost.

xjm’s picture

Regarding #23 - #25, beta already means "MVP featureset with upgrade paths and API stability", so we would not be changing that. Experimental modules with beta stability or higher should follow the same BC and deprecation best practices that we use in core generally. We might be more likely to err on the side of "no BC" if a BC layer is in itself risky, adds serious performance cost, etc., but this will be the same as it is under current policy. (We just haven't had that many experimental modules stay at beta for very long, aside from Migrate.)

Beta experimental modules could include string changes, UI changes, etc. between patch releases, so that could be a way to get feedback on them. We might also allow internal API changes between patch releases. (For stable code, changes to e.g. render array structure are only allowed in minor releases.) We could also potentially allow API additions between patch releases while in beta, so long as the API addition was not too disruptive.

larowlan’s picture

I'm +1 on this

yoroy’s picture

Status: Active » Reviewed & tested by the community

Thanks for all the responses, looks like there's strong consensus on making these changes and making them now.

As for the "getting feedback early" part, we need to improve there regardless of the specifics of how we handle experimentals. I think it's working for getting implementor feedback (developers mostly, a bit of site building). Real solid feedback from site builders, content creators on the usability and usefulness of these new features did not magically appear through this process. We should get serious again about usability testing, but that's another topic, in the mean time lets proceed with these changes here and (re)build trust in what we release to the masses :)

Marking this rtbc.

Cottser’s picture

+1, to me this is a big step in the right direction.

Sam152’s picture

Re: #26, makes sense. The movement as I've seen it has been towards @internal (implicit or explicit) for anything that hasn't proved itself as api-worthy, so this potentially mitigates my misgivings.

mpdonadio’s picture

Status: Reviewed & tested by the community » Active

IS says community feedback is open until 2017/09/06.

I think I am mostly +1 to this in spirit.

However, I think I am neutral to this in practice. My biggest worry is that outside the champions (ie, the people writing the code) we have limited resources who can get experimentals from alpha -> beta -> stable. We have added core committers, who are doing a great job with the RTBC queue, but there are other people with limited bandwidth who hold the keys to the core gates.

Rather than removing code, though, I would prefer something in settings.php analogous to the composer min stability setting, eg

$settings['min_module_stability'] = 'stable'; // or 'beta', 'alpha', 'dev'

We have something similar for the test modules, and this could also solve some concerns that people have with contrib. This will allow people to use alpha quality code w/o needing to patch, but make it harder for them to do it. I think this would allay my concerns.

xjm’s picture

@mpdonadio, I think that's a good suggestion for beta+ stability modules even if we implement the current proposal for alpha modules.

mpdonadio’s picture

Dries’s picture

+1 from me as well.

I like the suggestion in #31 too - it could be a 80% solution for alpha too.

xjm’s picture

Status: Active » Reviewed & tested by the community

Dries, catch, cilefen, effulgentsia, Cottser, larowlan, webchick, and I discussed this issue today and agreed we will go ahead and adopt this policy change for the 8.5.x cycle. (Something like #2907162: Add $setting to specify minimum module stability allowed to be installed could be added alongside it in the future.) Hopefully this will both resolve some community concerns with having experimental modules and allow us to have more confidence in them when we do change them. Thanks everyone for the feedback!

xjm’s picture

Updated the handbook page:
https://www.drupal.org/node/2657056/revisions/view/10627418/10630477

Leaving RTBC for final review, since some of the specifics about beta modules, removed modules, etc. were interpolated as it were rather than present in the original summary.

mpotter’s picture

This topic summary does a great job describing the current pain with experimental modules and I really like the proposed solution.

xjm’s picture

Status: Reviewed & tested by the community » Fixed

Marking fixed now. Thanks everyone! We'll use this process for experimental modules we add going forward.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.