Background

Drupal 8 has so far seen many major improvements to our code (OOP, Symfony, etc) and our configuration (Config Management Initiative etc). But what about the third pillar — content? The Entity API team have done some amazing work on making our content APIs a lot more solid. But this work has not yet brought any new features for actually managing content, such as workflows, preview and staging. We propose an initiative with goals to improve in this area by including functionality in core based on what contrib modules have done for many years.

The scope of the roadmap is big. Very big. The work is likely to stretch over multiple years. There will be a dedicated team that will drive and coordinate the work needed.

A big focus for this initiative will be to get the necessary sign-offs ahead of time in order to make it possible (and sustainable) to work toward the big end goals of this initiative. You can read about the specific sign-offs in each separate phase issue.

For some additional background, see Dries’ blog post: http://buytaert.net/improving-drupal-content-workflow

Goals

Bring major improvements to Drupal’s content workflow, preview and staging capabilities. It will be done by primarily extending and improving aspects of the Entity API in core.

Much of the functionality that will be implemented will take heavy inspiration from modules that already are being battle tested in contrib, such as Entity, Multiversion, Workspace, Deploy and Workbench Moderation.

The concrete end-user features that are prioritized by this initiative are:

Must have

  1. Revision history for all content entities in core (phase A targeting 8.2)
  2. Moderation API (with MVP UI) (phase C targeting 8.3)
  3. Trash bin functionality (phase E targeting 8.3)
  4. Workspace API (no UI) (phase F targeting 8.4)

Should have

  1. Full-site preview with Workspace UI (phase G targeting 8.5)

Could have

  1. Cross-site content staging (phase X)
  2. Auto-save functionality and offline support (phase XI)

Process and where to find it

Each phase in the roadmap is designed to be as simple and contained as possible to allow it to be committed/merged into core without dependency on the next phase. All issues should be tagged with Workflow Initiative.

Roadmap, should-have

Each phase below will likely be broken out to its own meta issue once the overall plan is clear.

Phase A: Use the revision API in more places

The idea of phase A is it to implement and use our great revision API for all content entities in core, except users. This will provide a necessary foundation for any moderation, workflow and staging capability. No new features or functionality is being added to the Entity API in this phase.

Plan issue: #2725433: WI: Phase A: Use the revision API in more places

Phase B: Extend the revision API with support for parents

Phase B is where we start adding some new small/medium size features to the Entity API in core, primarily adding parent revisions (for nonlinear revisions). Tracking where revisions come from is important in order to be able to solve conflicts between revisions made in different environments.

Plan issue: #2786133: WI: Phase B: Extend the revision API with support for parents

Phase C: Content Moderation module (with MVP UI)

Depends on: phase A.

This phase will introduce an experimental module for moderating publishing state on individual content entities. An MVP UI will be provided.

Plan issue: #2755073: WI: Content Moderation module roadmap

Phase D: Enable archive/purge storage

Depends on: phase A.

Some historic background: http://drupal4hu.com/node/233

Phase D will deprecate the traditional delete method in our Entity Storage API and introduce two new methods; archive and purge. This is primarily important for replication and conflict management (see phase H) but also enables things like Trash module (see phase D).

Plan issue: #2725447: WI: Phase D: Enable archive/purge storage

Phase E: Introduce Trash module

Phase E will leverage the new archive and purge methods and introduce an experimental module in core; the Trash module.

Plan issue: #2786135: WI: Phase E: Introduce Trash module

Phase F: Workspace API and entity indexes (no behaviour changes)

Depends on: phase D and B.

These phases will introduce the underlying Workspace API along with entity revision indexes to keep track of what entity revisions exist in what workspace etc. These phases will lay the foundation for full-site preview which will be introduced in phase I.

Plan issue: #2732071: WI: Phase F: Workspace API and entity revision indexes

Phase G: Full-site preview with Workspace UI

Depends on: phase F

Super big phase. Lots of discussion needed here. This phase will use all previous APIs and implement full-site preview with a fully developed UI.
Initial wireframes here: https://marvelapp.com/dhb53d#11624482

Plan issue: #2732081: WI: Phase G: Full-site preview with Workspace UI

Roadmap, could-have

Phase X: Cross site content staging

This phase would implement support for cross site content staging. See phase H where single site content staging is implemented.

Implement CouchDB compatible REST endpoints to support cross site content replication. See Relaxed Web Services module for the current contrib implementation.

Phase XI: Autosave (big end-user feature)

Big phase. This will be a separate module. Lots of details still to be figured out, below are some ideas:

  • This would be the first feature that starts utilizing concepts of storing content offline in the browser’s localStorage. Some background: #153313: Form data is lost when using the browser's back button
  • Revisions could be auto-saved to localStorage and continuously replicated to backend. Changes would then be auto-saved even if the Internet connection is down. Rely on PouchDB and HTTP APIs from phase X
  • Auto-saved revisions could be marked as such and later be purged. Rely on APIs from phase C (deleted flag, revision purging).
  • Auto-save might be implemented together with concurrent editing. Rely on APIs from phase G (conflict management).

Related contrib projects:

Not in scope

Content migration

Although content migrations (between Drupal versions) will not be in scope, having the replication API available from phase G, step 2 could still help this effort significantly.

Related work

Contributed projects

Much of the above functionality currently sits in a number of contributed modules outlined at: http://www.drupaldeploy.org

Team and resources

As with any major work in core, the work will be carried out by the community openly in the issue queue and anyone is welcome and encouraged to take part of the process. In addition, there will be a few people that are going to be working on this as part of their day-to-day job as well, funded by a few independent sources:

Initiative coordinator

UX and frontend

Backend

Senior advisors

Communication and planning

Signoffs

This section still needs work. More approvals will be needed, to be figured out.

Below are the major architectural changes that this plan is introducing to Drupal core.

Signoffs needed from release and framework managers

  1. Amending the BC policy to define how entity type definition changes should be handled

Signoffs needed from framework manager and Entity API subsystem maintainers

  1. Changing core to be Create Read Archive Purge instead of Create Read Update Delete
  2. Making revisions nonlinear by introducing one or many revision parents
  3. Workspaces as an architecture to encapsulate site content for full-site preview and staging

Signoffs needed from UX topic maintainers

  1. Workspaces as a concept for full-site preview and staging (existing contrib and mockups available for conceptual review)

Signoffs needed from contrib maintainers

All contrib modules featuring this functionality are maintained by this team.

Signoffs given

The product, framework, and release managers have all signed off on this being an active core initiative. Each phase will undergo maintainer review individually on the respective sub-issues (above).

Comments

dixon_ created an issue. See original summary.

dixon_’s picture

Issue summary: View changes
dixon_’s picture

Issue summary: View changes
dixon_’s picture

Issue summary: View changes
timmillwood’s picture

Issue summary: View changes
timmillwood’s picture

Issue summary: View changes
larowlan’s picture

Great write up - really game changing stuff.
I think you should elaborate in the OP regarding the organisations/funded capacity etc that are/will support this work - both during the initiative and longer term. Knowing that there are dedicated resources for building and maintaining the large volumes of code we're talking about here will help your case, in my opinion.

Lee

dixon_’s picture

Issue summary: View changes
dixon_’s picture

Issue summary: View changes
dixon_’s picture

Issue tags: +Workflow Initiative
dixon_’s picture

Issue summary: View changes
dixon_’s picture

Issue summary: View changes
dixon_’s picture

Issue summary: View changes
dixon_’s picture

Issue summary: View changes
dixon_’s picture

Issue summary: View changes
dixon_’s picture

Issue summary: View changes
markdorison’s picture

Where is the discussion/planning around this initiative happening? Is there an IRC channel or Slack team? Could that information be added to the issue summary?

dixon_’s picture

Hey Mark, we will very soon be setting up weekly IRC or Google Hangout meetings where we are going to do lots of planning and discussions.
The meetings will likely happen in #drupal-entity on IRC.

I will certainly add that information to this issue once we have figured out the what/when.

dixon_’s picture

Issue summary: View changes
dixon_’s picture

Issue summary: View changes
dixon_’s picture

Title: Content Workflow Initiative » Workflow Initiative
Issue summary: View changes
dixon_’s picture

Issue summary: View changes
agentrickard’s picture

Issue summary: View changes
agentrickard’s picture

I updated the team a bit, since I act as Product Owner for Workbench Moderation and have maintainer access.

dixon_’s picture

Issue summary: View changes
dgtlmoon’s picture

Do the goals cover "reverting to a revision" ?

dixon_’s picture

Issue summary: View changes
stevector’s picture

Issue summary: View changes

Changing "Entity Moderation" to "Content Moderation" to fit #2725533-7: Add experimental content_moderation module

dgtlmoon’s picture

Entity to Content? why? All content types are Entities, for this to really cover all the cases, we should focus on generic Entity support, not just content, or am I missing something here?

stevector’s picture

It will be generic for Content Entities, which in the Drupal 8 parlance include Taxonomy Term, nodes, custom blocks, etc. So in Drupal 8 the key distinction is between content entities and config entities (which are exportable to yml through configuration management)

In the issue linked above Fago suggested the change to be consistent with Content Translation module.

dixon_’s picture

Issue summary: View changes
catch’s picture

Must have:
Trash bin functionality (phase D targeting 8.3)

Phase C will deprecate the traditional delete method in our Entity Storage API and introduce two new methods; archive and purge. This is primarily important for replication and conflict management (see phase H) but also enables things like Trash module (see phase D).

Phase D will leverage the new archive and purge methods and introduce an experimental module in core; the Trash module.

Neither phase C nor phase D seem blocking for phase E. So why are they 'must have' and why are they blocking intermediate phases when they won't be used until phase H?

It might be easier to reason about this if the structure of the phases was more of a flow chart - then it'd differentiate between work that's blocking vs. work that theoretically could be done as separate, parallel pieces.

dixon_’s picture

Issue summary: View changes
Status: Active » Needs work

@catch

You are right, the plan doesn't clearly surface the dependencies. So for now, I have notated some dependencies between the phases in the description above and moved Content Moderation module to phase C, and bumped the archive/purge stuff to phase D and E (since they don't depend on each other).

Also marking this plan as "Needs work" since we're still working through a few things.

I've been talking to @AniG on the team about how we better can communicate or visualize the plan, something like a flow chart as you suggested. I think something like that would be quite helpful indeed, so that people doesn't need to read through a wall of text and issue links. We've got some ideas that we hopefully can publish sometime soon.

dixon_’s picture

Issue summary: View changes
catch’s picture

So the early phases look good to me, but I still have concerns with the dependency tree for the later phases and some specifics.

Phase D, E: Enable archive/purge storage and introduce Trash module
Depends on: phase A and B.

[..]. This is primarily important for replication and conflict management (see phase H) but also enables things like Trash module (see phase D).
[..]
Phase F, G, H: Workspace API and entity indexes (no behaviour changes)
Depends on: phase D and it's dependencies.

This looks like an editing issue with things moving around. As far as I can tell Phases F and G don't depend on phases D and E at all. This is where a diagram would be useful.

Not clear why conflict resolution (H) depends on deletion/trash (G). And from there not clear why full site preview with workspaces (I) depends on (H). Specific sub-features like having forked entity branches might depend on it, but you can do full site preview without that (CPS in 7.x supports full site preview but not conflict resolution, it warns you if you fork an entity that you'll need to resolve any conflicts between changesets/workspaces manually, which is not perfect, but it makes it a soft rather than hard dependency).

Separately to this, there are two issues that are major, but would be critical if we had a UI for creating forward revisions in core already #2708735: Creating a draft removes published node from taxonomy view and #2684213: Preview on forward node revisions throws an error. Additionally taxonomy_index as it currently stands won't work with full site preview for the same reason that it's not revision aware, although that feels more 'major bug'.

Would be good to link those from the relevant sub-issues. I'm not clear exactly where the 'create a forward revision' UI comes in the different phases - is that with workspaces?

One thing that doesn't appear to have been discussed is path aliases. To have genuine full site preview and workflow, path aliases would need to be versioned independently and linked to workspaces - this again was a major pain point for CPS (it has some support but there's outstanding work). #1553358: Figure out a new implementation for url aliases is probably the closest to a discussion of this, although we might need to start again with new issue. This also makes forward revision support very difficult - i.e. we can't have a situation where you create a new draft of a node in a workspace, change the path alias on the node edit form, and that then bleeds into the live site, becomes a data-integrity issue then.

catch’s picture

Oh one more thing. Alex Bronstein asked why comments need to have revisions.

I can see adding revision support for comments - i.e. we can edit comments on Drupal.org, but there's no audit trail for what was previously in them. However I can't see adding a UI for forward revisions, workflow and content staging support for comments - that's something that would normally be explicitly excluded from any sort of workflow. So comments in particular feels like it doesn't block the later stages in the same way that say taxonomy terms would, although it could still be done independently.

Similarly with users - this is excluded entirely at the moment, but I could see us actually wanting to add revisions support for some things on users, but definitely not forward revisions by default (forward revisions of a username or e-mail address is a bit frightening for a start). Might be worth an issue to discuss separately.

larowlan’s picture

In addition to what catch notes about path, the same would be true for menu, which doesn't use a real entity reference. Speaking of entity reference, does this mean we need revision support back for that?

catch’s picture

We don't need revision support for entity reference afaik. With workspaces, if you edit multiple different entities, including with references to each other, then all those edits are treated as a set and will be published at the same time.

In the end, there can only be one active revision of an entity at a time, so if node A references entity B, then on the published version of the site, it's always going to reference the 'published' entity B - not older or newer revisions.

In draft workspaces, entity B could be any number of different revisions, and preview works because you're loading that revision within the workspace. Similarly node A could reference different entities with different revisions and that'll be fine too.

Commerce has a use case for referencing revisions - so it can track the price of a product at the time it was added to a cart/ordered - even if the product then later changes, but that's a specialised use-case not relevant to previews.

One area that's tricky in Drupal 7 with forward revisions is field collections, field collections are versionable, but the field collection field itself also references specific revisions, so in CPS's case at least, the field collection entities had to be taken out of workflow support, and by virtue of the parent entity reference changing, the right one gets shown at the right time without being separately tracked. There's no way to edit field collection entities outside of the node form, so it works for that case, but it would fall down with any entity that can be referenced from more than one place - then you have multiple versions of an entity showing at the same time. Field collection might be different again in 8.x and we don't have that in core, but need to make sure it doesn't explode due to core changes.

Files is also tricky - in core we don't allow changing a specific file upload, you can only create a new file upload and change the reference. The file on disk is obviously not versionable, and it would need to have the same path for URLs. So while metadata like description/title might be versionable, the file itself can't be. But a specific entity referencing a file could reference a different, newly uploaded file and that'd work even if files aren't versionable.

yoroy’s picture

One of the tasks in #2725449: WI: UX prototypes for Trash module is figuring out which entity types should get revision (UI), and how to make that call. Looks like we're making a start with it in here :)

dixon_’s picture

Issue summary: View changes
yoroy’s picture

dixon_’s picture

Here are finally the notes from the meeting on Thursday June 9 with Tim Millwood, alexpott, catch and I:

Major points

  • Scalability/performance of the workspace replication was discussed a lot. Currently (in the plan, and in the contrib modules), all entities on the site are duplicated in order to exist in other workspaces. This is a simple solution that works around some challenges and potential bugs (e.g. avoiding altering all the queries, access bugs, revision bugs etc). That said, we should look into a solution similar CPS where the storage handler alters all the queries so that only changed entities need to be explicitly in the workspace. Unchanged entities would be joined/queried from the live workspace
  • Replication needs to be in a single transaction. Currently not possible/feasible because of the above scaling issues.
  • Move in Content Moderation (phase C) earlier as an experimental module, as it will help/force us to uncover many issues with forward revisions etc. Possibly even for 8.2
  • We can be aggressive with committing experimental modules. It's even ok (and even good) if the module surface known and major bugs in core.

High-level architecture, and scope changes

  • We agreed that as part of this initiative we will try to tackle some relevant part of the technical debt accumulated around the Entity/Revision API
  • We need a plan for path aliases, this will become a problem when forward revisions are used more
  • As already noted in the plan, we should try getting as much as possible into experimental modules
    • In experimental modules we can be aggressive and swap out services, storage handlers etc.
  • We will need a functionality for rebuilding indexes (solution for out-of-sync problems). However, the sequence index can't be rebuilt.
  • Entity types to NOT make revisionsable
    • Users - Document more clearly why we're not doing this (mainly security/privacy concerns).
    • Files - We can let them behave as by default, e.g. new files will be created as part of replication etc. Also, there's an initiative already touching this a lot.
    • Contact Messages - Really not needed. And there's an initiative that's already touching this a lot, so we'd avoid stepping on each others toes.
    • Comments - Use case debatable. Can as well skip this for now.
  • We should move the entity storage migration into the schema handlers and EntityDefinitionUpdateManager
    • Need to re-iterate on this discussion with @dawehner and @amateescu who are in favour of update hooks (I believe)
  • We should not go down the route that CPS did, i.e. trying to support archive/revert of workspaces. It's very messy, complicated, and comes with many hidden issues.
  • The "live" or "default" workspace should not be a content entity
    • This would ensure that the system works without any workspace content entities at all
    • Performance would improve slightly as we can avoid a condition on all entity queries etc.

Re-organizing phases

  • We need to split up phase D (the new archive/purge storage) a bit more. E.g. after phase A we should configure most entity types to save new revisions. This will slowly make people more used to having revisions around in their storage, even before we get to phase D.
  • We should make the conflict API (phase H) not depend on archive/purge storage (phase D).
  • Add step/phase for MVP conflict management, e.g. block replication if a conflict would be introduced.

Other notes

  • Config and workspace will need a lot of thought to not make things confusing. Kevin Oleary's suggestion from New Orleans to only allow users to edit config in the live workspace is good (to make it clear that config is not staged between workspace).
  • We should more clearly document the reasons why we are not adding users/accounts to the scope of this initiative
  • JSON API and GraphQL also needs something similar to entity indexes - coordinate with that initiative
moshe weitzman’s picture

Config and workspace will need a lot of thought to not make things confusing. Kevin Oleary's suggestion from New Orleans to only allow users to edit config in the live workspace is good (to make it clear that config is not staged between workspace).

It would be pretty nifty if you could change config in a non-default workspace and have that import into default later. I guess the changes would be implemented as config overrides when in a non-default workspace. This is out of scope for workflow initiative for sure.

dixon_’s picture

Issue summary: View changes
agentrickard’s picture

@moshe Yes, it's a side issue, and note the problems detailed in #2677124: Allow discovery of configuration UI. It does not appear that we can build a config workspace UI without core changes. (Unless we had some mechanism for determining the context of the configuration during the save routine.)

Feel free to move the discussion over there.

webchick’s picture

webchick’s picture

xjm’s picture

Issue summary: View changes
Issue tags: -Needs product manager review

It's probably kind of obvious by this point, but this evening @webchick, @catch, @effulgentsia, @alexpott, and myself all agreed that this is an active core initiative. ;) Hooray!

Since the proposed work has such diverse scope, each individual roadmap step will undergo maintainer review as it comes up. The current status of the first phases:

I'm also removing the product manager tag on behalf of @webchick and @Dries. @webchick stated verbally that this she agrees with this product direction for core, and Dries also stated that it was a strategic direction for Drupal in the DrupalCon New Orleans keynote.

Many thanks to the WI team and especially @dixon_ for responsiveness, dedication, and patience as we have worked through planning for this significant initiative and on blazing some trails in terms of how initiatives in general will work in Drupal 8 minor releases.

xjm’s picture

xjm’s picture

Removing issues already covered by specific phases from the summary/related issues as well as issues that were identified as not in scope for the initiative in the triage session discussed in #2725433-24: WI: Phase A: Use the revision API in more places.

xjm’s picture

Issue summary: View changes

During the triage session, we identified the following issues as things that were not part of phase A, but needed to be revisited for later phases:

Adding the first to the related work in the summary. The latter three are already included.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.0-beta1 was released on August 3, 2016, which means new developments and disruptive changes should now be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

dixon_’s picture

Issue summary: View changes

I have now split up all phases (and merged some) into separate plan issues to make it easier to review and provide approval on each phase of the initiative.

dixon_’s picture

Issue summary: View changes
webchick’s picture

Issue tags: +Drupal core ideas

Hi there! This is an issue that we'd ideally like to move to the new "Drupal core ideas" queue when/if #2785735: [policy, no patch] Agile process for bigger changes in core (Part A: Ideation) goes through (hoping for the next week or two). If you could read that issue and provide your thoughts on that, that would be great!

webchick’s picture

Dries asked me this question the other day, and I confess I did not have a good answer, so...

Is there a hard dependency on undo functionality prior to workspaces/site-wide preview? Of the two, the second is definitely a hugely in-demand feature, so it seems a bit unfortunate the current projection says we won't see anything user-facing for that until 8.5 (spring 2018).

timmillwood’s picture

@webchick - "undo functionality" (#2725463: WI: Add StatusItem field type to store archived state) is not a dependency of workspaces per-se, however when replicating content between workspaces we'd be unable to replicate "deleted" entities.

I think we can start working on many things in tandem, I opened #2784921: Introduce the concept of workspaces a few weeks back to start discussing the implementation details of workspaces. I think workspaces should be defined in an experimental module, but then we have the issue that it needs to be an override/alter rather than baked into the entity api.

The biggest issue I see with "we won't see anything user-facing for that until 8.5 (spring 2018)" is that even if we add workspaces in 8.3 there is no reason to make them user facing until content can be replicated between workspaces. Replication has a lot of dependencies, currently in contrib these are things like indexes and serializers. Also in contrib when we replicate we end up with one copy of each entity per workspace. This is a bit of a contentious issue already in contrib, so I'm not sure how well it will sit in core.

webchick’s picture

Thanks for the additional details!

The biggest issue I see with "we won't see anything user-facing for that until 8.5 (spring 2018)" is that even if we add workspaces in 8.3 there is no reason to make them user facing until content can be replicated between workspaces.

Hm. I wonder if that's true. I think for a number of use cases, just the ability to, say, preview changes to both a node *and* a block and/or menu at the same time on the same site would be light-years beyond what we can currently do. I don't know if we can just get that functionality without all the mountain of dependencies, though. But moving content across workspaces/sites is a far more advanced feature that I don't think most people know they need yet. :)

Anyway, not sure the above is even remotely feasible/accurate, but essentially the gist I'm trying to get across is we should always look for lower-hanging fruit with user-facing tidbits we can expose along the way, even if the penultimate functionality is going to take months and months. In other words, referencing https://cdn-images-1.medium.com/max/800/1*qINsG4WH_BDN-viMJUH6Ng.png, I think users would be super happy with getting a scooter in 8.3 and a motorcycle in 8.4 while we build the car for 8.5, if it's feasible. :)

dawehner’s picture

While having actual working workspaces would be super nice, I would like to put a small reminder here in place. Stuff needs time, and focusing and too many devices at the same time let's you end up with a bicycle with one wheel and a scooter without breaks. There are for example still a lot of tasks in #2725433: WI: Phase A: Use the revision API in more places which needs to be done, before adding workspaces really make sense.

balsama’s picture

Of the two, the second [site-wide preview] is definitely a hugely in-demand feature, so it seems a bit unfortunate the current projection says we won't see anything user-facing for that until 8.5 (spring 2018).

Related, the Lightning distro plans to release our own experimental module "Lightning Preview" bundled with Lightning it's 8.1.0 release this fall that will include what we're calling the "Workspace Preview System": #2677938: Implement a Workspace Preview System

webchick’s picture

Project: Drupal core » Drupal core ideas
Version: 8.3.x-dev »
Component: entity system » Plan

Moving over to the new Drupal Core Ideas queue while we get this into shape.

maxilein’s picture

I'd like to bring the idea of "Composite Entities" to everybody's attention: https://www.drupal.org/node/2822476

hchonov’s picture

Issue summary: View changes

Adding a reference to the new autosave_form module for D8.

xjm’s picture

Component: Plan » Active Initiative

This initiative is active and has been awhile. :)