Latest core conversations
- Baltimore: https://www.youtube.com/watch?v=wwh-3jVqaWU
- New Orleans: https://www.youtube.com/watch?v=_rAB8DJnc8Y
Background
Drupal 8 has so far seen many major improvements to our code (OOP, Symfony, etc) and our configuration (Config Management Initiative etc). But what about the third pillar — content? The Entity API team have done some amazing work on making our content APIs a lot more solid. But this work has not yet brought any new features for actually managing content, such as workflows, preview and staging. We propose an initiative with goals to improve in this area by including functionality in core based on what contrib modules have done for many years.
The scope of the roadmap is big. Very big. The work is likely to stretch over multiple years. There will be a dedicated team that will drive and coordinate the work needed.
A big focus for this initiative will be to get the necessary sign-offs ahead of time in order to make it possible (and sustainable) to work toward the big end goals of this initiative. You can read about the specific sign-offs in each separate phase issue.
For some additional background, see Dries’ blog post: http://buytaert.net/improving-drupal-content-workflow
Goals
Bring major improvements to Drupal’s content workflow, preview and staging capabilities. It will be done by primarily extending and improving aspects of the Entity API in core.
Much of the functionality that will be implemented will take heavy inspiration from modules that already are being battle tested in contrib, such as Entity, Multiversion, Workspace, Deploy and Workbench Moderation.
Process and where to find it
Each phase in the roadmap is designed to be as simple and contained as possible to allow it to be committed/merged into core without dependency on the next phase. All issues should be tagged with Workflow Initiative.
Kanban board: https://contribkanban.com/sprint/WorkflowInitiative
Roadmap, should-have
Each phase are broken out to its own planning issue. Phases aren't necessarily fully sequential. Some things can and will be done in parallell.
Phase A: Use the revision API in more places
The idea of phase A is it to implement and use our great revision API for all content entities in core, except users. This will provide a necessary foundation for any moderation, workflow and staging capability. No new features or functionality is being added to the Entity API in this phase.
Plan issue: #2725433: WI: Phase A: Use the revision API in more places
Phase B: Extend the revision API with support for parents
Phase B is adding support for tracking parent revisions (for nonlinear revisions). Tracking where revisions come from is important in order to be able to solve conflicts between revisions made in different environments. This phase only required in order to make progress with Phase H.
Plan issue: #2786133: WI: Phase B: Extend the revision API with support for parents
Phase C: Content Moderation module
This phase will introduce an experimental module for moderating publishing state on individual content entities. Plan issues:
Phase D: Enable archive/purge storage
This phase is no longer needed because of the implementation decision made for Phase E.
Plan issue: #2725447: WI: Phase D: Enable archive/purge storage
Phase E: Introduce Trash module
Phase E will leverage the new workflow module and implement Trash as a new status. Some historic background: http://drupal4hu.com/node/233
Plan issue: #2786135: WI: Phase E: Introduce Trash module
Phase F: Workspace API and entity indexes
Due to the way it's been decided to handle experimental modules in core, everything for Workspaces will be implemented as one experimental module in the next phase G.
Phase G: Introduce Workspace module
This phase is split into two sub-phases. The primary purpuse here is to introduce the ability to preview and stage changes, within multiple "workspaces" on a single site.
- Experimental workspace module with MVP UI: #2732071: WI: Workspace module roadmap
- Full UI implementation: #2732081: WI: Phase G2: Full-site preview with Workspace UI
Phase H: Workspace conflict management
This phase will extend the Workspace module to provide conflict management for content changes in separate workspaces.
Plan issue: #2867707: WI: Phase H: Conflict management and local workspace merging support
Roadmap, could-have
Phase X: Cross site content staging
This phase would implement support for cross site content staging. See phase G where single site content staging is implemented.
Implement CouchDB compatible REST endpoints to support cross site content replication. See Relaxed Web Services module for the current contrib implementation.
Phase XI: Autosave
Big phase. This will be a separate module. Lots of details still to be figured out, below are some ideas:
- This would be the first feature that starts utilizing concepts of storing content offline in the browser’s
localStorage
. Some background: #153313: ckeditor input is lost when using the browser's back button - Revisions could be auto-saved to
localStorage
and continuously replicated to backend. Changes would then be auto-saved even if the Internet connection is down. Rely on PouchDB and HTTP APIs from phase X - Auto-saved revisions could be marked as such and later be purged. Rely on APIs from phase C (deleted flag, revision purging).
- Auto-save might be implemented together with concurrent editing. Rely on APIs from phase G (conflict management).
Related contrib projects:
Not in scope
Content migration
Although content migrations (between Drupal versions) will not be in scope, having the replication API available from phase G, step 2 could still help this effort significantly.
Related work
- #2700261: allRevisions() entity queries ignore non-revisionable fields
- #2490136: Enable revisions by default
- #2707255: All content entities should inherit baseFieldDefinitions
- #2705433: Node should implement RevisionLogInterface
- #2705389: Selected content entities should extend EditorialContentEntityBase or RevisionableContentEntityBase
- #1812202: Add UUID support for entity revisions
- #2690747: [PLAN] Create an index of UUIDs
- #1776796: Provide a better UX for creating, editing & managing draft revisions.
- #2465901: [META] Make entity revision translation work
- #2068063: Change "Save and keep un-/published" buttons to a "Published" checkbox and an included "Save" button
- #2350939: Implement a generic revision UI
Contributed projects
Much of the above functionality currently sits in a number of contributed modules outlined at: http://www.drupaldeploy.org
Team and resources
As with any major work in core, the work will be carried out by the community openly in the issue queue and anyone is welcome and encouraged to take part of the process. In addition, there will be a few people that are going to be working on this as part of their day-to-day job as well, funded by a few independent sources:
Initiative coordinator
UX and frontend
Backend
Senior advisors
Signoffs
Each individual plan issue will list the required sign-offs.
Comments
Comment #2
dixon_Comment #3
dixon_Comment #4
dixon_Comment #5
timmillwoodComment #6
timmillwoodComment #7
larowlanGreat write up - really game changing stuff.
I think you should elaborate in the OP regarding the organisations/funded capacity etc that are/will support this work - both during the initiative and longer term. Knowing that there are dedicated resources for building and maintaining the large volumes of code we're talking about here will help your case, in my opinion.
Lee
Comment #8
dixon_Comment #9
dixon_Comment #10
dixon_Comment #11
dixon_Comment #12
dixon_Comment #13
dixon_Comment #14
dixon_Comment #15
dixon_Comment #16
dixon_Comment #17
markdorisonWhere is the discussion/planning around this initiative happening? Is there an IRC channel or Slack team? Could that information be added to the issue summary?
Comment #18
dixon_Hey Mark, we will very soon be setting up weekly IRC or Google Hangout meetings where we are going to do lots of planning and discussions.
The meetings will likely happen in #drupal-entity on IRC.
I will certainly add that information to this issue once we have figured out the what/when.
Comment #19
dixon_Comment #20
dixon_Comment #21
dixon_Comment #22
dixon_Comment #23
agentrickardComment #24
agentrickardI updated the team a bit, since I act as Product Owner for Workbench Moderation and have maintainer access.
Comment #25
dixon_Comment #26
dgtlmoon CreditAttribution: dgtlmoon commentedDo the goals cover "reverting to a revision" ?
Comment #27
dixon_Comment #28
stevectorChanging "Entity Moderation" to "Content Moderation" to fit #2725533-7: Add experimental content_moderation module
Comment #29
dgtlmoon CreditAttribution: dgtlmoon commentedEntity to Content? why? All content types are Entities, for this to really cover all the cases, we should focus on generic Entity support, not just content, or am I missing something here?
Comment #30
stevectorIt will be generic for Content Entities, which in the Drupal 8 parlance include Taxonomy Term, nodes, custom blocks, etc. So in Drupal 8 the key distinction is between content entities and config entities (which are exportable to yml through configuration management)
In the issue linked above Fago suggested the change to be consistent with Content Translation module.
Comment #31
dixon_Comment #32
catchNeither phase C nor phase D seem blocking for phase E. So why are they 'must have' and why are they blocking intermediate phases when they won't be used until phase H?
It might be easier to reason about this if the structure of the phases was more of a flow chart - then it'd differentiate between work that's blocking vs. work that theoretically could be done as separate, parallel pieces.
Comment #33
dixon_@catch
You are right, the plan doesn't clearly surface the dependencies. So for now, I have notated some dependencies between the phases in the description above and moved Content Moderation module to phase C, and bumped the archive/purge stuff to phase D and E (since they don't depend on each other).
Also marking this plan as "Needs work" since we're still working through a few things.
I've been talking to @AniG on the team about how we better can communicate or visualize the plan, something like a flow chart as you suggested. I think something like that would be quite helpful indeed, so that people doesn't need to read through a wall of text and issue links. We've got some ideas that we hopefully can publish sometime soon.
Comment #34
dixon_Comment #35
catchSo the early phases look good to me, but I still have concerns with the dependency tree for the later phases and some specifics.
This looks like an editing issue with things moving around. As far as I can tell Phases F and G don't depend on phases D and E at all. This is where a diagram would be useful.
Not clear why conflict resolution (H) depends on deletion/trash (G). And from there not clear why full site preview with workspaces (I) depends on (H). Specific sub-features like having forked entity branches might depend on it, but you can do full site preview without that (CPS in 7.x supports full site preview but not conflict resolution, it warns you if you fork an entity that you'll need to resolve any conflicts between changesets/workspaces manually, which is not perfect, but it makes it a soft rather than hard dependency).
Separately to this, there are two issues that are major, but would be critical if we had a UI for creating forward revisions in core already #2708735: Creating a draft removes published node from taxonomy view and #2684213: Preview on forward node revisions throws an error. Additionally taxonomy_index as it currently stands won't work with full site preview for the same reason that it's not revision aware, although that feels more 'major bug'.
Would be good to link those from the relevant sub-issues. I'm not clear exactly where the 'create a forward revision' UI comes in the different phases - is that with workspaces?
One thing that doesn't appear to have been discussed is path aliases. To have genuine full site preview and workflow, path aliases would need to be versioned independently and linked to workspaces - this again was a major pain point for CPS (it has some support but there's outstanding work). #1553358: Figure out a new implementation for url aliases is probably the closest to a discussion of this, although we might need to start again with new issue. This also makes forward revision support very difficult - i.e. we can't have a situation where you create a new draft of a node in a workspace, change the path alias on the node edit form, and that then bleeds into the live site, becomes a data-integrity issue then.
Comment #36
catchOh one more thing. Alex Bronstein asked why comments need to have revisions.
I can see adding revision support for comments - i.e. we can edit comments on Drupal.org, but there's no audit trail for what was previously in them. However I can't see adding a UI for forward revisions, workflow and content staging support for comments - that's something that would normally be explicitly excluded from any sort of workflow. So comments in particular feels like it doesn't block the later stages in the same way that say taxonomy terms would, although it could still be done independently.
Similarly with users - this is excluded entirely at the moment, but I could see us actually wanting to add revisions support for some things on users, but definitely not forward revisions by default (forward revisions of a username or e-mail address is a bit frightening for a start). Might be worth an issue to discuss separately.
Comment #37
larowlanIn addition to what catch notes about path, the same would be true for menu, which doesn't use a real entity reference. Speaking of entity reference, does this mean we need revision support back for that?
Comment #38
catchWe don't need revision support for entity reference afaik. With workspaces, if you edit multiple different entities, including with references to each other, then all those edits are treated as a set and will be published at the same time.
In the end, there can only be one active revision of an entity at a time, so if node A references entity B, then on the published version of the site, it's always going to reference the 'published' entity B - not older or newer revisions.
In draft workspaces, entity B could be any number of different revisions, and preview works because you're loading that revision within the workspace. Similarly node A could reference different entities with different revisions and that'll be fine too.
Commerce has a use case for referencing revisions - so it can track the price of a product at the time it was added to a cart/ordered - even if the product then later changes, but that's a specialised use-case not relevant to previews.
One area that's tricky in Drupal 7 with forward revisions is field collections, field collections are versionable, but the field collection field itself also references specific revisions, so in CPS's case at least, the field collection entities had to be taken out of workflow support, and by virtue of the parent entity reference changing, the right one gets shown at the right time without being separately tracked. There's no way to edit field collection entities outside of the node form, so it works for that case, but it would fall down with any entity that can be referenced from more than one place - then you have multiple versions of an entity showing at the same time. Field collection might be different again in 8.x and we don't have that in core, but need to make sure it doesn't explode due to core changes.
Files is also tricky - in core we don't allow changing a specific file upload, you can only create a new file upload and change the reference. The file on disk is obviously not versionable, and it would need to have the same path for URLs. So while metadata like description/title might be versionable, the file itself can't be. But a specific entity referencing a file could reference a different, newly uploaded file and that'd work even if files aren't versionable.
Comment #39
yoroy CreditAttribution: yoroy commentedOne of the tasks in #2725449: WI: UX prototypes for Trash module is figuring out which entity types should get revision (UI), and how to make that call. Looks like we're making a start with it in here :)
Comment #40
dixon_Comment #41
yoroy CreditAttribution: yoroy commentedComment #42
dixon_Here are finally the notes from the meeting on Thursday June 9 with Tim Millwood, alexpott, catch and I:
Major points
High-level architecture, and scope changes
Re-organizing phases
Other notes
Comment #43
moshe weitzman CreditAttribution: moshe weitzman at Acquia commentedIt would be pretty nifty if you could change config in a non-default workspace and have that import into default later. I guess the changes would be implemented as config overrides when in a non-default workspace. This is out of scope for workflow initiative for sure.
Comment #44
dixon_Comment #45
agentrickard@moshe Yes, it's a side issue, and note the problems detailed in #2677124: Allow discovery of configuration UI. It does not appear that we can build a config workspace UI without core changes. (Unless we had some mechanism for determining the context of the configuration during the save routine.)
Feel free to move the discussion over there.
Comment #46
webchickRaw notes from Dev Days Milan: https://docs.google.com/document/d/1-_AgTnZu9e2iAuEpnp262mguFInNaiIcSxsM...
Comment #47
webchickNote that a lot of the action is now happening over at #2755073: WI: Content Moderation module roadmap and #2725533: Add experimental content_moderation module.
Comment #48
xjmIt's probably kind of obvious by this point, but this evening @webchick, @catch, @effulgentsia, @alexpott, and myself all agreed that this is an active core initiative. ;) Hooray!
Since the proposed work has such diverse scope, each individual roadmap step will undergo maintainer review as it comes up. The current status of the first phases:
I'm also removing the product manager tag on behalf of @webchick and @Dries. @webchick stated verbally that this she agrees with this product direction for core, and Dries also stated that it was a strategic direction for Drupal in the DrupalCon New Orleans keynote.
Many thanks to the WI team and especially @dixon_ for responsiveness, dedication, and patience as we have worked through planning for this significant initiative and on blazing some trails in terms of how initiatives in general will work in Drupal 8 minor releases.
Comment #49
xjmI also updated the core roadmap for this:
https://www.drupal.org/node/2716063/revisions/view/9732467/9834825
Comment #50
xjmRemoving issues already covered by specific phases from the summary/related issues as well as issues that were identified as not in scope for the initiative in the triage session discussed in #2725433-24: WI: Phase A: Use the revision API in more places.
Comment #51
xjmDuring the triage session, we identified the following issues as things that were not part of phase A, but needed to be revisited for later phases:
Adding the first to the related work in the summary. The latter three are already included.
Comment #53
dixon_I have now split up all phases (and merged some) into separate plan issues to make it easier to review and provide approval on each phase of the initiative.
Comment #54
dixon_Comment #55
webchickHi there! This is an issue that we'd ideally like to move to the new "Drupal core ideas" queue when/if #2785735: [policy, no patch] Agile process for bigger changes in core (Part A: Ideation) goes through (hoping for the next week or two). If you could read that issue and provide your thoughts on that, that would be great!
Comment #56
webchickDries asked me this question the other day, and I confess I did not have a good answer, so...
Is there a hard dependency on undo functionality prior to workspaces/site-wide preview? Of the two, the second is definitely a hugely in-demand feature, so it seems a bit unfortunate the current projection says we won't see anything user-facing for that until 8.5 (spring 2018).
Comment #57
timmillwood@webchick - "undo functionality" (#2725463: WI: Add StatusItem field type to store archived state) is not a dependency of workspaces per-se, however when replicating content between workspaces we'd be unable to replicate "deleted" entities.
I think we can start working on many things in tandem, I opened #2784921: Add Workspaces experimental module a few weeks back to start discussing the implementation details of workspaces. I think workspaces should be defined in an experimental module, but then we have the issue that it needs to be an override/alter rather than baked into the entity api.
The biggest issue I see with "we won't see anything user-facing for that until 8.5 (spring 2018)" is that even if we add workspaces in 8.3 there is no reason to make them user facing until content can be replicated between workspaces. Replication has a lot of dependencies, currently in contrib these are things like indexes and serializers. Also in contrib when we replicate we end up with one copy of each entity per workspace. This is a bit of a contentious issue already in contrib, so I'm not sure how well it will sit in core.
Comment #58
webchickThanks for the additional details!
Hm. I wonder if that's true. I think for a number of use cases, just the ability to, say, preview changes to both a node *and* a block and/or menu at the same time on the same site would be light-years beyond what we can currently do. I don't know if we can just get that functionality without all the mountain of dependencies, though. But moving content across workspaces/sites is a far more advanced feature that I don't think most people know they need yet. :)
Anyway, not sure the above is even remotely feasible/accurate, but essentially the gist I'm trying to get across is we should always look for lower-hanging fruit with user-facing tidbits we can expose along the way, even if the penultimate functionality is going to take months and months. In other words, referencing https://cdn-images-1.medium.com/max/800/1*qINsG4WH_BDN-viMJUH6Ng.png, I think users would be super happy with getting a scooter in 8.3 and a motorcycle in 8.4 while we build the car for 8.5, if it's feasible. :)
Comment #59
dawehnerWhile having actual working workspaces would be super nice, I would like to put a small reminder here in place. Stuff needs time, and focusing and too many devices at the same time let's you end up with a bicycle with one wheel and a scooter without breaks. There are for example still a lot of tasks in #2725433: WI: Phase A: Use the revision API in more places which needs to be done, before adding workspaces really make sense.
Comment #60
balsamaRelated, the Lightning distro plans to release our own experimental module "Lightning Preview" bundled with Lightning it's 8.1.0 release this fall that will include what we're calling the "Workspace Preview System": #2677938: Implement a Workspace Preview System
Comment #61
webchickMoving over to the new Drupal Core Ideas queue while we get this into shape.
Comment #62
maxilein CreditAttribution: maxilein commentedI'd like to bring the idea of "Composite Entities" to everybody's attention: https://www.drupal.org/node/2822476
Comment #63
hchonovAdding a reference to the new autosave_form module for D8.
Comment #64
xjmThis initiative is active and has been awhile. :)
Comment #65
dixon_Comment #66
dixon_Comment #67
dixon_Comment #68
dixon_Comment #69
dixon_Comment #70
dixon_Comment #71
dixon_Comment #72
dixon_Comment #73
dixon_Comment #74
dixon_Notes and some decisions from the Hard Problems meeting for the Workflow Initiative at DrupalCon Baltimore today: https://docs.google.com/document/d/1-5B-Gndhklns20NZpGQul10525rFWuYB2NbT...
Comment #75
dixon_Comment #76
shubhangi1995can anyone please share the updated progress in conflict management related to handling of change in content simultaneously , as currently if a conflict gets recorded its not resolved until we delete the node.
Comment #77
nod_Issue summary is still up to date?
Comment #78
catchWorkspace module was added to core, and the official initiative generally is inactive now.
A more updated list of remaining tasks is in #2732071: WI: Workspace module roadmap.
I think things get marked fixed when they move to core implementation, so marking this as such.