Problem/Motivation
While using basic workspace functionality extensively at Pfizer we noticed that our editorial staff found it cumbersome to work on sites with lots of "living" content. Content would constantly have to be "pulled down" to draft workspaces in order to do full-site preview.
The solution is content inheritance between hierarchical workspaces by introducing the concept of sub-workspaces! The ability of having hierarchical workspaces has existed in the contrib Workspace module for a while, but content inheritance is new with this core patch.
Not only does this patch substantially improve the editorial experience of workspace sites, the functionality can enable many other very interesting things;
- Entirely new editorial workflows are possible – sub campaign workspaces, regional or country workspaces that inherit content, etc.
- Contrib can expose a JSON API per workspace and you suddenly have an incredibly competent content-hub solution!
- Domain module anyone?
Proposed resolution
Add a parent field to workspace entities, and track each workspace-specific revision in all its descendants.
Given the following workspace hierarchy, here's what's allowed to do with each workspace:
Live
|-- Stage
| |-- Dev
| | |-- Local 1
| + \-- Local 2
\-- QA
- Only top-level workspaces can be published to Live (i.e Stage and QA)
- Workspaces that have children can not be deleted (i.e. Stage and Dev)
- Workspace that have a parent are allowed to merge their contents into their parent (i.e.
Local 1 -> Dev
,Local 2 -> Dev
,Dev -> Stage
)
Remaining tasks
Review, commit.
User interface changes
When adding a workspace, the user will be able to select a parent workspace.
API changes
Nope.
Data model changes
- a new entity reference field on the Workspace entity type
- the workspace_association index (from #3062434: Track the workspace of a revision in a base field and convert the workspace_association entity type to a custom index) will also track a workspace-specific revision in all its descendant workspaces
Release notes snippet
The Workspaces module now provides the ability to create sub-workspaces, which automatically inherit all the contents of their parent workspace, and enables various use-cases like: new editorial workflows (sub campaign workspaces, sub-workspaces per country/region, etc.), possible integrations with JSON:API or GraphQL to instantly provide a content-hub solution, and many others.
Comment | File | Size | Author |
---|---|---|---|
#60 | interdiff-60.txt | 20.89 KB | amateescu |
#60 | 3062486-60.patch | 90.38 KB | amateescu |
#56 | 3062486-56.patch | 108.96 KB | amateescu |
#55 | interdiff-54.txt | 15.47 KB | amateescu |
#55 | 3062486-54.patch | 108.3 KB | amateescu |
Comments
Comment #2
amateescu CreditAttribution: amateescu for Pfizer, Inc. commentedHere's some initial work.
The patch needs to be rerolled (and parts of it rewritten) on top of #3062434: Track the workspace of a revision in a base field and convert the workspace_association entity type to a custom index, because I only realized we need to do that issue first while working on this :)
Comment #3
amateescu CreditAttribution: amateescu for Pfizer, Inc. commentedComment #4
pmelab CreditAttribution: pmelab at Amazee Labs commentedRebased the implementation on #3062434-7: Track the workspace of a revision in a base field and convert the workspace_association entity type to a custom index. The failing test on uninstall stems from there.
Comment #5
yogeshmpawarComment #7
pmelab CreditAttribution: pmelab at Amazee Labs commentedPorted over test fixes from #3062434: Track the workspace of a revision in a base field and convert the workspace_association entity type to a custom index
Comment #8
pmelab CreditAttribution: pmelab at Amazee Labs commentedAdded an API and basic UI for re-building the associations index.
Comment #9
pmelab CreditAttribution: pmelab at Amazee Labs commentedAdded missing link type definition for association forms.
Comment #10
pmelab CreditAttribution: pmelab at Amazee Labs commentedAs discussed with @amateescu, index rebuilding will move somewhere else to keep this patch smaller. Setting patch #7 as the current one.
Comment #11
pmelab CreditAttribution: pmelab at Amazee Labs commentedAdding intermediate 8.7.3 backport patch.
Comment #12
pmelab CreditAttribution: pmelab at Amazee Labs commentedFixed backport patch.
Comment #13
amateescu CreditAttribution: amateescu for Pfizer, Inc. commentedExcellent work in #4 to make this patch work on top of #3062434: Track the workspace of a revision in a base field and convert the workspace_association entity type to a custom index. Here's a few cosmetic fixes for now.
The interdiff is to #10.
Comment #14
blazey CreditAttribution: blazey at Amazee Labs commentedGreat effort on this one. I've reviewed the code, have done some testing and here are my findings.
The live workspace was its parent on a clean installation. This brought a few more issues. On this line in the
WorkspaceStorage::loadTree()
a vertex from 'live' to 'live' was added which caused the graph to no longer be acyclic. This violated the input conditions of the Graph component and results in unexpected output
The
_depth
property was zero regardless of the position in the tree.Here's a summary of the changes introduced in this comment. This patch:
::getAncestors()
,::getDescendants()
and::getDepth()
to the workspace class and interface.The combined patch contains is based on the most recent patch in #3062434: Track the workspace of a revision in a base field and convert the workspace_association entity type to a custom index (#16).
Comment #15
amateescu CreditAttribution: amateescu for Pfizer, Inc. commented@blazey, I'm sorry but I have to push back a bit on most of the changes from #14:
We already have an issue for this: #2280479: Add setting to EntityReferenceSelection plugins to prevent references to referencing entity, and I'd prefer not to try and sneak it into this one..
For Drupal 8.8.0, I would really like to get #2935780: Remove the concept of a 'live' default workspace done, so we won't have to care about the root workspace anymore.
The reason I didn't add these methods on
WorkspaceInterface
in the first place is that when we'll finally have a generic Tree API in core, we won't be able to re-use the methods names. We already have this problem withTypedDataInterface::getRoot()
andTypedDataInterface::getParent()
, which trickle their way up to the FieldItemList class, so I'd rather not make it worse.Also, adding these methods on the main entity interface will make it very confusing when an entity will be able to be part of multiple hierarchies, for example the one proposed in #2725523: Add a revision_parent field to revisionable entities. When someone calls a method like this in code:
$revision = $workspace_storage->loadRevision(5); $revision->getAncestors()
, how could they know if they're getting the ancestors of the main entity object (the workspace tree), or the _revision_ ancestors of that revision (the revision tree of a specific workspace entity).That's why I chose to make the
_depth
,_ancestors
and_descendants
properties as internal and as hidden as possible, and only document them in theloadTree()
method from the storage :)Comment #16
blazey CreditAttribution: blazey at Amazee Labs commentedWe had a discussion in slack with @amateescu and he said that none of the changes in the patch in #14 are worth pursuing, so reverting the display files back to the state from #13.
Comment #17
vijaycs85Here is an initial review/questions at code level (didn't apply and check manually yet).
don't feel it matches with what this field is for?
Do we have to do anything similar for $limit as well?
could we add a comment why we don't do anything special if $match or $limit set here?
It means, from the next revision update, these workspaces also considered tracking the entity?
Comment #18
amateescu CreditAttribution: amateescu for Pfizer, Inc. commentedRerolled the for-review patch to take into account the latest changes from #3062434: Track the workspace of a revision in a base field and convert the workspace_association entity type to a custom index and rolled a 8.7.x combined patch as well for those who need it :)
Comment #19
amateescu CreditAttribution: amateescu for Pfizer, Inc. commented#3062434: Track the workspace of a revision in a base field and convert the workspace_association entity type to a custom index was committed, so this is unblocked now!
Comment #21
amateescu CreditAttribution: amateescu for Pfizer, Inc. commentedFixed that test failure and the tree sorting.
Comment #22
amateescu CreditAttribution: amateescu for Pfizer, Inc. commentedDid a thorough self-review after demonstrating this patch at a local Drupal meetup, and fixed a few things:
- only top-level workspaces can be published
- workspaces that have children can not be deleted
- don't show the 'parent' field on the workspace publish form
This should be ready for final reviews now :)
Comment #24
amateescu CreditAttribution: amateescu for Pfizer, Inc. commentedFixed that test.
Comment #25
jeqqI've done some testing and it looks good in general.
My concerns:
Only top-level workspaces can be published.
. This is not a good user experience and after discussing with @amateescu, he said the plan is to make possible to deploy from child to parent workspaces. I think that would be the correct behavior.Comment #27
dixon_The issue that will address this is: #3070788: Add support for merging the contents of two workspaces
Sounds indeed like good tests to add.
Comment #28
amateescu CreditAttribution: amateescu for Pfizer, Inc. commentedBrought the patch from #3070788: Add support for merging the contents of two workspaces into this one and added tests for everything :)
Comment #29
amateescu CreditAttribution: amateescu for Pfizer, Inc. commentedUpdated the issue summary a bit.
Comment #31
dixon_Andrei and I walked through the entire patch and did a whole round of manual testing and found a few things to tweak. I'll be back in a few hours to give this one another review and full end-to-end test.
Comment #32
amateescu CreditAttribution: amateescu for Pfizer, Inc. commentedTweaked those few things ;)
Comment #33
dixon_Thanks @amateescu! The patch looks good and functionality works as expected!
It's not trivial to describe or let alone imagine all possibilities this patch brings to Drupal 8! Content inheritance between workspaces is simply AMAZING! But if I try summarise this in a few bullet points;
Apart from the solid test coverage in this patch I have done a lot of manual testing and everything works as expected. Below are some demos of a few notable things.
Basic demo
In this first video I demo a basic workflow that shows the content inheritance from Live down through two campaign workspace ("Top campaign" and "Sub campaign"):
https://www.youtube.com/watch?v=AjKPC3PYXVk
Conflicting drafts demo
This is an important (but logical) gotcha. A node can only have unpublished drafts in a single workspace. The user is currently prevented in the UI to create conflicting drafts. This is the simplest and most logical solution until we get this into Drupal 9: #2725523: Add a revision_parent field to revisionable entities
https://www.youtube.com/watch?v=UBuF33zc0ts
Deleting content demo
While revision changes are contained in separate workspaces, deleting a node affects all workspaces. Ideally deleting a node in a sub-workspace would create a new "softly deleted" revision that should be merged and published upstream as any other revision change. However, that's to be considered separate from this issue as we need to extend the delete API with some significant changes in order to support this in Drupal 9. For background on this see: #2786135: WI: Phase E: Introduce Trash module
https://www.youtube.com/watch?v=waKELEDkKSI
Comment #34
dixon_Comment #35
dixon_Comment #36
dixon_Just got off a call with @webchick where I walked her through the functionality end-to-end. She'll drop in her own comments later, but here's the main take-away from our conversation:
Webchick rightfully pointed out that deleting content in a workspace or sub-workspace which also wipes the content from all other workspaces is a no-go. Workspaces should be considered "a safe space" for making any changes and potentially destructive changes should not be allowed. I agree with this.
What we came up with is the following:
Comment #37
webchickWow, that's a great summary! :)
I will say, with my "product manager" hat on, that this truly seems like a really amazing feature, and I'd love to see it make it into Drupal 8 still, though we are now past alpha deadline so this isn't my call alone.
The entire point of Workspaces is to provide a "true" preview of changes before they happen. That currently is not possible in core, because once you've triggered a workspace, there's no way to update it with changes that happen subsequently. So while you can preview the new things you are adding within the workspace on their own, you can't preview them in context, like how they'll look on the front page (which now has so much new content it'll move yours out of the "featured" block entirely once you've deploy it, for example).
In contrib, @dixon_ mentioned that there's a solution that uses a manual "pull" content model, but editors find this a confusing model to work with (which makes sense; it's a developer's model). The current patch is much more intuitive, because it handles the "pulling" for you automatically.
But yes, unfortunately I do think the current delete logic is too dangerous, and works against user expectations. They expect a workspace to be a "private sandbox" for playing in, and if deleting content from there also deleted it from other peoples' workspaces or—even worse—from Live, that would be very unexpected/shocking behaviour, I feel. :\
I'm really hoping that since this "deny" logic is already figured out for editing, it will hopefully not be too much more work to do something similar for delete.
Also, I have to say that I really appreciate all of the thought/care that went into preparing the issue summary, preparing the video walkthroughs of how it works (and also the being up until stupid o'clock to run through it, but :P). That's SUPER helpful for orienting people who are new to the issue (like me!).
So yes, sadly back to needs work, but thanks so much for pushing on this!
Comment #38
Wim Leers😲🥳🤯
I'd sure love to hear more about this, and I think it'd really help if the issue summary explained this :)
Comment #39
amateescu CreditAttribution: amateescu for Pfizer, Inc. commented@webchick and @dixon_, thank you both for taking the time to review this patch and for the thoughtful comments/proposal :)
Here's an update that implements the suggested solution from #36 by enabling content deletion protection on three levels:
1. Users without the
bypass node access
permission will not be able to the see Delete operation at all;2. Users with the
bypass node access
permission are able to see the link (because they can bypass all node access rules), but the deletion form displays a helpful message and disables theDelete
button;3. Finally, for direct
$entity->delete()
API calls, we throw an exception with the same message from the delete form.Comment #41
amateescu CreditAttribution: amateescu for Pfizer, Inc. commentedFixed those fails by skipping the inherited test method, and explained why in the code comment.
Comment #42
dixon_Took this for a quick spin, and overall it looks really good. The functionality works as expected. I'll post a longer review tomorrow with a demo video.
There's one issue with the patch, it throws a type error in certain circumstances because WorkspaceManager::getActiveWorkspace() can return FALSE. Here's a quick fix I implemented while testing.
Comment #43
dixon_And attaching the fix I talked about as an interdiff to the patch in comment 41.
Comment #44
amateescu CreditAttribution: amateescu for Pfizer, Inc. commentedGood catch, @dixon_ :) Only one of the hunks from your interdiff is needed, because
\Drupal\workspaces\EntityOperations::entityPredelete()
and\Drupal\workspaces\EntityAccess::entityOperationAccess()
already check if there's an active workspace. Added a test for this as well.Comment #45
dixon_Gone through and done plenty of manual scenario testing on this latest patch, which also fixes the type error I discovered.
IMO we're good to go! :)
Comment #46
catchDid a review mainly for data integrity stuff, questions below:
This class isn't providing access control afaict.
Does this need to be in a transaction now it's more than one query?
Reading this through, I was really confused why we're checking all entities when we only need the information for one. This could do with an explicit comment that we're initializing the property for all entities up front for performance reasons or similar. Although also are we really checking deletability for multiple entities during the same request much?
Also what about moving the pre-load initialization stuff into a protected method?
Thanks for adding a transaction here.
Given we're merging entity changes from one workspace to another how does this remain true now?
Published workspaces get deleted still right? Just wondering how finite every workspace on the site is going to be.
Should this come from the entity type or similar?
Comment #47
plachThis is an awesome addition and code looks very good to me. There are a couple of fixes still needed and a few clarifications pending (mainly the deletion logic), but I think we're very close!
I didn't immediately realize this method is defined on the parent class: I'd add
parent::
here to make it more obvious.Don't we need a (post) update for this?
I might be wrong but I didn't spot any explicit test coverage for this. I think it would be important to have both functional and integration test coverage to check both the form and the validator itself.
Personally I'd prefer if this kind of changes happened in the storage: I realize that this may imply duplicated logic when writing an alternative storage backend, however we keep introducing dependencies from domain objects to services, which doesn't feel right. IMO these methods should be used only to manipulate the entity/field data/state.
Just saying, since we're already doing something similar in
::postDelete()
, so nothing really new added by this.Rant over :)
Is this change strictly related/required?
I'm wondering whether we could add a helper method to the storage to do this, something like
::loadSubTree()
. This would of course rely on::loadTree()
internally.Would it make sense to rely on
MemoryCache
for this?Nit: I'd call this
$tracked_revision_id
to improve readability.s/
'nid'
/$id_key
:)WorkspaceMerger
:)This seems an implementation detail to me, I'd move it into
::trackEntity
. If invalidating at each entity tracking is expensive, we could add a multiple version of track entity (::trackEntities()
?) invalidating only at the end of the process.It seems to me that this logic could be factored out to a (protected) helper method that both
::getDifferringRevisionIdsOnTarget()
and::getDifferringRevisionIdsOnTarget()
could rely on.(Even if we only have two of them ;)
Nit: I'd rename this to
$workspaces
and$id
to$workspace_id
to improve readability: in the middle of the review I was wondering whether there were also tracked entities involved in the process besides workspaces.Missing type hints.
I'd explicitly mention that roots are stored with a
NULL
parent: I had to read this a few times before grasping the logic.$child_id
:)Is this a BC break for API clients? Not sure what the policies are in this case.
Based on @webchick's feedback I was assuming this would work quite differently: I'd expect an entity to be deletable only if it was created in the active workspace and if descendant workspaces have no pending changes for it, regardless of its publishing state. Otherwise I could create an unpublished default article in
live
, plannning to edit it later, just to see it disappear because it was deleted in thestage
workspace by an editor assuming to act only in thestage
scope.Can we explicitly assert that deletion was successful?
This may seem minor, but it would have saved me a minute or so of head scratching: can we move this comment below, right before the assertion block, and replace it with something like
"// Check that there is no content in Stage that's not also in Local 1."
? I spent some time wondering why an empty result implied that the merge could not happen :DWe should do this only if the default storage class is set.
While manually testing the patch, I found this (minor) UX issue that seems to be pre-existing, since it happens also with the deploy action: operation links are not hidden when the operation is not allowed. Maybe there's already an issue for this, didn't check.
Edit: typos
Comment #48
catchI got a bit confused when reviewing for the wrong reasons - I was thinking about other workspaces, which of course we prevent editing in so that shouldn't happen, and thought 'oh OK then', but yes definitely creating an unpublished entity is still independently valid on live so seems like an extra check would be useful.
Comment #49
webchickThat is an accurate representation for how I expected the delete logic design to work.
Comment #50
amateescu CreditAttribution: amateescu for Pfizer, Inc. commentedReplying in separate comments to make it easier to follow :)
Re @catch in #46:
EntityWorkspaceConflict
validation constraint for disallowing changes to the same entity in other workspaces. With this patch, the constraint works like this:- given an entity that exists in Live;
- a user edits the entity in Stage, and the new revision is inherited by Dev. At this point, the entity can no longer be edited in Live, but it can be edited in Stage or Dev;
- if another user edits that entity in Dev, it can no longer be edited in Live nor Stage, until the Dev revision is merged into Stage, which brings us back to the second step above, where the entity can only be edited in Stage or Dev, at least until Stage is published.
Comment #51
dixon_:)
Comment #53
amateescu CreditAttribution: amateescu for Pfizer, Inc. commentedRe @plach in #47:
Also opened #3091481: Add explicit test coverage for EntityWorkspaceConflictConstraintValidator for the test coverage part.
WorkspaceRepository
service for the workspace tree stuff, mostly because I think the entity storage should only be concerned with CRUD operations.As for the specific part of the code you highlighted, we think that it actually belongs to the
workspaces.association
service, so I moved it there as a hook implementation.workspace.repository
service:getDescendantsAndSelf()
.::trackEntity()
is usually called when an entity is inserted or updated, so the cache is invalidated by the storage itself. This usage of::trackEntity()
in the workspace merger is a special case, so I think it's ok for it to do the cache invalidation itself.workspace.diff
" service specially for this, and we could do your suggestion in that followup issue.Opened #3091490: Hide the Publish and Merge operation links when they can't be performed for the UX issue.
In the next comment, I'll take care of #47.17 / #48 / #49 and the test fail. Leaving at NW because the patch will have the same failure as #50.
Comment #54
plachInterdiffs look great!
@amateescu, #53:
2: Ok, I hope it won't be forgot ;)
3: Good call!
6: Ok, we can always switch to
MemoryCache
later, if needed.10: Makes sense, thanks for clarifying
11: 👍
16: 👍
20: 👍
$this->initializeWorkspace()
:)Nit: we could restore this comment right before the final assertions.
Comment #55
amateescu CreditAttribution: amateescu for Pfizer, Inc. commentedFixed #47.17 / #48 / #49, #54 as well as the failing test (and added explicit test coverage for it). I hope it's ok to fix it here...
Re #54:
1. LOL, fixed.
2. Done.
Comment #56
amateescu CreditAttribution: amateescu for Pfizer, Inc. commentedThe interdiff from #55 contains some changes to
core/modules/workspaces/tests/fixtures/update/drupal-8.6.0-workspaces_installed.php
, which is considered as a binary by git, so the patch doesn't apply because it wasn't created with the--binary
option.Re-rolled the patch with
--binary
and here's a manual interdiff for that file. In order to test the upgrade path forworkspaces_post_update_update_deploy_form_display()
, we have to add thedeploy
form display and form mode to the database fixture which installs the Workspaces module:Comment #57
pmelab CreditAttribution: pmelab at Amazee Labs commentedReviewed all the changes since my initial patch and also ran the tests of our projects that build on top of it.
Good to go from my side!
Comment #58
amateescu CreditAttribution: amateescu for Pfizer, Inc. commentedThanks @pmelab for testing! Since all the feedback has been addressed, I think the patch is ready for committers to take another look :)
Comment #60
amateescu CreditAttribution: amateescu for Pfizer, Inc. commentedI think the "prevent entity deletion in a workspace" problem is becoming too hard to address in this issue.
I tried it until now because I thought the Workspaces module could get to stable in 8.8.0, but, since that's no longer the case, I'm going to open a separate issue for it, especially because it's a preexisting problem in HEAD and it's not made any worse by this sub-workspaces patch.
Comment #61
dixon_@amateescu It makes sense to address the delete behaviour in a separate patch now that we've decided to keep Workspaces module as experimental.
I gave this latest patch a whole new round of testing (now disregarding the fact that deletes affect all workspaces). All is looking good. All comments have been addressed. Back to RTBC!
Comment #62
amateescu CreditAttribution: amateescu for Pfizer, Inc. commentedOpened #3092247: Prevent content from being deleted when there is an active workspace and posted a test-only patch to demonstrate that the delete problem exists in HEAD :)
Comment #66
plachI spoke with @webchick and she is ok with committing this as-is, given that the issue is pre-existing, which #3092247: Prevent content from being deleted when there is an active workspace nicely proves. The follow-up should be a stable-blocker, though.
Given that, I went ahead and committed/pushed f10a397 to 9.0.x and friends. Congratulations!
Marking this as needing work, since the release notes snippet is missing as well as a CR announcing this new functionality to site builders and editors.
Comment #67
plachComment #68
dixon_Thanks everyone who's been involved here! 🎉 This is a major piece of functionality that just landed!
Thank you all!
Comment #69
amateescu CreditAttribution: amateescu for Pfizer, Inc. commentedAdded a change record and a release note snippet :)
Comment #70
webchickExciting stuff, folks! Thanks for all your hard work!
Comment #72
jibranI was wondering should we enforce
max_length
here as workflow ID has->setSetting('max_length', 128)
but the question is can we enforce lenght on ER field target_id field?