This meta issue tracks certain Entity API improvements as well as defines a roadmap for an Entity Serialization API needed to make web services (e.g. content staging) easier to accomplish in contrib. This meta issue won't cover the RESTful interface itself, only the components such an serialization interface that we'll need.

All of the things below was identified, discussed and sketched on during the WSCCI Web Services Format Sprint in Paris, June 3rd - 6th. Report from that sprint is available here: http://groups.drupal.org/node/237443

Definitions

We've already identified that people talk about some different things when mentioning 'content staging'. The two primary things are:

  1. Moving content between different physical environments (editorial staging > quality assurance > production)
  2. Content moderation and content preview on one and the same environment

While both are immensely important for large scale Drupal implementations and both sometimes can solve the same problem, they are technically different. The former has some more history as being referred to as 'content staging' and the later to just 'moderation' and/or 'content preview'.

Once we have a good framework for moving content between environments we can build more workflows on top of that. So this meta issue is going to focus on 'content staging' in the first sense.

Roadmap

Step 1: #1368394: Convert all core entities to classed objects Done!

A lot of the patches in those sub-issues are, or are quite close to RTBC, let's push those through first.

Step 2: Finish the UUID implementation

In order to identify entities across environments we need UUIDs.

Actions:

  1. #1252486: Low level UUID API in core has follow-up issues:
  2. #1637370: Add UUID support to core entity types

Step 3: Entity Property API

In order to do things like entity serialization we need an Entity Property API that should provide better introspection of entities. Done at #1696640: Implement API to unify entity properties and fields.

Step 4: Entity Reference module in core (optional)

#1801304: Add Entity reference field. This would replace the File Field and Taxonomy Term Reference Field in core, as well as all pseudo-references like node author, parent node for comments etc.

This step is not required, it would just add a lot more consistency to the whole API. But considering the time constraints we have, we might skip this.

Actions:

  1. Start a branch in the Entity API Improvements sandbox
  2. Create an issue in the core queue with initial patch once we have a working prototype with passing tests.

Step 5: Entity Serialization API

Lin Clark completed this work - #1832840: Enable fieldtype-specific JSON-LD normalization

During the WSCII sprint (mentioned above) we more or less decided to move forward with JSON-LD as the serialization format for core. We also identified that we need something similar to the AtomPub specification (definition of collections, support for paging etc.). We might end up replicating that in JSON-LD.

Step 6: Start work on Deploy for Drupal 8

After we're done with step 1-5 we can start doing some first test implementations in contrib to experiment with a UI and so on.

Other discussions

Related discussions around content moderation, content preview etc are here:

Comments

vuzzbox’s picture

I think there will be a lot of devils in the details of Step 3. And I think they will be an issue of increasing complexity for both the Staging support and Preview support.

These are some of the areas that I think need to be considered:

  • Dependencies can be multi-directional and ad-hoc.
  • The order of entity deployment matters.
  • Deployments can be bi-directional, i.e., they may need to be "unstaged" at some point.
  • When deploying, where the entity might appear in a site matters because it might need to appear exclusively.

Dependencies can be multi-directional and ad-hoc
So, for example, lets say that I was publishing a new "Case Study" on my website and when I publish a new Case Study I need the Case Study (primarily a PDF file) published along with content (text and images) that will appear on my home page, which will feature the Case Study. And assume this Case Study is about a new "topic" for my website and all my topics are represented through taxonomies, so I have a new taxonomy term that needs to published with it. Assume that for reasons that make sense for this site that these are all separately defined entities or nodes.

I might have a node that, when published, must be published along with three or four other separate, but dependent, entities, perhaps two image entities and a "resource" entity - the Case Study PDF file. It may be that once published, the Case Study entity (pdf file) will live permanently in the Resources section of my website, but the "Featured Article" content could all be removed.

So in the deployment of this content, I would want it all to be packaged together and dependent. If one of the entities could not be deployed, then none of them could be. This requires not only identifying dependencies, but also some kind of transactional deployment handling. And the transactions would need to be at both the database and file system level, so both data and file deployments would be rolled back on a failed attempt.

The deployment order matters
One example: If I am deploying that Case Study that needs to be categorized by a new taxonomy term, then I would need to ensure that the taxonomy term was deployed first or that both the term and the entity were deployed completely before the record tying the entity and term together was added to the table, otherwise the insert would (should) fail.

Deployments can be bi-directional
I assume that deployments can cause stuff to be removed or unpublished from a server, as well as added to the server. So that has to be done in such a way that an entities dependencies, as both a parent and child in dependency relationships, are taken into account.

To explore the Case Study example above further, let's say that at a later date I might decide that the feature content doesn't need to be published any longer. So I might want to deploy changes that would unpublish all those entities, but the Case Study that was part of the original package should be left on the site. So the dependencies for this deployment would need to change.

Consider also that maybe I want to unpublish the feature content that was published along with the Case Study but that one of the images from the original feature content was reused later for some other feature content and that featured content is still published on the site. In this case, I should not be able to deploy changes that would unpublish that image since it is part of a dependency with another entity.

When deploying, where the entity might appear in a site matters because it might need to appear exclusively
Example: If I have blocks on the front page that are used for presenting one item and one item only at any given point time (like my featured Case Study), then the Staging process needs a way to determine if deploying an entity would violate that exclusivity. Publishing rules need to be definable for blocks or regions or pages etc that would check the availability of a block or region for a given entity depending on a variety of conditions, not the least of which would be publication and/or expiration date.

Anyway, some food for thought.

dixon_’s picture

@edmcl Very good write up! You are right in everything you are saying. I'm fighting with most of these things when building the Deploy module.

I'm not sure we will solve all of them in core though. For step 3 we should really try to come up with a very good foundation and an API for working with dependencies. Once that's done, modules like Deploy can take care of them business logic around it.

But hashing out step 3 a bit more is probably a good idea, because that's where most of the complexity is hidden as you mentioned. @edmcl (and everyone else), please feel free to edit the original post and add in things to the plan! It's a collaborative effort :)

indytechcook’s picture

Status: Needs work » Active

Thanks for getting this organized @_dixon

From the perspective of "Content moderation and content preview on one and the same environment", there is already work being done in the following issues.

Steps 1 - 4 in this plan also are very valuable to both methods.

(Settings status to "Active" since there is no actual patch to work on)

stevector’s picture

@Indytechcook, did you intend to link to two separate issues?

stevector’s picture

Issue summary: View changes

Adding link to 218755

sun’s picture

Component: configuration system » entity system
Issue tags: +staging, +UUID, +Configuration system
sun’s picture

Issue summary: View changes

Added links to issues and the sandbox currently being worked on.

dixon_’s picture

Issue summary: View changes

Updated the description with more information and links.

dixon_’s picture

Issue summary: View changes

Clarified where work on the Property API are being done.

dixon_’s picture

Issue summary: View changes

Clarified the goal of this meta issue.

dixon_’s picture

Issue summary: View changes

Step 1 is done

dixon_’s picture

Issue summary: View changes

Refined the roadmap and clarified action points.

dixon_’s picture

Issue summary: View changes

Restructured headings.

dixon_’s picture

Title: [META] Better support for content staging in core » [META] Entity Serialization API for web services (e.g. content staging)
Issue tags: +WSCCI

I clarified the title and description of this meta issue a bit. The focus here should really be the Entity Serialization API since that's the foundation for web services (like content staging etc.).

dixon_’s picture

Issue summary: View changes

Added more details to step 5.

dixon_’s picture

Issue summary: View changes

Updated issue summary.

dixon_’s picture

Issue summary: View changes

Updated issue summary.

heyrocker’s picture

Linclark started a discussion about workflows on gdo here:

http://groups.drupal.org/node/246748

I wrote a long response about how Deploy works, and the pros and cons of its' approach. This is touching on a lot of the issues raise in #1.

linclark’s picture

Ahh, didn't realize this issue was here.

I've been working with the JSON-LD working group to get some issues fixed in the spec that will help us with our use case. I have posted a summary at http://groups.drupal.org/node/244158.

sun’s picture

FWIW, I've tagged a whole bunch of issues with UUID:
http://drupal.org/project/issues/search/drupal?issue_tags=UUID

And I also just created:
#1726734: Replace serial entity IDs with UUIDs in URLs, or even globally?

In light of that, it appears to me that we need to solve some much more low-level things first, in order to allow the higher-level content staging efforts to truly take off. That isn't really reflected in the outlined staging roadmap in this issue summary here.

(That said, I'm also not 100% sure whether this issue is actually the roadmap for content staging, or actually something else.)

Also, (no pun intended) is @dixon_ still leading this effort? I wasn't really involved in staging discussions, but I've the impression there wasn't much activity lately, so we overall didn't make much progress in the low-level UUID/entity system field yet :-/

Given the remaining timeline until feature freeze, we should really make sure that someone is actively taking on the lead for the low-level UUID system changes that are required for the staging efforts. I'm trying to push issues where I can (because we're going to use UUIDs for configuration, too), but I can't really take on that lead myself.

sun’s picture

Issue summary: View changes

Updated issue summary.

Cottser’s picture

Issue summary: View changes

Added links to UUID followup issues

Cottser’s picture

Updated the issue summary to point to the new UUID followup issues. Although I'm not sure we need the docs followup here - feel free to remove it.

dixon_’s picture

Issue summary: View changes

Putting UUID followup issues in a UL

dixon_’s picture

The roadmap is of course very rough and not yet fully thought through. But it's a start that we all can work on.

@sun
I think the really big thing we depend on is the new Entity Property API. It will be my focus to help on that during DrupalCon. When we can introspect entities easier then we can create a generic Entity Serialization API.
Below this serialization API there's of course a lot of things needed, like actual HTTP endpoints with routing, content type negotiation etc. But they can't exist before we have anything to serialize :)

This roadmap will not provide a content staging functionality/solution per see. It will just make it possible in contrib in a slightly less hacky way than before :)

I think I never was leading any efforts or initiatives, so no pun taken ;)
I might have pulled some more strings regarding this than others, but haven't had time for core lately. So it's true what you are saying @sun, we haven't made the progress we need.

Edit: Made the OP slightly more clear of the intention here.

linclark’s picture

Regarding Step 5: Entity Serialization API, we need to reconsider whether we want to move forward with JSON-LD or not. I have posted a discussion on groups.

sun’s picture

I've created #1757452: Support config entities in entity reference fields (but won't have time to work on it)

linclark’s picture

I've created #1784216: [META] JSON-LD support to collect issues around JSON-LD support and start trying to sequence them.

moshe weitzman’s picture

Step 3, Entity property API has just been committed! Would be awesome if someone could tackle Step 4, Entity Reference Field in core. Note that Paris WSCCI writeup has some notes.

bojanz’s picture

Here's an issue for Entityreference in core, from one of the co-maintainers: #1801304: Add Entity reference field.

tstoeckler’s picture

Note that there's already some sort of an entity reference field that went in with the properties patch: http://drupalcode.org/project/drupal.git/blob/HEAD:/core/lib/Drupal/Core...

bojanz’s picture

It's just an item type (for the metadata, or whatever it's called now). There's no reference field in there. Not sure why this is confusing people across the queue.

bojanz’s picture

Issue summary: View changes

Clarified the intention.

moshe weitzman’s picture

Issue summary: View changes

Updated issue summary.

moshe weitzman’s picture

We are making great strides here. Step 4 and Step 5 are in progress. I've updated the Issue Summary. Please help Amitaibu and Lin and perhaps architecture/planning should start for Step 6.

moshe weitzman’s picture

Issue summary: View changes

Updated issue summary.

moshe weitzman’s picture

Step 5 is done and Step 4 is nearly RTBC. Anyone up for focusing on Step 6? We are so close!

jibran’s picture

alexpott’s picture

Issue tags: -Configuration system

I don't think this is a configuration system issue

alexpott’s picture

Issue summary: View changes

Updated issue summary.

Berdir’s picture

Anything left to here for core? :)

moshe weitzman’s picture

Status: Active » Fixed

I think we can close this issue.

As a follow-up, we are not using entity reference fields in core at all AFAIK. Thats blocked by #2116551: Fix the UX of Entity Reference fields. So help there is welcome.

moshe weitzman’s picture

Another good task would be to start a fresh deploy module for D8, since so much has changed. See https://drupal.org/project/deploy

Berdir’s picture

@Moshe: Yes, taxonomy/file/image fields do not directly extend from the configurable ER field. That doesn't matter much for this, they all extend from the core entity reference field type and can be introspected about what they reference and access those references using a single API. Including base fields like the node author, node type and so on.

moshe weitzman’s picture

@Berdir - good point!. That means that the best place for interested folks to move to is #2112799: [meta] Port deploy to Drupal 8.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

moshe weitzman’s picture

FYI, @dixon_ made a terrific presentation at Drupalcon on content replication for D8. See https://austin2014.drupal.org/session/content-staging-drupal-8. Help him!