Implement Autosave

I'd like to point out that thinking in terms of published / unpublished as workflow states is a bit.. "outdated" in Drupal 8, and we should switch the terminology to be in terms of default and pending revisions. Default revisions are what's displayed to anonymous site visitors, while pending revisions can be seen as drafts, usually displayed only to site editors, and which may or may not end up being 'promoted' to a default revision state.

An example why this matters is to take the core Workspaces module as a concrete scenario: a piece of content edited (either created or updated) in a non-live workspace (e.g. stage) can be marked as published via the Publishing status field, however this doesn't mean that the content is "published" and displayed to anonymous site visitors. It is stored as a pending revision (draft), and it can get to a default revision state when the whole workspace is pushed/published to Live.

With the terminology stuff out of the way, I think it's a good idea to try and add the ability to not enforce entity validation when the entity will be saved as a pending revision, and only enforce it when we're saving a default revision.

Adding some tags to try and bring this into attention of core maintainers :)

we should switch the terminology to be in terms of default and pending revisions.

This is an excellent point, I've updated some of the github issues for the Admin initiative to reflect this. Thanks!

I'm hugely in support of this feature (both autosave + delayed form validation to make it possible), so big +1 there. Escalating to major, as this would have tremendous end-user impact.

I think my main concern stems around the timing of this... Drupal 8.8 is (probably? hopefully?) the last version of Drupal 8. We already have a pile of stuff to do to stabilize Workspaces module. Adding more "scope" to Workflow-based stuff in Drupal 8 therefore seems a bit spooky. :) But no harm in fleshing out a plan, esp. any contrib-space blockers for this kind of functionality, even if we (probably?) can't realistically see to it in core until D9.

This sounds like a great idea, but also quite tricky to implement.

One of the things to consider is, the validation system is used to protect the integrity of the published version of the content in some cases. A random example is: #2856363: Path alias changes for draft revisions immediately leak into live site. I think there are also other examples where validations are in place to stop major usability headaches with regards to non-translatable fields across translations in pending revisions, ContentTranslationSynchronizedFieldsConstraintValidator comes to mind.

So it seems validation shouldn't be enforced for some constraints for draft content, because the system as it stands is being used for a lot more than making sure user input is valid.

However, even if you chose a simple constraint like NotNull which is enforced when a field is marked required, disabling that would potentially cause db level constraint violations, resulting in data not being saved, since those fields can be marked as NOT NULL at the table level.

@Sam152, I also think that the NotNull constraint is the (first) critical piece of the puzzle and should be the main focus point at the beginning, at least for the technical aspect of this feature.

One possible solution is to extend the new "storage required" concept used by base fields and make it available for the properties of a field as well. Then we can decouple the two required and storage required concepts and make the first one only about "required at the UI/presentation layer".

This way we keep the ability to mark specific fields and/or some of their properties as storage required, meaning that the system as a whole can't function without actual data for them in the database (for example: the bundle of an entity type), and the "required for a published/default revision" would be just a presentation-level constraint, for example: the article content type needs some data in a summary field otherwise it can't be displayed in listings, or something like that :)

Aside from the validation and entity system challenges, I think it would be a good idea to use multiple layers for storing auto-saved draft revisions, because we shouldn't store a draft revision in the database every 5 seconds or so, which would translate to every few characters typed in an entity form. IMO, the first layer should use the local storage of the browser and "sync" with the database at a higher (potentially configurable) time interval.

Just a thought, other than validations we also need to make sure it should not create bunch of revisions on every autosave.

I totally believe autosave is a feature people expect these days. Much like many people not longer know the floppy disc icon, nor do they know the save functionality given the entire industry moves away from it.

I do have one basic question. @amateescu / @Sam152 talk a lot about issues happening with trying to save this kind of content in our tables. I'm wondering though whether this is actually needed. If we limit ourselves to storing the content in the browser a hole lot of questions disappear. For example we don't need to ask us whether/how to hide this information from other users. Maybe I'm missing something obvious here though.

@shahzad-anwar, see the last paragraph from #8.

@dawehner, storing a pending revision in the database means that a user is able to work on the same piece of content on different devices :)

I think this is a crucial feature. Anybody who's ever used a computer assumes that their input will never get lost.

However, let's not forget we tried to do this in the past. I already see localStorage mentioned a first time in #8 and again in #10.
That's what we tried to do in #153313: ckeditor input is lost when using the browser's back button. I was one of the people who worked on that. In a few weeks, it'll be six years ago that I worked on it: #153313-118: ckeditor input is lost when using the browser's back button. I felt that patch could be RTBC. Some agreed, but some were quite vocal against it:

• @webchick felt that automatically repopulating the form without some notification/message was a problem: #153313-121: ckeditor input is lost when using the browser's back button
• @Damien Tournoud was concerned about the node locking/concurrency handling then becoming useless: #153313-122: ckeditor input is lost when using the browser's back button
• @Damien Tournoud & swentel pointed out that this does not work at all with AJAXy form elements

I hope this issue learns from the discussions and prior work in #153313: ckeditor input is lost when using the browser's back button, so that we don't have to rehash those discussions. I know many things have changed since then, but many things have also remained the same.

JSON:API currently overwrites the current revision and does not automatically create a new revision. I wonder if:

1. This would be better than using localStorage
2. This might be a feature masquerading as a bug. For auto-saves, perhaps creating new revision isn't necessary. And whether the revision ID is being incremented or not could determine if validation is run. The problem is to find a reliable way of figuring out when an editing "session" starts or finishes so that a new revision can be saved so that a useful revision history exists.

> JSON:API currently overwrites the current revision and does not automatically create a new revision. I wonder if:

Yeah, you might think that, but try enabling content_moderation :) That forces a new revision on every entity save at the moment.

Are you aware of the autosave_form module presented in the session Autosave and concurrent editing in Drupal 8 @ Drupal Europe Darmstadt 2018? It is also available on youtube - https://www.youtube.com/watch?v=ixlLskzT9c4

I've been implementing the module with the intention of proposing it for core integration at some point.

The module takes a different approach instead of requiring the big struggle with the storage and introducing complexity, which is not necessaraly required. Instead of creating a new revision, a serialized entity will be saved to a dedicated autosave storage. During the development of the module I've tried various approaches and considered different options and for some time saving to an autosave revision was my favourite, but I came to the conclusion that either it is impossible or if so then the solution will increase the complexity of the storage incredibly.

Therefore I came with the idea of utilizing a dedicated autosave storage. One could use it and extend in such a way which would allow us to even perform entity queries on it. Currently it saves only the serialized entity, but the flexible interface allows for all kind of storage solutions. We could even create autosave tables "{$tablename}_autosave" for each table of the entity type, but remove NOT FULL or unique constraints.

P.S.
The module also already takes care of the validation of incomplete entities in the Form API.

Oh and one of the cool next features I am planing is saving to the browser's local storage in case of no internet connection - for example when during travelling there is no connection at all or it interrupts often. :)

I have just started trialling that autosave_form module and it works how I would expect it to, i.e. it doesn't save a new content revision each time and the user gets to choose whether to continue editing or discard the changes.

It hasn't been pushed out to many users here yet but it seems promising.

I think if there is any possibility that this feature will want to save new revisions that ideally that would be optional. Some content workflows would be made quite unpleasant for users with that many revisions cluttering up the place. Plus it's not a consciously created revision, which matters in some situations.

Another option could be that there is some way to identify whether or not a revision is an autosave revision.
In that case it could either save all as new revisions and the moderation UIs could filter out the autosave ones or the system could detect whether the most recent revision is an autosave revision and then if it is, reuse that revision, otherwise create a new revision.

Personally I like the idea of a separate autosave database table because I consider autosave and manually saving a new revision to be two very different things.

The other thing to be considered, if the data is stored separately, is whether there is a single autosave thread or different ones for each user.
If users can only see their own autosaved content, then it would potentially have to deal with merge conflicts if the original content has been updated since the autosave was created.

One difference that the autosave_form module does that other apps (e.g. google docs) don't seem to do, is it saves periodically based on a fixed time interval, whereas other apps I've used seem to more often trigger saves based on user activity (and possibly also fixed time periods).
Basing it on user activity is obviously more complex but would likely give a better user experience. Maybe a nice to have or a future addition.

Should we get rid of the explicit mentioning of revisions in the issue title? I got the feeling from the discussion mentioned by @ckrina that revisions are not the place to save some arbitrary state of form input. Everyone seemed to be very cautious about creating unnecessary revisions.
Additionally, it feels more logical to see this as a feature of forms, not of entities. In the end, you do not care what the form might be doing with the data. You just want to keep the form state. Thinking about storing the intermediate state independent of what will be generated by the form enables us to use this functionality for any kind of form. We could autosave e.g. config forms or completely different stuff as well.

@ckrina, why is the autosave_form module not mentioned?

It already has a storage for the entity and the form state. Also the module has a stable release now.

A backend implementation would create a pending revision that others can see.

Backend vs client side - benefits for multiple editors and people moving between devices.

Autosave is not content moderation. These are two different concepts.

There were comments here that doing this like this

1. Will make the entity DB storage explode, which is dangerous and will be really unexpected.
2. There are database constraints preventing saving an incomplete entity.

Have this been addressed during that call?

Additionally using the Entity API for autosave is extremely expensive. This will add a really big last on the servers especially when there are a lot of active users.

autosave_form has been implemented with the goal of minimizing the performance impact.

We should implement a server side solution that saves the form state into a temporary store for a single user and be able to rehydrate a form with this data.

This then leads to an additional database insert. IMHO the entity and its form state belong together, which is already offered by the entity autosave storage provided by autosave_form.

Concurrent editing - our decision about autosave should take this into account

This is already implemented in contrib and provided by the combination of the modules autosave_form and conflict 2.x. I've implemented both modules from the beginning with concurrent editing in mind and they successfully work together and already offer this great feature.

Please take a look at my session about this from Drupal Europe 2018 - Drupal Europe - Autosave and concurrent editing in Drupal 8, which I've already linked in #15.

Removing the specific mention of revisions in the issue title, we might end up deciding client side + tempstore is a better choice here, or client side + tempstore + pending revisions combined.

It would be good to have a summary of contrib solutions (at least module name and general approach to storage) in the issue summary.

We just had an online meeting with (sorted alphabetically by first name)

• Alex Pott
• Angela Byron
• Cristina Chumillas
• Daniel Bosen
• Gábor Hojtsy
• Hristo Chonov
• Sally Young
• Wim Leers

Thank you Cristina and Gábor for organising this meeting!

Unfortunately some of the attendees were unable to attend the whole meeting, however most of them were there from the beginning till the end.

A summary from the meeting and answers to some questions by me in the aftermath mostly regarding the autosave_form module:

1. We need to get more people into evaluating the autosave_form module. A screencast about its usage might be useful. Do we need to present more stuff than what is shown on the Drupal Europe recording? Tweeting about it might be an additional option as well. The module is currently being considered for integration into the Thunder distribution and into the Lightning distribution, which would allow it to reach to more users. As of today it has official 124 sites reporting using it, however this number might be higher as not all sites do report.
2. Storage
2.1. Should we make use of forward revisions - all the people on the talk were leaning towards a temp storage without forward revisions. Some of the reasons:

• a) we want to offer autosave for forms regardless of entities (autosave_form currently only works for entity forms),
• b) we want to cover config entities (autosave_form does not support config entity forms by default, but it is easy to add support by default),
• c) we want to cover non-revisionable content entities (autosave_form supports them),
• d) we don't want to add additional last on the backend servers by increasing the number of heavy requests (autosave_form has a very little impact, because it stores only in one table and prevents any hook invocation and form validation),
• e) we don't want to deal with saving partial entities and the problems that might arouse from there,
• f) we don't want to deal with additional custom constraints that might have been added on entity tables,

2.2. User-dependent autosave states - yes, for now we want that the autosave states are per user and not shared among users. Sharing among users is something that content moderation is doing. Sharing among users might trigger privacy issues. autosave_form already does it that way.
2.2. Swappable and form API independent storage, which could be used by decoupled frontends, which only save a serialized version of the form values for the entity. autosave_form offers a way to exchange the storage, however it is currently not necessary implemented for usage by a decoupled frontend, but this could be done.
3. Highlight what happens when a change in the field/form structure has been performed by an admin. Currently autosave_form will drop the autosave states under some circumstances. This perceived data loss should be highlighted, however we should consider that it is possible to perceive data loss even without autosave and with already completely saved entities - for an example when an entity field is removed/deleted. autosave_form will currently drop the autosave states in the following cases:

• a) if the entity is saved by another user without having the conflict module (2.x branch) enabled, which when enabled adds support for concurrent editing and conflict resolution.
• b) for an user when the set of roles assigned to an user changes
• c) when a form related config is modified or deleted - field configs, field storage configs, entity form display configs, user role configs. For those configs only form-independent properties such as weight, dependencies, label, description and so forth are allowed to change without affecting the autosave states.

We need the opinion of all entity API maintainers on this subject and also the opinion of the workspace module's maintainers, where one of the most important decisions that we all should agree on is whether we are going to use a dedicated autosave storage and not deal with issues that might be caused by using revisions instead. Having a swappable storage however makes it possible to offer a contrib storage that is saving in revisions, so we don't need to permanently close that door even if we decide to stick to a dedicated autosave storage.

Before organising another meeting it would be great if everybody has watched the recording of the session about autosave and conflict during Drupal Europe 2018 - https://www.youtube.com/watch?v=ixlLskzT9c4 . Some things have changed since that session, but there have been mostly additions.

I hope, that I haven't forgotten something.

Would autosaving provide any visual or semantic feedback to the user? If so I'd love to explore that in an a11y perspective but I appreciate that might be some time in the future yet!

@anevins, the autosave_form module already provides that visual feedback. It would be great if you could take a look at the session recording linked in the previous comment, where the module is demonstrated.

There are currently two use cases here.

1. Work is not lost: even if i don't press save, i anything i've typed, toggled, uploaded, etc. is there to be recovered.
2. Incomplete work can explicitly be saved: even if i haven't addressed all required fields, i can save a draft, find it again in a list of unpublished content (pending in the updated language preferred by amateescu), and publish it (make default).

Autosave form is the best in contrib at addressing the first.

Require on publish is the best workaround in contrib for the second.

(That i know of; would love to stand corrected!)

As was mentioned by dawehner, people don't expect to have to press a save button anymore. Once we're ready to take that step, the two use cases can collapse into one: some kind of lightweight revision is made for every change to content; content is saved in a pending state the moment anything is typed in a blank form, etc., and there's no concept of personal autosaved copy and intentionally saved see-able-by-others pending revisions, in that world conflicts are ideally avoided by pushing changes made by one person (or, more likely, in one browser tab) to any other open edits of the same content.

But as hchonov suggests, we're not ready to put that load on revisions and workflow just yet, so we will need to keep track of these two use cases independently. (Which as has also been noted allows autosave to work on more than just content, as Autosave form already does.)

This issue came up in a discussion in committer slack and it was pointed out that this issue should be in the core ideas queue. I am moving it there now.

And now the Ideas project is being deprecated. This issue is moved to the Drupal project. Check that the selected component is correct. Also, add the relevant tags, especially any 'needs manager review' tags.