Hello everyone,

I'm really getting excited for Drupal 8 and want to kick off the discussion (in the bug queue at least) about specific features of HTML 5 that could have a dramatic impact on how our drupal-based website behave. I'd like to focus first on Web/Local Storage

References:
http://www.w3.org/TR/webstorage/#storage
http://caniuse.com/#namevalue-storage

The HTML 5 Storage API already has over 80% support by web browsers and, if used wisely, could provide us the tools to dramatically change the user interface design of drupal content editing / creation.

With Local storage we could "pre-load" all of the content that appears on a page into local storage, then when the editor is in "edit" mode they could click an "edit' link and instantly be able to modify the content of that thing they clicked. When the editor is finished editing they could hit save and we could either initiate the persist-to-database action immediately or we could defer the action to later.

Anyone with a google account can see a similar workflow whenever they are creating an email or document.

With HTML 5 storage APIs, a little AJAX magic, and a slightly more intelligent context subsystem we can make a standard Drupal page come alive and make a few administrative pages redundant.

* Why would we use a block admin page if we could modify blocks directly on a page.
* Why would we use a node edit/create page if we could click a specialized box at the end of a node list and create a new node?

There are likely lots of other uses for a larger and more accessible key-value pair storage mechanism that lives in the browser. All of them awesome.

So should we make HTML 5 storage API a goal for Drupal 8?

Comments

cosmicdreams’s picture

Title: Decide how to handle Local Storage in Drupal 8 » Decide how to handle HTML 5 StorageAPI in Drupal 8

Changed the name to a more descriptive title.

ericduran’s picture

I'm trying to come up with a valid storage usage for core and can't come up with one.

I wont denied there isn't any potential in contrib, There is. I just can't think of anything that web storage is needed for in core.

cosmicdreams’s picture

Given the outline for the plan for configuration, ( http://groups.drupal.org/node/155559 ), I think that the HTML 5 storage api, local storage, or whatever else it's called out there, would be a good candidate for the second layer of storage, the active layer.

There are security considerations to handled for accessing this data. And perhaps an additional discussion on what kinds of data should and should not be in this layer. But once those discussions have been had we can talk about what we could possibly do with a fast client-side storage mechanism and how that might effect the workflows we currently provide by default with Drupal.

In general, any workflow that needs to temporarily store the state of something could benefit.

As experienced web developers we've grown accustomed to thinking about web applications as not having an inexpensive or easy to use state storing mechanism as for the longest time web pages have been stateless or have needed to use highly limited or expensive mechanisms like sessions and cookies (respectively).

Now we have are getting the ability to make our web pages come alive with the StorageAPI. This new feature can be our web site's short term memory to accompany the authoritative server storage.

Bonus Thought: What is the difference between a desktop application and a web application?

One answer is: robust and inexpensive state management.

cosmicdreams’s picture

I'll take this thinking another step forward.

Q: When I click on a link on a drupal site, why do I need to talk to the database running on the web server?
A: Because that's where the current state of things exist. If I don't read from the database there's a chance that someone may have changed something and I don't have the most current, authoritative view of things.

Q: When can I not care about that?
A: When it can be proven that there hasn't been a more recent change to the things your page load cares about.

So it could be the case that we can get a GIANT performance win (read: instant) if we are working things that have in its workflow the reading, editing, and saving of state to and from the database. Why even hit the database as long as you can prove that your client-side state manager has the most current state?

All we need to provide a mechanism that can mange that for us.

ericduran’s picture

@cosmicdreams most of the things you mention don't really seem related to local storage. They seem more about caching whether it be edge side/server side, or browser caching.

We really need to understand that local storage is not meant for building the drupal stack on the client side, is also great for caching in certain situations but is not meant to cache an entire page, there are other solutions for that.

cosmicdreams’s picture

Also, It appears that anything worth keeping private should never be in local storage:
http://stackoverflow.com/questions/3718349/html5-localstorage-security

So local storage should only ever contain things that we would ok having a hacker get a hold of. Perhaps things available to the theming layer. Perhaps parts of the FAPI.

@ericduran, I'm really just brainstorming here. Thank you for letting me know if I'm heading down the wrong train of thought. With that spirit in mind, can you tell me what would be wrong with representing a sort of "Drupal Document Object Model" in some kind of client-side storage mechanism? Is that entire idea not a good one? Or is it just using localstorage as the client-side storage mechanism?

Crell’s picture

The D8 Config API (still in early development) is not really suited for this. It's by design a very low level server-side system. Moving the entire configuration set client-side would involve shipping potentially over a megabyte to the client on the first page, where most of the data would be totally irrelevant anyway. Node type structure, for instance, is irrelevant on the client.

Drupal doesn't have a consistent Document Object Model or anything of the sort, so it cannot be shipped client side. :-) Even if it did, remember that you don't want to ship the client things it is not going to need. That's just a waste of bandwidth. Saving it into local storage doesn't waste any less bandwidth, it just bloats the browser memory.

cosmic, what you're describing (smart browser, dumb server) is an approach much more suited to a Google Docs like application where the server is purely a rest server. There's certainly a use case for that, and we want Drupal to get better at that than it is now, but it's still a minority case. Most sites are still page-centric, will remain page-centric, and just need to be faster and prettier.

Where I *could* see localStorage being useful is for multi-part FAPI forms, multi-step forms, and compound forms like Views uses. I do not know all of the details of those implementations, but right now they probably spend more time talking to the server than they should. Part of that is the server's fault for assuming everything is a page(I'm working on that with WSCCI, but it's a ways off), but we could probably get some optimization by making smarter use of the browser's local storage for form state.

Everett Zufelt’s picture

I think that a core API to check for local storage support, and to allow for putting and getting values would be a place to start, possibly as a contrib module.

Really the first place to start would be with some well defined use-cases against which the API can be built.

cosmicdreams’s picture

And the process of finding a well defined use cases for using the feature can start by finding all the use cases where it shouldn't be used. Here's a list of disqualified use cases:

  • Anything that uses private information (we should formally define what information qualifies as private
  • full page caching: An obvious waist of resources
  • many other scenerios I'm probably not thinking of

Cases where it could come in handy could be cases where storing the information locally (and temporarily) pays off.

  • Intelligent widgets that need to keep track of their own state:
    • Table based grids
    • Calendars
    • Slideshows
    • Tabbed blocks
  • node/node->nid/edit Edit<->Preview actions
  • Data intensive animations

There's actually a formal programming model for this kind of idea that derives from the traditional Model - View - Controller called Model - ViewModel - View (MVVM): http://en.wikipedia.org/wiki/Model_View_ViewModel

HTML 5's local storage capabilities has me exploring whether it's possible to implement this architectural pattern outside of tightly managed languages like Silverlight or Flash. Local Storage is one key to generalizing that pattern out of the XAML context. But there are many other supporting technologies that are going through standardization these days that will fill the missing gaps.

P.S.: It's noteworthy that Crell's criticisms are very valid and also noted in the wiki post for this. I wonder however, whether the benefit could ever outweigh the cost.

cosmicdreams’s picture

Status: Active » Postponed

Postponed this issue until we ready for phase 2.

Crell’s picture

Very clear use case: http://drupal.org/project/autosave

I'd love to trash the whole form post serialization nonsense in that module and just use localStorage. However, that would require working around Form API's extensive efforts to ensure that a form is coming from it, not from just the browser. That, in turn, causes a problem for dynamic forms (think multi-value fields on nodes) that MUST be build server-side. I do not know how to resolve that, but I suspect we need to make FAPI more client-side-activity aware. There's an action item for someone. :-)

cosmicdreams’s picture

Perhaps we should add a #storage property to FAPI. Which would allow localStorage to be an available setting for the property.

Crell’s picture

That's not the issue. The issue is that if a form is sent from the server with 2 text fields, and the user clicks the "add value button to add a third, that requires a round-trip to the server to update the form cache so that when it gets a form with 3 textfields it doesn't flip out and reject it on security grounds. Yet if we save form state on the client side in case of crashes then we need to trigger the appropriate rebuilds before we can even start repopulating the form.

I almost wonder if our form handling is too tight in this day and age...

Crell’s picture

Status: Postponed » Active

This is not postponed, since this is not blocked by CMI. It's blocked by FAPI being too tight. :-)

Jacine’s picture

Issue tags: +sprint

Adding this to the current sprint.

cosmicdreams’s picture

Ok, taking another stab at this:

What is localStorage good for?

1. You can do things that other programming frameworks have tried to create specialized solutions for in the past, like:

  • Storing the state of your web application (this is where Crell's idea of a state-aware text editor comes in)
  • Caching the basic structure of a web component that may be duplicated by user actions
  • Make a web component be able to store data into an intermediate store so that you can speed up the thing by not having to round-trip to the server

2. Storing anything that cookies aren't built for and that is less than 5MB. (Well I should say that it's possible to do this, but it likely is good if you do.)

Other references:
http://24ways.org/2010/html5-local-storage
http://htmlui.com/blog/2011-08-23-5-obscure-facts-about-html5-localstora...
http://paperkilledrock.com/2010/05/html5-localstorage-part-one/

Crell’s picture

So as I said in the HTML5 meeting, I don't think the issue is "how should we use localStorage in Drupal 8"; we don't need to throw any crazy Drupal-specific wrappers in the way. The question is "how might we want to use localStorage, and what do we need to do to get Drupal out of the way of that". Eg, FAPI state handling makes a localStorage-based autosave really really hard. What are some other specific use cases, and are there any blockers for them?

cosmicdreams’s picture

Sorry, I new I was going to miss important things by not making that meeting. I found an old issue that seems to be an early attempt to use localStorage for persisting application state. No, it's not a specific use case where FAPI makes usage of localStorage difficult but I thought I'd link it anyways to show an example of how we might use localStorage.

#65578: Remember state of collapsible fieldsets

I'll keep thinking about web application state and bring it up again if I can imagine a use case that matters.

nod_’s picture

Crell’s picture

There's probably cases where sessionStorage would be better, which shouldn't have any IO. #65578: Remember state of collapsible fieldsets may well be one of them, actually.

nod_’s picture

Yes but people we need some UX people around if we're to use sessionStorage.

How do we manage the expectation of the user that once he collapse the fieldset this state is saved but it's gone when he close his browser? People are used to cookies, not so much this stuff yet. I agree that it'd be a perfect use-case though.

sun’s picture

Title: Decide how to handle HTML 5 StorageAPI in Drupal 8 » Evaluate standardized usage of HTML5 Storage API (e.g. through a library)
Issue tags: +jQuery, +JavaScript

The question is not whether we want to use the Storage API. We have plenty of use-cases for that.

However, using the Storage API properly, detecting support for it, providing easy access to the different storages (sessionStorage, localStorage[, globalStorage][, Web SQL Databases]), and figuring out how to gracefully fall back (or at least, not break code) is non-trivial.

As mentioned over in #65578-18: Remember state of collapsible fieldsets, I am sure there must be a facility in jQuery core or at least some jQuery plugin that provides a standardized way to interact with the storage API/engines. The patch over there contains a Drupal.storage helper object that I've written myself, but it looks poor to me.

The purpose of this issue should be to evaluate any possibly existing libraries and methods for handling the Storage API.

To do so, we should revise the issue summary to provide an overview of approaches and libraries that exist on the web, and evaluate them.

nod_’s picture

Storage API is not really pretty:

  • sessionStorage very likely async because there is no data persistance (I don't have confirmation for this though).
  • localStorage sync = not good, as the article earlier pointer out but is supported but 80% of browsers.
  • indexDB = sqlite in the browser, hardly any support for this http://caniuse.com/indexeddb.

And existing libs will always rely on one of the three. The danger is that i'll end up in localStorage and if people start putting views result/data in there it'll come close to the 5Mb data limit and will slow everything down, which is what we don't want, because of mobile and all. And clearly real people won't drop IE7 any time soon so this needs to be taken into consideration even if core don't support it.

My point is that any js library won't get around the performance/coverage issue and we have to be careful about what to consider and what we make available for people to hurt themselves. Evaluating libraries, sure, but there are underlying issues that needs to be taken into account now or it will come back to bite later.

nod_’s picture

Component: theme system » javascript

And how what that ever a theme issue? :þ

cosmicdreams’s picture

As per sun's comment, here's a list of jquery plugins that address handling localStorage. I suppose the problem is that there are so many. I don't know which is most mature:

http://archive.plugins.jquery.com/plugin-tags/localstorage

Jacine’s picture

Just noting that RobLoach had a good use case in our last HMTL5 IRC meeting:

Could replace stuff we do with jQuery Cookie (toolbar open/close) with localStorage.

andypost’s picture

node/node->nid/edit Edit<->Preview actions

Also this could help to store defaults a-la $form_state['node'] on client-side to reset to defaults for forms that have a lot of options - for example fieldUI and system settings provided forms

nod_’s picture

@andypost, that's the way to go to run into the localStorage performance issue that is highlighted in the article above. We need benchmarks to see how that affects page load, mobile devices are slower than desktop and D8 try to be as mobile friendly as possible.

For this kind of things IndexDB would go very well though.

nod_’s picture

Some new insight from Mozilla based on the article linked in #19: http://hacks.mozilla.org/2012/03/there-is-no-simple-solution-for-local-s...

Jacine’s picture

Can we please get can updated issue summary here?

webchick’s picture

Tagging.

nod_’s picture

Ok, let's reduce the scope.

The bare minimum we can do is completely remove the use of cookies and replace that with localStorage, if we keep all that stuff around a few kb that shouldn't hurt perfs to much.

Preloading a big chunk of pages into localStorage is not a good idea yet. We could use sessionStorage for that though but I'm not exactly sure how that'll make things faster since it's using JS that makes things slow in the first place, especially on mobile (I'll present data in a couple of weeks). Unless we ajaxify the hell out of drupal core admin for mobile it's not worth it.

Also, localStorage may be the only JS API that doesn't suck, let's not deal with the overhead of a library unless there are really nasty bugs to take care of.

cosmicdreams’s picture

@nod_ I like the ideas here but I disagree on what the bare minimum of what we can do here is. The bare minimum is that we provide a plugin that when used by developers would store data that would be bound for cookies into the localStorage instead.

Once the work to allow Drupal to have plugins lands, we might even be able to implement this as a contributed module.

sun’s picture

FWIW, this is a more complete library along the lines I'd imagine:

jQuery jStore: http://code.google.com/p/jquery-jstore/

I'm not suggesting to use this library, but merely want to point out the built-in multiple storage engine and fallback mechanism in this one.

ericduran’s picture

If we're evaluating libraries here's a list of client side storage libraries: https://github.com/bebraw/jswiki/wiki/Storage-libraries

Personally I've found lawnchair to be the nicest

nod_’s picture

#1569648: Follow-up: Use localStorage to store tableDrag.showWeight value just pointing out a possible real world use, could help the discussion. As you can see in the other issue, it's not so horrible we have to use a library to deal with it.

I'm still not convinced we need a pluggable JS storage thing.

nod_’s picture

Issue tags: -html5

tag

nod_’s picture

Title: Evaluate standardized usage of HTML5 Storage API (e.g. through a library) » Evaluate standardized usage of HTML5 Storage API through a library
Version: 8.x-dev » 9.x-dev
catch’s picture

Version: 9.x-dev » 8.1.x-dev
Status: Active » Postponed

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.0-beta1 was released on March 2, 2016, which means new developments and disruptive changes should now be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.0-beta1 was released on August 3, 2016, which means new developments and disruptive changes should now be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.3.x-dev » 8.4.x-dev

Drupal 8.3.0-alpha1 will be released the week of January 30, 2017, which means new developments and disruptive changes should now be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.0-alpha1 will be released the week of July 31, 2017, which means new developments and disruptive changes should now be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.0-alpha1 will be released the week of January 17, 2018, which means new developments and disruptive changes should now be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.6.x-dev » 8.7.x-dev

Drupal 8.6.0-alpha1 will be released the week of July 16, 2018, which means new developments and disruptive changes should now be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

nod_’s picture

Version: 9.3.x-dev » 10.0.x-dev
Status: Postponed » Closed (won't fix)
Issue tags: -JavaScript +JavaScript

Cross browser support is good, no strange behaviors. I don't think we need a library for consuming it in core, or provide an special API that contrib can use.