Problem/Motivation

There's huge resistance to adding front-end libraries (e.g. jQuery, Backbone.js, etc.) in core, because our current policy is to never update them once a stable release of Drupal core comes out. This means we're constantly lagging behind other front-end communities on compatibility, reducing the utility of these libraries, and create a janky experience for sites in the process (e.g. "deprecated" warnings from using out-dated versions of jQuery).

Proposed resolution

@sun proposes the following in #7:

  1. Commit multiple versions of a library over time, and have consuming code specify the exact version it is compatible with.
  2. Once there's a new version in core, the compatibility definition of consuming code would turn into a min/max version range.
  3. As soon as all consumers are compatible with a new version, the new one is loaded and the old one no longer.
  4. This means that consuming code can (should) progressively advance to use new features/APIs of the new library version — however, either using fallback/polyfill code for the case when the old version is loaded, or loading an entirely different consumer/integration file per version.

This might sound similar to Libraries API's (/Wysiwyg's) approach, which turns the situation entirely around and says "The library is always supplied by the user, your consuming code must be able to work with whatever version the user has downloaded." — however, the situation is not the same with libraries that are shipped/bundled/distributed with core (or a module), because in that case, there is a given contract that the bundled library is known to exist in that version and variant; it cannot and must not simply change under the hood without careful planning, notice, and updates of consuming code.

Therefore, the above proposal takes that into account, by allowing consuming code to progressively improve/renew.

If you're running on core only, then you will immediately use the new library versions (assuming that core immediately adapts its consuming code for updated library versions). If you're running 100 contributed modules, then the community needs to patch those modules for a new library version; and as soon as all dependencies on the old version are resolved, the new version is used.

Now that we're using JS only via libraries, it essentially boils down to additionally specifying compatible versions and/or version-dependent integration files per library.

@effulgentsia further follows up in #38:

  • Any JS added to drupal that has a dependency on anything else must be added via drupal_add_library(), not drupal_add_js(). Therefore, all dependencies can be registered in hook_library_info(). I think we're already doing this in D8.
  • Within hook_library_info(), we specify 'js' and 'dependencies' on a per version basis. E.g.:
    $libraries['jquery_ui']['version']['2.0'] = array(
      'js' => 'path/to/jquery-ui-20.js',
      'dependencies' => array('system', 'jquery', '2.0'),
    );
    $libraries['jquery_ui']['version']['2.1'] = array(
      'js' => 'path/to/jquery-ui-21.js',
      'dependencies' => array('system', 'jquery', '2.1'),
    );
    

    Exact syntax on allowing ranges TBD.

  • Calls to drupal_add_library() don't specify version, just the library. If you want a specific version, register your own custom library in hook_library_info(), list the dependency that JS code has, and then call drupal_add_library() of that custom library.
  • When Drupal renders the scripts for the page (e.g., within drupal_get_js() or similar), it looks at the entire list of libraries added to the page, and runs some algorithm to determine which version of each library to provide: requiring compatibility, and among options that are all compatible, choosing the maximum. Details of this algorithm TBD.
  • If it's impossible to satisfy version compatibility for all libraries (e.g., Library A depends on jQuery 2.1 and Library B depends on jQuery 2.0), Drupal starts dropping top-level libraries (ones that no other library on this page depends on) until it can output a set that are version compatible. [EDIT: To be replaced with AMD's methodology when/if #1542344: Use AMD for JS architecture happens.)
  • Dropping libraries? Is that ok? Yes, because page markup needs to work without JS anyway. It should always be possible to drop a library (that no other library depends on) and still have a functioning (accessible) site, just with a degraded UX.

Make sure to read and understand the consequences of this concept outlined in #51.


Original report by effulgentsia

In #1774312-40: Create.js: yay or nay *and*: when?, nod_ said:

core needs to decide on a strategy about library updates. If we can't/won't update during D8 stable release, [adding Create.js to core] will not work.

It's very unfortunate if our current approach of freezing JS library versions for an entire major Drupal version prevents us from adding really good libraries that can benefit core. At the same time, we need to be conservative about introducing BC breaks into minor releases: otherwise, people will stay on outdated minor releases and not be able to receive security updates.

How can we solve this? #586146: [policy, no patch] Decide if and how to implement semantic versioning for Drupal 8.x might be one way. Any others?

Note, this isn't just about Create.js and its stack. Recent comments on #1252178: Add Modernizr to core are wanting Modernizr in core too.

Comments

RobLoach’s picture

Title: [meta] Strategy for updating vendor JS libraries within a major stable version » Figure out strategy for updating vendor JS libraries within a major stable version
Priority: Major » Normal
Status: Reviewed & tested by the community » Active
Issue tags: -Needs architectural review, -Libraries

Would love to have Bower, JamJS, EnderJS, or Composer handle the JavaScript packages for us.

seutje’s picture

Issue tags: +Composer

Not sure how a package manager would help us here.
For instance, if an update in library x fixes a bug module y depends on, we wouldn't catch that, and personally, I don't know how we could.

RobLoach’s picture

For instance, if an update in library x fixes a bug module y depends on, we wouldn't catch that, and personally, I don't know how we could.

Had that conversation over at #1451056: [policy] How to handle unforeseen diversion of Symfony code in stable/API-locked Drupal core?. If you need to run a fork of a package, you can. But we really should work with the projects that we use in order to make their code for us.

If we update Library X, and that update breaks module Y, then we should work with the Library X community in order to make it work for module Y. Either that or fix module Y... We're already working very closely with both the Symfony and Composer communities, and I'd like to see it done in the JavaScript space as well. Scott Gonzalez from the jQuery UI team is regularily in #drupal-contribute with Drupal-compatibility questions too. I'm sure the Create.js team is open to communications if they arn't already.

For package management with Create.js specifically, it would probably need a package.json file.

catch’s picture

There was a fair bit of discussion of this thread, although only related to jQuery in http://groups.drupal.org/node/210973.

I really hate the situation we have with jQuery, it does more harm than good when sites are running 4 year old versions of jQuery since it directly affects site visitors in terms of crappy front end performance (like all those chrome deprecated console errors).

So my current position is that we fix this issue or just don't add any external js libraries to core any more, so it'd be good to fix this... I don't have great ideas on how to do that, although we could potentially change our BC policy for stable releases to only be 'best effort' for JavaScript - i.e. if you have js in your contrib module, you're expected to keep that up-to-date with the library it depends on rather than the major core version?

effulgentsia’s picture

I added a note about Modernizr to the issue summary to make clear that this issue is not motivated solely by Create.js.

catch’s picture

Priority: Normal » Major

This blocks several other issues, bumping priority for some more visibility.

sun’s picture

Issue tags: +Libraries

we could potentially change our BC policy for stable releases to only be 'best effort' for JavaScript - i.e. if you have js in your contrib module, you're expected to keep that up-to-date with the library it depends on rather than the major core version?

I perfectly understand that it's only one idea spoken out loudly, but I have to strongly disagree with this, because it will add a barrier for people to update their modules. I'm specifically thinking of security updates -- if you cannot be sure that your site still works when updating modules/themes, then you will delay updates "until there's time for extensive manual testing."

I thought a lot about this problem space for Libraries API already. From my perspective the only proper way to address this would be:

  1. Commit multiple versions of a library over time, and have consuming code specify the exact version it is compatible with.
  2. Once there's a new version in core, the compatibility definition of consuming code would turn into a min/max version range.
  3. As soon as all consumers are compatible with a new version, the new one is loaded and the old one no longer.
  4. This means that consuming code can (should) progressively advance to use new features/APIs of the new library version — however, either using fallback/polyfill code for the case when the old version is loaded, or loading an entirely different consumer/integration file per version.

This might sound similar to Libraries API's (and Wysiwyg's) approach, which turns the situation entirely around and says "The library is always supplied by the user, your consuming code must be able to work with whatever version the user has downloaded." — however, the situation is not the same with libraries that are shipped/bundled/distributed with core (or a module), because in that case, there is a given contract that the bundled library is known to exist in that version and variant; it cannot and must not simply change under the hood without careful planning, notice, and updates of consuming code.

Therefore, the above proposal takes that into account, by allowing consuming code to progressively improve/renew.

If you're running on core only, then you will immediately use the new library versions (assuming that core immediately adapts its consuming code for updated library versions). If you're running 100 contributed modules, then the community needs to patch those modules for a new library version; and as soon as all dependencies on the old version are resolved, the new version is used.

Now that we're using JS only via libraries, it essentially boils down to additionally specifying compatible versions and/or version-dependent integration files per library.

EDIT: @tstoeckler is actively working on an entirely rewritten Libraries API for D8, so please talk to him, before anyone starts to work on this.

casey’s picture

What if we define our own Javascript interfaces and require external libraries to provide their own implementation. That way we don't have to include these libraries in core, just a package manager, config for the default implementations and something like symfony's dependency injector but then in Javascript.

Drupal.register('dom', jQueryDrupalDOM);
(function ($) {

}(Drupal.get('dom')));
sun’s picture

@casey: The problem is not "library." The problem is "version."

casey’s picture

Yes, but isn't it the case that there are lots of issues popping up were new external libraries are wanted? What I propose is to not include these libraries but instead include na interface (probably mostly matching the methods of the external library) and ask the maintainers of the external library to provide an implementation.

Custom core code should rely on these interfaces rather than the external libraries. As time passes new faster/better implementations may arise. Website owners can update the libraries without worries.

sun’s picture

...and ask the maintainers of the external library to provide an implementation.

That's utopian.

casey’s picture

You're probably right. Also on your #9 statement.

webchick’s picture

So I will bite. Why is the current approach of "jQuery Update"-esque modules in contrib not sufficient for dealing with this issue? Because Spark is targeting D8, we add a dependency to jQuery Update in our various modules' info files to get jQuery 1.7 instead of whatever Drupal 7 ships with. Done and done.

catch’s picture

@webchick: http://drupal.org/project/usage/jquery_update

- less than 20% of sites use it
- this means 80% of sites are using an extremely outdated jQuery, with the front end performance, deprecated console log issues and the rest that this entails.

attiks’s picture

The problem is that other contrib modules are assuming a certain version, and you'll have end up with problems, we added several fixes to clientside_validation, because they aren't part of jQuery 1.3.2, but now we start running into problems like #1801764-2: Clientside Validation breaks AJAX file uploads.

jquery_update might solve some problems, but it would be nice if core just was using a 'recent' version and all contrib can live happily together.

steveoliver’s picture

sun, in #7:

As soon as all consumers are compatible with a new version, the new one is loaded and the old one no longer.

seems ideal to me as a developer.

moshe weitzman’s picture

- less than 20% of sites use it
- this means 80% of sites are using an extremely outdated jQuery, with the front end performance, deprecated console log issues and the rest that this entails.

I don't think that either of these is a significant problem. Those sites are also using a an old Drupal, and old MySQL, etc. Thats how they roll. And there isn't any shame in that.

I agree with webchick that our current approach is decent. If someone wants to add the library API that sun describes in #7, thats great. Thats the only real solution to this. FYI, D7 module dependencies can already be version specific. The fact that hardly anyone uses this feature is telling. The version mismatch problem may be overstated. I know that JS libraries are different from modules, but still. I don't think fear of version mismatch or version decay should inhibit us from using all the great JS code thats out in the wild..

effulgentsia’s picture

How about a simpler variant of #7:

  • Ship 8.0 with whatever the most recent versions of vendor js libraries are out at the time.
  • Create contrib projects for each one. e.g., "pinned_jquery", "pinned_backbone", etc., where each of these implements hook_library_alter() to use its version rather than core's, and has a d.o. project version release (e.g., 8.x-1.0) containing the same library version as what's in core 8.0 (making it a useless but harmless module to start with, but keep reading).
  • When a core-included vendor library has a new release available, file a core issue to update core to use it, including fixing core js code that uses it to work with the new version while still also working with the old version. When that issue is resolved (i.e., the new version is committed to 8.x, replacing the old version), also add the new version to the corresponding pinned_* project and create a release for it (e.g., pinned_jquery 8.x-1.1).
  • When the next core release is shipped (e.g., 8.4 or whatever number we're on at that time), add release notes saying which library versions have been updated, and point out that for any site maintainer concerned about the update, that they can install a version of pinned_* that keeps them on what they're used to. In other words, there is no reason not to install the updated core.
  • Meanwhile, contrib modules can choose to list particular ranges of pinned_* as dependencies, using the normal dependencies[] feature of MODULE.info.

What this approach allows:

  • Core only needs to ship with one version of each library, though its consuming code needs to work with all versions since whatever X.0 shipped with (similar to when PHP and browser updates happen).
  • Most simple sites won't bother installing the pinned_* projects if they don't need to, allowing them to by default stay current, addressing #14.
  • We'll be able to track statistics on what sites are actually using via the drupal.org/project/usage/pinned_* page.
  • For sites using an old pinned_* version, Drupal's already existing Update Status feature will gently nag admins to update to the newest versions, hopefully prompting them to file bug reports (and ideally, contribute to fixing those bugs) with whichever contrib project is keeping them on the old version.

This approach requires us to write no new code. It's just a policy decision.

If #1542344: Use AMD for JS architecture gets figured out in a way that cleanly allows multiple versions on the same page, then we can potentially switch to that approach at that time.

moshe weitzman’s picture

Wow. A comprehensive solution that requires only policy change. Great thinking, effulgentsia!

attiks’s picture

As Moshe said and Sun explained in #7, the only solution is the support version specific versions. When only using core, or one set of modules there isn't a problem, but from the moment you start using independent modules, each using javascript you'll run into problems.

Are there any plans to move Libraries (as described in #7) into core?

effulgentsia’s picture

When only using core, or one set of modules there isn't a problem, but from the moment you start using independent modules, each using javascript you'll run into problems.

This isn't a JS-specific or new problem. If a particular version of Views has a dependency on a particular version of CTools, and a particular version of Rules has a dependency on a different version of CTools, then you can't run a site on that version of Views with that version of Rules. It becomes each module's responsibility to be compatible with broader ranges, or at least the newest one, if that module wants to integrate well with others. And it's the site builder's responsibility to select modules whose dependencies are reasonable and mutually compatible.

As I understand it, #18 is conceptually similar to #7, but using capabilities we already have in MODULE.info and d.o. project versioning as the implementation. And the only way to allow module X and module Y to both be enabled and simultaneously using different versions is with something like #1542344: Use AMD for JS architecture, and if that happens, great, but I don't think we should hold up everything on that, when #18 is better than what we did for Drupal 5, 6, and 7.

attiks’s picture

#21 You're right and never intended to imply we should wait, but I was just wondering if somebody was working on this. The solution in #18 is indeed nice (and an improvement) in the mean time. So +1 on the proposal in #18.

jessebeach’s picture

+1 to the proposal in #18. You're a clever fellow effulgentsia!

RobLoach’s picture

This was the original intent of hook_libraries_info(). Ability to move things to contrib and alter them if needed. We already have the architecture in place for it, so let's use it.

Crell’s picture

#18 sounds great to me.

Crell’s picture

Issue summary: View changes

Updated issue summary.

webchick’s picture

Title: Figure out strategy for updating vendor JS libraries within a major stable version » [Policy, no patch] Figure out strategy for updating vendor JS libraries within a major stable version
Status: Active » Reviewed & tested by the community

Sounds like we're heading towards consensus. I've updated the issue summary w/ effulgentsia's proposal; marking RTBC to see what others think.

nod_’s picture

+1 to really nice solution for a painful issue.

About the AMD thing even if it's possible I don't really want to make it work, it'd encourage people to load 3 versions of jQuery or whatever on the same page. Not a good thing.

Grayside’s picture

jqmulti contrib module was a best case of a bad situation because D6 hadn't moved forward far enough for even jquery_update to catch it up to yesteryear. This proposal sounds solid, easy to understand, and has that light-going-on feeling.

Advance notifications of library updates to give contrib time to get ready would be good, as would working with external communities to make sure anything on the deprecation chopping block is understood as such.

sun’s picture

I'm not able to leave a longer comment.

I don't think #18 solves any problem. It moves the problems to a more complex space instead. Semantically, it introduces and shares the same defects as #4, obfuscating the problem that core breaks an API compatibility contract, which will have the consequence of users not updating their sites.

Version-specific module dependencies are their own can of worms. There are some good cases for using them, and there are good reasons for why they are used sparingly. Module dependencies are acting on a global, site-wide level. Libraries are not. It is not a good idea to use them to work around this problem space.

I also think the assumption that consuming code can be adjusted to be compatible with all versions of all libraries is misleading/wrong. This is why I was rather thinking of version-specific consumer/integration files in #7.

seutje’s picture

This sounds like a solution that could actually work, and it would also mean we'd have maintainers for each of these 3rd party libraries, but I guess most of them already have contrib modules anyway.

sun’s picture

Title: [Policy, no patch] Figure out strategy for updating vendor JS libraries within a major stable version » [meta] Strategy for updating vendor JS libraries within a major stable version
Status: Reviewed & tested by the community » Needs review

Improving issue title for mobile sanity.

RobLoach’s picture

This issue is becoming much less of a problem as more projects adopt the CommonJS standard. Projects are starting to declare their own dependencies in a package.json file, which encourages decoupling and becoming less dependent on a specific version of their dependencies. Once there's a package.json file, it can not only be used in an AMD architecture like RequireJS, but also be consumed by a package manager like JamJS or Bower.

When Drupal is bringing in a JavaScript project, we should:

  1. Work with the project to introduce a package.json file
  2. Use that package information in hook_library_info()
  3. Use hook_library_info_alter() in contrib to allow upgrading the package

Again, the package.json file isn't necessarily a requirement, but it does encourage the project to think about their dependencies, and decouple themselves from a specific version of the libraries. The fact that we could use that information is of benefit to us.

effulgentsia’s picture

obfuscating the problem that core breaks an API compatibility contract

No contract is broken. Even in #7, you suggest that consuming code be *allowed*, not required to declare min/max versions of a dependent library. Similarly, #18 allows consuming modules to declare dependency on particular versions of a pinned_* module. If they do so, the core update does not result in a change to the version of the library that the site uses. If no modules declare a dependency on pinned_*, then there's no contract preventing a version update of the library.

which will have the consequence of users not updating their sites

Even if no modules declare an explicit dependency on a particular pinned_* version, #18 suggests that our release notes mention the libraries being updated, and pointing the site maintainer to the corresponding pinned_* projects if they want to be extra cautious and stay on their existing library versions. Why would this not be sufficient removal of barriers to applying the latest core updates?

Module dependencies are acting on a global, site-wide level. Libraries are not.

Can you clarify where the difference is something we want to consider? Here's differences that I question the practical validity of (using jQuery as an example, but discussion applies to any library):
- A given module has 2 different features/widgets/whatever. Feature 1 has js code that depends on jQuery version X. Feature 2 has js code that depends on jQuery version Y. I don't think we need to support this: module maintainers should at least be internally consistent in what library versions their code works with.
- Module 1 has code that depends on jQuery version X and Module 2 has code that depends on jQuery version Y, but these consuming scripts are for entirely different UI pieces (e.g., Views UI vs. Rules UI), so #18's effect of disallowing both modules to be enabled on the same site is too constraining, whereas #7 would allow for this. However, I question the wisdom of relying on different UI pieces never interacting on the same page, since one of the benefits of Drupal is the ability to easily remix and recombine UIs.
- Module 1 has code that depends on jQuery version X and Module 2 has code that depends on jQuery version Y and both exist on the same page. #18 disallows that. I could see the benefit of a solution that would allow this properly, but per #21, I think it's better to make that a separate issue than hold up a resolution on this one.

I also think the assumption that consuming code can be adjusted to be compatible with all versions of all libraries is misleading/wrong. This is why I was rather thinking of version-specific consumer/integration files in #7.

So, as an example, are you saying, module Foo wants to work with any version of jQuery, but to do so, needs to supply two (or three or four ...) different consuming JS files and therefore, also make its hook_library() definition dependent on the actual version of jQuery used by the site? This sounds like a nice capability, but how much of this do we need to figure out / implement before we allow new libraries (e.g., Backbone) into core?

RobW’s picture

Questions/ extrapolations on the practical consequences of #18 -- please correct me if I'm wrong:

Core would never be able to use any new syntaxes or completely new features of library versions released after 8.0, but would benefit from performance enhancements of the new library, etc. Unless the pinned_* module implemented something like jQuery update module in reverse (jQuery downgrade?), providing altered js to maintain functionality with its version of JavaScript.

If a module added a dependency on pinned_*, it requires a previous version of the library. How does that work sitewide? For instance, if module B requires the latest version of that library, does pinned_* load its library just when the module dependent on its code runs, or does it load site wide, breaking module B? How does this work when there's no explicit dependency on the latest library?

It would be all module maintainers' responsibility to maintain up to date js, or a core update could break a user's site. It seems like there needs to be a grace period once a new version of a library is released before it's bundled in core so that module maintainers can catch up -- maybe it goes into core -dev immediately and is pushed out no less than 1 month later. This is definitely a shift in responsibility but I'm all for it. if pinned_* versions and new versions can exist on the same site (or it looks like the same page once an AMD architecture solution goes in), pinned_* will be an easy fix for modules that don't have time/ expertise to update their js, at the expense of overall site performance. We can't have our cake and eat it too, and this seems like a good balance between not breaking things and incentivizing keeping up to date. Hopefully, a pinned_* dependency would be a call for front end developer help, attracting the people who currently work on modules like jQuery update.

catch’s picture

It seems like there needs to be a grace period once a new version of a library is released before it's bundled in core so that module maintainers can catch up -- maybe it goes into core -dev immediately and is pushed out no less than 1 month later.

Something like that is entirely reasonable and I doubt we'd be capable of updating any quicker anyway.

webchick’s picture

Right, we try and do that with other changes in a stable release, like string fixes, which impact contributors. But AFAIK, the other option is "lazy contrib" just adds dependencies[] = pinned_jquery 1.0 (or whatever) to their .info files, so they don't actually need to worry about updating their code unless users start filing complaints (and patches) about conflicts with module X.

effulgentsia’s picture

#34 made me realize some stuff that's now making me appreciate #7 more. Will write up a comment with my new understanding shortly.

effulgentsia’s picture

From #29:

I also think the assumption that consuming code can be adjusted to be compatible with all versions of all libraries is misleading/wrong. This is why I was rather thinking of version-specific consumer/integration files in #7.

From #34:

Core would never be able to use any new syntaxes or completely new features of library versions released after 8.0

Ok, after reading these two quotes side by side, I think I finally got what sun was pointing out is wrong with #18. The following example is likely bogus, as I don't think the following libraries will actually be this restrictive (and I'm using future library versions that don't yet exist), but consider the possibility that we ship Drupal 8.0 with jQuery 2.0, jQuery UI 2.0, and Aloha editor 0.30.0. Suppose also that some time later, jQuery releases 2.1 which is sufficiently different from 2.0 that jQuery UI 2.0 doesn't work with jQuery 2.1, and so jQuery UI 2.1 is released which only works with jQuery 2.1 and not with jQuery 2.0. Suppose further that this makes Aloha 0.30.0 not compatible with jQuery UI 2.1, and so Aloha 0.31.0 is released which is compatible with jQuery UI 2.1, but not with 2.0. In other words:
- jQuery UI 2.1 depends on jQuery 2.1; jQuery UI 2.0 depends on jQuery 2.0
- Aloha 0.31.0 depends on jQuery UI 2.1; Aloha 0.30.0 depends on jQuery UI 2.0

The above would make #18 a problem, because we can't make something in core have a dependency on something in contrib, so we can't update core to the new Aloha editor and give it a dependency on a particular version of pinned_jquery_ui. But, if we update to the new Aloha editor without giving it that dependency, then a site can be running a version of pinned_jquery pinned to jQuery 2.0, with nothing preventing that, and with Drupal suddenly broken.

So indeed, I think #7 is the correct idea. Specifically:

  • Any JS added to drupal that has a dependency on anything else must be added via drupal_add_library(), not drupal_add_js(). Therefore, all dependencies can be registered in hook_library_info(). I think we're already doing this in D8.
  • Within hook_library_info(), we specify 'js' and 'dependencies' on a per version basis. E.g.:
    $libraries['jquery_ui']['version']['2.0'] = array(
      'js' => 'path/to/jquery-ui-20.js',
      'dependencies' => array('system', 'jquery', '2.0'),
    );
    $libraries['jquery_ui']['version']['2.1'] = array(
      'js' => 'path/to/jquery-ui-21.js',
      'dependencies' => array('system', 'jquery', '2.1'),
    );
    

    Exact syntax on allowing ranges TBD.

  • Calls to drupal_add_library() don't specify version, just the library. If you want a specific version, register your own custom library in hook_library_info(), list the dependency that JS code has, and then call drupal_add_library() of that custom library.
  • When Drupal renders the scripts for the page (e.g., within drupal_get_js() or similar), it looks at the entire list of libraries added to the page, and runs some algorithm to determine which version of each library to provide: requiring compatibility, and among options that are all compatible, choosing the maximum. Details of this algorithm TBD.
  • If it's impossible to satisfy version compatibility for all libraries (e.g., Library A depends on jQuery 2.1 and Library B depends on jQuery 2.0), Drupal starts dropping top-level libraries (ones that no other library on this page depends on) until it can output a set that are version compatible.
  • Dropping libraries? Is that ok? Yes, because page markup needs to work without JS anyway. It should always be possible to drop a library (that no other library depends on) and still have a functioning (accessible) site, just with a degraded UX.

Example: suppose in addition to the jQuery, jQuery UI, Aloha stack above in core, a site is also running contrib modules A and B. A defines Library A 1.0 that depends on Aloha 0.31.0, and B defines Library B 1.0 that depends on jQuery 2.0 (only, not 2.1). On this site:
- when someone visits a page that uses Library A only: gets A 1.0, Aloha 0.31.0, jQuery UI 2.1, jQuery 2.1
- when someone visits a page that uses Library B only: gets B 1.0, jQuery 2.0
- when someone visits a page that uses Library A and Library B: incompatible dependencies on jQuery. Drupal drops either A or B based on some logic TBD, and then outputs one of the above two.

Is this a correct understanding of #7? If so, I'm on board with it. How about others?

RobW’s picture

I think #38 is a little more robust than the pinned module idea (although I really liked the elegance and mental model of pinned). Something not mentioned yet that I would like to see in any solution is documentation and possibly Drupal set message warnings (attached to a permission) that make it clear that including different versions of a library on one site is extremely discouraged -- I mean, if you use jQuery 1.8 and 1.9 on the same site some shaming is in order. Loading multiple versions is a stopgap that keeps your site working while module maintainers fix the problem. That it's possible sometimes leads people to think they should use it, which we should avoid here at all costs.

Also, how would themes work with this? A good percentage of JavaScript is in the theme layer -- maybe not backbone level stuff, but jQuery and other ui libraries certainly. Will themes declare dependencies as well, or do we assume that they work with whatever happens to be loaded on a particular page?

Wim Leers’s picture

#38: nice synthesis :)

The only thing I disagree with is this:

- when someone visits a page that uses Library A and Library B: incompatible dependencies on jQuery. Drupal drops either A or B based on some logic TBD, and then outputs one of the above two.

Neither A nor B should be dropped. Thanks to AMD, we will be able to load multiple versions of jQuery on the same page. Yes, as @RobW indicates in #39, "some shaming is in order" for doing this, but IMO *this* is the grace period. Temporarily sacrifice performance until everything is updated to the latest & greatest.

I think you worded it the way you did because we're not yet using AMD? (I.e. #1542344: Use AMD for JS architecture.)

Kiphaas7’s picture

Loading multiple (possibly incompatible) versions of a library should be possible, but really, some sort of warning should be given. It's just screaming 'wrong' all over.

Something like "Look, I'm awesome code and I keep your site running, despite that you want me to load 5 incompatible versions of the X library, but really, get it together (wo)man."

Anyway, +1 to #9 but also +1 to AMD.

rooby’s picture

I also like the idea of #38 & #40.

I think that is probably the best we could do when requiring different versions.

If we don't end up with AMD it still needs to be as easy as possible for developers to use multiple versions at once if they want to.

sun’s picture

Yes, I think #38 is a proper understanding of #7 and #29.

As long as AMD isn't in core yet, I'd keep the incompatibility resolution process KISS: FIFO. If an earlier library adds jQuery 2 and a later library needs 1.9, the later is ignored.

If and when there will be AMD in core, we can possibly remove the ignorance.

This approach resolves each and every multidimensional dependency and compatibility issue that exists.

And, yes, this means that we will potentially commit many versions of libraries to Drupal core over time (especially given a total maintenance cycle of ~6 years). But all of that is mainly dead code, and whoever measures the quality of software by its download package size is stupid and to be ignored anyway.

Lastly, there is still room for us to say at a later point in time:

Look, it's 2016. All jQuery 1.4 - 1.6 releases are completely broken, it's impossible that anyone can reasonably still use this on any site, and our awesome new library usage stats on drupal.org indicate that, indeed, no one is using them anymore. git rm that shit.

Alas, without having the advantage of library usage stats, Wysiwyg module is going to do exactly that for TinyMCE 2 in its next release, because, you know, that version is full of bugs and stone-age and cannot reasonably work for anyone anymore.

sun’s picture

Issue summary: View changes

Updating the issue summary with proposal from #18, which seems to be building consensus.

webchick’s picture

Status: Needs review » Reviewed & tested by the community
Issue tags: -Libraries

Ok, I've updated the issue summary again on what I believe is the current consensus. Marking back to RTBC to see if this is in fact the case.

webchick’s picture

Issue tags: +Libraries

WTF, Drupal.org?

nod_’s picture

Status: Reviewed & tested by the community » Needs review

That sounds good to me, what more do we need to get started on the implementation?

nod_’s picture

Status: Needs review » Reviewed & tested by the community

(edit) guess I have my answer, sorry about the status change :p

seutje’s picture

In the case of dropping libraries or loading multiple versions of the same library, a message on the status page definitely wouldn't hurt.

catch’s picture

This looks great to me.

tstoeckler’s picture

Yup, totally makes sense. This is in essence, what jquery_update.module does, I think, and it totally makes sense to put the "burden" of updating core JS on core devs. Since that will need to be done for the next Drupal version anyway, we're not really adding any "burden", though, anyway. In short, +1. :-)

sun’s picture

I think that the strategy is pretty straightforward, and the required technical implementation relatively simple.

(Given how simple it is, it's a huge bummer that it can't be retroactively applied to D7... but alas, it required some years of thinking through the Libraries API problem space to get a better understanding of it first.)

However, what I believe to be most controversial in this proposal is the following part, and we should make sure that everyone understands it and its consequences:

yes, this means that we will potentially commit many versions of libraries to Drupal core over time (especially given a total maintenance cycle of ~6 years). But all of that is mainly dead code, and whoever measures the quality of software by its download package size is to be ignored anyway.

  1. D8 in 2013 will most likely ship with:
  2. D8 in 2014 will most likely ship with:
    • jQuery 2.0
    • jQuery 2.6
    • jQuery 1.9.0
    • jQuery 1.9.8
    • jQuery UI 1.9
    • jQuery UI 2.3
    • Backbone 1.0
    • Backbone 1.2
    • Underscore 1.6
    • Underscore 1.12
  3. D8 in 2015 will most likely ship with...

    [...add +.6 to each library, and perhaps even new major versions...]

  4. D8 in 2016 etc.pp.
  5. ...
  6. D8 in 2019 will end with [all of the above cumulated].

That's not the only consequence though. The other one, and perhaps more interesting for contributed modules and themes as well as site builders is very potentially this:

/modules/admin_menu/js/admin_menu.j2.0,b1.0.js
/modules/admin_menu/js/admin_menu.j2.6,b1.0.js  # Assuming a BC break in jQuery 2.6
/modules/admin_menu/js/admin_menu.j2.6,b1.2.js  # Assuming a BC break in Backbone 1.2
...

Separate compatibility/consumer/integration files are not guaranteed to happen, but definitely a desired and wanted possibility and consequence of the concept.

Not all BC breaks and library improvements will need separate and new support/integration files in modules - quite some of them can most probably be handled with conditional runtime logic.

And of course, lastly, there's the idea of using AMD, which would essentially cause every module to load its latest and greatest integration files at all times. I agree that sounds exciting... However, given my experience with libraries, I have my own reservations on the multidimensional problem space of contributed modules; i.e., how quickly each and every contrib project will adapt its code to the newest library versions — as long as that does not happen in a speedy way, and when your typical site contains ~100 modules, this can potentially and easily mean that, in 2017, you will load 5 different versions of jQuery on a single page with AMD (2013 + 5 = 2017). At that point, it might make more sense to search for the common denominator of all libraries attempted to be loaded for a single page and try to ignore everything else (which might make more sense than the FIFO processing I mentioned earlier).

Anyway, the last 3 paras are technical and real world details to figure out. What matters is to understand the architectural concept and its consequences.

Wim Leers’s picture

Thanks for describing the worst-case scenario, @sun! It's indeed important to grasp the full consequences.

P.S.: Personally, I'm slightly more optimistic. More often than not, BC breaks don't imply *huge* differences. I think that even if module maintainers don't upgrade, that *users* of those modules will contribute patches, so that they have to load less ridiculous amounts of JS. Whereas right now, module maintainers have less of an incentive, since "upgrading" implies e.g. supporting both core's jQuery and jquery_update's jQuery. Future will tell, of course.

sun’s picture

Tagging. I don't expect objections/pushback, but some more feedback from the wider community would be good to have.

tstoeckler’s picture

As a contrib module author, if you have to choose for which version of jQuery* you write your code, you will always want the version that satisfies two conditions:
1. It has as many of the latest and greatest features as possible. (Self-explanatory)
2. It is supported by the largest share of consumer code, i.e. core and other contrib JavaScript. Reason:
- We have already established that in the case of conflicting versions (i.e. contrib_module_a requires jQuery 1.9 and contrib_module_b requires jQuery 2.0), some consumer code will simply not get loaded. We require modules to degrade gracefully anyway, so that's OK. But you want to provide the best possible experience for your users. So if you know that e.g. Rules module doesn't work with jQuery 2.0, you might not want to depend on jQuery 2.0 yet, because users of your module will have a degraded (and perhaps poor**) experience with either your module or Rules module. Or if jQuery 2.1 is new, and core JavaScript doesn't support it yet, you likewise might not want to go there yet, because your users won't have vertical tabs and overlay.

I think we can achieve both, at least to an acceptable degree, if we have a set of Recommended JS library versions at any given time. Any Drupal 8.x will always support the versions that Drupal 8.0 shipped with, but maybe with Drupal 8.3 we would rather recommend newer versions of the libraries. Most likely, these recommended versions will simply be the latest versions Drupal core supports (since that will presumably not change every month), but I think we need to very explicit about documenting
- A: What the recommended versions are at any given time
- B: When we change those versions. We already have the ability to add change notices for minor versions, so that should also be managable.

* This obviously applies to any major JS library we have, but with jQuery this problem is faced most often.

** This is purely a hypothetical example. In no way do I want to insult the no-js version of the actual rules.module. I have never even tried that out, TBH.

sun’s picture

@tstoeckler:
Perhaps I'm missing something, but the scenario you describe is exactly what this proposal will avoid:

in the case of conflicting versions (i.e. contrib_module_a requires jQuery 1.9 and contrib_module_b requires jQuery 2.0)

The entire point is that both modules can support both versions by providing integration files for both. There's no need to wait for Rules or any other module to become compatible. Your module just simply makes sure that it provides integration files that are compatible with all supported versions.

If you're lucky, library versions won't matter at all and you will live with a single integration file forever. Slightly more advanced modules will need to check that their code is either compatible across versions or that they provide different integration files for varying min/max compatibility version ranges.

sun’s picture

Issue summary: View changes

Updating issue summary to reflect new consensus.

sun’s picture

Issue summary: View changes

Updated issue summary.

tstoeckler’s picture

Re #55:

The entire point is that both modules can support both versions by providing integration files for both.

That is correct, and I'm aware that that very point will mitigate most cases of version incompatibilities. I'm also aware that version incompatibilites are, for the most part, easily overcome. There is no guarantee, though, that Rules, or any other module, actually does provide multiple versions of its JS. And (only!) in that case, will you come across problem, if at least two (major) modules provide only a single version of its JS, and the two do not match. That is why I proposed to actively encourage module authors to update their JS from time to time by updating the current recommended version, and by documenting that this is expected.

Maybe I should have been more clear that this not a technical deficiency in the proposal at all.

jessebeach’s picture

As I understand the discussion in this issue, there's general consensus on the approach. The next step is to write code to support version detection of libraries as outlined here:

http://drupal.org/node/1342220#libraries-detection

We would also want some support for modules to declare their supported version or version range for drupal_process_attached. A flag should be raised on the modules page when the latest version of a core library != or is > that the max range value of a module's library's requirements.

I did a search for issues similar to these and didn't come up with exact hits, so I've added them.

The followup issue to effulgentsia's #38 bullet point "When Drupal renders the scripts for the page (e.g., within drupal_get_js() or similar), it looks at the entire list of libraries added to the page, and runs some algorithm to determine which version of each library to provide: requiring compatibility, and among options that are all compatible, choosing the maximum. Details of this algorithm TBD."
#1810352: Support version detection of core libraries when implemented through drupal_process_attached (drupal_add_library)

The followup issue to #1810352 that would make library incompatibilities known to a user on the modules page.
#1810366: Warn site maintainers on the modules page that a module's library requirements are incompatible or older than libraries in Core

darvit’s picture

If possible, perhaps as a separate issue, could the status report page (admin/reports/status) be redesigned to provide a more organized and complete listing of any installed 3rd party libraries?

Could show installed and recommended versions, links to download, license info, description, etc. and have groupings such as core /contrib , JS/PHP.

Finally, has there been official announcement to have libraries in core (and would it be used for core libraries ), and does this intiative need to be resolved before code freeze?

With so many “Proudly Found Elsewhere” components, libraries and a detailed reporting would be helpful to maintain version management.

Some in, some pending, some JS, some PHP:
Symphony , Aloha, Angular JS, ArchiveTar.php, Assetic, Backbone, Composer, Create JS, guzzle, JamJS, jQuery, jquery form plugin, jQuery UI, JSHint, knockout JS, Lootils Archiver, Twig, Underscore, VIE, etc.

tstoeckler’s picture

Finally, has there been official announcement to have libraries in core (and would it be used for core libraries ), and does this intiative need to be resolved before code freeze?

Note that the issue of external libraries is different to the issue of core- (or contrib-)shipped JavaScript. This particular issue is related to Libraries API, though. I am currently revamping Libraries API for D8, but I wouldn't consider it likely that I'll/we'll make it until December 1st at this point.

Dries’s picture

I need to spend some more time on this issue before I'm ready to approve this.

I worry about the potential performance issues (having to load multiple libraries from one page to another), stability issues with this patch (many modules are not regularly updated/maintained), and maintainability issue (we'd have to update and test multiple versions/branches of jQuery when security releases are made).

Long story short, it feels like we may be trading of end-user experience in favor of developer experience.

I also don't agree this issue is 'major' yet. It feels more like a 'normal' issue but will leave it at 'major' for now. It's not truly blocking any patches in my mind.

RobLoach’s picture

Title: Figure out strategy for updating vendor JS libraries within a major stable version » [meta] Strategy for updating vendor JS libraries within a major stable version
Priority: Normal » Major
Status: Active » Reviewed & tested by the community
Issue tags: +Needs architectural review, +Libraries
sun’s picture

The interpretation that this would only impact developer experience is not exactly right.

Newer versions of frontend libraries usually tend to contain performance improvements, ranging from small to massive. For example, the current version of jQuery is significantly faster than jQuery 1.4.4 that D7 ships with. In addition, it is considerably more stable and more compatible with older as well as newer web browsers. That affects user experience.

In fact, I'm inclined to say that this is primarily about user experience. I'm not even sure whether there will be a very big developer experience aspect to this in the real world. That is, because I assume that the majority of existing code in stable modules will only be ensured to be compatible with newer library versions, but I wouldn't expect major rewrites of consuming/integration code.

Major rewrites of consumer/integration code only tend to happen when there is a new major version of a library that either contains larger API changes or allows to significantly simplify the integration code. At least that's what we're experiencing in Wysiwyg module (which happens to use the concept of version-specific integration files already).

Nevertheless, the updated base/framework libraries will improve the frontend/user experience due to their own/built-in stability, compatibility, and performance improvements.

That said, adding support for new major versions is expressively encouraged with this concept; i.e., if core will add jQuery 2.0 at some point, contrib will be able to adopt it in a progressive way. Contrib will not remove the integration files for <2.0, but only add new ones for 2.0 (if needed, or just adjust the version compatibility range).

The main benefit is that we're detaching our release cycle from the release cycle of included libraries. This means that it doesn't matter anymore when exactly jQuery 2.0, jQuery UI 2, Backbone 1.0, and whatever will be released.

webchick’s picture

The performance trade-off is a good point to make.

But I think the scenario Dries is worried about is that http://drupalcontribstatus.com/ continues to show 9 outstanding modules that are still not yet ported to a full stable point release, despite the fact that Drupal 7.0 was released nearly 2 years ago, and Drupal 8 is less than a year out (we hope). And these aren't random silly modules, these are the top 60 modules that the most Drupal 6 sites relied on. So the assertion that contrib is going to "chase head" in terms of libraries during stable releases is most likely incorrect; it's going to highly depend on the maintainer.

For any of my own personal modules, for example, I would be far more inclined to peg them at whatever version of jQuery D8.x shipped with at the time I ported my module, because that means I never have to bother updating them again and can go work on other life pursuits, like committing core patches. :D

So that means as Drupal 8 ages, the likelihood that jQuery 1.9, 2.0, 2.2, etc. all get loaded on a single page increases as the life of Drupal 8 increases. This will kill off any performance benefit we would've received by shipping with a new default library in the first place.

webchick’s picture

Status: Reviewed & tested by the community » Needs review

Given #60 btw, it seems like we still need some more time to discuss this.

sun’s picture

I partially agree. That's why I'm not convinced of the egg-laying-wool-milk-pig that AMD is promoted to be by others.

I.e.: I don't think that concurrent loading of multiple library versions on a single page/response is desirable in any way.

In fact, it's more realistic to assume that:

  1. At the time of D8 release, all contrib will depend on the initial versions in core.
  2. After core added new versions, it will take time for contrib to get compatible. (again, either by adjusting version compatibility ranges, or adding new integration files)
  3. A particular site will only switch to new library versions when all consumer/integration code is compatible. (the less modules are installed, the less need to be updated, the earlier new versions will be used)

Where I (strongly) disagree is the part on minimally maintained contrib modules. There's not much hope for these anyway, and they impose problems for the community already today, especially when people are using jquery_update, jquery_ui, and other library-upgrading modules. In other words, the problem exists already, and it does not change with this concept. The only part that changes is the origin of possible library upgrades, and as a bad maintainer you are going to say "Screw core." instead of "Screw jquery_update." — Of course, disrespecting core has a different meaning and impact than to disrespect another contrib module. In turn, if it is the "Screw core." that bugs you, then you actually disagree with the idea of updating libraries in core at all. — which in turn makes you disagree with the entire problem space of this issue, and of course, everyone is free to disagree.

nod_’s picture

I'm with sun on the AMD side of this issue. It can make loading several version of a libs concurrently possible but really, nobody wants that.

About contrib and outdated libraries version I'll just give an example: In the D8 cycle the upgrade from jQuery 1.7 to 1.8.2 speed-up page load by 10% across all/most of the admin pages (they dropped a number of check in support.js and sizzle got optimized), this is the kind of reasons why you'd want the latest version of a lib and why it's a good reason to push contrib to update. D8 being mobile first this is a very serious issue to consider. For mobile we never want to load several version of an already slow library, which is why the possible AMD solution shouldn't be relied upon too much.

I also want to point out that It's likely that core will include views+aloha which are both a big pile of JS that will need porting and debugging from the core team for each update of a third party lib. A lot of Ctools utilities are already in core as well, it'll be indirectly updated too. Meaning that a huge amount of panels and related will be taken care of in the process.

An other thing to consider is that D7 had 0 JS maintainer for it's dev and maintenance cycle. Since DrupalCon, D7 and D8 has 3 of them (and possibly an other one on the way). Maintainers are not an absolute guarantee that all will be well, but it's a damn good indication that someone will care that something broke in core or contrib.

nod_’s picture

sorry, dup.

effulgentsia’s picture

A particular site will only switch to new library versions when all consumer/integration code is compatible.

That might address part of Dries's performance concern. My assumption (and what is therefore written in the issue summary) is that which library is added to any given page only depends on the requirements of that page. So if module Foo has JS that requires jQuery 2.0 (not higher), but core has moved to jQuery 2.4, then all pages to which module Foo doesn't attach its JS would get jQuery 2.4, but on pages that module Foo does attach its JS would get jQuery 2.0. Thus, across a site visit (but not on the same page), a user would download 2 versions of jQuery. #65 indicates we might want to make the logic site wide, which would address the performance concern, but would increase the odds of incompatibility, since if module A requires one particular version and module B requires a different particular version, then even if the 2 modules never attach their js to the same page, we still have to deny adding the JS of whichever module is deemed lower priority by the algorithm that determines how to deal with the incompatibility.

jessebeach’s picture

When D7 was introduced, we didn't have robust make-file and profile support. Now, there are examples of custom distros running off make files and the process of including a patch on top of D8 or a contrib module is more formal. If some key contrib module needs a patch to run on a point release of D8 that increases a version of a core library, that's not a terrible issue to resolve ahead of a module up-versioning.

For contrib authors in D7, using an up-to-date version of jQuery meant adding a dependency to jquery_update or simply packaging a custom jQuery file. Because the core version was guaranteed not to change, one needed to assume that other contrib modules would still be running 1.4.4. The pressure in D7 is to use the old core version and not be the outlier module.

With a D8 strategy that pushes library versions forward, contrib authors can assume that the latest core version is safe. The pressure here is to upgrade module compatibility because other modules maintainers might be doing this as well.

It's a subtle difference in incentives, but an important one.

effulgentsia’s picture

stability issues with this path (many modules are not regularly updated/maintained)

Since Dries raised stability as a concern, I wonder if the concern was related to the possibility of modules/themes *not* explicitly specifying the maximum library version they're compatible with. For example, if modules/themes get in the habit of setting 'max' to some very high number (e.g., '99' for jQuery) under the mistaken belief that their limited use of it is very unlikely to break in future versions, then a core update could cause a site to break. I don't know to what extent we want to discourage / disallow this.

RobW’s picture

For the repeated general worry about loading two versions of jQuery on one page, as sun mentioned minor versions updates of js libraries tend to keep the same API. Breaking a module with a jQuery 2.0 to 2.1 upgrade in core is an edge case, unless the module is very jQuery heavy and not conservative in it's implementation. On the other hand, the performance, security, and stability gain with that minor version jump will affect every visitor to a jQuery using Drupal site. *That's* serving our users. For amateur site builders and module maintainers AMD provides a safety net. It seems like this proposal is directly in line with the site visitors > site builders > professional developers user priority, without ruining things for any of them. I'm all for it.

sun’s picture

re: #68: site-wide vs. page-specific:
Nothing prevents us from investigating both. I'd even be tempted to say "Make it a damn setting on admin/config/development/performance." However, if time/timing will become an issue (and it will, the longer we debate), then I'd rather recommend a site-wide dependency resolution.

In any case, while it's slightly techy, I'd love to see a page on, say, admin/reports/libraries that informs you about which extension requires/supports which library (+ version), giving you a clue about which extensions you need to patch/update in order to leverage the new version of a library. (poor example)

re: #70: stability vs. max-version-hacks:
Off-hand, I don't know whether a max-version hack would actually impose problems. I guess that overly simple Drupal.behaviors and other integration code along the lines of that complexity could happily do that (i.e., when only relying on stupidly simple public API functionality that simply cannot reasonably change within the same major version without breaking the entire Internet).

AFAIK, experience with e.g., jquery_update in contrib follows exactly these lines — we never had to update/replace stupidly simple stuff in core like node.js, but only slightly more advanced things like ajax.js, tabledrag.js, states.js, and so on. I'm not exactly sure how sophisticated and battle-tested the current replacements are (6.x-2.x certainly contained more), but overall, what you generally see in there are replacements for more advanced scripts in core only.

Therefore, many extensions would certainly be able to apply a max-version hack; e.g.:

'dependencies' => array(
  // We don't know about jQuery 2 yet, but anything until then can
  // reasonably only be compatible, since our usage is lame. :P
  // @todo Consider to depend on Underscore instead. ;)
  array('system', 'jquery', '1', '1.99'),
),

This obviously involves "a risk of wrong predictions", so I'm not sure whether we'd want to officially endorse the max-version hack. For core itself, I'd object to it. Contrib, OTOH, can always do whatever it wants, and such a hack is reasonable when carefully applied.

Lastly, I think this entire concept and proposal needs to be amended with a library update/upgrade policy, which most probably deserves its own issue, but before I forget it again, I'll put it here:

  1. Core will not add new [frontend] library versions "immediately". The ecosystem should expect new library updates in intervals of 6 or 12 months only. The exact interval depends on the maturity of the particular library.
  2. Within stable releases, core will only ever add .1 releases of [frontend] libraries. The risk of encountering bugs in .0 releases is too high and we want to prevent the entire ecosystem from potentially having to provide compatibility with .0 and shortly after with .1. This applies to both major and minor versions.
  3. When a new major version of a library is available and included, core will try to also include the latest minor version of the library at the same time, so as to allow early adoption of the new major, but also provide more stability for the old major. (e.g., when jQuery 2.0.1 is added, the latest jQuery 1.9.x is added at the same time)

Exact details to be discussed, but something like this should be reasonable and work out. As mentioned, the discussion belongs into a separate issue, so if you want to discuss or reply to this, let's create a separate issue first :)

catch’s picture

I'm not really concerned about the performance trade-off here. For a new Drupal install (or one that's updated and doesn't have a JavaScript dependency soup), then instead of running versions of jQuery that are several years old with known performance issues long fixed in the stable releases, they'll be running the latest version. That's a massive front-end performance increase for all of those sites.

For those who are afflicted by JavaScript dependency soup, they'll be the more complex sites with a load of JavaScript heavy modules. Some percentage of those complex sites are exactly the ones which have the resources to test contrib module upgrades and/or provide patches for them. There'll also be a lot more incentive for contrib maintainers to upgrade in step with core compared to keeping up with jQuery update.

I also agree entirely with sun that this issue is primarily about end-user experience. Keeping up with the latest releases is not about making new APIs available to developers, it's about making new bug fixes and performance improvements available to end-users - since all that code is loaded/executed by the browser each request.

sun’s picture

Status: Needs review » Reviewed & tested by the community

Tentatively moving back to RTBC.

effulgentsia’s picture

Assigned: Unassigned » Dries

Given #60, needs follow up feedback from Dries.

webchick’s picture

Just a note that this is on Dries's radar, but he's traveling and most likely won't be able to respond until next week (or we could just hash it out at BADCamp on a whiteboard. ;))

YesCT’s picture

any news on this?
or anything we can do to make this easier for Dries to look at?

webchick’s picture

Component: javascript » documentation
Status: Reviewed & tested by the community » Active
Issue tags: +JavaScript

Spoke with Dries about this just now (he had to run, so paraphrasing). While he still has some concerns mentioned in #60, he recognizes that this approach is likely necessary in order to use some of the "cutting edge" libraries we wish to add to core, such as Modernizr, Create.js, etc. which are likely to evolve rapidly during Drupal 8's lifecycle. So we can consider this approach signed off on.

Seems like we probably need a handbook page to document this, so moving component to "documentation" and marking "active."

webchick’s picture

Assigned: Dries » Unassigned
catch’s picture

We're going to need some code written to handle the version dependencies in hook_library_info(), but maybe open a new issue for that?

corbacho’s picture

if we already passed feature freeze, what's the current valid strategy to update javascript libraries in D8 ?

andypost’s picture

at least we need a "revisit before release" tag for issues that updates each used library

effulgentsia’s picture

if we already passed feature freeze

This issue is a task, so I think that makes #80 a task as well, so we're not yet passed the deadline for making this work.

effulgentsia’s picture

Issue summary: View changes

Updated issue summary.

webchick’s picture

Wondering how this proposal intersects with #2135189: Proposal to manage the Drupal core release cycle, which was just recently approved.

I *think* how it works now is we stick to the latest versions of various libraries every minor release until LTS, and then we would introduce this policy until LTS went into "security fix only" mode. Does that jive with others' thinking?

sun’s picture

Dang, I forgot about this issue when designing the new library declaration syntax/format/structure.

In case we want to do the bare minimum and only prepare D8 for the possibility of multiple versions, then we should (1) agree on a plan and (2) create an issue to change the library (dependencies) definitions along the lines of #51 and #72.

nod_’s picture

I'm hoping the faster release schedule of drupal will mean we can get away with updating in point releases. We'll still tie the lib version to a drupal version but no more 4 years old lib version with your Drupal.

catch’s picture

I'd hope we can update everything feasible for each minor release (so 8.0.x, 8.1.x, 8.2.x).

Not sure about keeping the old libraries around during minor releases or only LTS. Partly depends on what the actual new releases are like for the different libraries.

Either way with the minor releases, assuming we're able to update versions months before the actual release comes out, we've got plenty of time to discover unexpected API incompatibilities. Of course this would be a lot easier with core/contrib js testing available.

webchick’s picture

+1 to #87. I imagine leaving the old libraries prior to LTS we'd probably do on a case-by-case basis, depending on how badly they broke BC.

sun’s picture

Not sure I'm able to follow the recent comments about "we can keep the old LTS libraries around"

With the current library declaration and dependency resolution model, a single library can only exist once. Thus, we can only update a library OR keep the old release.

The only workaround to that would be a hook_library_info_alter() implementation in core (or contrib), but that will essentially run into all the painful issues that we know from the contrib jQuery Update + jQuery UI + Co modules already.

→ IMO, there are really two choices only: (1) Either everything can rely on the same version of libraries at all times (possibly updated over time, with potential BC breaks). (2) Or there can be multiple versions of a library, and every consumer may state a particular version [range] it is compatible with.

asad.hasan’s picture

+1 for bower in drupal 8 core

jhodgdon’s picture

Component: documentation » other
Category: Task » Plan

This is not documentation (just came across this in an issue search). I thought we had a component for project management questions? Maybe not. Anyway, not documentation.

jhodgdon’s picture

Issue tags: +Project governance

Oh maybe it's a tag.

jhodgdon’s picture

Component: other » javascript

In which case the component is probably javascript.

Wim Leers’s picture

Do we still need this issue?

Version: 8.0.x-dev » 8.1.x-dev

Drupal 8.0.6 was released on April 6 and is the final bugfix release for the Drupal 8.0.x series. Drupal 8.0.x will not receive any further development aside from security fixes. Drupal 8.1.0-rc1 is now available and sites should prepare to update to 8.1.0.

Bug reports should be targeted against the 8.1.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.2.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.9 was released on September 7 and is the final bugfix release for the Drupal 8.1.x series. Drupal 8.1.x will not receive any further development aside from security fixes. Drupal 8.2.0-rc1 is now available and sites should prepare to upgrade to 8.2.0.

Bug reports should be targeted against the 8.2.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.6 was released on February 1, 2017 and is the final full bugfix release for the Drupal 8.2.x series. Drupal 8.2.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.3.0 on April 5, 2017. (Drupal 8.3.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.3.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

andypost’s picture

Version: 8.3.x-dev » 8.5.x-dev
xjm’s picture

So what we did with jQuery 3 was just straight up updating the library in #2533498: Update jQuery to version 3, since the old version has disclosed security issues that were not receiving bugfixes. In retrospect, though, I wish we'd tried to adopt something a bit less disruptive with a bit more advanced notice to a wider audience than the core issue queue. Some things we could have done would have been:

  • A change record before #2533498: Update jQuery to version 3, announcing that the change was coming and recommending developers test their code with jQuery 3.
  • A g.d.o/core announcement ahead of time and again as soon as the change was made.
  • Maybe even status report info message of some sort, backported to the previous release.
  • Committing the change as early as possible in the minor development cycle, which could even have meant to committing it to 8.5.x only at the end of July.
james.williams’s picture

If jQuery was upgraded because it was a security issue, why was 8.4.0 not tagged as a security release?

effulgentsia’s picture

Drupal 8.3 was not triggering the conditions that would escalate jQuery 2's disclosed security issues into something that would have warranted tagging 8.4.0 as a security release. That said, every minor release triggers the end of life of the prior one, so there might be future discoveries and disclosures of vulnerabilities in 8.3.

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.0-alpha1 will be released the week of January 17, 2018, which means new developments and disruptive changes should now be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.6.x-dev » 8.7.x-dev

Drupal 8.6.0-alpha1 will be released the week of July 16, 2018, which means new developments and disruptive changes should now be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Mixologic’s picture

Issue tags: -Composer

cleaning up tag. unrelated issue.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.