Problem/Motivation

During the Managing Drupal sites with Composer BoF at DrupalCon Baltimore 2017, the topic of contributed module front-end library management came up.

Here's the current problem:

  • Webform (and other similar contrib projects) requires a 3rd party front-end library.
  • Modules can't rely upon libraries being available inside [drupal-docroot]/vendor/[library/here], because the project vendor directory isn't guaranteed to be in the docroot.
  • Current practices seem to indicate that a libraries folder in the docroot is the best place for 3rd party front-end libraries.
  • It's hard to get contributed module dependencies to be installed to [drupal-docroot]/libraries/[library/here] instead of the vendor directory. Especially if the library doesn't care about Drupal and doesn't have a custom type of drupal-library

There are ways to hack around this:

  • Add a custom and project-specific repository for every 3rd party library in your composer.json file.
  • Use the composer-installers-extender library to allow the specification of the install path per-library.
  • Ditch Composer and just download and commit all the 3rd party libraries into the [drupal-docroot]/libraries folder.

Proposed resolution

It would be really helpful if Drupal core could enable contributed modules to do things like specify a specific path into which a 3rd party dependency will be placed.

One solution we discussed would be adding code that responded to a Composer hook and moved a downloaded dependency into a specified path (which could be indicated via the project's composer.json file).

So a contributed module maintainer (e.g. Webform, Font Awesome Iconpicker, etc.) would add something like (pseudocode):

    "extra": {
        "drupal-install-library-path": "libraries"
    }

And then when you composer require that contributed module, something in Drupal core's composer configuration would pick up that path and place the libraries for the 3rd party module into the specified path.

This is totally open to change (in terms of implementation), but the general idea is: everyone is solving this problem in a different way, and it would be nice if core could provide an official mechanism for doing this 'the Composer way'.

Or, at a minimum, we should document a recommended solution.

Remaining tasks

  • Choose an approach (custom Composer hook? Add another library to help? Use/recommend Robo?)
  • Implement the approach (this part's easy, right?)
  • Document / announce the change so Contrib modules and themes can use this new approach

User interface changes

N/A

API changes

There may be some additional API functionality, but probably not.

Data model changes

N/A

Comments

geerlingguy created an issue. See original summary.

geerlingguy’s picture

And to add a little bit more color:

  • Yes, I know that themes can manage libraries via NPM / separately—this issue is talking about contributed modules and any other non-theme components of a Drupal project (e.g. Webform, Font Awesome... probably many other modules and install profiles at least)
  • Tangentially-related: many front end libraries are not on Packagist; if you just want to be able to reference a front end library, you might find Asset Packagist helpful!
mglaman’s picture

Is this just for front end libraries? Because I do not see how this would work for Open Social, Commerce, Search API Solr, etc.

geerlingguy’s picture

@mglaman - yes, exactly. This the pseudocode; since things like CKeditor JS and CSS can't be autoloaded like PHP can, modules need a way of getting libraries in a web-accessible path. Traditionally sites/all/libraries but /libraries seems to be most common in D8.

How does core put front end stuff in the codebase? Is it all somewhere inside the core dir? (I'm on mobile, can't check right now.)

Mile23’s picture

OK, so the problem is: We don't know where vendor/ is.

The solution in the IS is to have core move the library from vendor/ to modules/mymodule/[extra:drupal-install-the-thing-here].

I disagree with this solution. I think core should have a way to know where vendor/ is, so that then you could just write a composer.json file for contrib, have Composer get the library, and then your contrib can use it. And core's composer implementation wouldn't have to deal with moving anything around.

mglaman’s picture

So it seems like we need support for Bower here, correct? Someway to define front-end assets and know where they will be accessible in the web docroot.

geerlingguy’s picture

@mglaman - I'm not sure about that. I typically think of Bower, Gulp, NPM et all as being helpful frontend tools for themes, where themes need to pull in frontend libraries, but for something like a Drupal module (e.g. Font Awesome), the only solution that's been floated is using Composer to manage dependencies.

If we want to say, as a project, that developers should use Composer for PHP-based dependencies, and [bower|npm|gulp|tool-of-the-month] for frontend... that doesn't seem like it would fly. It would require people trying to use modules like Webform to use two different command line tools (one requiring PHP, the other Node.js) just to gather dependencies for their codebase.

Aside: adding #2605130: Best practices for handling external libraries in Drupal 8/9 and 10 to related issues... but that seems like it might be a broader exploration of this topic (or maybe this issue should be marked duplicate?).

andypost’s picture

lpalgarvio’s picture

How about allowing the LIBRARIES path be specified in settings.php?
Why not tell Drupal to just look up libraries at ../vendor or ../assets or any custom directory? A symlink is the best option currently.

This problem needs to be tackled in a universal way, not in a Drupal way.

Don't expect tools like Bower, Browserify and Yarn to adapt to Drupal.

Btw,
- Component.js is deprecated.
- Composer Component future is uncertain at best
- It makes no sense either to create custom repos/packages just to add hundreds of libraries to Drupal with Composer Component
- It makes no sense either to do this on your custom composer files
- Composer isn't meant to manage non-PHP code, libraries nor projects
- Robo, like all task runners, is meant to change behaviours disconnected from dependency manager. If you interfere with the dependency managers, these start to loose value (think composer.lock)
- Hacking Composer is not a solution and will only complicate things further

mglaman’s picture

Composer isn't meant to manage non-PHP code, libraries nor projects

Exactly. Composer is for managing PHP. Just because we can get it to put a library somewhere does not mean it's the right tool for the job.

[bower|npm|gulp|tool-of-the-month] for frontend... that doesn't seem like it would fly

Gulp, Grunt are task runners like Robo. There is an issue of Bower vs NPM. I think more and more Bower is being sidestepped in favor of NPM. I just hate nested dependencies. The real people to ask here are the front end developers. These are our users who interact most with NPM and Bower.

Yarn is stacked on top of NPM to give it lock files and caching (it makes it familiar to Composer.)

These dependency management for the frontend tools are not new either. They are also established practices. Drupal is just reactionary to modern tooling.

tormi’s picture

If we want to say, as a project, that developers should use Composer for PHP-based dependencies, and [bower|npm|gulp|tool-of-the-month] for frontend... that doesn't seem like it would fly.

There are non-PHP libraries that doesn't support Composer, Leaflet for example: https://www.drupal.org/node/2774237#comment-11701099

Mile23’s picture

There's some work towards normalizing on yarn: #2809281-17: Use ES6 for core JavaScript development

So maybe this issue is going to be the 'how will we use yarn for contrib?' issue.

SKAUGHT’s picture

IMO:
#667058: Add a libraries folder with a README.txt in it to DRUPAL_ROOT needs to be.
anyone developing a theme, profile or module in drupal (and may be adding it to a contrib project, where they can't commit that to the project repo..) needs be able able to let others know where to setup that related folder. then using a hook_requirment() to verify plugin is there...

for those building, and not contributing it, who don't need to worry about slamming stuff in their theme/module folder and can use *.library.yml in that context -- will just do so to attach js/css..

that is now...the Drupal Way. (:

for sure, the libraries API (in drupal 8) really confused things when I was ramping up about *.library.yml files abilities. When i was working #2605130: Best practices for handling external libraries in Drupal 8/9 and 10 as the document to help others that was the biggest reality i got to -- Libraries API is kinda dead.

SKAUGHT’s picture

and once again:

Some Aside Notes

  • On Tools of the trade:
    • Have users run a post-install script like "bower install" from your module directory, and include your libraries relative to your module directory. (ex: bower_components/my_library/my_library.js) IE: File Entity Browser.
    • Let's not focus on how a developer/site builder downloaded to their file system, as there is a variety of tools to do so and will change in time. (ie: no one uses MAKE files anymore)
  • Linking from 3rd party Servers: Use a CDN like cdnjs.com to ensure the correct version is being used and reduce install complexity.
    • Not recommending module developers linking CDN's in their *.libraries.yml file. This is a poor development practice, you can't guarantee your module/theme's functionality this way.
    • Problems with 3rd services (CDN's or general linking to other live Servers) with 404 & maintenance realities or just slow load times..
    • Some issues on the effects on js/css caching can occur with CDN use.
  • Let's not focus on how a developer/site builder downloaded to their file system — it's irrelevant.

    gordon’s picture

    The biggest problem with installing js libraries into a directory other than vendor is the amount of junk that follows along with it. in the form of examples, docs, etc.

    For 1 project I am using robloach/component-installer which allows the installation of components type composer packages.

    I did the following:-

    {
        "require": {
            "components/backbone": "1.0.0",
            "components/modernizr": "2.7.2",
            "components/underscore": "1.5.2",
            "robloach/component-installer": "*",
       },
       "config": {
            "component-dir": "docroot/sites/all/libraries",
            "component-baseurl": "/sites/all/libraries"
        }
    }

    Then in my libraries files I just have want is needed an no rubbish.

    The biggest PITA is that when using the Drupal packages (https://packages.drupal.org/8), if there is an exisiting composer.json file in the repository it doesn't merge in the require or the require-dev into the composer.json so it doesn't then load the dependencies that have been specified.

    Version: 8.4.x-dev » 8.5.x-dev

    Drupal 8.4.0-alpha1 will be released the week of July 31, 2017, which means new developments and disruptive changes should now be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

    i.bajrai’s picture

    Would like to add 2c to this, lots of passionate people here :D

    I tried to install this module: https://www.drupal.org/project/css_editor

    However it needs the codemirror javascript library: https://github.com/codemirror/CodeMirror

    Master breaks the drupal plugin ( I think you need to compile the javascript in that version) so I had to revert to a v3 branch.

    So I download that version, stripped out all the test and irrelevant code to make it lighter (eg only needs the css language), created a repo on bitbucket and added a composer.json to say that its a Drupal library.

    Then required that the repo in my main composer.json.

    Seems to work so if anyone has a better way to do it using this perhaps as an idea or if this is terrible and should never be done this way, love to hear it!

    What the composer.json looks like in the bitbucket repo.

    {
      "name": "my-modules/my-module-css-editor-library",
      "type": "drupal-library",
      "require": {
        "composer/installers": "~1.0"
      },
      "extra": {
        "installer-name": "codemirror"
      }
    }
    KarenS’s picture

    I like the solution in #15 a lot. I tried it out locally and it works very seamlessly and without any extra steps to end up with a specified version of a js library in /libraries. Even though I understand that composer is for PHP I think it's too much to expect two separate workflows, first do composer require/install/update then do something else for the javascript libraries.

    ressa’s picture

    amateescu is experimenting with an approach, where a service on drupal.org handles the Composer dependency resolution, and it looks quite promising: #2910136: Experiment: package PHP libraries in a single Phar file

    KarenS’s picture

    While the Phar file approach looks interesting, it feels like a pretty cumbersome solution compared to #15, which pretty much just works right now. I am coming at this from the standpoint of how a contributed module could add javascript libraries that core doesn't provide, so am particularly interested in figuring that out.

    heddn’s picture

    Related issues:

    I think for a lot of use cases, it's a fine thing to do #15. I used it about a year ago for a while. But then I found that many of those libraries are for the theme and my themer wanted to also run his gulp tasks against them. So we reverted to using a proper js dependency system provided by Bower. I need the js and CSS in pattern lab too. And I don't want to fight against composer or duplicate composer into Bower for pattern lab as well.

    So, whatever we pick as the official method, don't make it a so knowledgeable dev cannot do his own thing. And not use composer.

    Mile23’s picture

    So if, for instance, we could use yarn to build out a libraries directory somewhere in core, and yarn knew where to look in modules for package.json files, and then your module.libraries.yml file could tell core to optionally look in this central libraries directory, that would solve the problem, right?

    Like:

    $ cd /path/to/drupal/
    $ yarn install
    # yarn finds modules and merges their package.json files and builds them.
    $ ls libraries/
    # big list of all the JS dependencies.....
    
    anavarre’s picture

    The Lightning distro now supports https://asset-packagist.org/ which is an alternative solution. See blog post: https://dev.acquia.com/blog/round-up-your-frontend-javascript-libraries-...

    heddn’s picture

    re #23: that's the reason I posted my comments in #21. I played around with that approach for a while, but then found it wasn't what we needed for our themers. I much prefer the solution in #22. It is much less of a drupal-ism or a hacky composer-ism. JS (and CSS) tends to fall into the themer camp pretty heavily. And themers are much more likely to use gulp/grunt to compile css and js.

    KarenS’s picture

    That's true, for dev teams with themers. But, the people who are most likely to have trouble figuring out how to add a javascript library to a contrib module are probably not using build tools. They do, however, already need to know how to use Composer to install a number of the D8 contrib modules, so the Composer solution is one they could use.

    The tricky thing is coming up with a solution that works for everyone, big teams with build processes and individual, non-developer, site builders. Maybe that's not possible, but we do have both audiences.

    acbramley’s picture

    While I agree that some JS libraries do belong in the theme, it can't be said for all. Things like leaflet or chosen aren't theme specific and don't belong in the theme. They aren't dependencies of a site's look and feel, rather they are dependencies of the functionality of a site (i.e they belong in a module).

    Mile23’s picture

    OK, so given that we have a contrib module that needs leaflet.

    And given that we have a build tool that puts libraries in a global libraries location, including leaflet.

    And given that the build tool would not put leaflet in the global libraries location if the contrib module wasn't in the codebase.

    And given that the build tool could rebuild the global libraries location without leaflet if the contrib module was in the codebase but since has been removed.

    Would that solve at least some of the problem of torturous 3rd party library management?

    SKAUGHT’s picture

    still, IMO
    a. having a /libraries that actually exists in docroot.
    b. regardless of whatever tool downloads/installs it (make/composer/bower/yarn...git/submodules..) drupal need to be able to "know it's there" -- ie, auto-reading package.json discovery. so that a library_list() and library_load. both for php & twig loaders (perhaps a theme setting to load a given library should 'themer not know how to do better..) can make them 'online/active' for actual use..

    This, could make for a happier library driven world.

    #26
    true enough. site builders often only can handle things 'in a theme' which is a wield part of the learning curve when we fail to see when a feature is actually a module, versus: "i just need to put this thing somewhere" and have a behaviour to trigger the script. people want "a theme to do everything" -- still.

    Version: 8.5.x-dev » 8.6.x-dev

    Drupal 8.5.0-alpha1 will be released the week of January 17, 2018, which means new developments and disruptive changes should now be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

    balbuf’s picture

    To throw another hat into the ring, I've created a simple composer plugin which allows referencing required libraries via a URL to a distributed ZIP file: https://github.com/balbuf/drupal-libraries-installer.

    The thinking here is that, for simplicity, modules will often list out their library dependencies as direct links to ZIP files or to the library's GitHub repo (where finding its associated ZIP file is just an extra step), so listing out libraries as such is a natural extension of the process. An end user can simply list out their required libraries as a hash, similar to how composer-patches works.

    The libraries are listed in the extra section of the composer.json file and are considered type drupal-library when installed, so controlling the installation path is as simple as configuring composer/installers, e.g.:

    {
        "extra": {
            "drupal-libraries": {
                "flexslider": "https://github.com/woocommerce/FlexSlider/archive/2.6.4.zip",
                "chosen": "https://github.com/harvesthq/chosen/releases/download/v1.8.2/chosen_v1.8.2.zip"
            },
            "installer-paths": {
                "web/libraries/{$name}/": ["type:drupal-library"]
            }
        }
    }
    

    Admittedly, this leverages composer in a way that is outside of its scope as a PHP package manager. At the same time, for Drupal composer packages, external library dependencies are just as important as other PHP / Drupal package dependencies to use the package as intended.

    Using asset-packagist is a similar solution, but the major difference is that the required library must be listed in the npm and/or bower registries, and as an end user, you'll need to track down and identify the correct package. While most of these libraries should be listed, in the event that any are not, you're back at square one, necessitating a mixture of solutions for pulling in all of your libraries.

    Using npm/yarn or bower is not an ideal solution IMO because they do not support placing packages in different installation paths like composer does (which requires an additional step to symlink or move packages into place) and they require the use of an additional tool on top of composer.

    Right now the plugin does not recursively search into dependencies to identify drupal-libraries listed therein, meaning all libraries must be listed in the user's root composer.json file. However, it's certainly possible that we could add this functionality if people were interested in adopting this tool.

    Anyway, please have a look and let me know what you think!

    ressa’s picture

    Drupal core idea #2940733: Site Builder Tool/Project Browser initiative was recently initiated, which might make 3rd party library management easier.

    Jon Pugh’s picture

    I think it needs mentioning that one of the reasons we need better ability to add libraries using composer.json is because it's been supported by drush make for a long time.

    You can download any file with drush make, and the power of that is you can build your entire codebase with a single command.

    libraries:
      jquery-chosen:
        version: '1.6.1'
        download:
          type: get
          url: 'https://github.com/harvesthq/chosen/releases/download/v1.6.1/chosen_v1.6.1.zip'
        directory_name: jquery.chosen
        destination: libraries
    

    Right now you have to add to composer repositories and requirements to replicate this.

    composer is a great tool for building your entire stack in one go. I think we should think about how to extend the packagist for 'drupal-libraries' types.

    We already have an approval process for 3rd party libraries in makefiles for the distribution building system. Perhaps we can leverage that data to allow 'drupal-library' types to be added to composer simply with composer require drupal/jquery.chosen

    moshe weitzman’s picture

    Lightning, Thunder, and Commerce distros are using Asset Packagist. This approach allows for downloading any npm package and putting it where you need, just with Composer. This approach is available to all composer projects, so its no longer true IMO that "Drupal core should help make 3rd party library management not torturous". Perhaps all we need is blessing that this is the preferred approach and then Contrib can rely on it.

    acbramley’s picture

    I've been using Asset packagist on a recent project and it's been working really nicely. I guess the only problem with using it in contrib is that it requires configuration in the root composer file that would have to be documented in the README of the contrib module since you need:

    1) The repositories entry for https://asset-packagist.org
    2) The composer/installers library so you can put the lib in the right place
    3) The installer-types and installer-paths keys set up properly in extra so it's installed in the right place

    Jon Pugh’s picture

    Excellent, I didn't realize Drupal distros are already using this.

    Thanks Moshe!

    Jon Pugh’s picture

    Asset Packagist covers things stored in NPM and bower, but what about projects that aren't?

    I guess adding to repositories for a few libraries isn't that bad, most things are in there.

    sun’s picture

    #34 is true.

    To resolve that remaining question, we should add it to https://github.com/drupal-composer/drupal-project/blob/8.x/composer.json

    This PR https://github.com/drupal-composer/drupal-project/pull/286 proposed it already a longer time ago.

    phenaproxima’s picture

    The composer/installers library so you can put the lib in the right place

    This is incorrect. You need oomphinc/composer-installers-extender, which requires composer-installers and allows it to work with arbitrary package types like bower-asset and npm-asset.

    rodrigoaguilera’s picture

    In my team we have been using https://asset-packagist.org/ for a while and is working real nice. We have been missing a workflow like that since the drush make days ;)

    I repeat here what I said in the PR:

    Maybe in the future contrib projects can start assuming the asset repository is added to your root composer.json just like https://packages.drupal.org/8

    or since the software that manages the https://asset-packagist.org endpoint is BSD licensed it can be integrated into packages.drupal.org.

    joegl’s picture

    I am still not entirely clear on what I should do to add a simple CSS library dependency. I want to use FontAwesome in two of my modules, and have it required as a dependency, stored in the same place, etc., Is there really no way to do this now? I have included the fontawesome.css file in both modules libraries.yml files separately for now.

    I am also seeking a non-composer solution.

    sun’s picture

    @joegl: You're right, it's a big mess currently, sorry. Let me try to summarize the current best practice I'm aware of:

    1) To enable Composer to download any frontend packages, you need the additional repositories definition in the root composer.json of your project, as patched here: https://github.com/drupal-composer/drupal-project/pull/286/files#diff-b5... — The "repositories" definition is only supported in the root and not inherited from any children; see https://getcomposer.org/doc/04-schema.md#repositories

    2) To make Composer automatically download additional dependencies for your module, you need a composer.json in your module; e.g. as patched here: https://www.drupal.org/project/photoswipe/issues/2855483#comment-12561174

    3) To instruct users without Composer on how to manually download & place additional dependencies for your module, your module needs to depend on https://www.drupal.org/project/libraries, and your module needs to implement hook_libraries_info(); e.g., as patched here: https://www.drupal.org/project/photoswipe/issues/2832405#comment-12525953

    Mile23’s picture

    I think core and contrib should be buildable.

    That is, if you run npm install in the extension root directory, you'll get the things you need where they should be.

    Other tools, even a composer plugin or script, could execute npm install for each contrib, project-wide.

    If we normalize that across contrib, then a build tool could perform this task in a repeatable way with reduced complexities. #2940733: Site Builder Tool/Project Browser initiative

    Jaypan’s picture

    I ran into some of the issues in this thread today, my solution was to create the Vendor Stream Wrapper module, that creates a vendor:// stream wrapper that points at the vendor directory. This allows for creating URIs like vendor://vendor_name/project_name/some_file.css, and the vendor prefix can be used in *.libraries.yml files to link to vendor stylesheets and scripts.

    Version: 8.6.x-dev » 8.7.x-dev

    Drupal 8.6.0-alpha1 will be released the week of July 16, 2018, which means new developments and disruptive changes should now be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

    Jaypan’s picture

    pabloveintimilla’s picture

    #41 great solution.

    jwilson3’s picture

    Aside from the known limitations of Asset Packagist only working with NPM and Bower packages, is there any reason why that methodology shouldn't be added to docs page for using composer to manage Drupal site dependencies?

    Jaypan’s picture

    Which methodology?

    voleger’s picture

    jwilson3’s picture

    I've added a section to document Downloading third-party libraries with Composer.

    jwilson3’s picture

    Similar to ^ I've also added a section to document Requiring third-party libraries with Composer targeted specifically for module, theme, and profile maintainers.

    Perignon’s picture

    xx

    Jaypan’s picture

    This is not the right place for that question as it is not the current topic.

    Mixologic’s picture

    Okay, so https://asset-packagist.org/ somewhat sorta works for some people's use cases, but I dont think we can start documenting it as an official way to do things, for several reasons.

    Its a build service, and as such, we need some sort of assurances that its going to:
    1. Be available. AFAICT, this service is small, probably running on one server somewhere, and suits its creators needs. If we start recommending that the entire drupal ecosystem use this as a standard, it'll see considerably more traffic, which could cause it to fail.

    2. Be long term. Once you start adding metadata for things like this, they dont go away. If the maintainers of asset-packagist decide its not worth it to continue running this service (because theres no revenue model, yet services cost money to run), then we'd be in a bind in the future if it goes away.

    3. Be flexible/responsive : if this service needs updates or packages added removed, is that easily doable?

    So, while yes, it does *work* right now, we haven't really got any assurances that it'll continue to work, or that its part of an ecosystem such that it seems likely to live in perpetuity. packages.drupal.org has grown steadily since becoming "official" and now we see about 60,000+ hits a day, to packages.json (which happens on every composer request). We've got Fastly, and multiple webnodes, so we can handle that, but we've also got funding to pay for it.

    packages.drupal.org / packagist.org both have funding and staff to ensure their continued existance. I dont know what the situation is for asset-packagist.org.

    One alternative may be to fold its functionality into drupal's infrastructure, and host our own instance: https://github.com/hiqdev/asset-packagist.dev

    Then again, theres the question of whether or not we should be using composer as a comprehensive build tool to create a whole site, or whether we should expect it to be able to do such things. Jordi (the maintainer of composer) has mentioned several times that you probably should not do that, and that its specifically for building php projects, and that npm and tools like it are much more suited to handling other parts. But sometimes pragmatism outweighs design.

    geerlingguy’s picture

    I think the main difficulty is that a lot of Drupal modules (which traditionally would be all PHP code) take on a lot of theming responsibilities, so you have a mix of PHP and JS library dependencies. For themes, it's exceedingly rare you'll find a theme with other PHP library dependencies, but most people are used to using npm/yarn (or sometimes even bundler) to install theme dependencies.

    For many PHP projects there's a better separation of 'PHP' vs the 'theme', and it is rare any of the PHP libraries/extensions would also toss in some node.js or plain .js libraries. But in Drupal/Wordpress/other traditional CMSes the landscape is different.

    So to your point:

    But sometimes pragmatism outweighs design.

    I think the statement from Jordi comes more from the perspective of a Symfony-style developer, and framework-based development, where there is a very strict divide between PHP backend stuff, and theme layer which is segmented out pretty cleanly.

    Drupal is—for better or worse—in a much different situation due to our legacy.

    jwilson3’s picture

    To add to @geerlingguy's point, I'd already added a caveat emptor message in the docs linked in #50 explaining that typically Themes would not need to use Asset Packagist, particularly if they're already using a packaging tool like npm webpack or yarn.

    @Mixologic: You bring up some great points about reliability and availability. I totally agree that point #52.1 and #54.2 should be considered before any of those community docs move into the official documentation. However, I have a question about what is official documentation versus what is community documentation?. Is there any standard here that says what can and cannot be written in community documentation? Most of the composer stuff lives in the community documentation, not in the official restricted-edit-access User Guide yet.

    Additionally, the fact that a number of popular distros are using Asset Packagist makes me feel slightly more confident about suggesting that we can in fact document this pattern for Drupal site building, distro building, and contrib module landscape. And as such it feels extremely slow moving to specifically not tell people about useful recipes on the suspicion that it might one day stop working. Would the potential harm be mitigated by adding a caveat emptor message about "use at your own risk" and then continue to observe how things play out and — as is the nature of open source — let it grow organically until we grow out of it, and into a custom solution on drupal.org infrastructure?

    Thank you for explaining in the discussion thread that the code I copied — directly from what @sun linked to above, and what a contrib module had already committed — was wrong. However, instead of removing the code snippet, the community would be better served by having this recipe properly documented *somewhere* so people can refer to it when implemented improperly. I can't think of a better place to iterate on what works vs what doesnt work than in the community documentation. To your point about requiring more steps to get it to work, you're correct, however, I did include instructions there about how there is more work to be done by end users to leverage Asset Packagist, and for module maintainers that want to use that method, they could point to the other documentation piece that you removed. *sigh*

    Finally, in the site builder focused doc page, I'd provided a link to an example of CKEditor plugins, as as a fallback solution when Asset Packagist doesn't suit your needs. This technique of adding custom entries to the "repositories" section of your project's root composer.json works very well, and doesn't suffer from any of the points you mentioned in #52, so potentially that could be genericised and brought into the main documentation, to clarify that this is a valid option, and potentially as part of the caveat emptor from above.

    Hopefully someone can pickup the baton here. Thanks

    jwilson3’s picture

    I'm sorry if my post above has come off harsh. I guess there is something about ending the week on a low note, having the documentation you spent time writing up and thinging about to be flat-out removed. For that I apologize, and can only quote something I found particularly relevant this week, linked from Dri.es blog post about governance:

    Community members in multiple forums express frustration, believing the DA should be doing more to support the community. We also heard that people don’t feel empowered to create change in the community without DA permission or approval.

    What is clear is that there is a community need here. The community is jumping on board to solve a problem. Solutions have emerged and are used by large orgs (read Acquia Lightening distro). The fundamental question now is can D.o infra team get behind the solution too, and host asset-packagist on their own infrastructure, or not?

    Mixologic’s picture

    flat-out removed

    Apologies, as thats more a limitation of our toolset for documentation than it was a reflection of "we dont want this, at all". Ideally we'd have a way to unpublish a section for further review, which is really what I was aiming to do there. I only removed that particular edit because of its inaccuracy (and also because I've had to help solve many problems with other contrib maintainers making that exact mistake). I did leave in the rest of the changes documenting how to use asset packagist on https://www.drupal.org/docs/develop/using-composer/using-composer-to-man.... The key thing that was incorrect in that docs was what @sun posted in #41-1,

    The "repositories" definition is only supported in the root and not inherited from any children;

    - i.e contrib maintainers cannot put "repositories" into their composer.json.

    Ideally, we would have contrib maintainers avoid including asset packagist requirements in their modules, but if they must, then those modules need to detail that in order to install that module with composer, a site owner must *first* follow steps 1-4 as you outlined in https://www.drupal.org/docs/develop/using-composer/using-composer-to-man.... Which, honestly, is a lot of hoops to jump through, and certainly doesnt seem any less 'torturous' than the existing methods using the libraries api or giving instructions to download a tarball to a specific directory.
    Which is why I dont think we want to advise published contrib maintainers to use this mechanism, unless we were to add asset-packagist to drupal core so that contrib maintainers could rely on it always being there. But that goes back to what I said in #54-1/#54-2. If asset-packagist goes away for whatever reason, the consequences are pretty dire for anything relying on it.

    We're used to the open source way of doing things where we can add in a library to our code, and if, in the future, that library becomes abandoned, at least what we have keeps working, and we have the option of maintaining it ourselves. But when you add a dependency to a 3rd party service, then that requires a lot more assurances and confidence in its continued existence, because if it becomes 'unmaintained' things break *hard*. So if module maintainers start taking on the 'risk' of using asset packagist, the end user who wants to use that module doesn't get to evaluate that risk, they'll inherit it. If a popular enough module like webform adopts this, a fair number of sites ability to update may depend upon the financial stability of a small web dev shop.

    So, even if we 'organically' suggest people use this service in their modules, it exposes the whole project to the same amount of risk. Which is why we should probably reach out to the asset-packagist folks to find out their intentions for the service, find out whether we should run our own version of it, or perhaps even skip a service dependency entirely by exploring the composer plugin that asset-packagist is based on and recommending that instead (https://github.com/fxpio/composer-asset-plugin).

    Side note, despite the fact that I work for the DA, this really has nothing to do with the DA, I just also happen to be the maintainer of the composer docs section, but in that capacity Im really no different than any other community member. The only power a maintainer really has over the docs section is the ability to create new sections. Otherwise, its kinda like a git repo that everybody has push access to.

    Version: 8.7.x-dev » 8.8.x-dev

    Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

    rodrigoaguilera’s picture

    I have been digging a bit in to the related issues to try to move this forward.

    First I think that we can split the discussion into two different concerns:

    • The source of the assets: adding extra repositories to composer, adding extra composer plugins, packagist.org or a infraestructure supported by the DA.
    • Once the library can be retrieved from the original source how to place in proper place like /libraries so it can be served properly and the modules can rely on its location.

    For the second bullet point I like the approach of the vendor stream wrapper
    #3002647: Add a vendor:// stream wrapper
    and I think we should reconsider it as a way to make the 3rd party library management a bit less tortuous even at the cost of stretching composer for different a purpose then initially thought.

    I agree about not relying on asset packagist. I have been posting a few issues on their github and I don't think they are reliable enough to recommend that service to the entire Drupal community. Despite this I still use asset packagist very happily.

    Composer asset plugin is abandoned in favor of Foxy. Anyway my experience is that makes composer very slow when it has to check repositories.

    Foxy requires npm to be installed as far as I understand (I haven't tried) so using it will bridge composer and npm by only using composer. Not a lot of value in my opinion.

    It is also true that there is already many frontend libraries available on packagist.org so some approach like the vendor stream wrapper will make it easy for libraries like jquery, bootstrap, font awesome, etc.

    What do you think?

    geek-merlin’s picture

    What about this:
    * provide a standard to describe asset sources and where they should go, BUT goes into mymodule/composer.json => extra/drupal-assets key.
    * We can then have different post-install handlers that do the actual downloading.

    That handler may use stream wrappers or symlinking (note that in probably most installations /vendor is not in docroot). That handler may add vcs repositories to the root composer.json, OR do the download themself.

    This approach is in fact what D7 libraries module did, but on the composer level (no bootstrapping drupal needed).
    Allowing pluggable handlers should allow us to settle on a libraries specification standard without prescribing exactly how libraries are fetched and integrated into the build.

    Jaypan’s picture

    Allowing pluggable handlers should allow us to settle on a libraries specification standard without prescribing exactly how libraries are fetched and integrated into the build.

    Libraries often come with insecure files - HTML/JS etc, that are not needed, and could become a security risk. By keeping them in the vendor folder and out of the webroot, then linking to specific files in the vendor folder, this becomes more secure, instead of having a bunch of web accessible files in the /libraries folder, which was a problem in D5.

    Mile23’s picture

    Folks interested in this issue should probably look at and understand the scope of #2982684: Add a composer scaffolding plugin to core

    kpv’s picture

    Hello, we've implemented a fully functional infrastructure for handling external library dependencies and use it for our projects. It consists of VisualN Libraries module and a custom packagist with basic packages. A package contains a plugin that provides info about a library (as in *.libraries.yml) and assets dependencies (via composer.json). It can be used even for existing projects/modules out-of-the-box (due to libraries override feature) in a non-breaking fashion (if you later decide to stick to a different approach, you can just disable the module).

    composer configuration example:

    "repositories": [
       {
           "type": "composer",
           "url": "https://packages.drupal.org/8"
       },
       {
           "type": "composer",
           "url": "https://asset-packagist.org"
       },
       {
           "type": "composer",
           "url": "https://drupal-libraries-packagist.example.com/8"
       }
    ],
    
    "extra": {
       "installer-types": [
         "bower-asset",
         "npm-asset",
         "drupalbase-module",
         "visualn-library-plugin"
       ],
       "installer-paths": {
           "web/visualn-libraries/{$name}": [
             "type:drupal-library",
             "vendor:npm-asset",
             "vendor:bower-asset"
           ],
           "web/components/visualn_libraries/src/Plugin/VisualN/Library/{$name}": [
             "type:visualn-library-plugin"
           ]
       }
    }
    

    The main idea is that what really should be cared about is Drupal libraries info but not js/css assets, the assets themselves are just dependencies. Actually Drupal doesn't use or care about js/css assets at all (except in aggregation and tests), all it uses is libraries info. Some analogy could be made with Linux packages systems, e.g. deb and rpm packages have different structure and distribution systems though may contain the same program, and the primary thing for management is packages metadata.
    Check project page for more info.

    A production-ready/community-driven libraries packagist can be discussed if there is an interest, maybe implemented as part of Drupal infrastructure.

    geek-merlin’s picture

    @Jaypan #62

    By keeping [libraries] out of the webroot, then linking to specific files, this becomes more secure.

    Yes +1 for not making every lib demo folder web-accessible, by whatever means.

    @Mile23 #63

    Folks interested in this issue should probably look at and understand the scope of #2982684: Add a composer scaffolding plugin to core

    You mean this component should be used to do the build process for libraries too, not only core root-level files?

    #64: Interesting!

    kpv’s picture

    The key point is that we should decide how to handle libraries registration info first. And it is where the main efforts should be focused for now.

    It includes:
    * libraries contents metadata (assets files locations) and their versions (to resolve possible conflicts)
    * a common namespace that could be used across all Drupal projects

    Using plugins for that may seem excessive at first, though it provides some additional advantages, e.g. for dev and prod environments it can register different versions of assets (some-library.min.js and some-library.js) which can be helpful for debugging purposes.

    It would be nice to have your guys thoughts on the subject.

    Jaypan’s picture

    The key point is that we should decide how to handle libraries registration info first. And it is where the main efforts should be focused for now.

    I think they are separate issues if anything. The fact is, there are CSS/JS libraries that can be managed through Composer. Drupal has been integrated with Composer. At the moment, Drupal does not allow having the Vendor directory outside of the webroot if/when managing a library with Composer. Therefore, we need to solve this problem - allowing the vendor directory to be moved out of the webroot, and providing a method of using any CSS/JS files in said library by providing a web-accessible URL. This will allow said libraries to be managed in a consistent manner with the rest of the codebase of the site.

    Now, if we also want to provide additional library registration info, similar to a hook_libraries() in D7, allowing for libraries to be managed in a non-Composer way, would allow for libraries that do not offer any Composer integration to be included on the site. So I can see the advantage of this as well, though it's personally not my focus, and I don't feel it's a solution to the issue we are discussion, as it is not library management through Composer.

    That said, maybe I'm misunderstanding - is registering libraries something to do with Composer integration, and I'm not realizing it?

    kpv’s picture

    That's correct, resolving the issue with vendor directory is a different one.
    My point is that handling 3rd party libraries is a broader issue then just handling js/css assets libraries only. Registering libraries can also be handled by Composer (as described in #64). Just wanted to attract attention to that since it is never discussed. The libraries api doesn't solve it either.
    Probably, this should be discussed in a separate thread though..

    markhalliwell’s picture

    The title "3rd party library management" is rather ambiguous here I think; it isn't very clear what "library" means.

    Using Composer for 3rd party PHP dependencies is an appropriate solution, one that many projects/sites already implement.

    Using Composer for CSS/JS based libraries/assets is not, as I have explained in #3002647-19: Add a vendor:// stream wrapper.

    ---

    One of the problems, I feel, is that some module developers want their cake and be able to eat it too.

    They develop a cool module and need "dynamic" or "cool looking" UIs to go with their vision of what think is an appropriate UX decision.

    Unfortunately, that is almost always far from the case especially when client requirements are involved.

    People often forget why we have distinct extension types; in fact, they often mistake calling themes "modules".

    They're not.

    Modules are not themes and themes are not modules.

    They have very different, specific tasks and priorities.

    Allowing modules to make unilateral decisions for FE assets is a dangerous and irresponsible precedent.

    Not only does it make assumptions of which libraries are used, based solely on the personal discretion of the maintainer, but it also makes it rather difficult to override later in a theme when client requirements dictate that a different plugin be used.

    Modules should be focused, primarily, on the "backend" functionality of their module. When they need to provide markup, they should use existing templates as much as possible and provide unique theme hook suggestions.

    If they have to provide JavaScript (to get their module to actually work), then they should do so using Drupal's existing coding standards and with theme abstraction in mind (i.e. don't use existing markup selectors, introduce your own unique attributes to target and use Drupal.theme anytime there is actual markup introduced).

    These ideas are anything new. In fact, it's why themes exist in the first place. This is why they're the last in the chain to allow overrides of, almost, anything markup/library related.

    Why are we suddenly going down the path of "merging" these unique responsibilities?

    It is kind of baffling to me.

    There's a reason why modules have a harder time implementing FE assets: because they're modules.

    I'm not saying that I have a perfect solution here or really any at all (at the moment).

    I just feel that this needs to be said so we create solutions that don't muddy the waters any further than it already is.

    Jaypan’s picture

    Title: Drupal core should help make 3rd party library management not torturous » Improving core management of 3rd party CSS/JS libraries

    The title "3rd party library management" is rather ambiguous here I think; it isn't very clear what "library" means.

    I've changed the topic to reflect what the discussion is about.

    One of the problems, I feel, is that some module developers want their cake and be able to eat it too.

    They develop a cool module and need "dynamic" or "cool looking" UIs to go with their vision of what think is an appropriate UX decision.

    Unfortunately, that is almost always far from the case especially when client requirements are involved.

    I personally am not an early adapter. I prefer only to use well established technologies that I feel have longevity ahead of them. I was not a fan of Composer integration with core when D8 was released, but over time, I have seen that Composer integration is part of what makes Drupal 8 awesome. When used properly, it solves a lot of code management issues that were present in earlier versions of Drupal, and it allows Drupal 8 to act more like a hub, rather than having to rebuild the wheel in Drupal.
    So I don't feel Composer integration is a matter of being 'cool', rather it's an extremely powerful tool that has solved a lot of issues, and brought a lot of power along with it. It also brought some new issues - mainly, being a command line tool which is daunting for developers not familiar with the command line. But these new issues should be fixed on their own merits (for which initiatives exist), in order to continue the evolution of Drupal as the powerful framework it is.

    Not only does it make assumptions of which libraries are used, based solely on the personal discretion of the maintainer, but it also makes it rather difficult to override later in a theme when client requirements dictate that a different plugin be used.

    This is a ship that has already sailed. Core already uses multiple third party libraries based on the discretion of core. jQuery being the most long term example. And developers have always been able to choose specific libraries to integrate with. No one has to use those modules, and that's why there is a diverse ecosystem.

    The fact is however, that modules provide functionality on Drupal sites. The idea that CSS/JS libraries can/should only be part of the theme does not make sense; there are CSS and JS frameworks that can be installed at a module level, then implemented at the theme level. Also consider that Field API fields are created at the module level, not in the theme layer, and many of these fields require 3rd party libraries to do what they do. An example is the jQuery Colorpicker module of which I'm a maintainer - this module provides a field widget. This widget could not work without the JS library that backs it - without that library it's just a textfield. The library replaces that text field with a JS widget. This both could not, and should not, be done at the theme level, as the form element is something to persist regardless of theme, not dependent upon theme.

    Why are we suddenly going down the path of "merging" these unique responsibilities?

    See my field example. CSS/JS is not unique to themes at all. It's part of core, part of modules, and part of the theme.

    The fact is, we have a framework for managing code: Composer. Composer is used by core, and used by 3rd party modules. It is specifically for managing code, so having to manage CSS and JS assets in separate manner requires more work, and the lack of consistency opens up more possibility for problems such as conflicts.

    markhalliwell’s picture

    So I don't feel Composer integration is a matter of being 'cool', rather it's an extremely powerful tool that has solved a lot of issues, and brought a lot of power along with it.

    Please don't take what I said out of context. The "cool" I was referring to was the FE assets that modules can come with because they wish to add, most of the time, superfluous features to "solve" their UI/UX blandness.

    Often times these decisions and implementations of FE assets are completely unnecessary.

    This is a ship that has already sailed. Core already uses multiple third party libraries based on the discretion of core. jQuery being the most long term example.

    No, it hasn't. In fact, jQuery is probably a perfect example of core imposing a specific library on sites whether they wanted it or not.

    It hasn't been until 8.x where jQuery became less of a dependency for other core JS.

    Another perfect example is jQuery UI (for core dialogs). In a base theme like Drupal Bootstrap, which comes with its own modal plugin, the need for jQuery UI Dialogs is completely unnecessary.

    Supporting core's Ajax/Dialog API is, currently, rather difficult (not impossible), due to the severe coupled nature and assumption that jQuery UI will be used.

    No one has to use those modules, and that's why there is a diverse ecosystem.

    Except in cases where it isn't as "diverse", like with webform.

    The idea that CSS/JS libraries can/should only be part of the theme does not make sense; there are CSS and JS frameworks that can be installed at a module level, then implemented at the theme level.

    I never said "only themes" should provide CSS/JS. I said that modules should focus on their primary functionality.

    When they do need to provide CSS/JS, they should do it in a way that is theme abstract.

    That is, however, rarely the case.

    An example is the jQuery Colorpicker module of which I'm a maintainer - this module provides a field widget.

    This module exists for a very specific and sole purpose: a singular jQuery plugin.

    However, this is exactly what I'm talking about. This kind of functionality shouldn't be a module in the first place.

    It's a FE library/asset that should live in the theme. Would the textfield technically work without the plugin, yes (just a hex value).

    What happens when the client wants to use a different FE library for color picking. Granted, the likely solution would be to just uninstall the module and replace it with something else.

    I get the desire/idea that people want "one-click" solutions to enhance their site; often because they aren't jQuery/JS experts.

    This both could not, and should not, be done at the theme level, as the form element is something to persist regardless of theme, not dependent upon theme.

    Except that it is actually entirely dependent on the theme, especially when external FE frameworks are being used.

    Themes have the ultimate power at the end of the day. They are the last chain in the alter process. They can choose which libraries are used and which aren't.

    The fact is, we have a framework for managing code: Composer. Composer is used by core, and used by 3rd party modules. It is specifically for managing code, so having to manage CSS and JS assets in separate manner requires more work, and the lack of consistency opens up more possibility for problems such as conflicts.

    This topic is, or rather should be, about how to collectively add FE assets for all extension: profiles, modules, [theme engines], and themes.

    However, more specifically, also establishing how to implement said FE assets so that they can be extended/overridden properly down the extension chain and not impose its own static implementation... client/site requirements be damned.

    Furthermore, as I stated in #3002647-19: Add a vendor:// stream wrapper, the issue with installing FE assets using Composer is that it does so in a global vendor space.

    I don't know why you keep ignoring this fact, but this presents a major issue when you have a multi-site installation.

    While, yes, one site may use the version of the FE asset installed globally, another site may need an entirely different version or even an entirely different implementation altogether.

    How is this one site supposed to "use composer" to install said alternate version/implementation (aside from really shady composer hackery)?

    ---

    The true fact of the matter is that use of libraries, regardless of where they initially come from, is ultimately decided on a client/site/theme basis: not modules.

    I think any real solution here would have to take that into account.

    The reality is that people are only using Composer for FE assets because there really isn't any other viable solution currently available to them.

    This doesn't automatically make it the "right" solution.

    In fact, I would argue that a "correct" solution would be to finally implement some sort of CDN based library asset management API.

    There are tons of FE specific CDN libraries out there.

    This problem isn't unique to just core or modules but also themes as well.

    I actually sort of started abstracting this idea last year in the form of https://www.drupal.org/project/cdn_library and https://www.drupal.org/project/jsdelivr.

    Granted, it is nowhere near perfect or even working for that matter.

    However, it's a possible solution that I think is more appropriate than using (and abusing) Composer for something it was never originally intended for in the first place.

    kpv’s picture

    @markcarver, if I correctly understand the problem, the module mentioned in #64 should solve it. It will require a couple lines of code though: you'll still need to register a custom library in your_module.libraries.yml with css/js assets you want instead of original ones. Then use it to override the original module library (via UI). After that, at any time the original library is attached, your custom assets will be used instead.

    markhalliwell’s picture

    No, it does not.

    Mile23’s picture

    Here's a user story for you:

    "As a contrib maintainer, I want Drupal or some native tool to compile my module's ecmascript to JS when the site codebase is built."

    This implies that my contrib module would have a packages.json file, and a script for how this build process is performed.

    (As it turns out, core does this, but not automatically in any way.)

    So we have a contrib module with packages.json and whatever we call the build process, like maybe drupal_build.js.

    Now we have to put our compiled JS somewhere. We could put it into the contrib module filesystem. We could put it into libraries/. It could be inside the docroot or not inside the docroot (depending on where we put libraries/). I suggest we put it in a subfolder of private://.

    We need to be able to use the render system to attach this generated JS. So we'd make a render array that has ['#attach']['library'][] = 'my_contrib/generated_js';

    Then we'd need to be able to define the my_contrib/generated_js library within the library API in core, so that it knows to look in the build-generated part of the filesystem for the library. This might mean adding a key to the YAML schema that says 'build-generated' or something. Then the library API knows to look in our configured generated code directory.

    Now the generated JS file can be attached by the render system, and then the caching systems for presentation to the user.

    But in order for that to work, we have to have a build phase.

    In fact, there's an initiative to make a build tool: #2940733: Site Builder Tool/Project Browser initiative

    What should happen is that some people decide to actually put in the effort to make the build tool. :-)

    Mile23’s picture

    Jaypan’s picture

    No, it does not.

    Yes it does. I’ve been using for exactly that in multiple modules.
    And honestly, your comment was not constructive whatsoever.
    It seems you have a personal issue with this rather than looking at it practically.

    Jaypan’s picture

    We need to be able to use the render system to attach this generated JS. So we'd make a render array that has ['#attach']['library'][] = 'my_contrib/generated_js';

    Then we'd need to be able to define the my_contrib/generated_js library within the library API in core, so that it knows to look in the build-generated part of the filesystem for the library. This might mean adding a key to the YAML schema that says 'build-generated' or something. Then the library API knows to look in our configured generated code directory.

    Or you could even point at the directory with a stream wrapper. Like the private stream wrapper, but to the vendor directory instead. Like, a vendor stream wrapper! Then you wouldn’t need to introduce a build process into the system.

    markhalliwell’s picture

    Yes it does. I’ve been using for exactly that in multiple modules.

    Great! Let's start collecting data from more people about how this contrib module may help or hurt the architecture of a site and its development workflow.

    And honestly, your comment was not constructive whatsoever.

    That's because I didn't feel like repeating what I already said in #71.

    It seems you have a personal issue with this rather than looking at it practically.

    Actually, it's quite the opposite.

    I'm primarily looking at this topic through the eyes of a FE developer, namely one who has successfully built and maintained the #1 most installed base theme on d.o: Drupal Bootstrap.

    I have years of experience dealing with and integrating 3rd party libraries; especially ones imposed on FE developers by modules with little or no regard to ensuring the implementation of said 3rd party library is properly theme abstracted (see #2803407: Refactor form element and JS to work with any theme).

    I'm also looking at this topic critically, to ensure we don't have a repeat of jQuery Update. People often use this project as "proof" that we can and should implement 3rd party libraries via modules.

    This module's creation was simply a direct result of the slow development process of core in the past; nothing more. In fact, the introduction of this module has since caused numerous issues and confusion amongst developers and less technically (JS) inclined site-builders for years.

    Nevermind the ever-increasing incompatibility between modules which still being developed, but based on much newer versions of jQuery. This effectively "bumps" the entire site's minimum jQuery version and causes even more compatibility issues with older and now less maintained projects.

    Suffice it to say: it's a mess; one I'd like to avoid at all costs in the future.

    ---

    @Mile23, I've actually given the "use a build process" approach some thought, hence my slow reply to #74.

    I'm still not done mulling this over though and will need to come back so my thoughts are consolidated into something a little easier to digest.

    Mile23’s picture

    markhalliwell’s picture

    Title: Improving core management of 3rd party CSS/JS libraries » Implement core management of 3rd-party FE libraries

    Re: using a build process

    I don't think this will work either, if only for similar reasons to the multi-site Composer single library issue; just more pronounced and easier to create conflicts.

    What happens when two different projects depend/develop based on a specific version?

    Let's use jQuery as an example (hypothetically as if it wasn't part of core and didn't have/use noConflict mode):

    If I understand #74 correctly

    • module_a depends/builds on jQuery 1.5
    • module_b depends/builds on jQuery 3.1
    • Both projects would be packaged with each of their respective libraries as part of the project's tarball after d.o "builds" the project's dependencies.
    • A generated_js folder would be included with the project which the project could target to create its own Drupal "library" (i.e. module_a/jquery and module_b/jquery)
    • Both projects attach the library in a page alter because it's a global implementation and needs to be used on every page.
    • Browser JS engine fatals due to vastly different and incompatible APIs of the same library.

    I'm using jQuery here as a very extreme example if only to exaggerate the issues we'd ultimately run into with this kind of approach.

    ---

    The main issue I think is made more clear just by re-reading the title:

    Improving core management of 3rd party CSS/JS libraries

    This implies core actually manages these 3rd party libraries in the first place; which it doesn't. Projects can implement, via any number of ways, their dependency on 3rd party libraries. Core has absolutely no centralized management system in place to deal with 3rd-party FE libraries. I'm changing the title to better reflect what this issue should actually be about.

    #71: implement some sort of CDN based library asset management API

    Currently, the core's Library API is completely arbitrary and prone to easy duplication (see module_a and module_b example above). This isn't "management", it's just a basic registry. There's already a number of services out there that do this: NPM for instance.

    Core just needs to adopt some sort of external FE library registry that is authoritative and that can be queried against, like https://registry.npmjs.org/-/v1/search?text=jquery. Ultimately, some sort of CDN integration (assets are actually served from some CDN) would be nice, but it's not technically necessary. Although using a CDN like jsDelivr (which has a better API IMO) could allow core to actually download and install to /libraries based on the registry search above:

    https://data.jsdelivr.com/v1/package/npm/jquery@3.4.1
    https://cdn.jsdelivr.net/npm/jquery@3.4.1/dist/jquery.min.js

    Of course, a manual installation should still be possible if a CDN isn't wanted or unable to be used (regional/government/security issues).

    Granted, I don't think we can ever get a perfect solution that will actually "prevent" any conflicts (that's ultimately the job of the site-builder/developer), but this would at least give us better control in the long run.

    We could even create some sort of UI that can help search for and install new external libraries and even help visualize which extensions depend on certain libraries/versions. Core could even go as far as tracking which libraries are used on routes or allow developers to specify manual overrides of libraries/versions for these routes.

    This isn't a perfect solution and would definitely need to be flushed out, but it basically boils down to: lets make core actually manage assets and then let a site actually manage the FE assets loaded.

    I feel like this is just another area that we need to "Get off the island" still and utilize other services that already do this job fairly well.

    mxr576’s picture

    Even though Composer has never intended to be anything else than a tool for PHP dependency management, a few weeks ago I had an idea about introducing meta-packages for assets and depending on them in consumer PHP libraries instead of their providers (ex.: npm-asset, bower-asset, etc.). I am leaving a reference here, maybe this thread adds some new aspect/value to this conversation.

    blazey’s picture

    Many of the problems described here can be solved with the webpack module. It uses a node-native way of declaring and storing dependencies and works fine for custom and contrib development. Theoretically, it could also be used in core to bring js (and possibly css) bundling and preprocessing capabilities to all drupal libraries.

    markhalliwell’s picture

    Interesting module! Thanks for adding ideas to the conversation :D

    I'm not entirely sure Webpack is the right solution for the job here though.

    Webpack is a packaging system, not a management system.

    Also, just thinking this through a bit.

    Bundling all of core + contrib + custom seems like it would be massive and likely have extreme performance issues.

    Even if we were to somehow figure out asynchronous chunk loading, we'd need a very complex system in place to determine when to do this for which route, condition, etc.

    Ultimately, I think webpack is out of scope for this issue. Might be far better off for someone to just build an entirely decoupled FE at that point.

    blazey’s picture

    Thanks for comments @markcarver. Looks like you didn't have a chance to check out the module yet, so I'll try to briefly describe what it does.

    Webpack is a packaging system, not a management system.

    Webpack is a bundler and preprocessor. We already use a preprocessor in core (the es6 scripts). If we want to be able to import npm packages and use the modern js dependency management techniques (e.g. tree shaking), we need a bundler. This is the state of the art in JavaScript. Without one we will be adding a system that is outdated straight from the start.

    Even if we were to somehow figure out asynchronous chunk loading, we'd need a very complex system in place to determine when to do this for which route, condition, etc.

    Actually that's solved already. Each drupal library is a separate webpack entrypoint. The module keeps track of them and loads all the chunks instead of the original library (source) files. Linking libraries with routes is done by Drupal.

    Mixologic’s picture

    If webpack is a bundler, that probably will not work for Drupal, because of licensing concerns at the very least.

    i.e.

    If you're building a contrib module and you don't want to force the end sites to have to set up webpack you can bundle a single library along with all its npm dependencies. The library will work on a plain drupal installation.

    You can't/should not do that unless you know that the library and all of its npm dependencies are licensed gplv2-or-greater, otherwise you cant submit your module to drupal.org as a contrib module.

    If it can be set up such that the modules have a manifest (packages.json?), and a tool to resolve those (npm/yarn - do you need both?) Then FE libs can be installed the FE way, and php libs can be installed the php way (with composer). That would still leave any old archaic libs that are not in npm to be manually installed or something.

    But, thats a pretty big product decision to declare that "end users of drupal, now require npm" -> not saying it shouldnt be made, just that its got the same kind of impact as introducing composer has for php dependency resolution.

    The primary impact is that 'modules' are no longer really standalone things that we can just install without an extra build step, and we really ought to see about making that build step painless for those end users who are already decrying how 'simple' drupal used to be before the management tools ruined everything.

    A perfect world would probably be a commandline tool in core (not external like drush or console), that is responsible for installing extensions and running all the build tools required to fully 'install' a new extension.

    blazey’s picture

    You can't/should not do that unless you know that the library and all of its npm dependencies are licensed gplv2-or-greater, otherwise you cant submit your module to drupal.org as a contrib module.

    True, the option of including the bundle should be used with care. I've added a sentence about that to the module page.

    In general, the webpack suite (Webpack Babel, Webpack React, etc) follows a pattern that is very similar to how composer manages php packages. Each module has a package.json file with its own dependencies (similar to composer.json in modules). That file can then be referenced from the site-wide package.json with

    yarn add file:./web/modules/contrib/webpack_babel
    

    This way there's a central node_modules/ holding all the packages (similar to vendor/).

    Contrib modules that aren't allowed to bundle their dependencies can use this technique as well. It requires a manual post-install step (yarn add file:...) but it's still easier than downloading files manually. The libraries that come with core already meet the licence criteria, so bundling them should be fine.

    If it can be set up such that the modules have a manifest (packages.json?), and a tool to resolve those (npm/yarn - do you need both?) Then FE libs can be installed the FE way, and php libs can be installed the php way (with composer). That would still leave any old archaic libs that are not in npm to be manually installed or something.

    Webpack requires yarn now. The process of building webpack.config.js, which is the most important part, doesn't have any external dependencies though.

    The primary impact is that 'modules' are no longer really standalone things that we can just install without an extra build step, and we really ought to see about making that build step painless for those end users who are already decrying how 'simple' drupal used to be before the management tools ruined everything.

    The extra install step would only be required in the non-gpl2 scenario (and right now such modules need manual action anyway). In core there already is a "build" step (yarn run build:js). With a bundler it would just be a more powerful one.

    A perfect world would probably be a commandline tool in core (not external like drush or console), that is responsible for installing extensions and running all the build tools required to fully 'install' a new extension.

    Adding pure shell support is possible but it would have some limitations compared to the current modus operandi. Processing the static libs (defined .libraries.yml) should work fine though.

    Mixologic’s picture

    In my experience, the dependency chain in npm land is gigantic, and thus the odds of not hitting a non GPLV2-or-greater license is pretty high (any apache license, for example).

    core *does* have that yarn build step, but the output of that is js that gets committed, none of the actual dependencies do.

    So, the ideal FE tool has a way to express dependencies in a metadata format (packages.json) that can be shipped with a module,
    And the end user needs to run a tool that can retrieve and assemble those dependencies and make them usable by the system. (yarn, in this case). Even if module maintainers were able to ship their dependencies all packaged up, they probably shouldn't because the mechanism/methodology should be the same/similar for all installations -> Add an extension, run yarn to get the FE setup and/or composer require and extension, and trigger yarn to run from a composer script.

    It does sound like webpack might have some potential here, if it can behave similar to composer and not actually attempt to bundle the artifacts, but leave that to the end users to do.

    blazey’s picture

    It does sound like webpack might have some potential here, if it can behave similar to composer and not actually attempt to bundle the artifacts, but leave that to the end users to do.

    That's what the webpack module does now in the default mode (see Building for prod). Module dependencies are declared in package.json files which are then added to the project-wide package.json (so yarn sees them as "local npm packages"). Neither node_modules or the resulting bundle files should be commited to the repository (although bundles can be set to be commited if there's no other way - that's for sites that can't run build steps).

    Then the build itself is a two step process

    1. yarn install
    2. drush webpack:build

    The latter generates a webpack config file, runs the build process and parses the cli output to determine which chunks are needed for which entrypoints (drupal libraries). The chunks go to the public files directory (by default) and are loaded instead of the original files by the DecoratedAssetResolver.

    Basically yes, webpack could theoretically be used as the "composer" for js (and css). The difference would be that composer needs just the install. Webpack would need an install and then a build, with the ability to store some additional metadata on top of the output bundles. Would it work for core? I have no idea :). It might be worth a shot though.

    Version: 8.8.x-dev » 8.9.x-dev

    Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

    fgm’s picture

    Another data point: we've been using asset-packagist for many months too, and I noticed two issues with it:

    • it slows down the composer install step immensely
    • due to its operation requiring a specific format, packages will have invalid names starting with Composer 2.0, which is likely to be a blocker until either side moves, which in turn may imply having to redo composer files for projects relying on it.

    Version: 8.9.x-dev » 9.1.x-dev

    Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

    prudloff’s picture

    due to its operation requiring a specific format, packages will have invalid names starting with Composer 2.0, which is likely to be a blocker until either side moves, which in turn may imply having to redo composer files for projects relying on it.

    This is no longer an issue: https://github.com/composer/composer/commit/960fa4b205bc336caac3b15d437e...

    fgm’s picture

    Excellent news ! The speed issue remains, but is not such a blocker.

    Version: 9.1.x-dev » 9.2.x-dev

    Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

    jungle’s picture

    https://github.com/drulibs/drulibs, https://drulibs.com: an easier way managing Drupal 3rd-party FE libraries with Composer, read more at https://jungleran.com/drulibs

    BTW, I've built a custom composer repository with Statis for managing libraries and I am using it myself currently.

    If it could be added to the Drupal core's root composer.json file as the following

    {
        ...
        "type": "project",
        "repositories": [
            {
                "type": "composer",
                "url": "https://packages.drupal.org/8"
            },
            {
                "type": "composer",
                "url": "https://drulibs.com"
            }
        ],
        "require": {
        ...
    

    Adding a library could be as simple as adding a Drupal module.

    Version: 9.2.x-dev » 9.3.x-dev

    Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

    Version: 9.3.x-dev » 9.4.x-dev

    Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

    effulgentsia’s picture

    I found this issue as part of researching #3145958-48: [META] Re-evaluate use of Backbone.js in core. I haven't read through this issue summary or comments yet, but I quickly noticed that Webform is mentioned, but I'm not sure if its use of composer.libraries.json has been mentioned as how it does it, so linking that here for reference in case that helps. I don't know if this is necessarily the best practice we want to adopt, but perhaps it's useful prior art.

    mxr576’s picture

    Priority: Normal » Major

    Nowadays there were so many major vulnerabilities reported in Node packages. Those ppl who are suggesting/inventing solutions for using Composer as a tool for managing JS dependencies: How do you image _proper_ security scanning of those non-PHP dependencies?

    Why not just figure out and document how to use NPM/Yarn for managing JS dependencies in Drupal - and rely on existing tools like npm/yarn audit for security audits and package updates?

    I am increasing priority by hoping this gets addressed in Drupal 10.

    mxr576’s picture

    mxh’s picture

    How do you image _proper_ security scanning of those non-PHP dependencies?

    I don't rely on Composer on projects that make heavy use of NPM packages (for example using React for frontend). I've built the Entrypoints module for my projects where my clients can see in the status report when used packages are out of date.
    In the last couple of years, we face a fraction of multiple contrib projects addressing that, all of them are barely used. That seems either the awareness of this concern is very low in the Drupal community, or simply there is no such need.

    prudloff’s picture

    Those ppl who are suggesting/inventing solutions for using Composer as a tool for managing JS dependencies: How do you image _proper_ security scanning of those non-PHP dependencies?

    We install several JS dependencies from Asset Packagist with Composer and we were worried about this.
    So we developed a Composer plugin that calls the NPM audit HTTP API (the same one used by NPM or Yarn audit command) for these packages and warns us if they have known vulnerabilities.

    But I agree that is totally something to keep in mind when choosing a JS dependency management solution for Drupal.

    kreynen’s picture

    @effulgentsia regarding #98 and Webform, I wasn't able to figure out a solution to use any of the recommended Webform dependency management options when sites are built with an upstream where wikimedia/composer-merge-plugin isn't supported (like Pantheon). Where we landed was, that while Webform itself is included in the upstream, sites that want to use Webform need to enable it, see the CDN warning and either run drush webform:composer:update to update the root composer.json file or ignore the warning. Unlike adding a module with composer require drupal/[PROJECT] which only requires git clone and not a functioning Drupal instance, drush webform:composer:update requires Drupal and Webform to be running where the command is run. Using DDEV, Lando, Docsal or Terminius to add Webform's FE dependencies to a project is beyond what most site owners/builders are capable of.

    I tried #3260159: Allow drush_webform_libraries_composer to target composer.json other than root before realizing that the Repository element of the Composer schema is root-only.

    I'm now back to looking at what @datvance is doing with ALL composer.libraries.json in their build process with https://gist.github.com/datvance/09dfb274c77e2a8104e415f8f1854557

    When adding the repository locations for Webform dependencies to the root composer.json, #3266007: Include licensing information in drupal-libraries helps with track/confirm the licensing of the Webform dependencies, but it really doesn't help with security. We're really left relying on Webform to monitor the security of its dependencies and include a check for outdated dependencies. At this point, we are almost back to projects including 3rd party dependencies in the Drupal project again.

    @prudloff THANKS for sharing insite/composer-npm-audit. We will definitely look into that.

    geek-merlin’s picture

    Some while ago i built something that is not a solution for the challenge stated here, but may be inspiration or a building block:

    AssetFetcher:

    This module downloads all external CSS & JS assets (e.g. from a CDN) and serves them from the site.

    Initially built to ensure GDPR compliance, it may be useful for other purposes, too.

    nod_’s picture

    Issue tags: +JavaScript
    mxr576’s picture

    OMG :) I really like how creative minds are thinking (like #102 and #104), but again, is it just spending time on newer and newer fixes (band-aids) for problems that were created by previous band-aids on band-aids...?

    There is some ongoing discussion related to this on Drupal Slack, I am copy-pasting some part of the conversation here before it gets lost.

    Aaron McHale 8 days ago
    Composer is not a (JS) asset mgmt solution
    Sure it's not the target audience, but fundamentally whether it's Composer, NPM, Apt, or whatever, Package Managers do the same basic things - they download things, put them in the right place, and manage the dependencies - does it really matter whether it's Composer or NPM. The advantage of asset-packagist is that the library gets installed when the module is installed, no extra dependencies required.

    mxr576 14 minutes ago
    > does it really matter whether it's Composer or NPM.
    Let me kindly disagree, that reasoning is just covering dust with sand. Language-specific package managers are adjusted for the language and the ecosystem.
    Would you use NPM to install PHP packages, no you would not... but you use Composer to install NPM packages because that is easier than solving the actual problem, introduce a process in Drupal to bundle other types of dependencies with Drupal components than PHP ones.
    An additional problem is that whatever is hosted on asset-packagist is fetched on-demand AND dependency trees are not fetched automatically.
    So just try to install asyncapi/react-component with Composer and you will see how it fails to install due to missing dependencies (of dependencies of dependencies of dependencies...).

    $ composer req npm-asset/asyncapi--react-component
    
      Problem 1
        - npm-asset/asyncapi--react-component[1.0.0-patch.1, ..., 1.0.0-patch.27] require npm-asset/isomorphic-dompurify >=0.13.0,<0.14.0 -> satisfiable by npm-asset/isomorphic-dompurify[0.13.0].
        - npm-asset/isomorphic-dompurify 0.13.0 requires npm-asset/jsdom >=16.5.2,<17.0.0 -> satisfiable by npm-asset/jsdom[16.5.2, 16.5.3, 16.6.0, 16.7.0].
        - Conclusion: don't install npm-asset/jsdom 16.5.2 (conflict analysis result)
        - Conclusion: don't install npm-asset/jsdom 16.5.3 (conflict analysis result)
        - Conclusion: don't install npm-asset/jsdom 16.6.0 (conflict analysis result)
        - Conclusion: don't install npm-asset/jsdom 16.7.0 (conflict analysis result)
        - Root composer.json requires npm-asset/asyncapi--react-component ^1.0 -> satisfiable by npm-asset/asyncapi--react-component[1.0.0-patch.1, ..., 1.0.0-patch.27].
    
    You can also try re-running composer require with an explicit version constraint, e.g. "composer require npm-asset/asyncapi--react-component:*" to figure out if any version is installable, or "composer require npm-asset/asyncapi--react-component:^2.1" if you know which you need.

    mxr576 6 minutes ago
    Symfony solved this problem with Flex in one way, half of the community loves it the other hates it... that is part of our job.

    mxr576’s picture

    Just for the record, this has happened and Asset packagist can easily become a security risk some time.

    It is definitely not about blaming the maintainers of the service, especially not in these times, but it perfectly highlights the risk of being "lazy" and depending on tools that can be gone anytime because they are maintained with good will and there is no community and/or big enterprises behind them.

    https://github.com/hiqdev/asset-packagist/issues/146

    geek-merlin’s picture

    As mentioned in #60, Asset Packagist is based internally on Composer Asset Plugin, and by using that, local composer can do the work.

    Can anyone provide experiences with that?

    sinasalek’s picture

    fgm’s picture

    FWIW, Foxy above self-describes as "A fast, reliable, and secure NPM/Yarn bridge for Composer"

    effulgentsia’s picture

    Why not just figure out and document how to use NPM/Yarn for managing JS dependencies in Drupal

    If there's a viable way to do it without requiring npm/yarn (or anything else requiring Node.js), that would be better for Automatic Updates and Project Browser, for Drupal sites on shared hosting or other environments without Node.js.

    https://github.com/fxpio/foxy requires Node.js.

    https://github.com/fxpio/composer-asset-plugin does not, but I don't know if/how it deals with the dependencies of dependencies problem in #106.

    Version: 9.4.x-dev » 9.5.x-dev

    Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

    lauriii’s picture

    Discussed this with @justafish at DrupalCon Prague 2022 to try to address #3277748: Drupal 10 compatibility in a way that doesn't require using asset-packagist. @justafish had a great idea that we could try to utilize yarn/npm workspaces for implementing this. Each module/theme would be able to ship a package.json to create a yarn workspace. This would allow installing all JavaScript dependencies for all modules with a single yarn install call. It also seems like that yarn workspaces is smart enough to not trigger warnings for directories in the workspace directory without a package.json. This allows package.json to remain optional for modules. We could also add core as a workspace and this could theoretically allow sites to update core JavaScript dependencies without waiting for a core release.

    We will still need to do some further research on what is the preferred way to move the 3rd party assets to their correct destination ([drupal-docroot]/libraries/[library/here]. Drupal core already implements some core specific tooling for this. Ideally we could do something similar for contrib in a slightly more flexible manner. This would allow modules to specify the assets it needs from its dependencies and where those should be located. With that information, the 3rd party assets could be moved to their correct destination either in a post install hook or a separate script that the developer would have to run.

    This would make some contributed modules introduce a dependency on Node.js for building the module (still not a runtime dependency meaning this could be done in a local environment or CI). That is a major drawback on this approach and we may need to try to come up with solutions to that. We could try to come up with documented steps for running the command locally without having a full installation on local environment. We could even try to figure out if we could install install npm/yarn + Node.js with composer when it's not available globally.

    effulgentsia’s picture

    If we want to go with a Yarn solution per #113, then I wonder if the way we can solve the Project Browser on shared hosting use case (e.g., use Project Browser to install a module that has front-end dependencies, but the shared hosting account can't run Node.js) is to run the Yarn command in the browser, and then have the browser submit the artifacts to the server. I don't know how feasible this would be, but https://blog.stackblitz.com/posts/introducing-webcontainers/ seems pretty intriguing.

    mxr576’s picture

    If we want to go with a Yarn solution per #113, then I wonder if the way we can solve the Project Browser on shared hosting use case (e.g., use Project Browser to install a module that has front-end dependencies, but the shared hosting account can't run Node.js)

    Just a bold idea and tbh I haven't thought too much about it because it felt so natural... what if Drupal.org would support dist and non-dist builds. The dist builds would contain the precompiled JS/CSS based on how the package maintainer described/implemented in the build process (in a DrupalCI configuration?). That way we can easily eliminate the need for compiling anything on the consumer side, yet we can leverage "modern" and _native_ JS asset management tools, like Yarn or Webpack, etc.

    geek-merlin’s picture

    Great debate: How do we enable modern JS asset management, without requiring JS builds on the server.
    The idea "do the build somewhere else" sounds intriguing.

    The #115 idea boils down to
    a) "Do the build only for a single module, and package the final JS build results."
    let's call it "local-to-build".

    OTOH we can say
    b) "Do a global build (resolving all package.json or equivalent of modules, spit out a package-lock.json, and process only the final package-lock on the target server by downloading stuff."
    let's call it "global-to-lock".

    The up-and downsides are straightforward, pick your poison.

    lauriii’s picture

    For what it's worth, I tested https://github.com/thecodingmachine/nodejs-installer on a php:8.1-apache container which doesn't include Node.js. After installing zip package which was required by composer, I was able to run composer. With composer, I was able to install the nodejs-installer which downloaded node and npm bin files to the vendor directory. After these steps I was able to install npm dependencies using a local version of npm and node 🤯

    Wim Leers’s picture

    … and that's despite https://github.com/thecodingmachine/nodejs-installer receiving its last commit on Nov 4, 2020, nearly two years ago 🙃

    Version: 9.5.x-dev » 10.1.x-dev

    Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

    mxr576’s picture

    #117

    Sorry for being a bit passive-aggressive this Monday morning... but I'd call it "install a potential backdoor/security risk on your production server" solution :D That approach would never pass on any security review conducted/requested by any enterprise company.

    (I am saying this with my full respect to the thecodingmachine guys, they have plenty of good stuff on the Internet)

    effulgentsia’s picture

    Re #120, is running yarn on a production server at all the thing that would not pass security review? Or are you saying that https://github.com/thecodingmachine/nodejs-installer specifically has security issues that you're concerned about?

    If the latter, then if you're aware of a specific vulnerability, please reach out to them to ask how to report that to them. If it's a theoretical risk that's okay to discuss publicly, what is it?

    If the concern is about running yarn at all on production, then enterprise users wouldn't need to (and shouldn't) use Project Browser or Automatic Updates in production. Assuming they have code deployment processes already in place that include running security scans, then they'll be able to use those modules on a dev environment, and then deploy from dev to prod as normal.

    Also, https://github.com/thecodingmachine/nodejs-installer wouldn't be needed on sites that already have Nodejs and NPM installed on the environments where they install/update Drupal modules. Assuming our solution to managing front-end dependencies requires Nodejs and NPM, then https://github.com/thecodingmachine/nodejs-installer (or some other approach like #114 or #115) would only be needed when whichever environment is installing/uninstalling/updating modules doesn't already have those tools.

    effulgentsia’s picture

    Another possibility for a yarn-based approach: if Yarn can't be called from the environment on which someone is using Project Browser or Automatic Updates (e.g., nodejs isn't on that server and the site doesn't want to depend on https://github.com/thecodingmachine/nodejs-installer either, or if we or the site owner just decides that having the Drupal site run a yarn command is not secure enough), could we then fall back to a CDN, with a status report warning for the libraries being included from a CDN and instructions that running the needed yarn command can resolve those warnings? We could have a setting for turning off the CDN fallback, in which case if a site isn't able to run yarn and also has turned off the CDN fallback, then if Project Browser or Automatic Updates is used to attempt an installation or update of a module that brings in package.json changes that require something different than what is already in the existing yarn.lock file, then we prevent that installation and instead show a message that the site owner needs to perform that particular installation or update manually, on an environment where they can run yarn.

    This would then cover the following scenarios:

    • Enterprise site: don't use Project Browser and/or Automatic Updates, or only use them on dev/staging environments where it's okay for the Drupal site to run composer and yarn.
    • Site on shared hosting without any dev to prod workflow, option 1: If Nodejs is already on the server or can be installed via https://github.com/thecodingmachine/nodejs-installer or similar, Project Browser and Automatic Updates work fine, because they can execute yarn when needed.
    • Site on shared hosting without any dev to prod workflow, option 2: If Nodejs isn't on the server, and installing https://github.com/thecodingmachine/nodejs-installer is not acceptable, you can still use Project Browser and Automatic Updates, but they might start causing some of your libraries to be fetched from a CDN. You can decide to either not care about that, or at some point when you're able to, you can run a yarn command on your local machine (or within a GitHub action or wherever) to clear that up.
    kay_v’s picture

    And another possibility for a yarn-based approach (building on #122) is the "zero install" workflow supported by Yarn ^2.0.

    With Zero install, a developer runs yarn install <package> on a dev machine. That command not only downloads packages but caches them. The cache gets committed to the repo. Thereafter the dependency tree is accessed from the cache, meaning that the dependency tree is built for the last time before it lands on the production server.

    Here's an excerpt from the "modern Yarn" docs:

    Zero install ... remove[es] the main source of entropy from the equation: Yarn itself.

    [... W]e believe this workflow to be a defining feature for professional-grade projects

    Would Zero install potentially address concerns raised above?

    andypost’s picture

    Issue tags: -JavaScript +JavaScript

    Even if yarn^2 will be used each asset library will need extra script to remove test/demo files from installed package like VendorHardeningPlugin is doing for composer.

    I see no general solution for contrib/custom code as each package upgrades may need to tune "hardening" settings for it.
    Also not clear how to manage dependencies of assets if any appear.

    Version: 10.1.x-dev » 11.x-dev

    Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

    prudloff’s picture

    Symfony tried to solve a similar problem so if we want to download NPM packages without adding a dependency on Yarn/NPM, we might be able to leverage the Symfony AssetMapper component.
    It is in PHP and is able to download NPM packages and their dependencies:

    They also implemented an audit command to check for known vulnerabilities: https://github.com/symfony/symfony/blob/6.4/src/Symfony/Component/AssetM...

    The main caveat is that it adds a dependency on the jsDelivr API to get the dependency tree of each NPM package (https://github.com/symfony/symfony/blob/6.4/src/Symfony/Component/AssetM...).
    Also they are basically reinventing NPM in PHP, which is something I have mixed feelings about.