Create a mirror for external library dependencies for composer support

Thanks @pameeela for introducing this.

Happy to help out with this, if maintainers agree to proceed with this. Here is what we've done in other places like Fullcallendar or Klaro:

• Create a general project on d.o as a mirror of the remote library
• Add a composer.json to that repository, set the name to drupal/NAMEOFLIB and set the type to drupal-library, also add the installer-name in the extra section to the name of the directory into which this library needs to be installed
• Push that code and create a release, ideally with the same version number as the referencing upstream library

The packaging system from d.o will publish that project to Packagist and you can now add drupal/NAMEOFLIB to require section of the composer.json of this project. From now on, the library will be installed automatically without the user having to do anything manually.

Of course, this needs to be done for each external library and yes, there will be some overhead to keep the mirrors up-to-date. From experience, I can tell, this doesn't happen too often and should be manageable.

This problem and solution for external libraries makes sense to me.

What will be the namespace for these JavaScript projects?

Key in mind there is https://git.drupalcode.org/project/webform/-/blob/6.3.x/composer.librari...

Could we move this file to something webform_libraries/composer.json and include it via a path?

The below example is entirely untested.

{
    "repositories": [
        {
            "type": "path",
            "url": "webform_libraries/composer.json"
        }
    ],
    "require": {
        "webform_libraries": "*"
    }
}

The namespace for such libraries will be drupal/NAMEOFLIB.

After that change, I think that the extra webform libraries composer.json file won't be required anymore.

I think this makes it more urgent that Webform sub-modules that depend on external libraries be moved into separate projects. That way, the library would only get installed if the site wants to use it. The sites I work on only use a very few external libraries that we have specifically chosen to add to our Composer files.

@liam morland do you mean sub-modules or separate modules? For the former, this won't make any difference, because only the main module comes with a composer.json, and that requires everything that itself and its packaged sub-modules potentially need. Only if some modules were moved into stand-alone separate modules, the list of required libraries would be reduced.

I am suggesting that some modules that are currently sub-modules of the Webform project should be moved into their own projects. If you choose to install the module, its Composer file will bring in the corresponding library.

Thanks for the clarification, so that would be separate modules then. You're correct, that would work. But I doubt this could be done in the short timeframe that's available in light of the Drupal CMS requirement.

Perhaps we should separate them out for Webform 6.4.x.

I'm concerned about adding all those libraries to the Webform Composer file. This would mean that all Webform users would suddenly get libraries installed that they are not using. Could you have the Drupal CMS Composer file require the libraries? Once the Webform sub-modules are moved into being top-level modules in their own projects, then the libraries would get installed from each of those.

I tested my suggestion in #5 and using the a path repository won't work as expected.

I don't think we need to require the new webform external library projects in webform/composer.json but Drupal CMS could require them.

The webform external library projects would be another (recommended) way to install external libraries.

We are going to run into namespacing issues unless we add a prefix or suffix. For example, there is already a https://www.drupal.org/project/select2 and https://www.drupal.org/project/codemirror project.

We could suffix the namespace, below are some example.

https://www.drupal.org/project/select2_library
https://www.drupal.org/project/select2_lib
https://www.drupal.org/project/select2_ext

I think select2_library would be good, pretty clear. Does Drupal have a pattern for this?

There is no established pattern AFAIKT, we've used the suffix *_js in other places before.

Adding the dependencies to Drupal CMS' composer.json does work, too. I just wonder why other users of webform shouldn't benefit from this improvement as well. The fact that a dozen JS libraries get downloaded and stored in the libraries folder, doesn't sound too bad. At least compared to the gain factor in convenience.

*_js is perfect because it is the shortest and easiest-to-understand suffix.

Adding the dependencies to Drupal CMS' composer.json does work, too. I just wonder why other users of webform shouldn't benefit from this improvement as well. The fact that a dozen JS libraries get downloaded and stored in the libraries folder, doesn't sound too bad. At least compared to the gain factor in convenience.

The added benefit is that the site gets notified of a new version or security issue. The one downside is files in /libraries are publically available, and there have been test *.html pages in J /libraries with XSS issues.

there have been test *.html pages in J /libraries with XSS issues.

That's true, but we could easily clean-up the mirrored libraries and remove those docs and example directories.

Also, another benefit is that webform will then be installable with project-browser without the user having to bother about anything.

Even if the test pages are gone, I would much rather not have any JS library available on the server unless I am using it.

Drupal CMS, we should create and require the *_js external library projects.

We can also rework the webform module to recommend these external library projects.

At a later point, we can decide to require these external library projects via the webform module's composer.json.

Does this seem like a reasonable way forward?

Yes, that makes sense. I don't think we should require any external library unless Webform cannot operate without it.

Still, I wonder how users who just work in the UI and install the webform module with project browser would ever get that up and running. It sounds like this isn't possible right now, or am I missing something.

As for requiring the libraries by the recipe in Drupal CMS, yes, that possible. The original intent of this issue has been to make this a webform improvement in general. But if that's not what the maintainers want, then @pameeela needs to make the call how Drupal CMS wants to deal with that.

As it currently is, there is no way to install the libraries from Project Browser; it would be CDN-only.

We could also create a webform_libraries project with a composer.js file that requires all the external libraries. The Webform Libraries project could be used via Project Browser.

If the Drupal community agrees that external libraries should be mirrored as projects (w/ *_js suffix ) on Drupal.org, I will be 100% committed to adopting this solution for the Webform module.

That could work, it should be a recipe rather than yet another module, though.

That could work, it should be a recipe rather than yet another module, though.

I am not used to thinking in terms of recipes but that it is perfect.

To help move us along, I updated the UPDATE_LIBRARIES.md and exported libraries

@see https://git.drupalcode.org/project/webform/-/blob/6.3.x/docs/UPDATE-LIBR...
@see https://git.drupalcode.org/sandbox/jrockowitz-2941983/-/tree/6.3.x

Below are all the libraries

• choices
• codemirror
• jquery.chosen
• jquery.image-picker (sub-module)
• jquery.inputmask
• jquery.intl-tel-input
• jquery.rateit
• jquery.select2
• jquery.textcounter
• jquery.timepicker
• popperjs
• progress-tracker
• signature_pad
• svg-pan-zoom
• tabby
• tippyjs

Deprecated

• algolia.places (deprecated)
• jquery.geocomplete (deprecated)
• jquery.hotkeys (deprecated)
• jquery.icheck (deprecated)
• jquery.toggles (deprecated)

How would we use composer.json to ensure the project with *_Js is placed in the correct directory?

It is also worth noting that the progress-tracke only contains CSS, so I suggested using an *_library project extension. Using *_js is also okay.

Lastly, we also need to account for #3489932: Move deprecated modules into webform_deprecated. I also noted which libraries are required for sub-modules.

How would we use composer.json to ensure the project with *_Js is placed in the correct directory?

The most reliable way is to give each package the type of drupal-library.

It seems we have two options.

OPTION 1: Create a webform_libraries recipe based on https://git.drupalcode.org/project/webform/-/blob/6.3.x/composer.librari...

OPTION 2: Mirror external libraries as projects (w/ *_js suffix ) on Drupal.org

Option 1 is much easier to implement and maintain.

Don't do a recipe - do a metapackage. I don't see how a recipe is useful here. If it's just to have it show up in Project Browser, we have ways to make that happen. Make it a metapackage. Don't repurpose the recipe package type for something it's not meant for.

But the repositories key has no effect in any composer.json except for the top-level (project-level) one. Source: https://getcomposer.org/doc/04-schema.md#repositories and https://getcomposer.org/doc/04-schema.md#root-package

So with that in mind, I'm not sure how this is supposed to work.

Leave this one with me for a bit. I have a few mad-science ideas that might bear fruit here.

@phenaproxima Thanks

Here's an idea -- what if we had a Composer plugin which could handle this? For the sake of description, let's call it phenaproxima/librarian.

So, first, you'd enable this plugin in your top-level composer.json: composer config allow-plugins.phenaproxima/librarian true

Then, in its top-level composer.json, Webform would depend on this plugin. And it would add the contents of composer.libraries.json's repositories section to its own extra.librarian section. Example:

{
  "extra": {
    "librarian": {
      "choices": {...},
      "codemirror": {...},
      etc.
    }
}

So what would happen here is: when Webform is about to be installed, the librarian plugin kicks into action. If it finds a package (Webform, in this case) that has a set of libraries defined this way, it exposes them to Composer as packages, generated on-the-fly, and adjusts Webform's requirements dynamically so that it depends on them. All of this information should enter Composer's dependency solver so that it can resolve and install them like normal packages.

I need to try this out, but I suspect it would work.

@phenaproxima the architecture of only supporting repositories on the top-level composer.json has been implemented for security reasons, and for years, the composer maintainers refused to change that, ever. When dependencies were allowed to extend the list of source where packages can come from, then the user will be out-of-control, finally. Supply-chain attacks will become just that much easier.

There are already plugins around that do this, though. But it has already been turned down, especially for security reasons.