Add a vendor:// stream wrapper

If this is a major security bug, it should not have been posted here in public.

On the security angle, are you referring to site admins unzipping entire library directories into the docroot when in fact they need only one component? Even if that is the case, I don't actually think this is a bug report, but rather a much-needed feature request that by the way is also a security hardening.

Also, this should be the asset library system, not composer.

Generally I use asset packagist - like the lightning and commerce distros (although I started using asset packagist before I knew those distros were using it) And set the location for them to libraries just like modules go into modules/contrib.

I would say that this is a minor not major issue btw, since there are multiple contrib or composer project level solutions. Or probably better, feature request with normal priority.

Just to point out that one of the goals of #2958021: Proposal: Composer Support in Core initiative is to allow for placing Composer-based dependencies outside the docroot. We also have a third-party solution for that already: https://github.com/drupal-composer/drupal-project

Then as you say it's a problem of getting assets where they belong. A Composer-package-specific file wrapper seems like a good idea ('composer://vendor/package/path/to/thing.js'). Technical challenges would include knowing where the vendor directory really is.

I agree with Jaypan. I also have to access libraries from the vendor folder, which is above the drupal core folder. So I don't see any other useful way than suggested by Jaypan.

I'm using a drupal-composer/drupal-project.

my project folder structure:

project root:
example.com

private
example.com/private

vendor
example.com/webroot/vendor

drupal core root:
example.com/webroot/web

I think this is a great solution to many headaches that exist managing frontend assets. The security hardening I feel is also a big plus.

For example it will allow to remove the core/assets/vendor from the drupal core repository once this is in core and ensure that all frontend libraries that Drupal depend on are in packagist.org.

This PR can be simplified
https://github.com/drupal-composer/drupal-project/pull/286
Since the frontend libraries con be downloaded into /vendor.

I feel it shouldn't be a module that can be enabled optionally but simply a part of normalizing drupal library definitions.

Can take many baby step for implementation:

- First enable the vendor steam wrapper
- Then enable the route for downloading assets outside the web root.
- Then allow the translation of vendor:// to the proper path (using the route if it is outside the web root)
- Then modify core.libraries.yml to use the vendor path and add those libraries to composer.

One possible drawback would be to have PHP serving the assets instead of the web server but it can be mitigated by checking if the vendor folder is outside the web root.

I volunteer to move this forward once is approved.

Do we agree on the way forward?

I think this is a great solution to many headaches that exist managing frontend assets.

I think this is a horrible solution for managing FE assets.

Composer was, originally, intended as a way to manage PHP; not CSS or JS.

Over the years, people have abused this and suddenly creating packages that actually have nothing to do with PHP.

Nevermind the architecture this imposes on FE assets and how they relate to how core actually operates with themes.

I don't know why, likely because very few people actually do this, but people often forget that Drupal has the ability to create multiple site instances using a single code base.

This means that if you use composer to install FE assets, then all sites/themes have to use the singular version of the FE asset.

This isn't always possible for seemingly obvious and various reasons (e.g. new sites are often developed at different times and have different requirements).

No, Composer is not a great solution for FE assets.

NPM (or Yarn) is. Its assets are relative to the actual theme that's used on that specific site, not the global Drupal install base.

---

I'm tempted to close this as won't fix if only for the above reasons. However, I'll simply mark this as needs more info.

A couple of additional issues I have with this:

First, this implies that Composer is a requirement, when in fact it isn't.

Core has made great strides to improve the integration of Composer, but it by no means requires it to be used.

Second, building on above, what actually happens with vendor:// when there is no actual vendor directory?

I personally believe this should remain as a contrib project, if only due to the many reasons stated above.

Architecturally, this lends to anti-pattern and abuse of Composer.

I also feel that using composer for downloading FE assets seems a bit weird but I think is one of the few ways forward to manage FE assets better in Drupal. Especially for modules adding some kind of JS or CSS functionality. Themes can still provide their own assets within the theme directory.

How do you feel about other stream wrappers like node_modules:// or libraries:// ?

I think the stream wrappers can be a good way to finally have full FE libraries in the docroot and serving only the assets that Drupal specifies.

Do you think we should have the Drupal core repo without third party FE assets and downloading them with npm for development just like composer dependencies?

I think composer might be a good candidate to avoid having another tool required for development like npm or yarn.

Second, building on above, what actually happens with vendor:// when there is no actual vendor directory?

I don't know Drupal can run without it.

Anyway I would keep this open for discussing further. I don't care if the issue is repurposed without composer or vendor, maybe having additional stream wrappers can be proven to be useful.

what actually happens with vendor:// when there is no actual vendor directory?

Yes, there's always technically a "vendor" directory... I misspoke.

I was actually asking what happens when the "vendor" directory isn't actually a [composer] vendor directory? Meaning: the user installed Drupal using d.o's packaged tarbar and not via composer:

Just because this stream wrapper points to the vendor directory, doesn't mean it will actually work, for example:

1. User downloads and installs https://ftp.drupal.org/files/projects/drupal-8.7.3.tar.gz
2. User downloads and installs a module that has a composer requirement
3. User finds that the module doesn't actually work because said composer requirements were not installed into the existing vendor directory (because Drupal doesn't do this, Composer does).

Thus, the module's reliance on vendor:// is unreliable between two types of installation: manual vs. composer.

While the directory may technically exist, the library it points to may not.

Adding this now, without composer actually being a requirement, will only introduce further confusion and complicate matters.

---

For the record, I'm not against the idea of this issue. In fact, I'm very much in favor of it (edit: not for the purposes of FE assets though). However, there seems to be misinformation out there that implies Drupal "requires" composer. It does not.

I recommend reading, specifically the first section which covers d.o's packaging process:

https://www.drupal.org/docs/develop/using-composer/using-composer-with-d...

While Composer certainly makes for easier implementation, it is by no means a "requirement" for Drupal to actually function. Until that is the case, I believe adding this will only further fragment the communities understanding surrounding this topic.

Again, I believe this type of functionality should [currently] live in the contrib space so that sites that are actually built with Composer can use it.

users without Composer will not be able to install these modules in the first place, due to the missing library dependencies

in other words anything that requires the library will not be able to be installed regardless of whether there is a vendor stream wrapper due to that requirement.

Not true.

Depending on libraries !== requirements that prevent the installation of an extension.

See: #2494073: Prevent modules which have unmet Composer dependencies from being installed and #2830880: Warn site admins when composer dev dependencies are installed inside of docroot for attempts to tell the user that they need something installed via Composer.

The reason we'd want a vendor:// stream wrapper is so that we can get the location of packages that were installed using Composer.

The reason we'd need the location of those Composer-based packages is because Drupal does not have a standardized build process, and we need the location of those files for the render and caching systems.

If Drupal had a build process we could put a packages.json and a script named something like drupal_build.js in extensions. We'd have a standard place to put assets where they could be accessible by the libraries API (the core one).

If we did that, then we start to rely heavily on not only Composer but also npm/yarn.

This all seems reasonable and good to me, but perhaps not to others. But if we did that, we wouldn't need a vendor:// stream wrapper.

If we did that, then we start to rely heavily on not only Composer but also npm/yarn.

This all seems reasonable and good to me, but perhaps not to others. But if we did that, we wouldn't need a vendor:// stream wrapper.

Exactly. The only reason this issue exists is that there is currently no formal process on how external libraries should be added/packaged.

Thus, people have resorted to abusing Composer for adding FE assets. It's no one's "fault", just that it is, currently, the only easiest solution.

This doesn't mean that it's the correct long-term solution though.

IMO, this issue is merely a side-effect of #2873160: Implement core management of 3rd-party FE libraries.

A well programmed module would include hook_requirements to prevent the installation of the module.

Except that implementing this hook isn't actually a requirement (ironically). Novice developers may overlook this and that can still allow the extension to be installed. Also, this is only true for profiles and modules, not the theme engines or themes which aren't allowed to participate in this hook (which is an entirely different and complicated issue).

It's ridiculous to think of this as an abuse of composer.

Why? Because you say so? I'm pretty sure using something the way it wasn't originally intended to be used is the very definition of "abuse".

That's like saying using Entities in Drupal is an abuse of Drupal, because they weren't part of the original design.

No, it's not. This isn't even a valid comparison. You're comparing a single component of a CMS to an abstract external PHP dependency manager, which has nothing to do with Drupal in the slightest.

The fact is, systems improve, and evolve. Composer can be, and is, used for managing CSS and JS libraries.

Agreed. They can improve and evolve. I'm not disputing that. Nor am I disputing that Composer can technically be used for FE assets (because people are abusing it).

What I am disputing is this specific architectural decision/justification to actually do so. Just because we can technically do something, doesn't mean we should in the first place.

Being able to manage the entire codebase of a Drupal site with Composer is a benefit, as it keeps everything consistent, and able to track conflicts.

If you have a simple, singular site.

I don't know why this is being repeatedly ignored, but the fact is that there can be multiple sites using a single Drupal codebase.

What happens when one site needs version 1.0.1 of a library and another site needs version 3.2.3 of the same library?

How does Composer handle installing two separate versions of the same library, on two separate sites with a single Drupal codebase?

Short answer, it can't... short of some major advanced hackery and even then, it's likely to be a very unstable solution.

What this issue is proposing restricts, limits and actually increases the chance of conflicts within a site (in regards to FE assets).

Now, if you think there should also be other methods of managing code, then that's fine, and for another discussion.

There is. This is why I've been postponing this issue.

#2873160: Implement core management of 3rd-party FE libraries needs to be figured out first before introducing this feature; please stop changing the issue's status.

Mainly because the sole purpose behind this issue, thus far, has been in regards to FE assets.

Doing this now would only open the flood gates and lead to further fragmentation and confusion as to best handle FE assets (as I have stated numerous times before).

But the idea that Drupal 8 does not require Composer is a misnomer, as core requires libraries that depend upon it, and plenty of contributed modules also require Composer to be installed, and even list it in their installation instructions.

No. The misnomer is you believe that it does.

Installing core does not require Composer, which is only a dependency manager, not a runtime manager/library.

You can download the tarball from d.o and install it right now, all without Composer.

This is mainly because d.o obfuscates Composer and packages the tarball with the installed dependencies of the vendor directory.

This is not disputable, this is a fact (no matter how many times you say its a "requirement").

Considering this under it's own merits - CSS and JS libraries can be managed with Composer, and Drupal does not provide an effective means of using these libraries when they are outside the webroot.

This issue, on its own, does have merit. However, not for the reasons you are claiming.

Being able to target the vendor directory is necessary in some cases, but more so to be able to scan for raw packaged files that aren't loaded via the autoloader. An example that comes to mind is parsing a version from some sort of JSON file that is typically packaged with a library.

What is being proposed here, though, affects the entire community and will have major implications once it's introduced.

That is why I said this stream wrapper should continue to remain as a contrib module for now.

No one is denying anyone from using that contrib module if they really want to use it.

I'm cautious of proceeding with this issue until the other one is finally resolved, in fear that this one will be abused by developers looking for a quick and easy fix to their "FE asset" issue.

This is a core Drupal problem, of poor integration with Composer.

This is part of the ongoing integration with Composer. You seem to have this idea that Drupal just decided one day to "require" Composer and so it's the end all be all.

It's not.

It merely used Composer as a way (initially) to offload the inclusion of dependency management and reduce the committed codebase. Over time it has slowly added more and more integrations to support the use of Composer, but it by no means actually requires it to function during runtime.

Maybe one day it will, but we're definitely not there yet.

I really recommend reading #2002304: [META] Improve Drupal's use of Composer, part 1 and all it's child issues.

The solution to which is to introduce a vendor stream wrapper.

No. This is just one solution and a hastily preemptive one at that.

Again, as I stated above, this "solution" is a direct symptom/side-effect of the lack of proper FE asset/library management in core.

This is an issue that has been known for years and far predates even the first inklings of "Composer" into the picture.

Simply slapping a vendor stream wrapper and telling people to manage FE assets via Composer won't solve anything.

In fact, it will only make things worse.

Why? Because core has no official API (aside from libraries... meh) of properly dealing with external 3rd party assets.

That needs to be figured out first.

Well, let's just leave it broken then

We're not doing that. :-)

In terms of Composer: https://www.drupal.org/about/strategic-initiatives/composer

In terms of a build tool: #2940733: Site Builder Tool/Project Browser initiative

It might be that having a vendor:// wrapper is a good idea, but it should be an implementation detail for a larger strategy.

Let's set this postponed on #2873160: Implement core management of 3rd-party FE libraries because we might well need it and it will be easier to find if it's not closed.

This is ridiculous that people are still trying to pretend that Composer isn't a requirement of D8, and that providing a way to effectively with Composer is seen as something that shouldn't be done.

That's because it isn't (at least not yet). This is just a fact, not my "opinion".

Well, let's just leave it broken then, so that people can bikeshed about useless things in favor of making a system that works.

That isn't a solution.

I'm simply being cautious and examining the motives behind this issue.

Currently, in its present form, the sole purpose of this feature is to bundling 3rd party FE assets via Composer.

As this topic is highly complex, it must first be dealt with before implementing something that can potentially cause even greater unforeseen issues.

Let’s not. The maintainer wanted more info. All the info has been given, and it’s been decided to leave it broken.

The primary reason I marked this as "needs more info" is that it is only focusing on a single use case: 3rd party FE assets.

There hasn't been much research into the security implications this may bring either.

Furthermore, there has really only been a smattering of people really discussing this issue; the majority of which is @Jaypan (the OP).

We're going to need more information from many more people and real-world use cases.

This can easily be accomplished using the existing contrib module.

No use wasting any more thought with this thread. Developers will just have to find hacks for the broken system. We’ve hacked around it until now. We’ll just have to keep doing it.

I get that you're frustrated, so am I.

Not at you or this issue, but with the pure nature of complexity that 3rd party assets bring to the system and our lack of any proper management of them.

I'm actually on your side.

I believe though that this topic (3rd party assets) needs further discussion first (at a higher level) before this specific issue can be implemented correctly.

Otherwise, we'll just be opening the flood gates for something more catastrophic and fragment our community even further.

Maybe because you're not even reading what I'm saying?

@Japyan:

Adding a vendor:// stream wrapper could easily be accomplished in contrib, and could then become a dependency of any Drupal extension.

So if you made a module that used Composer to download some CSS or whatever, then you could also make it depend on this stream wrapper implementation.

Here's an example of how a stream wrapper can be implemented in a module (complete with tests): https://git.drupalcode.org/project/examples/tree/8.x-1.x/stream_wrapper_...

Here's a patch that implements a service that looks for the vendor directory, which could be adapted to the vendor wrapper contrib module: #2830880-58: Warn site admins when composer dev dependencies are installed inside of docroot

If there is a time constraint, you could just build the thing you want and then use it.

@Mile23 https://www.drupal.org/project/vendor_stream_wrapper

Hello together!

I just stumbled over this issue while searching for a solution of a really specific problem. I haven't read the whole thread but probably the most important parts. As you asked for more information, I cannot give you information about security implications but use case related.

Background information

I'm developing a library which contains and handles all logos of an international NPO, which in total are approx. 570 logo variants in at least three styles (full color, white, black) and maybe different sizes. Branches of the organization (which can be countries or "sections" on local level) can be identified by a code (e. g. AT stands for Austria, AT-WIEN-UNI for the University of Vienna). The library should contain all possible versions of all logos. This at first looks like an asset library which should be managed with NPM or Yarn. But it should be possible for the backend to automatically select the correct logo based without knowing where it actually is saved. The backend should just provide the code, style and size of the logo and receive a path (might be needed to embed that logo in a generated PDF) and relative url, the code can be data-driven.

Example:

// @var LogoServiceInterface $logoService Logo service implementation retrieved from the service container
// @var LogoInterface $logo The logo meta information
$logo = $logoService->getLogo('AT-WIEN-UNI', Style::FullColor(), Size::Large());
$path = $logo->getFilePath();
$url = \Drupal\Core\Url::fromUri($logo->getRelativePath(), ['absolute' => true]);

To sum this up, the library is a mixture of image assets and PHP logic. I cannot really separate that because the logic will need to search for the logo file in the file system and fall back to a default if a specific logo doesn't exist. Thus I need to use Composer and not NPM. #2873160: Implement core management of 3rd-party FE libraries says

It's hard to get contributed module dependencies to be installed to [drupal-docroot]/libraries/[library/here] instead of the vendor directory. Especially if the library doesn't care about Drupal and doesn't have a custom type of drupal-library

I must admit that I do care about Drupal, but still I cannot give it a custom type of drupal-library because this library must be able to used by Drupal, Wordpress, Symfony, Dokuwiki and maybe others.

Solution

For me there are possible solutions for the use case above, one of them is using a vendor:// stream wrapper, which of course needs specific implementations for the other CMS mentioned. Another possible solution could be PHP Composer Asset Manager or a similar solution which as part of installing or updating the library with Composer copies the libraries' assets to a proper location. But as in this case we are talking about approx. 5000 image files that might not be the best idea, so at the moment I'm in favor of the vendor:// stream wrapper solution.

I'm open to your input and suggestions on my specific use case, but my motivation to write this here is showing you that there might be use cases which are PHP libraries that also contain some FE assets that need proper handling. Of course they will be much simpler than this one and the relation between the amount of PHP code and assets will be way different.

Any news about this topic?

Brought this up in #contribute

@mondrake brought up there is a contrib for this https://www.drupal.org/project/vendor_stream_wrapper
@catch mentioned - covered by https://www.drupal.org/docs/develop/using-composer/manage-dependencies#i... (see libraries mentioned in there).

If still a valid feature request please reopen updating issue summary for D10

Thanks!

With composer installers you can install css and js libraries inside the webroot, and PHP libraries outside the webroot, per https://www.drupal.org/docs/develop/using-composer/manage-dependencies#i...

@catch:
> With composer installers you can install css and js libraries inside the webroot, and PHP libraries outside the webroo
Whether #54 is a striking point for your #52, can be debated.

As the IS states, having third-party libraries outside of the webroot is a big security hardening (eg. examples-contained-in-libraries are a well-known attack vector).

@jaypan there examples on the page I linked, see https://www.drupal.org/docs/develop/using-composer/manage-dependencies#i...

Quoting from the link, it is right there on the page:

In addition to the package type-based installation locations you can use vendor specific ones, like this:

"extra": {
"installer-paths": {
"web/libraries/ckeditor/plugins/{$name}": ["vendor:ckeditor-plugin"]
}
}
Or package specific ones, like this:

"extra": {
"installer-paths": {
"web/libraries/{$name}": [ "enyo/dropzone" ]
}
}
Note: If a particular package matches multiple installer-paths entries, the first one that matches will be used.

No you could have read the documentation provided before typing out two full paragraphs that were incorrect. Given that's been settled, moving this to more info.

Wanted to give this one more bump or if it should be closed?

Saving some credit for the work/discussion thus far.

Personally I think the OP states a need that is still valid. In contribspace the Vendor Stream Wrapper module exists and covers the need; here I think the point is whether we want to have a core-based implementation instead.