Add a “General” project type to Drupal.org

@klausi considering I'd have called it "library" and in the Issue Summary you describe it as such a number of times, why not use that name? Or "Shared Library", or 'Vendor Library'?

Good idea! If it walks like a Composer package and it quacks like a Composer package, why not call it a Composer package? That would also allow us to display or link to Composer instructions on such project pages.

I am not sure if I understand the visibility reason completely. I agree that it is easier to find Drupal related code on drupal.org but such code would typically be modules. Code which is not strongly coupled (like a library) would find better home on packagist. Are we talking about hosting the repository for such projects and listing the entry on packagist?

Basically, I am asking what would the workflow look like and what would be the output. If there is such a project, would there be associated tar/zip files? Would there be something special required to use them with PHP applications (not necessarily Drupal)? Is there an advantage to tie it with composer_manager module, for example?

See: #2401519: [policy] Decide on Composer Package Names

Basically you want a repo with issues and a way to test patches using the testbot, but don't necessarily need e.g. drush being able to confuse the library for a module.

Sounds like a sandbox, really. :-)

+1.

It would be super-awesome if this could also be useful for D8's Component namespace, where those components could just be dependencies rather than in the D8 codebase.

#1826054: [Meta] Expose Drupal Components outside of Drupal

I feel like we need input from relevant Working Groups to make a decision here. Moving to Technical Working Group queue, as they are about all things technical including Git repositories, projects, etc. One of their specific duties is "Provide recommendations to the Drupal Association, Drupal.org working groups, and infrastructure teams regarding proposed tool changes that will help accelerate contribution or otherwise enable the community."

The Technical Working Group discussed this issue and agree that having a new type of project differentiated from Module, Theme, or Profile would be a welcome and useful addition to help to differentiate these new shared PHP components and foster continued collaboration on Drupal.org. This need will increase as we move into Drupal 8 but we, the committee, have already seen compelling use cases for this new feature including the API client library for the upcoming DrupalCI projects.

@tvn do you have a feel for what group it should move to next?

Thanks, Howard. I would say the Drupal.org Software Working Group, which I am a member of. I'll make sure to discuss with the group asap.

Adding related issue.

Setting back to active for patch writing.

This will need to add the content type to the Drupal.org Projects Feature. More configuration, such as in the Drupal.org SearchAPI Issue Views Feature, may be necessary.

What about using "component library" or "library project" or just "library"?

As that is usually what composer has, also we have already registered "drupal-library" as type on composer/installers, so we should probably stick with something like that?

Given #1 I re-titled to "library" for now - as its short sweet and as effective as module, theme and distribution.

Tagging "Needs patch" for now.

--

Even though this has been answered, some more use cases are:

- e.g. while modules are not auto-loaded until enabled, libraries typically are, so for example memcache could use a sub-tree split and add a memcache_lib project for Drupal 8, then the cache class would be automatically available, etc.
- Pure libraries that are very Drupal specific, but especially help several modules out, it does not need state of its own. (and such not being installed)
- etc.

+1 on this.

Previous Drupal versions have kinda had libraries, as in code in modules used by other modules but useless on their own. For example, Ctools. These have historically been crafted like modules just so Drupal sees and loads them. With D8 this doesn't need to be the case anymore.

ps. Don't care what the content type is called.

I'd love to be able to move memcache to this.

Wait, really? Isn't memcache a straight-up module, vs. a shell script?

The memcache backends are just a collection of classes. If it was a core component, it would end up in a subtree split as part of the cache system once we do that, which is also not a module.

There are two modules in the memcache project - they could stay there and depend on the library, but you don't need either of them for the cache backend.

Gotcha; makes sense.

Some questions about how we'd handle libraries on d.o.

If a site is using a library that we host, would we need to change the current updates system to allow for us to alert users of a vulnerability in that library? Right now the updates system in core assumes things to be either modules or themes (https://api.drupal.org/api/drupal/core%21modules%21update%21src%21Update...) , and have a status of enabled or uninstalled. Would we somehow also keep libraries in the system table? where would we display library security updates? do we have 'installed' libraries? Do we not offer security updates of libraries at all, and let it exist like the composer dependencies do?

Sorry, just thinking out loud here and this seemed to be a place to do it.

We're considering using a composer-plugin type Composer package for #2494073: Prevent modules which have unmet Composer dependencies from being installed. Ideally we would host that ourselves, but pull it in through packagist.org, so we can add it as a dependency to drupal/core so it can be used without developers having to alter their application's own composer.json file.

TLDR; We'd love to see this happen soon, with the possibility to submit these repositories to packagist.org.

Im actually going to mark this postponed until we hear from the security team on how we want to proceed with hosting library types on drupal.org.

What I *think* we should do, is host library types on drupal.org, with the caveat that 'buyer beware' until we have a process in place like the pending PSR9 and PSR10.

I, personally, don't think we should wait until we have a security notification process in place to proceed with this, but I do want security team buy in, and some idea of what the process should eventually look like before moving forward.

@Mixologic: have you notified the security team you'd like a review? Note that the original issue reporter is a security team member.

Why do we need a security review? Whether we host the libraries here or on a platform like Github, there is no generic security notification process that's not specific to Drupal anywhere, as far as I know.

Currently, everything that we host on drupal.org is covered by the security team's vulnerability disclosure process, and when modules and themes receive a security release, end users of those vulnerable extensions are notified via the updates process. Right now, an end user has essentially comprehensive coverage for their drupal sites, an implicit feature that security is taken very seriously in the Drupal community.

Ctools had been suggested earlier in the thread as an example of a library type, and ctools has definitely had vulnerabilities in the past (e.g. https://www.drupal.org/node/2554145), so there is definitely a need to understand how this would work.

What I'm looking to establish is a plan on how we would intend to communicate to end users if they have a vulnerability in a library that they add to their drupal site. Maybe all we need to do is have a special warning on library pages that says "in order to ensure you are notified of security vulnerabilities in libraries, please subscribe to the security notifications mailing list". Perhaps we add some security metadata to the extra section of composer.json's for libraries, and have a "library security module" that parses that info, and notifies/warns end users. In any case, this is a policy/planning question.

What I would like to avoid is having a popular library published on drupal.org, having a critical vulnerability discovered in that library, and trying to figure out how to handle that in the moment instead of having a plan in place ahead of time.

My plan was to ask drumm (who is also on the security team) how to best get the right eyes on this after our Holiday here in the States (tomorrow).

Thank you for taking the time to write this down. I understand now that this has more to do with our Update State reports in Drupal itself (since a library would not be a Drupal extension) as a means of communication than with security problems themselves. I agree that getting @drumm's and/or @mlhess' input on this matter is important.
I do want to note that most Drupal 8 sites will make use of plenty of generic PHP packages (libraries) that are not hosted on drupal.org and about whose security issues the Drupal security team does not (cannot) communicate to site owners. This is primarily caused by many (major) modules requiring Composer-based dependencies.

There is a security standard being worked on for external libraries, both a reporting mechanism (PSR9) as well as a discovery mechanism (PSR10), and @mlhess has been participating in shaping that standard (https://github.com/php-fig/fig-standards/blob/master/proposed/security-d...). So while there does not exist a mechanism *now* for disclosing a vulnerability short of checking the php security advisories database, http://fabien.potencier.org/the-php-security-advisories-database.html, work is being done in this area, so it'd be good to ensure we've got alignment in how we should handle that aspect in the future.

@Mixologic: any word on the security team chipping in on this?

Yes. I had a chat with @drumm, and have a meeting next week with @mlhess. General direction so far is we'd proceed with implementing this and security wouldn't be a blocker.

So, what we should probably start to think about is all of the things we need to do to make this happen, and how everybody expects something like this to behave.

I'm going to throw out a disorganized thoughtstream off the top of my head notes on what we may need to figure out.

In a re-read of the thread I see that we'd want to use this for a number of things:

• Standalone Libraries
• Subtree splits of existing projects
• Potentially "other" non-library things like drush extensions.

Need to make sure that subtree splits do not get double counted in commit counts on user pages

Do libraries have a category like modules?

We have to consider all the places that a module or theme is displayed : search results, the Download and Extend page, 'https://www.drupal.org/download'

is there such a thing as a library sandbox?

How do libraries fit into the new project application process? is there a warning on library pages that there isnt security disclosure coverage? Or is there an application process to get coverage?

Libraries will not, and should not, fit the 7-x/8-x pattern of compatibility, so we'll have to ensure that everything that filters or checks platform in the code doesnt affect libraries.

Determine where the composer package metadata lives:

• On packagist means nobody has to fuss with adding our composer repository to their project, and drupal libraries just work in other projects, but we have to add integration with packagist.org (not too big a deal)
• On packages.drupal.org means we can have usage stats, and additional custom metadata that *does not require a commit* (i.e. security info in extra section perhaps)
• Maybe both?

Do we package libraries? Is there even a need for a tarball/zip? edit: of course we do. --prefer-dist is a thing.

Libraries should have support for semantic versioning - drupal.org supports it for 8.x core, but gotta make sure that is extended to everywhere we'd work with a library version.

What about development releases in a world of semver - do we list those on a project page? Example: a library is currently version 3.1.4. 3.1.x-dev is where patch level changes would go - bugfixes and security patches. 3.2.x-dev would be for adding new features without breaking backwards compatibility, and maybe 3.2.x-dev creates a 3.2.0-alpha1 release. At that point a maintainer may need 3 dev branches, 3.1.x-dev for security patches during a modules' beta phase, 3.2.x-dev for fixes to the alpha version, and 3.3.x-dev for new features that arent going to make it into 3.2.0. Finally there would be a 4.0.x-dev for building out api breaking features.

When 4.0.0 comes out, do we keep 3.0.0 on the project page? does it follow the same "recommended releases" kind of pattern as modules?

Testing: out of the gates, drupalci doesnt support anything but run-tests.sh - we're working on that separately, so I think we can ignore testing for now, until drupalCI can run phpunit separately.

What about namespaces? is the url https://www.drupal.org/project/libraryname? I assume we dont want to allow conflicts between a library name and a module name, so memcache, for example would need to be memcache-lib or something.

Are there any other additional fields required by a library that we don't need for modules/themes etc? like library type (drush extension, subtree split/core-component, php lib etc)

There's probably more to consider.

Thanks!

Need to make sure that subtree splits do not get double counted in commit counts on user pages

Can we count by unique commit hashes?

Testing: out of the gates, drupalci doesnt support anything but run-tests.sh - we're working on that separately, so I think we can ignore testing for now, until drupalCI can run phpunit separately.

Can Drupal CI even run composer install? Many projects are tested elsewhere already. This is not a new issue.

What about namespaces? is the url https://www.drupal.org/project/libraryname? I assume we dont want to allow conflicts between a library name and a module name, so memcache, for example would need to be memcache-lib or something.

We'll need to find another URL path structure. What about https://drupal.org/library/ohai? We must also take note that with libraries we're getting into a realm in which URLs and package names may not match up. Composer uses vendor and package names, but that's just Composer. What if we want to publish other types of packages with different naming schemes? I don't think we should enforce the URL to match any of that. It should be up to the individual maintainers, just like versioning.

Can we count by unique commit hashes?

The commit hashes are different in a subtree split since it is a different codebase as far as Git is concerned. We’ll simply not consider subtree split projects when counting up commit credit.

Can Drupal CI even run composer install? Many projects are tested elsewhere already. This is not a new issue

Yes, it can, and it does. The build isnt the issue, the job type is the issue, because libs are going to want to run phpunit tests and we do not have a phpunit runner on drupalci, that can then communicate its results back to drupal.org. Yet.

I think the only part of the URL that we are going to enforce, is that the end of the url is going to be the project machine name. And that project machine name *will* have to be unique amongst projects. So, even if we do not enforce any package naming standards as far as composer is concerned (ie. maybe vendor isnt 'drupal'), we *do* have to enforce project name uniqueness amongst projects.

What if we want to publish other types of packages

Are there other types beyond composer packages, drush extensions, and subtree splits of existing projects?

I suspect you actually want feedback from the Security Working Group (https://www.drupal.org/governance/security-working-group) here, more than the Security Team itself...

However I'm not sure there's much to discuss? To the best of my understanding, Security Team policy is already that we support projects on drupal.org based on certain criteria (https://www.drupal.org/security-advisory-policy), and there's nothing in those criteria that says the project has to be a module/theme/etc. So for example, something like https://www.drupal.org/project/registry_rebuild is already on drupal.org, and is not a module or theme - as far as I know this is currently supported by the Security Team, and if a security issue were found in it there would be a Security Advisory issued as a result.

Obviously it would be very nice if the Drupal Update Manager could warn people about security issues in those kinds of projects too, but I don't see how it would be a blocker for this issue, since those projects already exist... This issue really just seems more about providing a better structure on drupal.org for those kinds of projects, so they're easier to find and distinguish.

(The above is not an official opinion, but it is heavily colored by my experience on the Security Team and I'm pretty sure it's accurate.)

Yes, it can, and it does. The build isnt the issue, the job type is the issue, because libs are going to want to run phpunit tests and we do not have a phpunit runner on drupalci, that can then communicate its results back to drupal.org. Yet.

Assumptions like this one are exactly why projects have moved to Github and Travis CI in the first place. We don't know what package maintainers need.

Are there other types beyond composer packages, drush extensions, and subtree splits of existing projects?

Possibly, just like we did not know about Composer packages a few years ago.

Is the goal to limit these library node types to PHP vs. all the other libraries Drupal projects have had to violate or work around Drupal.org's policies to include with their code for years?

@kreynen - Im not sure I understand what you're asking. This is not intended to be used so that code, that exists somewhere else, can be hosted on drupal.org so that its 'easier' for end users install process or anything like that. This is for code, in any language, that does not currently fit the 'module', 'theme', 'distribution', or the seldom used 'templating engine'. It is also for subtree splits of core, and new code not hosted elsewhere.

This is for code, in any language, that does not currently fit the 'module', 'theme', 'distribution'...

@mixologic Do you intend to limit these "libraries" original code? That's what I was getting based on the discussion, but creating a content type called "library" is going to be very confusing to a large part of the community. We've had issues since Drupal started dealing with all types of libraries projects want to include. If you create something called a library project that could be used to solve these long standing issues, it will be used that way.

Why whitelist a distribution dependencies when you can can just create a library project, commit it, and actually enforce a .info dependency in D7?

If that's not what you want, I don't think "library" can be used unless you also add some criteria for exactly what can be committed to these library projects and some method of enforcing it... which we don't have a great track record for being able to do.

Without that, we are going to end up with copies of every 3rd party library that can be re-licensed as GPLv2 or uses a GPL friendly license on Drupal.org. Life would be so much easier for both maintainers and users if something just created library projects for all of https://www.drupal.org/packaging-whitelist

@klausi's original suggestion to name the content type "Components" is going to create much less confusion than calling them "Libraries".

If I understand correctly, your concern is that if we name them "Libraries" then people will abuse that by uploading duplicates of externally hosted code onto drupal.org?

I don't think we should delay implementing a feature like this because of the risk that people will abuse it. There should be guidelines in place to express that this is for hosting original code, and not "things your distro might need".

The consequences of people abusing this and violating our policies are relatively minor even if they *do* use it wrong, and our track record of cleaning up those abuses in the past is a reflection of how dire those consequences are.

Additionally, these types of libraries will be *much* easier to police than some random subdirectory of a module that contains imported code.

This is needed for #3187061: Drupal.org/GitLab Tooling Requirements for Decoupled Menus Inititaive

Small suggestion:

The naming discussion all happened before Drupal.org built its GitLab CI integration. Since GitLab CI lets a project write it's own tasks, maybe we should just call the new type "General project" so that anyone can make of it what they want.

Then, "Module project" and "Community projects" become projects that have special quality-of-life improvements. E.g. a "module project" has pre-written CI tasks to make testing in a Drupal environment easy.

Obviously, I'm late to the party. I don't think the name is critical, but if you like the idea and it's not too late to change the name... it might be worth it.

Updating the issue title, since “Library” is not currently under consideration.

Now is indeed the best time to get this right, as soon as this is deployed, it is effectively permanent.

Drafting the content type description may be of some help figuring out which label seems more-correct. My current draft is:

Components that are not Drupal modules, themes, or other more-specific project types. For example, JavaScript components, Drush extensions, and libraries.

(Projects can be moved from one type to another, mis-categorization happens. Currently “Community projects” gets an occasional module. Any name will catch the occasional mis-typed project, which is okay.)

From my point of view:

The advantage of something like 'General projects' (or 'other projects', etc) is that it is truly generic to whatever kind of project lives in that space.

The disadvantage of something like 'General projects'... is that it's generic - and we may wind up with projects uploaded that actually belong in the 'module', 'theme', or 'distro' spaces that have their specific tooling. That's okay though, we can move projects if needed.

I personally felt that 'Component' was generic enough to represent the variety of other kinds of things we wanted to support (drush extensions, db drivers, install profiles, js components, whatever else), but I'm happy to bow to 'General projects' if we think that may save us some headaches with some folks feeling they still don't have a place to put their work.

So my two cents... let's go ahead with 'general projects' so we can get this deployed.

What the heck, let's give an RTBC too 😊

This is now deployed: https://www.drupal.org/project/project_general

🎉