Store composer.json if present

Having the description from composer.json, or fallback to .info file, is also something we want saved and readily available.

Moving to composer.json for Drupal contributed projects has an appeal since it follows PHP general trends instead of having a custom solution for Drupal. I do not think it would be difficult to use composer.json, if present, and fall back to the .info.yml files otherwise. But I have already run into problems with a short sighted implementation #2660992: Due incorrect dependency information, the module can not installed via composer.

Projects versus modules

Composer looks for a vendor/project:version. Drupal dependencies are based on the module name and not the project. If we switch to specifying dependencies on projects instead of modules then this will greatly expand the number of dependencies. If you just use one module from a project then you do not need a lot of the stuff that will be pulled in by other modules in that project. It might be useful to consider an example. The private_taxonomy project (for Drupal 7) has the private_taxonomy and private_solr modules. The private_solr module is an interface to the apachesolr project. If you do not use Apache Solr then private_solr is not required and apachesolr is not pulled in. It might be possible to switch to projects but it would require a lot of changes in contributed Drupal projects. The problem is a bit easier for distributions and that may be a starting point (or a dead end).

Module names are not unique

I keep repeating this but it does not seem to be understood. Module names are not unique. Project Dependency prefers project names with the same name as the module. For example, Project Dependency will choose the webform module from the webform project instead of the wbform module from the webform_patched project. This seems reasonable. But Drupal 8 core has a module with a dependency on the link module. There is a link project. Now we have Drupal 8 core dependent on the Link project instead of using the link module in the Drupal 8 core. Now this does not have any practical effect but there are numerous cases where it does. With #2205271: Project namespace for dependencies there is now the possibility to specify project:module but this is still optional and not widely used. It might be possible to specify the dependency in Composer in some form such as vendor/project(module):version. I just do not know. But if we are going to continue to base dependencies on modules then this needs to be figured out before we can use composer.json. I think we also will need to make the project namespace required since running Composer with such an ambiguity seems risky.

The questions above need to be resolved and I need a sample composer.json before I can really proceed. I am worried that the questions will not be resolved and I will need to deal with a mess that just cannot be implemented.

We'll have a hybrid for awhile, maybe indefinitely.

Composer isn't set up to handle components within the main project and doesn't have a concept of disabled modules, everything required is loaded all the time. I expect .info.yml will be needed for Drupal-specific things for some time.

What we can do in #2576285: Drupal.org Composer Service (façade) is use data from both .info files and composer.json to expose dependencies to users. So .info.yml dependencies will be translated and exposed to the Composer world.

@cweagans and @drumm: That is very helpful. So I just need to consider the vendor/project and not worry about all the extra projects pulled in. And then the second question is not relevant as far as this issue is concerned. @cweagans: The example you cite should be very helpful.

I think my coworker Mixologic has started in on this, at least some planning. I pinged him about updating here.

We were thinking of recording the description to the DB for possible use. If a project has multiple components, which don't match the project's machine name, I don't know a good way to choose which .info file to grab the project's primary description from.

@cweagans: I think I will have to write more code if you just want the immediate dependencies.

@drumm: Good point. I would guess a module that matches the project name and then choose the shortest module name. Often supplemental modules will tack something onto the main module name and so they tend to be longer.

The first step that we're aiming for is to merely collect the data provided in a composer.json if it exists in the repository. We do not plan on basing any logic on that data (it wont affect project_dependency_get_external_release_dependencies() or project_dependency_get_external_component_dependencies(), nor will it be used by drupalCI to determine which dependencies to download). We're wanting to use project_dependency for this since it is already doing all of the heavy lifting of cloning the release and getting the data from the .info and .info.yml files, so it seems like the most logical place to have it *also* gather the data in the composer.json files if they exist.

We would like to store all of this data, regardless of whether it duplicates data, so that we have flexibility going forward when implementing the façade #2576285: Drupal.org Composer Service (façade)

As far as *how/where* to store that data, Im a little torn. On one hand, adding a composer_json column to project_dependency_component that held the serialized contents would be short and simple, and would allow us to pull out and unserialize that data for use on the façade, on the other hand, we wouldnt be able to do anything fancy with it on the rest of drupal.org (like showing a projects dependencies on its module page). Although, if we start with a single column, we could always do some database magic to turn it into something more valuable as new requirements emerge.

As a separate aside, it does not appear that we store the drupal "Package" data field from the .info or .info.yml files - I can imagine that we'd probably find value in having that data in the database for module discoverability purposes (i.e limit searches to just commerce modules for example).

Here is what I have for adding data to the table:

MariaDB [drupal]> SELECT * FROM project_dependency_component WHERE release_nid = 2614182\G
*************************** 1. row ***************************
component_id: 1618971
 release_nid: 2614182
        name: samlauth
       title: SAML Authentication
 description: Allows users to authenticate against an external SAML identity provider.
     package: User authentication
    composer: a:1:{s:7:"require";a:1:{s:17:"onelogin/php-saml";s:4:"~2.5";}}
1 row in set (0.00 sec)

I am storing data for the keys require, require-dev, conflict, replace, provide and suggest. The length of the composer field is 255. Is that too small?

For output I think I can check the vendor to see if it is drupal and then map require to dependencies and require-dev to test dependencies. Any thoughts?

I have the code hacked up so it works locally but when I reach a point where I can clean it up I will post a patch.

Sweet, thanks for working on that!

255 is definitely too small, and ideally, we want the entire file, not just the data keys for the deps. We can probably keep them as serialized json -> converting to php and back again could have unintended translation issues.

For output I think I can check the vendor to see if it is drupal and then map require to dependencies and require-dev to test dependencies. Any thoughts?

We should not need to intermingle composer dependencies in with the dependencies specified by the info files, and in fact it would be desirable to avoid that. We're planning on using project dependency data to feed into the composer facade, and the facade holds the responsibility of producing json that composer itself can utilize for managing dependencies.

That will let the .info files continue to specify the within-drupal dependencies and the composer facade to translate those into composer dependencies and merge with the external dependencies specified in the project composer.json (if any).

Once the facade is in place, we'll be able to use it to provide the metadata that composer needs to perform all of its own dependency calculation. Since the dependencies are no longer just drupal modules, but php packages stored across the web, we'll have to use composer to do all the recursion and metadata gathering from us as well as packagist.org to be able to package distributions and also feed drupalci all the dependencies it needs to check out and test modules.

As far as output goes, we'll probably want to eventually get to where we have all 6 categories on release nodes/project nodes

The composer dependencies are at the project level while the info dependencies are at the module level. So they do not intermingle. My responsibility is to avoid prescribing how the information is used but make it available for others to use in ways they think is best. Perhaps there will be several ways that will be implemented and then if one way is best in all cases the other methods can be deprecated or, if an entire module, will fall out of use. That is the way open source works.

There are three possible obvious reasons for using composer for Drupal dependencies:

1) Using Composer for external dependencies and info files for internal dependencies requires two approaches and some people will rather have one common approach.

2) Currently distributions are packaged together as one tar ball. There may be advantages to not packaging all those Drupal packages within the distribution but have them downloaded separately.

3) There may be some improvements when used with Drupal make or in place of Drupal make.

In addition Drush might be able to leverage this although they probably can do this directly.

The advantage of the info files is that the dependencies are finer grained since they are on the module level and can result in fewer dependencies.

Attached is a patch that adds the package and composer.json to the data that is saved when something is released. An example of the result is:

*************************** 1. row ***************************
component_id: 1611627
 release_nid: 2614182
        name: samlauth
       title: SAML Authentication
 description: Allows users to authenticate against an external SAML identity provider.
     package: User authentication
    composer: s:461:"{
  "name": "drupal/samlauth",
  "type": "drupal-module",
  "description": "Allows users to authenticate against an external SAML identity provider.",
  "keywords": ["Drupal"],
  "license": "GPL-2.0+",
  "homepage": "http://drupal.org/project/samlauth",
  "minimum-stability": "dev",
  "support": {
    "issues": "http://drupal.org/project/samlauth",
    "source": "http://cgit.drupalcode.org/samlauth"
  },
  "require": {
    "onelogin/php-saml": "~2.5"
  }
}
";

Is there a need for a Project Dependency function that returns the contents of the composer column? Are there any other things that I can add that might help people use this information?

Trobey++

That looks great. I was going to mention that composer.json can indeed exist at the module level and not just the project level (example: http://cgit.drupalcode.org/monolog/tree/) But it looks like the patch would handle that scenario regardless, so perfecto.

Is there a need for a Project Dependency function that returns the contents of the composer column

That would be great. I may have another patch to submit later on as there are some db queries I've got in project_composer that rightfully belong wrapped in a function in project_dependency.

Yes, composer.json can exist at the module level but it cannot point to a module as a dependency. It can only point to a project.

+++ b/project_dependency.drupal.inc
@@ -436,6 +445,18 @@ function project_dependency_info_batch_process_release(array $release) {
+    if ($composer['type'] == 'drupal-module') {

Why do we need this condition? There are other package types which we should support e.g. drupal-project, drupal-theme, drupal-drush, library etc.

I think we should remove the condition. Its up to composer to deal the different package types.

We used to exclude themes. Looking through the code it looks like in a recent rewrite I removed that restriction. The intention here is to limit this to Drupal code. I do not know if we have any cases where that happens but I only have one example. I will remove it unless someone can point out types of composer.json files that should not be processed.

Composer installers supports six different types, look at https://github.com/composer/installers/blob/master/src/Composer/Installe... for the full list.
The base class in composer installers adds the drupal- prefix to all types. module becomes drupal-module, profile becomes drupal-profile etc.

Attached is an updated patch. This adds a first cut at accessing the new information from the composer.json files. The first function just returns the information verbatim.

$composer = project_dependency_get_composer_json(2614182);
dpm($composer);

Array
(
    [samlauth] => {
  "name": "drupal/samlauth",
  "type": "drupal-module",
  "description": "Allows users to authenticate against an external SAML identity provider.",
  "keywords": ["Drupal"],
  "license": "GPL-2.0+",
  "homepage": "http://drupal.org/project/samlauth",
  "minimum-stability": "dev",
  "support": {
    "issues": "http://drupal.org/project/samlauth",
    "source": "http://cgit.drupalcode.org/samlauth"
  },
  "require": {
    "onelogin/php-saml": "~2.5"
  }
}

For a release with no composer.json files this results in

$composer = project_dependency_get_composer_json(2607858);
dpm($composer);

Array
(
    [private_taxonomy] => 
)

It might be useful here to provide a basic translation of the information in the info files as a fallback.

The second function retrieves the dependencies. For a project with composer.json files.

$dependencies = project_dependency_get_composer_dependencies(2614182);
dpm($dependencies);

	•	Array
	•	(
	•	    [samlauth] => Array
	•	        (
	•	            [require] => Array
	•	                (
	•	                    [onelogin/php-saml] => ~2.5
	•	                )
	•	
	•	        )
	•	
	•	)

For a project that does not have any composer.json files.

$dependencies = project_dependency_get_composer_dependencies(2607858);
dpm($dependencies);

•	Array
	•	(
	•	    [private_taxonomy] => Array
	•	        (
	•	            [require] => Array
	•	                (
	•	                    [drupal/taxonomy] => *
	•	                )
	•	
	•	        )
	•	
	•	)

Requesting test dependencies.

$optional = TRUE;
$dependencies = project_dependency_get_composer_dependencies(2607858, $optional);
dpm($dependencies);

Array
(
    [private_taxonomy] => Array
        (
            [require] => Array
                (
                    [drupal/taxonomy] => *
                )

            [require-dev] => Array
                (
                    [drupal/pathauto] => *
                )

        )

)

There also is a parameter to request the Drupal dependencies recursively instead of just the first level requirements but I have not implemented that yet.

Attached is an updated patch.

I *finally* got to where I could give this a review. Sorry it took so long, was buried in production issues, vacation, and building out the rest of the facade.

So a few things: what we need for the composer façade is the pure composer json - we dont need any parsing or logic to take place. If a component has an accompanying composer.json file, we'd like to store that in the db wholesale, as json.

Attached is a simpler patch that reflects that need. The 2 added functions in project_dependency.module in patch in #26 were also somewhat out of scope - we don't actually want to use information in composer.json to calculate the dependencies that project_dependency provides. Drupal modules should still express their dependencies in their .info and .info.yml files, so there shouldnt be any additional information in there. But if there *is* a discrepancy, project_composer will be able to detect those discrepancies and display that information on the project page.

I appreciate simpler but the original issue was for Drupal distros using composer to assemble their dependencies. While the latest patch address internal Drupal needs, I do not think it allows others to go with a composer build process. Are we addressing the use case in the original issue?

I believe that it will satisfy the use case.

In order for distributions to be properly packaged with their composer dependencies, we will have to use composer as part of the packaging step (like we do with drupal core now). Composer lacks the ability to add patches (unless using cweagans plugin here: https://github.com/cweagans/composer-patches), so we cant really use *only* composer to build distributions yet, and will have to use a combination of drush make and composer.

We wont officially support using composer.json in a distribution to specify drupal module dependencies until we have the Façade in place, and having the composer.json in the db is a blocker for that.

Establishing patterns for distributions built with Composer does indeed need to wait for Drupal.org’s basic Composer support. I’ve adjusted the issue summary.

I think it is far too early to think much about distribution support. Distributions are firmly entrenched in drush make. It will awhile to get clear best practices established for using Composer, official Drupal.org support will go a long way. Once there’s enough momentum around a consistent way of using Composer to build sites, then we can look at how that could be used for building distributions.

+++ b/project_dependency.drupal.inc
@@ -76,6 +76,12 @@ function project_dependency_info_parse(array $info_files, $tag) {
+    if (file_exists($composer_json)) {

I think we should validate the content of the file. Otherwise we to deal with invalid json later down the pipe.

We're hoping to address that in drupalci, but that would only catch modules with testing turned on: #2708751: Run composer validate --strict if a composer.json exists .

What should we do if the file isn't valid? Skip saving in the db?

Occasionally, the package information is not set, and thus, should be confirmed exists before inserting.

We've deployed #34 on Drupal.org and it is working well.

Adding support for Composer is a significant change and could eventually change the API for Project Dependency. So I started a new branch for these changes. I do not think there will be significant usage of the old branch but it will provide a way to track the changes for Composer.