In parallel with #2547385: [PR effort] Contrib maintainers should add composer.json I think we need to start thinking about how we're going to push Drupal 8 contrib modules to packagist. After a little research into this it seems we simply need a node add/update hook to push projects and project releases to packagist.

Packagist has an API but it only handles updates. Therefore we will need to submit a pull request to packagist to add a create endpoint. This has already had a small amount of discussion with the packagist maintainers, who said:

Anyway I guess we can just add API-token support to the package submit URL, or just add a new endpoint, shouldn't be too hard.

There are a few items I would like to note:
- Drupal modules via composer is already possible via https://packagist.drupal-composer.org/ but this has a number of issues:
-- Another Drupalism ie. yet another thing people new to drupal need to work with.
-- Not something that large scale drupal users may want to depend on
- Manually adding drupal modules to packagist under the drupal namespace is not possible unless you are currently a maintainer. Maintainers look to be people who added a module under that namespace before May 7th when packagist pushed a code change to lock down namespaces.

Comments

timmillwood created an issue. See original summary.

timmillwood’s picture

Status: Active » Needs review
FileSize
1.85 KB

Slow work has started on a pull request for packagist https://github.com/timmillwood/packagist/commit/06b9bb72fab482e82abe6819...

Attached is also my first pass at adding the drupal.org component.
Note: this depends on the packagist pull-request being completes to add the 'create-package' endpoint.

timmillwood’s picture

The pull request for Packagist is making pretty good progress, any feedback on that is welcome https://github.com/composer/packagist/pull/553

Also on the Drupal side I think we need to check the project is at least D8 and has a composer.json, can I do a file_exists on the git repo?

Crell’s picture

Do we want to do this on node insert? I'd say node update, since someone could easily add a composer.json file well after the module is published. (Like, all existing modules. :-) )

A file_exists() check seems fine to me if we can get at the repo. I don't know the infra well enough for that part.

timmillwood’s picture

I'm doing this on node update and insert, but I see what you mean, if a module adds a adds a composer.json after the release is created it will need to ping the create endpoint on node update.

Also I might need to think about the opposite. if a project release is created when the packagist project already exists it will need to do an update, not a create. I think we need to ping packagist for the package before doing the create / update then call the right endpoint accordingly.

webchick’s picture

Tagging to bring up on the DSWG meeting next week so this is on the DA's radar.

Crell’s picture

Tim: You may be correct. IIRC, Packagist has a "go fetch this repo and update your stuff" command, and you're adding a create command. Thus, I think the on-git-push operation on Drupal.org would be:

Ask Packagist if project X exists
If No {
Push project create
}
Send Packagist "things changed, do your updates" message

(Depending on how the create works, the latter part could be in an Else statement.) Also, I think we do need to make it on git push, not anything triggered by the node being edited. All Packagist is going to care about is the tags it finds, which it can get directly from the repo.

timmillwood’s picture

Does Drupal.org know about commits?

I was thinking package update would be a good compromise. Dev packages would get uodates via cron (iirc) and new proper releases (tags) would create new project release nodes.

Mixologic’s picture

Not sure if duplicate, but adding a related issue.

timmillwood’s picture

Title: Push drupal modules to packagist » Notify Packagist about repository update

Rewriting patch to use hook_versioncontrol_code_arrival

timmillwood’s picture

Latest patch uses hook_versioncontrol_code_arrival, pings packagist, if 404 it submits to 'create-package', else submits to 'update-package'.

Mixologic’s picture

Thanks Tim, this is great!

  1. +++ b/drupalorg_project/drupalorg_project.module
    @@ -916,6 +916,55 @@ function drupalorg_project_node_view($node, $view_mode = 'full') {
    +    $anon_project_url = "http://git.drupal.org/project/{$project_url_name}";
    

    Git server urls should use the git_base_url variable: see http://cgit.drupalcode.org/drupalorg/tree/drupalorg_project/plugins/rele...

  2. +++ b/drupalorg_project/drupalorg_project.module
    @@ -916,6 +916,55 @@ function drupalorg_project_node_view($node, $view_mode = 'full') {
    +  return drupal_http_request("https://packagist.org/packages/drupal/{$project_url_name}.json", [
    ...
    +    return drupal_http_request("https://packagist.org/api/{$endpoint}?username={$packagist_username}&apiToken={$packagist_api_token}", [
    

    These packagist urls should be a variable as well, that way we can set them for dev/staging environments when we're testing git functionality.

  3. +++ b/drupalorg_project/drupalorg_project.module
    @@ -916,6 +916,55 @@ function drupalorg_project_node_view($node, $view_mode = 'full') {
    +    drupalorg_project_packagist_submit('create-package', $repository); ¶
    ...
    +    drupalorg_project_packagist_submit('update-package', $repository); ¶
    

    remove end of line spaces

skwashd’s picture

@timmillwood I don't think this should be limited to D8, modules should be able to implement composer.json in 7 (or even 6) and be included on packagist under the Drupal namespace.

In terms of checking for composer.json, it should be possible to extract the branch from VersioncontrolEvent object. Then a drupal_http_request() HEAD request could check for a 200 response from http://cgit.drupalcode.org/[module]/tree/composer.json?h=[branch].

timmillwood’s picture

@skwashd - Thanks for the review, however I fear checking http://cgit.drupalcode.org is too early for hook_versioncontrol_code_arrival.

I believe in VersioncontrolEvent we can check if a repo has composer.json being added in *this* commit, but not if it has a composer.json in general.

What I am looking for do is...

if (file_exists($repo_path . '/composer.json')) {
  $packagist_project = drupalorg_project_packagist_get($repository);
  if ($packagist_project->code === 404) {
    drupalorg_project_packagist_submit('create-package', $repository);
  }
  else {
    drupalorg_project_packagist_submit('update-package', $repository);
  }
}

It seems like the only way to do this might be the clone the repo, but this would mean using exec() and will add quite an overhead.

timmillwood’s picture

Sorting items from #13.

skwashd’s picture

Status: Needs review » Needs work

@timmillwood should sandbox projects be filtered out?

And a nitpick

+++ b/drupalorg_project/drupalorg_project.module
@@ -916,6 +916,58 @@ function drupalorg_project_node_view($node, $view_mode = 'full') {
+    $project_url_name = check_plain(drupal_encode_path($repository->name . '.git'));

check_plain() is only for escaping markup in the UI and so I don't think it is needed here.

timmillwood’s picture

should sandbox projects be filtered out?

To throw this back at you, should sandbox projects be installable via composer?

webflo’s picture

Yes they should be installable via composer, but you can add the git repo in your composer.json directly. It not necessary to expose every sandbox to packagist. Besides that packagist uses the package name in the composer.json as identifier. First come, first serve that way i could push my "views" sandbox as the original package to packagist.

Maybe we should perform some basic validation the composer.json and package url should match before we register new packages in packagist.

skwashd’s picture

I'd say they shouldn't because under the current project approval process they can't have any releases. Under the revised process proposal (see #2453587), they can only have dev snapshots, not tagged releases. Given this will make packagist a mirror of the d.o releases, the same policy should apply.

timmillwood’s picture

ok, good, I'm kinda just playing devils advocate. I agree sandboxes shouldn't be on packagist, and think #19 is a good suggestion, that repos can just be explicitly added. Also maintainers can add sandboxes to packagist on their own, but just under a different namespace.

derhasi’s picture

Maybe we should perform some basic validation the composer.json and package url should match before we register new packages in packagist.

I think so to. We have a Composer package naming convention. So we should make sure, that the convention is applied appropriately beforehand.

timmillwood’s picture

@derhasi - think the issue is accessing the composer.json to check it even exists, let alone validating it.
The best suggestions I have so far is cloning the repo into /tmp after each commit and doing the validation, but doing the exec for this is frowned upon.

naveenvalecha’s picture

+++ b/drupalorg_project/drupalorg_project.module
@@ -916,6 +916,58 @@ function drupalorg_project_node_view($node, $view_mode = 'full') {
+  $project_url_name = check_plain(drupal_encode_path($repository->name));

Need to remove the used variable at the hook_uninstall

timmillwood’s picture

FileSize
5.33 KB

Added admin page for updating variables (and hook_uninstall to remove them).

timmillwood’s picture

FileSize
5.3 KB

Removing all the whitespace.

timmillwood’s picture

The create endpoint has now been deployed to packagist.

cweagans’s picture

What happens if Packagist is down when drupalorg_project_packagist_submit() is called? Do we have some way to reconcile the Drupal.org project data with what's in Packagist?

As mentioned in the other issue, I don't think we should push our projects to packagist at all. See https://www.drupal.org/node/2547385#comment-10219351 for reasoning. A very viable alternative is to serve the package JSON data directly from Drupal.org. Mixologic created a dev environment for me, and I should have it done by Sunday at the latest (depending on if I end up going to a family thing tomorrow).

timmillwood’s picture

FileSize
31.34 KB

It looks as though Packagist downtime is minimal.

It's defiantly the right place to allow Drupal modules to be downloaded via composer. It's known to every PHP developer from outside (and mostly inside) the drupal community. Any other option doesn't make sense!

cweagans’s picture

Well, that problem only has to happen once for it to be a problem, and with 99.97% uptime, that's 2 hours and 38 minutes per year where a request might not get through. What is the solution to that problem?

I'd appreciate a real response to the arguments against Packagist that I presented, rather than a blind assertion that we should move forward with pushing things to Packagist. The Composer docs themselves point to Wordpress plugins as an example of a good thing to have in a separate repository because they simply aren't useful to people outside of the Wordpress community, and the same thing applies to Drupal modules. Panels is useless to anyone not using Drupal, so why force it on the rest of the PHP community? Again, I'm not arguing against making Drupal extensions available via composer - I want that. I'm arguing against pushing them to packagist.org.

timmillwood’s picture

@cweagans

I feel the main reason to get the modules listed on Packagist is because Packagist is familiar, people in the PHP world know how it works and know how to use is. A custom drupal.org solution adds another learning curve which is frankly not needed in the Drupal world. Also it means no repository needs to be added to composer.json, this then makes nesting modules / packages easier. It also means this code you are planning to write for Drupal.org or to allow composer to work with it doesn't need maintaining.

Like you states with the Wordpress example it's often suggested not to use Packagist, but I think with the right support we can help Packagist, we (as a community) are already committing back to Packagist and with LSD (Large Scale Drupal) organisations making use of things like Toran Proxy we can push funding back into composer and packagist. We will also grow packagist's stats (which is a good thing) and consolidate the PHP ecosystem. A similar argument can also be made about symfony bundles, but they are on Packagist.

Yes, there are downsides to moving to Packagist. Like what if it's down when we try and push to it or pull from it, but that can be said about anything. Most people push from github to packagist, what happens then if packagist is down? nothing, people just carry on. Making sure you use composer.lock or Toran Proxy should mitigate issues of pulling from packagist if it's down, but this is not an excuse to not push drupal modules on packagist because core and a growing number of contrib modules depend on other php packages which themselves are listed on packagist and often hosted on github, maybe we should start worrying about what if github is down? that seems to happen more often than packagist going down.

In all I think listing Drupal modules on Packagist is the right thing to do. This is both from a technical point of view and also from a PHP ecosystem point of view.

derhasi’s picture

I totally second #31.

A very viable alternative is to serve the package JSON data directly from Drupal.org.

Instead of that, we even could contribute back to Packagist and provide an API callback to do that, for cases someone wants to deal with projects not containing a composer.json at the moment. That would open doors to even include packages information from tools like bower, npm or components. But first let's make composer.json-projects work.

@timmillwood: big thanks for your work in the Packagist issue!

markcarver’s picture

Panels is useless to anyone not using Drupal

@cweagans, I completely understand with what you're saying about the fact that if we were to push Drupal projects to packagist, they wouldn't actually work for any other PHP project.

While this may be mostly true (currently), it doesn't have to be the case, nor would it be is most instances. Something as complex and unique as Panels is, in fact considered as a whole, a Drupalism. This doesn't mean, however, that the entire project has to be this way though. I'm sure there are components of Panels that could become more abstract.

In all reality, I actually see most Drupal projects becoming more abstract (so their solutions could work on any PHP platform). While a project may have initially started out as "Drupal" in nature (i.e. providing the YML/PHP files too hook into Drupal), there is nothing preventing a project from actually breaking up some of these "Drupalism" into more abstract components. These components could, in theory, could be used to support other platforms as well (e.g. wordpress, joomla, etc.).

In fact, I can actually see "themes" taking advantage of this. Themes are not, inherently, PHP in nature; they only have to be to "plug in" to said CMS. Adapting themes to work with any CMS... that would be a site to see.

Composer's and packagist's entire existence and purpose is to help break down these PHP silos we have built. To allow ideas and solutions to be exchanged interdependently. Yes, there still may be a few projects on Drupal that truly wouldn't make any sense to use elsewhere. However, I suspect that a bulk of what we "do" isn't all that unique and could, in fact, be made more abstractly.

We actually have an opportunity to join the PHP community as a whole, not continue to alienate ourselves.

---

Some feedback on this issue, specifically:

Given that we have a ton of self-hosted projects, we have undoubtedly accumulated a lot of "cruft" along the way. Even in our own community, newer versions of core are continually ostracizing existing projects with newer/better concepts and subsequent projects.

I think it would make sense to exclude any project marked as obsolete/abandoned from this process. There is no sense attempting to keep parity with something we, ourselves, have deemed unnecessary. This could actually benefit us in the long run to help weed out projects that are no longer active and effectively allow d.o to become more of a "archive" of abandoned projects.

cweagans’s picture

#32: You misunderstand. I'm not talking about serving composer.json from Drupal.org. I'm talking about the package repository data, like what you see at https://packagist.org/packages.json. We can generate that dynamically from the data already in the database.

#33: Currently, there aren't very many modules that are useful outside of Drupal (especially given that the vast majority of our extensions right now are for Drupal 7). I'd guess the number is close to zero. I also recognize that can change over time, and when that time comes, I'd be elated to push the packages to packagist. It just doesn't make sense to do it right now, IMO. There are good technical reasons not to do this, and in all the issues I've come across, the only two reasons that I could come up with FOR using packagist are "we should because we should" and "the packagist maintainers are willing to accept PRs". In any case, I attempted to summarize the arguments over in #2551607: [meta] Path forward for Composer support, and I think that we should come to a consensus on the path forward before the patch in this issue is committed or deployed. In the internal D8 work we've been doing, most of the Drupal-specific code that we've been writing really is Drupal specific, but it's glue code between Drupal and a library from Packagist. In that way, we really are reducing the amount of Drupal-specific code we're writing, but it's still Drupal specific. If the panels team can abstract components of panels out into useful libraries, that's cool, but there's still a lot of Drupal glue code that will need to exist, and that still won't be useful to anyone but Drupal users.

derhasi’s picture

Why does a package provided on Packagist have to work for any other Framework? Packagist is about sharing package information from different sources of the web: github, bitbucket, public gitlab instances. And drupal.org would be simply another source. It really does not matter, if the package is reusable in another framework per se.

Packagist is simply a tool. One that everyone in the PHP world uses. We can build upon that and improve its reliability, instead of building a separate solution, which would need to take Drupal specific prerequisites into account and will bind ressources for improving infrastructure specific to Drupal.

Btw: I helped to build Drupal Packagist, because we justed wanted to start to use composer right ahead, without the need of "politics". But as we now know, Jordi Boggiano is very supportive of making Drupal packages work directly on Packagist.org, I do not see any reason, we should not support him and the Packagist team, to make the service even better.

cweagans’s picture

It doesn't, but given the volume of packages that we'd be submitting (something like 20k-25k packages including themes and distros and drush extensions), I think it requires some thought. We have the potential to cause problems for packagist if they aren't adequately prepared for that volume of extra traffic, and if we don't need to do that, then we shouldn't.

We can serve the *exact same* package data directly from Drupal.org and avoid those problems, and again - there's no real reason *for* pushing to Packagist, other than "we should".

Let's continue this in #2551607: [meta] Path forward for Composer support, though.

timmillwood’s picture

We would only be submitting to packagist on git commit, and (eventually when I work out the best way how) only if the module has a composer.json.

hestenet’s picture

tvn’s picture

drumm’s picture

Status: Needs work » Closed (won't fix)

Drupal.org now runs a Composer endpoint: https://www.drupal.org/node/2718229

webflo’s picture

https://packagist.org/packages/drupal/drupal is a thing on Packagist and it uses the Drupal.org Repo. I think we still need some kind of push notification for this Packages. Packagist pulls package informations every few weeks. The last update is 10 days old, i think this is not good enough. We should keep the package up to date specially around wednesday for security updates. The package is out there and people may rely on it.

Proposal: Setup a Jenkins-Job which pings Packagist every few hours. @timmillwood wrote the code in #26 already. It could be a simple call with curl.

+++ b/drupalorg_project/drupalorg_project.module
@@ -915,6 +921,114 @@ function drupalorg_project_node_view($node, $view_mode = 'full') {
+function drupalorg_project_packagist_submit($endpoint, VersioncontrolRepository $repository) {
...
+    return drupal_http_request("{$packagist_api_url}{$endpoint}?username={$packagist_username}&apiToken={$packagist_api_token}", [
+      'headers' => array('Content-Type' => 'application/json'),
+      'method' => 'POST',
+      'data' => json_encode(['repository' => ['url' => $anon_project_url]]),
+    ]);
drumm’s picture

Status: Closed (won't fix) » Needs work