Notify Packagist about repository update

Do we want to do this on node insert? I'd say node update, since someone could easily add a composer.json file well after the module is published. (Like, all existing modules. :-) )

A file_exists() check seems fine to me if we can get at the repo. I don't know the infra well enough for that part.

Tagging to bring up on the DSWG meeting next week so this is on the DA's radar.

Tim: You may be correct. IIRC, Packagist has a "go fetch this repo and update your stuff" command, and you're adding a create command. Thus, I think the on-git-push operation on Drupal.org would be:

Ask Packagist if project X exists
If No {
Push project create
}
Send Packagist "things changed, do your updates" message

(Depending on how the create works, the latter part could be in an Else statement.) Also, I think we do need to make it on git push, not anything triggered by the node being edited. All Packagist is going to care about is the tags it finds, which it can get directly from the repo.

I think the entry point is somewhere in Version Control API. Maybe http://cgit.drupalcode.org/versioncontrol_git/tree/vcs_hooks/post-receive => http://cgit.drupalcode.org/versioncontrol/tree/versioncontrol.drush.inc => http://cgit.drupalcode.org/versioncontrol/tree/includes/worker.inc#n112 ?

Not sure if duplicate, but adding a related issue.

Thanks Tim, this is great!

1. +++ b/drupalorg_project/drupalorg_project.module
   @@ -916,6 +916,55 @@ function drupalorg_project_node_view($node, $view_mode = 'full') {
   +    $anon_project_url = "http://git.drupal.org/project/{$project_url_name}";
   

   Git server urls should use the git_base_url variable: see http://cgit.drupalcode.org/drupalorg/tree/drupalorg_project/plugins/rele...

2. +++ b/drupalorg_project/drupalorg_project.module
   @@ -916,6 +916,55 @@ function drupalorg_project_node_view($node, $view_mode = 'full') {
   +  return drupal_http_request("https://packagist.org/packages/drupal/{$project_url_name}.json", [
   ...
   +    return drupal_http_request("https://packagist.org/api/{$endpoint}?username={$packagist_username}&apiToken={$packagist_api_token}", [
   

   These packagist urls should be a variable as well, that way we can set them for dev/staging environments when we're testing git functionality.

3. +++ b/drupalorg_project/drupalorg_project.module
   @@ -916,6 +916,55 @@ function drupalorg_project_node_view($node, $view_mode = 'full') {
   +    drupalorg_project_packagist_submit('create-package', $repository); ¶
   ...
   +    drupalorg_project_packagist_submit('update-package', $repository); ¶
   

   remove end of line spaces

@timmillwood I don't think this should be limited to D8, modules should be able to implement composer.json in 7 (or even 6) and be included on packagist under the Drupal namespace.

In terms of checking for composer.json, it should be possible to extract the branch from VersioncontrolEvent object. Then a drupal_http_request() HEAD request could check for a 200 response from http://cgit.drupalcode.org/[module]/tree/composer.json?h=[branch].

@timmillwood should sandbox projects be filtered out?

And a nitpick

+++ b/drupalorg_project/drupalorg_project.module
@@ -916,6 +916,58 @@ function drupalorg_project_node_view($node, $view_mode = 'full') {
+    $project_url_name = check_plain(drupal_encode_path($repository->name . '.git'));

check_plain() is only for escaping markup in the UI and so I don't think it is needed here.

Yes they should be installable via composer, but you can add the git repo in your composer.json directly. It not necessary to expose every sandbox to packagist. Besides that packagist uses the package name in the composer.json as identifier. First come, first serve that way i could push my "views" sandbox as the original package to packagist.

Maybe we should perform some basic validation the composer.json and package url should match before we register new packages in packagist.

I'd say they shouldn't because under the current project approval process they can't have any releases. Under the revised process proposal (see #2453587), they can only have dev snapshots, not tagged releases. Given this will make packagist a mirror of the d.o releases, the same policy should apply.

Maybe we should perform some basic validation the composer.json and package url should match before we register new packages in packagist.

I think so to. We have a Composer package naming convention. So we should make sure, that the convention is applied appropriately beforehand.

+++ b/drupalorg_project/drupalorg_project.module
@@ -916,6 +916,58 @@ function drupalorg_project_node_view($node, $view_mode = 'full') {
+  $project_url_name = check_plain(drupal_encode_path($repository->name));

Need to remove the used variable at the hook_uninstall

What happens if Packagist is down when drupalorg_project_packagist_submit() is called? Do we have some way to reconcile the Drupal.org project data with what's in Packagist?

As mentioned in the other issue, I don't think we should push our projects to packagist at all. See https://www.drupal.org/node/2547385#comment-10219351 for reasoning. A very viable alternative is to serve the package JSON data directly from Drupal.org. Mixologic created a dev environment for me, and I should have it done by Sunday at the latest (depending on if I end up going to a family thing tomorrow).

Well, that problem only has to happen once for it to be a problem, and with 99.97% uptime, that's 2 hours and 38 minutes per year where a request might not get through. What is the solution to that problem?

I'd appreciate a real response to the arguments against Packagist that I presented, rather than a blind assertion that we should move forward with pushing things to Packagist. The Composer docs themselves point to Wordpress plugins as an example of a good thing to have in a separate repository because they simply aren't useful to people outside of the Wordpress community, and the same thing applies to Drupal modules. Panels is useless to anyone not using Drupal, so why force it on the rest of the PHP community? Again, I'm not arguing against making Drupal extensions available via composer - I want that. I'm arguing against pushing them to packagist.org.

I totally second #31.

A very viable alternative is to serve the package JSON data directly from Drupal.org.

Instead of that, we even could contribute back to Packagist and provide an API callback to do that, for cases someone wants to deal with projects not containing a composer.json at the moment. That would open doors to even include packages information from tools like bower, npm or components. But first let's make composer.json-projects work.

@timmillwood: big thanks for your work in the Packagist issue!

Panels is useless to anyone not using Drupal

@cweagans, I completely understand with what you're saying about the fact that if we were to push Drupal projects to packagist, they wouldn't actually work for any other PHP project.

While this may be mostly true (currently), it doesn't have to be the case, nor would it be is most instances. Something as complex and unique as Panels is, in fact considered as a whole, a Drupalism. This doesn't mean, however, that the entire project has to be this way though. I'm sure there are components of Panels that could become more abstract.

In all reality, I actually see most Drupal projects becoming more abstract (so their solutions could work on any PHP platform). While a project may have initially started out as "Drupal" in nature (i.e. providing the YML/PHP files too hook into Drupal), there is nothing preventing a project from actually breaking up some of these "Drupalism" into more abstract components. These components could, in theory, could be used to support other platforms as well (e.g. wordpress, joomla, etc.).

In fact, I can actually see "themes" taking advantage of this. Themes are not, inherently, PHP in nature; they only have to be to "plug in" to said CMS. Adapting themes to work with any CMS... that would be a site to see.

Composer's and packagist's entire existence and purpose is to help break down these PHP silos we have built. To allow ideas and solutions to be exchanged interdependently. Yes, there still may be a few projects on Drupal that truly wouldn't make any sense to use elsewhere. However, I suspect that a bulk of what we "do" isn't all that unique and could, in fact, be made more abstractly.

We actually have an opportunity to join the PHP community as a whole, not continue to alienate ourselves.

---

Some feedback on this issue, specifically:

Given that we have a ton of self-hosted projects, we have undoubtedly accumulated a lot of "cruft" along the way. Even in our own community, newer versions of core are continually ostracizing existing projects with newer/better concepts and subsequent projects.

I think it would make sense to exclude any project marked as obsolete/abandoned from this process. There is no sense attempting to keep parity with something we, ourselves, have deemed unnecessary. This could actually benefit us in the long run to help weed out projects that are no longer active and effectively allow d.o to become more of a "archive" of abandoned projects.

#32: You misunderstand. I'm not talking about serving composer.json from Drupal.org. I'm talking about the package repository data, like what you see at https://packagist.org/packages.json. We can generate that dynamically from the data already in the database.

#33: Currently, there aren't very many modules that are useful outside of Drupal (especially given that the vast majority of our extensions right now are for Drupal 7). I'd guess the number is close to zero. I also recognize that can change over time, and when that time comes, I'd be elated to push the packages to packagist. It just doesn't make sense to do it right now, IMO. There are good technical reasons not to do this, and in all the issues I've come across, the only two reasons that I could come up with FOR using packagist are "we should because we should" and "the packagist maintainers are willing to accept PRs". In any case, I attempted to summarize the arguments over in #2551607: [meta] Path forward for Composer support, and I think that we should come to a consensus on the path forward before the patch in this issue is committed or deployed. In the internal D8 work we've been doing, most of the Drupal-specific code that we've been writing really is Drupal specific, but it's glue code between Drupal and a library from Packagist. In that way, we really are reducing the amount of Drupal-specific code we're writing, but it's still Drupal specific. If the panels team can abstract components of panels out into useful libraries, that's cool, but there's still a lot of Drupal glue code that will need to exist, and that still won't be useful to anyone but Drupal users.

Why does a package provided on Packagist have to work for any other Framework? Packagist is about sharing package information from different sources of the web: github, bitbucket, public gitlab instances. And drupal.org would be simply another source. It really does not matter, if the package is reusable in another framework per se.

Packagist is simply a tool. One that everyone in the PHP world uses. We can build upon that and improve its reliability, instead of building a separate solution, which would need to take Drupal specific prerequisites into account and will bind ressources for improving infrastructure specific to Drupal.

Btw: I helped to build Drupal Packagist, because we justed wanted to start to use composer right ahead, without the need of "politics". But as we now know, Jordi Boggiano is very supportive of making Drupal packages work directly on Packagist.org, I do not see any reason, we should not support him and the Packagist team, to make the service even better.

It doesn't, but given the volume of packages that we'd be submitting (something like 20k-25k packages including themes and distros and drush extensions), I think it requires some thought. We have the potential to cause problems for packagist if they aren't adequately prepared for that volume of extra traffic, and if we don't need to do that, then we shouldn't.

We can serve the *exact same* package data directly from Drupal.org and avoid those problems, and again - there's no real reason *for* pushing to Packagist, other than "we should".

Let's continue this in #2551607: [meta] Path forward for Composer support, though.

Drupal.org now runs a Composer endpoint: https://www.drupal.org/node/2718229

https://packagist.org/packages/drupal/drupal is a thing on Packagist and it uses the Drupal.org Repo. I think we still need some kind of push notification for this Packages. Packagist pulls package informations every few weeks. The last update is 10 days old, i think this is not good enough. We should keep the package up to date specially around wednesday for security updates. The package is out there and people may rely on it.

Proposal: Setup a Jenkins-Job which pings Packagist every few hours. @timmillwood wrote the code in #26 already. It could be a simple call with curl.

+++ b/drupalorg_project/drupalorg_project.module
@@ -915,6 +921,114 @@ function drupalorg_project_node_view($node, $view_mode = 'full') {
+function drupalorg_project_packagist_submit($endpoint, VersioncontrolRepository $repository) {
...
+    return drupal_http_request("{$packagist_api_url}{$endpoint}?username={$packagist_username}&apiToken={$packagist_api_token}", [
+      'headers' => array('Content-Type' => 'application/json'),
+      'method' => 'POST',
+      'data' => json_encode(['repository' => ['url' => $anon_project_url]]),
+    ]);