Many projects lack dependency information

I suspected this was happening. The vast majority of releases are never used by other releases so if there are reported problems then it would seem to be the tip of the iceberg. But it also seems that many projects have dependency information. This is important because it means that the release code is working. That means the probable causes are either the function call to process the release is not getting called or that Git is failing. If the function is not getting called then there is nothing that Project Dependency can do to fix the problem. If Git is failing then this probably is because the projects are updated instead of a clean clone of the project from the Git repository, This could be a permissions problem (current user does not have write privilege because the code was checked out under a different user) or a corruption of the checked out Git repository. (It also could be a problem with connecting to the Git repository but again that is not a Project Dependency problem.) Issue #2659782: Additional repositories are not installed as expected. changed from persisting the checked out code to a clean Git clone every time. This is in 7.x-1.0-beta12 release of Project Dependency which was released February 17. I do not have a way to test to make sure but I suspect that this latest release may have taken care of the problem. So the relevant question is whether you are still seeing project releases that are not getting processed.

There is code in the component block view code that checks the number of components from the project_dependency_component table. If the number is zero then I wrote code that processes the release. This is "self correcting" code since just visiting a release page with a problem fixes the problem. This code is commented out because it only worked when the persisted checked code was on the same server and it was decided we do not want to process releases on the webserver. But with the clean checkout of the code in the recent release this code probably would work. It would not cause significant server load because it only gets called when the components are zero which is only when a release is found that has a problem.

The other approach that I have considered is to use a query like you have in a cron job and process a few of the missing releases each time. There would probably have to be some safety so a release that will not process for some unknown reason does not get called again and again. But this requires processing releases on the webserver so it is not an option if we do not want to have the webserver processing releases.

I rebuilt dependencies for the most recent release in this list, fblikebutton 7.x-2.4 https://drupal.org/node/2673212. That completed normally:

Processing dependencies for fblikebutton
Processed fblikebutton: 7.x-2.4
Processed fblikebutton: 8.x-1.x
…

But there weren't components found for that release. I did see this this on a few other projects when I processed all releases that had been made in 2016 that had been missing dependencies. Some releases consistently aren't finding their components. I haven't dug into it to see the pattern yet.

There are three project_themes and three project_distributions that are skipped. The last three releases do not exist in my copy in addition to the missing radiogrid release. The two destination_alter releases are not typical projects with info files. special_taxonomy_tagging_in_body has Drupal 7 code in the Drupal 8 branch so the info file has not been converted yet into a info.yml file. That leaves 26 out of a total of about 138 over this period of time. It would appear that most releases are getting processed although there is significant failure rate.

I can manually process all 26 of these projects. I checked the logs and there are no messages about Git failing. So that would tend to indicate that the code is never getting called. That is consistent with what I have seem to have observed lately. Another observation is that the failures tend to cluster in certain projects although it could be a coincidence.

I found a recent release which did seem to be skipped. There isn't anything project_dependency-related in the logs, either it didn't run, or ran "successfully" at the time.

watchdog logging of the array as it is sent into project_dependency_info_batch_process_release() would be helpful to know what it is running.

Packaging passes the release node object straight from node_save() into project_dependency_project_release_create_package(). I could see this being a caching issue.

That sounds like a promising angle. Do you want me to add a call to watchdog in a formal commit? I think this is one of those very rare instances where just adding it manually would be okay since it is just for temporary debugging.

Here's the patch I'm adding. I think it would be good to keep around permanently. I'll go ahead and apply as a patch to Drupal.org so we don't need to tag a new release for now.

Removing an extra word in the log message.

An interesting missed release at the moment is leaflet_markercluster 8.x-1.0-alpha2, https://www.drupal.org/node/2673690. Re-processing dependencies for the projects does not populate the components there.

At the moment, 8.x-1.0-alpha2 is at the same place as 8.x-1.x, http://cgit.drupalcode.org/leaflet_markercluster?h=8.x-1.x. 8.x-1.x does have components successfully parsed, https://www.drupal.org/node/2068493.

Strange. Here is the command that is running to clone the code.

git clone --depth 1 --branch 8.x-1.0-alpha2 git://git.drupal.org/project/leaflet_markercluster.git '/tmp/leaflet_markercluster_1456354048' 2>&1

Here is what is checked out.

$ ls
leaflet_markercluster.api.php    leaflet_markercluster.make.example
leaflet_markercluster.drupal.js  leaflet_markercluster.module
leaflet_markercluster.info       README.txt

When I run the same command on my local computer I get

$ ls
leaflet_markercluster.api.php        leaflet_markercluster.make.example
leaflet_markercluster.drupal.js      leaflet_markercluster.module
leaflet_markercluster.info.yml       README.txt
leaflet_markercluster.libraries.yml

So this is not getting processed because the wrong code is getting checked out. Here is more information. On the system with the dev version of drupal.org we have

$ git --version
git version 1.7.1

My local computer (Ubuntu 14.04)

$ git --version
git version 1.9.1

It looks like both versions of git support --depth and --branch but I do see something about shallow clones having limited functionality before version 1.9. But the limited functionality should be sufficient. Maybe --branch does not support using a tag in the earlier version of Git?

For Git 1.7.1 the man page has

       --branch <name>, -b <name>
           Instead of pointing the newly created HEAD to the branch pointed to
           by the cloned repository’s HEAD, point to <name> branch instead. In
           a non-bare repository, this is the branch that will be checked out.

The man page for Git 1.9.1 has

       --branch <name>, -b <name>
           Instead of pointing the newly created HEAD to the branch pointed to
           by the cloned repository’s HEAD, point to <name> branch instead. In
           a non-bare repository, this is the branch that will be checked out.
           --branch can also take tags and detaches the HEAD at that commit in
           the resulting repository.

So it looks like cloning at a tag was added. What version of Git are we using on the appropriate server?

It looks like using --branch for a tag was introduced in Git 1.7.10.

Aha, we're on an older version of Git. (The versioncontrol* module integration parses output from git commands, so it gets pinned as we fear git output/behavior changes.)

I changed to use Git commands that work for older versions of Git. It is a lot less efficient. The more efficient version is commented out for use in the future. I did some testing but skim to make sure I did not miss anything. I also added the watchdog statement that already is deployed. I made a new release with these changes.

I did a bit more testing and found:

• A code comment that could use some clarification.
• Update the watchdog message for #9.
• --branch {tag} does get a checkout, it falls back to warning: Remote branch 7.x-1.13 not found in upstream origin, using HEAD instead, so the initial command can be kept.
• For static (project_release's term for tagged) releases only, do the git checkout fallback. This works well, even with --depth 1 on the initial checkout.

Looks like an improvement mostly in that the more efficient code will automatically work with newer versions of Git rather than having to create another release. The changes are in 7.x-1.0-beta14.

I deployed this and it is working well.