Download and parse tgz files as a data source

I'm not sure this is all that feasible. Let's say a given project has a new tgz file. Does that mean we would need to parse all of that project's contents again? Because I'm looking at the api project's tgz for 7.x-1.x-dev, and every single file in there has a date of May 18 at 14:25, whereas I'm certain that not every file in there was really updated for that release. So it doesn't look like we can tell which files actually need parsing. If we parse every file all the time, we'll be way overloaded on parsing...

I don't think so. I just downloaded api-7.x-dev.tar.gz and extracted it using the Linux command-line
tar xvzf api-7.x-dev.tar.gz
Every file says it was last updated on May 18 at 14:25, except LICENSE.txt (which I think is put in by the packager).

Adding new issue tag for issues pertaining to getting all contrib modules on api.drupal.org.

I just tested this again with the latest 7.x tar.gz from the API module, and now all the files say they were last updated June 12 at 14:32, except again LICENSE.txt (dated Sept 17 2011). Definitely a good number of those files were not updated between may 18 and June 12. So this is not going to be a viable way to track updates unless the d.o packaging scripts start putting better dates on the files. I'm postponing this accordingly, since it doesn't seem like it will work out at all to try this method... feel free to open if you have a different method for getting proper updated dates.

If the problem is to see what files changed, can't we introduce sha1 hashes as a way to verify file content changes. We can use http://php.net/manual/en/function.sha1-file.php. This would probably need a hash field in the api_documentation table.

It might help also with deleting old documentation. You don't need the dids that way because the sha1 is unique and it lowers the batch size for node deletion.

RE #7 - using a hash to see if a file changed does seem like a good idea.

I don't see how hashes would help with deleting the old documentation though. You *do* need the dids to delete documentation, because you have to delete information in quite a few tables, and also you have to delete the fake nodes that are used to implement comments (the did is the node's nid).

When a file is updated the hash changes with it. So the old files have the old hash. When the file is parsed the updated information get the new hash while the information that is legacy can be deleted based on the old hash. Because its unique you don't need have the full array of dids to delete them. This way you can quickly delete the appropriate records. At least thats what I think. If that is really a + on execution is something different.

RE #9, nope! Thanks for the suggestion though because they may help with the file downloads issue.

In thinking about this, doing a md5 or sh1 hash of the files will *really* slow down the process of checking to see if files need to be reparsed. Right now it is only checking file times, which is fast (quick system directory query). If we have to do a file hash, each file in each project will need to be read in and hashed, which will not be as fast. I really don't think it will be feasible... the test it's doing now, just comparing file modified times, in the Drupal 8.x repo is already fairly time consuming. Multiply that by thousands of projects and I don't think we can afford to add to the time the process will take.

I'm running a test on this and will provide some info back later. It looks like it slows it down just a little now in such a major way that it is problematic I think.

Thanks, that will be helpful! If you can test using the Drupal 8.x code base, that would be the worst case.

(I mean, use the 7.x version of the API module, but test it on scanning the Drupal 8.x code base.)

I took another look at this today:
- The file modified date for every file in the tar.gz and zip files is the date the archive was created (with the exception of LICENSE.txt).
- If you do a git clone in a new directory, the file modified date for every file is the moment that you do the git clone.
- If you do a git checkout to switch branches, the file modified date for every file is the moment you run the command.
- If you do a git clone, and then sometime later do a git pull to get updates, updated/new files will be dated as the moment you run the command.

So my conclusion from this is that:
- Git does not set the file modified times to the last time the file had a commit made. It lets the OS set it to the current time when the file is changed on disk.
- The packaging scripts are most likely doing a git checkout into a clean space, adding the license.txt file, and then zipping/tar.gzipping it all up. The only alternative to doing it this way would be to maintain directories for each branch of each project, which isn't practical in the least.

So. If we actually want to run the API module against downloaded zip or tgz archives, we would definitely need to do some kind of hash to see if the file contents for files in the archive have changed, rather than relying on last update times as we do now.

The problem is... Let's say we download a new Drupal Core tgz file for 8.x, and we realize that 125 files have changed since the last download (which is certainly possible). If we were using the file system as normal, we would just mark those 125 files as "needs parsing", and then over the next few cron or cron-queue runs, we would get through parsing them. But if the only location of the files is in an archive that we've downloaded temporarily, ... to say the least, the current parse queue system would not work well for this situation. I guess maybe we could unzip just the changed files to some temporary storage and set up a parse job that reads the temporary file? It sounds like a pain, but I guess it might be doable. We'd also need to set up a system where if, before we got around to parsing the temporary file we saved, the file changed again -- we would in that case want to queue another job and if possible, realize when we got to the first job that it was obsolete. Still, should be doable.

It does look like we could use the ArchiveTar class (which is now in Drupal 8.x) to do the extraction. You can construct an ArchiveTar (giving a file path or a URL I think), and then call the listContent() method to make a list of all the files it contains (not sure of the format but I'm sure we can figure that out). Then for each file in the archive, we could use extractInstring() to get the file contents into a string, check the hash, and save that string as a temporary file and a parse job if it has changed.

Interesting ideas...

{api_documentation}.code is currently storing the parsed/formatted code, not the raw source code, so that might be a bit difficult (for instance, in between noticing file A needed parsing and actually doing it, we wouldn't want to have overwritten the already-formatted code that we had).

But we could find another spot in the database. Putting it in the queued job itself (as you suggested) would be the most logical. I think that the data for the job is already a blob anyway.

We need to un-postpone this.

New possible idea of an approach:

a) We would probably only want to do this on contrib module branches. For Drupal Core, there are additional things we do besides just a git clone (such as running composer to get external dependencies), and these are not in the branch TGZ files I think.

b) For projects that we're using TGZ on, when the queues come across a job that needs the files, we could trigger a process that would download the TGZ to a temporary space, and unzip it.

c) The temporary space could have a time to live on it, so that it could be cleaned up later when it's not in use.

d) For branch update jobs, we could skip actually checking the individual files to see what is updated, by looking at a hash or checksum of the TGZ itself. If the TGZ has not changed, nothing in the branch has changed.

e) We would definitely need to switch from using file update times to say "this file needs reparsing" to using some kind of hash or checksum.

The "trigger a process" could be a hook, and we could have a separate sub-module that manages the TGZ download, extraction, and temporary file cleanups. There could also be a hook that asks "do we even need to process this branch update", which would check the hash on the TGZ.

Adding an issue summary...