Auto-generate some release nodes

Here's a start on this. This currently just generates a few releases for the Drupal project itself. The code if quite hackish because I couldn't get it to work any other way. I really hate drupal_execute().

Still to do:
1. Make this less hackish (if possible).
2. Generate more releases for more projects. Also create some snapshot releases.
3. Figure out a way to deal with files. Right now, I've manually created files in my /files/project directory for each of the files mentioned in the release nodes. It would be much better if this was handled automatically.
4. Once we get this working, we'll probably want to revisit the automatic creation of project issues and start assigning those issues and comments to actual versions so that we have project issue and project releases working together in this respect.

Any suggestions on how to accomplish any of the above would be appreciated.

Ok, with these two patches we're pretty close to nirvana as far as automatically generating project releases is concerned.

Let me explain what these patches do.

First of all, two new files are added:
* drupalorg_testing_build_releases.php: This is a script that is not designed to be run from within Drupal, but instead from the command line. It takes a list of projects and releases (hard coded into the script) that are used in the drupalorg_testing profile and contacts updates.drupal.org to get information about each release on d.o for each project and API term pair. Included in this information is the size, path, and hash of each release file. This script takes all of this information and dumps it (in the form of a serialized array) into another file

* drupalorg_testing_releases.inc: Contains a single function that contains a serialized array written by the file above. This file is included by the drupalorg_testing.profile file. It's pretty large, but it's designed to store data more than be code. Once in a while someone can run the file above on their machine to get new releases information, and then we can commit the changes in this file back to the project. When reviewing this patch, you can pretty much ignore this file--it's almost all data.

The other patch attached here is for the project issue module. It needs to be committed at the same time as or after this patch is committed or I think it'll break functionality in the project issues generation module. It basically just assigns issues and comments that are automatically generated to an appropriate revision. I also noticed a bug in the project_issue_generate_issue_comments() function. When the project was changed in a comment, a new array of categories needs to be created to make sure that a valid category is being used. I'm not sure that we've added any categories in any of the project nodes beyond the default categories, so this bug might not have actually shown up.

One thing I'd still like to do is to use the output of the XML data retrieved from updates.drupal.org and add that to {project_release_supported_versions}. Right now there is data written to that table, but it's not necessarily the same information that's on drupal.org.

Attached patch now generates rows in {project_release_supported_versions} as on d.o, with the exception that the snapshot column is set to 1 for all rows because I couldn't think of a way to pull this information from the XML data from updates.drupal.org.

One other thing, in case anyone is curious. I thought it's best to have this (rather large) data file that's created by the script included in this patch instead of fetching this data each time the profile is used so that it's possible to use the profile when offline. I suppose we could have it fetch fresh data if the user is online and otherwise fall back to the data file, but that's probably not worth the trouble.

I think this is working pretty well now, so I'm setting to CNR. This patch should be very useful when working on views conversion of project*.

Yeah, it does feel wrong to commit this big data file, but I can't think of any way around this unless we are ok with this profile being useless without internet connectivity (which I'm not). At least I didn't try to included the release node bodies as well :)

If it's the size of the data file that worries you I can probably figure out a way to compress it more, at the expense of readability. Storing a big array as a serialized string is obviously not that space efficient, but at least it keeps the data file somewhat readable.

If you can suggest another way to do this I'm willing to take a look.

what i'd like to see is:

• drupalorg_testing_build_releases.php generates an actual .inc file, ie, a normal <?php file with the necessary data in a normal php array
• this file is require_once'd in the profile
• the drupalorg_testing_build_releases.php can be used to regenerate the actual real .inc file periodically if we like

ok, the .inc file may be a bit large, but it's totally readable, it's done as a regular .inc file, and it's periodically updatable via the script.

Attached patch prints the $releases and $supported_releases in a pretty format instead of serialized.

I've also added a check to the script that generates the drupalorg_testing_build_releases.php file that checks to make sure that the SimpleXML extension is available. It's included in PHP 5 and greater by default.

This might need to be modified to not try to add release files and such if touch() is not permitted due to file system settings.

Fixed a bug where the script that generates the data file would fail if SimpleXML *was* present, not if it was absent.

Also, a comment is now added at the top of the data file with the date and time it was generated.

couple of things:

1. i'm a bit confused by the need for the manual queries -- what stage of bootstrap are we in during the install?
2. i'm pretty sure there is a php function that we can use instead of the custom get_file_name() function you created
3. does casting the to array work, or do we really need object_to_array()?
4. this looks a little odd to me $releases = array_merge(dot_get_releases($project, $api_term, $supported_releases), $releases); -- would it be cleaner to do both $releases and $supported_releases by reference?

re #10

1:

Are you talking about the code following this comment?

+      // CHEESY HACK: Because Drupal is not fully bootstrapped at install time,
+      // we have to do raw DB manipulation to add the terms and cvs related stuff. Sigh...

Since I don't typically sigh in my comments, I think that came from somewhere else in the profile, but I don't recall if I tested that the comment is still true.

2:
Yeah, it looks like parse_url(), possibly in combination with basename(), should do the trick.

@dww: By my reading of the issue you linked to, it sounds like, at the moment, this hack would still be necessary, right? Or has this been fixed in the project module and I'm not aware of it?

First, I want to point out here that before this code can be committed, #267313: Allow .inc files in contributions/profiles needs to be addressed, because otherwise the .inc file created by this patch won't be allowed to be committed to the repository.

Regarding hunmonk's comments in #10:
1. I added a code comment with a link to the issue dww pulled up in #12 to explain why the manual queries are still necessary.

2. Fixed.

3. Casting does work, but it's not quite that simple since $terms is a SimpleXML object that contains nested SimpleXML objects, so the casting to array has to be done several times. However, it's quite a bit less code than it was with the object_to_array() function there.

4. Good point. Fixed. By passing these by reference, the projects and releases are handled in a different order than they were previously, so the .inc files from before I made the change and the new one included in the patch will not be identical, however as far as I can tell the data for each release is the same, just the order is different.

I also fixed a few other things in this patch:
a) Improved handling of release nodes on d.o that have no file information in the downloaded XML (old releases from the Drupal project). Now, no value for the file path will be stored to the database, whereas before the function was storing the path without the file name.

b) Improved code in _drupalorg_testing_create_content_project_release() so that the /files and /files/project directories are created. If this isn't successful, then the function doesn't attempt to create the empty files associated with releases, and also doesn't add information about the files to {project_release_nodes}, because doing so would cause PHP errors when releases are viewed but the file associated with the release doesn't exist. This also takes care of my concern from comment #8 above.

@dww: Good idea in (c) about changing the names of these two files. I agree that the names you proposed are better.

I also fixed the notices you reported in (g) and (h) above.

In addition, I modified a few of the comments very slightly.

I've attached the final patch, as well as a diff of your patch vs. my patch, so it's easy to see what I changed.

I also (briefly) tried again to get drupal_execute() to work in the places it's failing and had no success. I agree that it's not worth spending more time trying to get that fixed, since this patch makes the profile much more useful.

I'll be committing this soon.

Committed to HEAD
http://drupal.org/cvs?commit=145958

Thanks for the reviews on this.

I've created an issue in the project_issue queue at #320278: Assign automatically created issues version information to get the patch from comment #2 committed there.