Project side of new d.o usage processing method

Feel free to make commits to a feature branch for this, named like 2575425-usage-parsing. That will be a bit easier to work with, and we might even merge parts like the project_usage_use_mongodb variable into 7.x-2.x branch earlier.

Looks good so far. A few things that could be better:

define('USAGE_STATS_COUNTS_PATH', '/var/log/updatestats/counts');

Let's not extend the pattern of maybe configuring with constants. --count-file-path can either be a required option; or remove the option and always use variable_get('project_usage_count_file_path', '/var/log/updatestats/counts');

project_usage_drush_command() had a small whitespace issue, I committed the fix to the branch.

Since Mongo use will be effectively unsupported, let's default it to off:

$use_mongo = variable_get('project_usage_use_mongodb');

The DB transactions can use Drupal's db_transaction(), https://www.drupal.org/node/355875.

Aside, why are the variables like mongo stuff not configurable in a settings page? It's really difficult to analyse the module code just to find out there is a hidden setting. Additionally there is no doc how to setup mongo.

This issue is deprecating Mongo use. If you don't have it set up already, I don't recommend starting.

I should point out that this doesn't actually deprecate mongo use - we're solving a specific usage stats issue for drupal.org that is not applicable to anybody outside of drupal.org. The sheer volume of data that we're processing, coupled with the fact that our data comes to us in a very specific, non-NCSA format, predicated us eliminating reformatting and counting that log data in php/mongo and replacing it with awk/count/sort/uniq.

http://cgit.drupalcode.org/infrastructure/tree/stats/fastlyusagestatpars... is our script responsible for reformatting the data each day, and http://cgit.drupalcode.org/infrastructure/tree/stats/update_aggregate_co... is the weekly script responsible for generating the counts.

We should probably eventually pull out drupal.org's usage stat processing from the project module itself and move it to drupalorg_customizations so that site specific code doesnt pollute a publicly used repository, but project_usage is just not used on enough sites other than drupal.org to warrant generalizing it before we solve the immediate problem of getting update stats working again.

That is not true. I have themes I am not allowed to host/develop on d.o. I'm running such a project site to provide my customers the same expierience like d.o. And i'm also interrested how much my themes are used. Therefore the module should be reusable.

According to the submodule reporting, project_usage is used on 26 sites - some of those may be drupal.org preproduction sites. That does not warrant development time to make our site specific code generalized enough for others to use. I've linked all the relevant code there, so others are welcome to build on what we've got.

Since this *adds* an additional drush command and function code that does not need to be used by any third party sites, there is no impact on third party sites and they can continue to either use the mongo implementation or reformat their statistics to match the api this introduces.

I've went ahead and cleaned up the commit so it wasnt mixing a bunch of apples and oranges.

D5 and D6 versions have not used mongo and there is no docu how to use set it up. I'd like to use what you are building and not any other code. Reinventing the wheel is nothing I prefer to do.

If you would like to use what we are building, then your log data has to be in the exact format that we have configured for fastly e.g.
2015-06-27T00:00:00Z cache-ams4140 fastlyupdates[310]: 144.76.104.230 | "-" | "-" | 2015-06-26 | 23:59:59 +0000 | GET /release-history/metatag/7.x?site_key=IPfiGkPnKUfj6HIFgEQXFo0JyCXwfP5R9QQZxMCLJJA&version=7.x-1.5&list=metatag%2Cmetatag_context | 200 | (null) | Drupal (+http://drupal.org/)

OR, you'll need to make your a modified version of http://cgit.drupalcode.org/infrastructure/tree/stats/fastlyusagestatpars... - bearing in mind that all of the paths either need to be changed to whatever you have in your environment.

Additionally, you'll need to run http://cgit.drupalcode.org/infrastructure/tree/stats/update_aggregate_co... , and probably change multisort to be just sort (we've got a more modern version of sort that runs on multiple processors, but it's just the most recent version of gnu sort)

Once you have your data in the format

Count projectname|majapi
 779494 views|7.x

for the weeknum.projectapicounts
and

Count projectname|releaseversion|majapi 
408797 views|7.x-3.11|7.x

for the weeknum.releasecounts files

Then you can use the drush command this adds. drush import-usage-stats

(be sure to set the project_usage_count_file_path variable if you put those files somewhere other than /var/log/updatestats/counts.

Hope that helps.

2575425-import-aggregated-usage-stats-rb1 looks good, merged to 7.x-2.x

Now deployed to Drupal.org. (Anyone coming here from #2509574: Project usage stats have probably gone bad (again), this is just getting the drush command ready to use, it will still be some time for us to run it, etc.)

Ran the drush command. Stats all the way back to Feb 7th have been updated.

Has this change here fixed #2402175: Versions like '7.x-1.0+10-dev' might not be counted for '7.x-1.x-dev'?

Yes, this change did fix that issue.