So I have this bandwidth stats stuff here. We're parsing logfiles. We need to keep track of what's the last time we parsed a specific logfile. I could settle for saving an integer (the timestamp) but ideally I would save an array of file -> timestamp mappings.

I tried saving the data directly:

d()->bwstats_last_record = $foo;
d()->write();

That doesn't actually do anything.

I tried calling provision_save_server_data(), that saves to .drush (wrong place), I am removing that function completely.

I tried calling provision-save itself, and that doesn't work either: 

hostmaster:/var/aegir/.drush/provision$ drush   --context_type='server' --server='@server_master' --name='@server_master' --bwstats_last_record='1' provision-save @server_master
hostmaster:/var/aegir/.drush/provision$ grep bwstats ../server_master.alias.drushrc.php
hostmaster:/var/aegir/.drush/provision$

I suspect this is because there is no property of that name set on the server. But here's the thing: i'm not creating a subclass of the server (or service) here: I just want to parse logfiles and don't want to patch core.

How the heck am I supposed to do this?

Comments

anarcat’s picture

anarcat CreditAttribution: anarcat commented

Here's the actual code:

  $options = d()->options;
  $options['bwstats_last_record'] = json_encode($last_record);
  drush_log("option: " . $options['bwstats_last_record']);
  drush_backend_invoke_args('provision-save', array(d()->name), $options);

I'd like this to be programatically done without having to call a subshell, btw... Seems like a waste to just write one file.

anarcat’s picture

anarcat CreditAttribution: anarcat commented

I tried all of this:

  $options = d()->options;
  $options['bwstats_last_record'] = $last_record[$filename];
  d()->bwstats_last_record = $last_record[$filename];
  d()->setProperty('bwstats_last_record', $last_record[$filename], 0);
  drush_backend_invoke_args('provision-save', array(d()->name), $options);

Note that I'm not saving $last_record because that's an array and I suspect this is just not passed to provision-save.

anarcat’s picture

anarcat CreditAttribution: anarcat commented

and this (reordering the above):

  d()->bwstats_last_record = $last_record[$filename];
  d()->setProperty('bwstats_last_record', $last_record[$filename], 0);
  $options = d()->options;
  $options['bwstats_last_record'] = $last_record[$filename];
  drush_backend_invoke_args('provision-save', array(d()->name), $options);

I'm giving up. Provision-save just hates me right now and i'm tired.

mvc’s picture

mvc
they
Primary language English
Location Montréal, CA
CreditAttribution: mvc commented

subscribe

Anonymous’s picture

Anonymous (not verified) CreditAttribution: Anonymous commented
Category: support » bug

I think this is a bug.

I've been running it from CLI and at the end of the day, this is what i've learnt:

1) I can't save new attributes to the drush alias file at all
2) I can modify existing attributes as a cli switch, i.e --http_port=8000, and it will be changed in the alias file.

Anonymous’s picture

Anonymous (not verified) CreditAttribution: Anonymous commented

Also, if I save a new alias altogether and add --mig5=fail in my command, a new alias file is created, but is missing the mig5 attribute. Instead it sticks in a bunch of defaults:

php /var/aegir/drush/drush.php provision-save @server_master2 --context_type=server --mig5=yo

<?php 
$aliases['server_master2'] = array (
  'context_type' => 'server',
  'server' => '@server_master',
  'remote_host' => 'localhost',
  'aegir_root' => '/var/aegir',
  'script_user' => 'aegir',
  'ip_addresses' => 
  array (
  ),
  'backup_path' => '/var/aegir/backups',
  'config_path' => '/var/aegir/config/server_master2',
  'include_path' => '/var/aegir/config/includes',
  'master_url' => NULL,
  'admin_email' => 'admin@localhost',
);
anarcat’s picture

anarcat CreditAttribution: anarcat commented

adrian, can we get help on this? What are we doing wrong?

Anonymous’s picture

Anonymous (not verified) CreditAttribution: Anonymous commented

17:36 <@Vertice> provision-save you need to write an extension for it
17:36 <@Vertice> and at the moment that means a service
17:36 <@Vertice> like the example service
17:37 <@Vertice> it's not like before where the file just magically saves attributes out of the ether
17:37 <@Vertice> you need to store them explicitly
17:37 <@Vertice> and load them

I realised got this to work by accident in my log_directory branch because I (probably incorrectly) add it as an attribute of the service, so it gets written into the alias.

anarcat’s picture

anarcat CreditAttribution: anarcat commented
Category: bug » support
Status: Active » Needs work

Okay, so I have a problem here: either I subclass all HTTP services to add logging capabilities (which will be a maintenance and usability nightmare), either I patch the core HTTP service to add a logfile and timestamp array mapping, which will not be useful unless stats are enabled.

A third option would be to allow arbitrary options to be saved to a context from outside the context definition.

Adrian, what is your take on this?

anarcat’s picture

anarcat CreditAttribution: anarcat commented
Title: how do i save data to a context from outside? » ability to save new arbitrary data to a context from outside the service
Category: support » feature
Issue tags: +API, +aegir-1.0, +wtf

So we're going to work around the issue by storing data in the frontend, in the variables table. It's rather unfortunate, because I don't feel it's the "right" way, but then again it makes sense: it's data that's changing regularly (every night), contrarily to context data, which is mostly static. Also, it means that the backend script are not fully autonomous: they can parse the logs, but don't remember where they are... which is fine because it's the frontend that collects the stats anyways.

Regardless, I believe this is a major wtf for people that try to implement things in the backend... I think this should be fixed just for sanity's sake.

Steven Jones’s picture

Steven Jones CreditAttribution: Steven Jones commented

This mostly defeats the purpose of having the alias in the first place. If I'm a module that wants to alter the behaviour of a site/server/platform in drush, my task has to be run from the frontend otherwise I may not have all my drush options kicking around.

Steven Jones’s picture

Steven Jones CreditAttribution: Steven Jones commented

So, I guess the trick here is that you must have a service for any frontend thing that needs to pass data into the backend. Guess we need to stop thinking like the frontend is running the show and remember that the backend is.

bgm’s picture

bgm CreditAttribution: bgm commented

I'm having the same issue for provision_civicrm. I need to save the "site key" that civicrm stores into the civicrm.settings.php, used for its cronjobs. Although maybe if the drupal is bootstrapped, I can just get the old value from the civicrm.settings.php before it gets rewritten. Not very clean though.

Steven Jones’s picture

Steven Jones CreditAttribution: Steven Jones commented

Yeah, you'll need a service to store data in a context, I floated the idea of having a variable storage service, but I don't think many people liked it :( I'm going to work on it in contrib though.

anarcat’s picture

anarcat CreditAttribution: anarcat commented

I'd like to see this land in core, not contrib.

Steven Jones’s picture

Steven Jones CreditAttribution: Steven Jones commented
Version: » 6.x-1.1
Assigned: adrian » Unassigned

anarcat, what do you reckon here? Do we actually need something in Aegir core to help devs store random things in contexts, or should we document contexts and services to the point that devs can implement their own services to store such data.

Note that I've written such documentation here:
http://www.computerminds.co.uk/articles/storing-data-aegir

anarcat’s picture

anarcat CreditAttribution: anarcat commented

See #9 - basically subclassing is not feasible in the case of a webserver, as there are already multiple subclasses. Because PHP doesn't support multiple inheritance, we get an exponential explosion of classes... doesn't make sense.

Steven Jones’s picture

Steven Jones CreditAttribution: Steven Jones commented

Maybe we need to rethink how we extend these classes then, inheritance sucks for a lot of this stuff.

What methods would you need to override in the http class?

anarcat’s picture

anarcat CreditAttribution: anarcat commented

It's not functions we need to override, it's about storage. We need to store data not defined in the class, more specifically:

  $options['bwstats_last_record'] = $last_record[$filename];

That doesn't work, because it's not defined as a setting... See #1 to #3 for the various failed attempts at adding a new storage from outside the class.

sfyn’s picture

sfyn CreditAttribution: sfyn commented
sfyn’s picture

sfyn CreditAttribution: sfyn commented
Assigned: Unassigned » sfyn

Apologies, I mistakenly commented twice to the wrong issue.

sfyn’s picture

sfyn CreditAttribution: sfyn commented

I have found a way to work around this in the back-end. I rely on post_ hooks for drush commands and the drush contexts. In this case, I wanted to save values to the site's drushrc.php file.

I added the following code to hook_post_provision_verify:

if (d()->type == 'site') {
  $val = drush_get_option('my_custom_option');
  drush_set_option('my_custom_option', $val, 'site');
}

This makes sure that my_custom_option perpetuates through these potentially destructive operations. Since it is provision-verify that writes the drushrc.php file for the site, options set to the site context in hook_post_provision_verify will be written to that file.

The final step is to make sure my verify hook is run with the appropriate options at the end of my migrate task, so I added this to hook_post_provision_migrate.

if (d()->type == 'site') {
  $options = array('my_custom_option' => drush_get_option('my_custom_option'));
  $target = drush_get_option('target_name');
  provision_backend_invoke($target, 'provision-verify', array(), $options);
}

I got this idea from looking at the drush_provision_drupal_provision_migrate function in platform/migrate.provision.inc.

I used this approach to create my patch in #1282200: Site key and civicron user/pass is lost after migration

sfyn’s picture

sfyn CreditAttribution: sfyn commented

I have updated my comments in #22 to reduce the amount of extraneous code used. I also created a page in the aegir handbook documenting this approach: http://community.aegirproject.org/node/935

ergonlogic’s picture

ergonlogic
he/him
Primary language English
Location Montréal, Québec 🇨🇦
CreditAttribution: ergonlogic commented
Version: 6.x-1.1 » 7.x-3.x-dev
Assigned: sfyn » Unassigned
Status: Needs work » Active

See: #2103173: Provide an example of saving data to a site context for an example of implementing a minimalist service to save data into a site context.

ergonlogic’s picture

ergonlogic
he/him
Primary language English
Location Montréal, Québec 🇨🇦
CreditAttribution: ergonlogic commented
Status: Active » Closed (works as designed)

Actually, I guess we can close this? You currently need a service to write to a context. Changing that will probably require us to #2043419: [meta] Re-think our object models & design patterns.

  • Commit fa5f25b on dev-ssl-ip-allocation-refactor, dev-1205458-move_sites_out_of_platforms, 7.x-3.x, dev-subdir-multiserver, 6.x-2.x-backports, dev-helmo-3.x by anarcat: 
    #1068660 - disable the site during migration

because of #955018,...
  • Commit f91244a on 6.x-1.x, dev-ssl-ip-allocation-refactor, dev-1205458-move_sites_out_of_platforms, 7.x-3.x, dev-subdir-multiserver, 6.x-2.x-backports, dev-helmo-3.x by anarcat:
    #1068660 - disable the site during migration

because of #955018,...

  • anarcat committed fa5f25b on 
    #1068660 - disable the site during migration

because of #955018,...
  • anarcat committed f91244a on 
    #1068660 - disable the site during migration

because of #955018,...

  • anarcat committed fa5f25b on 7.x-3.x-1966886-context-to-entity 
    #1068660 - disable the site during migration

because of #955018,...
  • anarcat committed f91244a on 7.x-3.x-1966886-context-to-entity
    #1068660 - disable the site during migration

because of #955018,...

  • anarcat committed fa5f25b on 6.x-2.x-1995506-profile-option 
    #1068660 - disable the site during migration

because of #955018,...
  • anarcat committed f91244a on 6.x-2.x-1995506-profile-option
    #1068660 - disable the site during migration

because of #955018,...