Tasks in queue aren't running in the Master Instance

I'm experiencing this as well so far on 2 out of 2 servers it's happening

I've just upgraded to 2.2.1 and tasks still aren't running.

The time that they are expected to run just keeps incrementing.

We can't reproduce this, but please try to run manually and let us know if this results with any visible errors on command line:

$ bash /var/xdrago/run-USER (replace USER with your instance system username)

I've the same problem after upgrade to 2.2.1

Never mind! After running barracuda, my queues had stopped getting processed. But I hadn't actually run the instance upgrade with:

octopus up-stable o1

I guess I'd assumed running barracuda upgraded instances too, but the docs clearly state what to do.

Yep, you have to upgrade all Satellite instances after running Master instance upgrade. We have tried to minimize major upgrade side effects, so not yet upgraded Satellite instances continue to work after Master upgrade (the websites), but things like backend tasks will stop working until you will upgrade also Satellite instance(s).

I think we can close this?

This is debug output from a server that was updated to 2.2.0 and then 2.2.1 where tasks are still not running

drush '@hostmaster' hosting-dispatch -d
Bootstrap to phase 0. [0.07 sec, 1.6 MB]                             [bootstrap]
Drush bootstrap phase : _drush_bootstrap_drush() [0.08 sec, 1.72 MB] [bootstrap]
Load alias @hostmaster [0.08 sec, 1.72 MB]                                                                      [notice]
Loading drushrc "/var/aegir/host_master/014/sites/a01.realityloop.com/drushrc.php" into "site" scope. [0.09  [bootstrap]
sec, 1.73 MB]
Bootstrap to phase 5. [0.25 sec, 5.69 MB]                                                                    [bootstrap]
Drush bootstrap phase : _drush_bootstrap_drupal_root() [0.25 sec, 5.69 MB]                                   [bootstrap]
Loading drushrc "/var/aegir/host_master/014/drushrc.php" into "drupal" scope. [0.25 sec, 5.69 MB]            [bootstrap]
Initialized Drupal 6.30 root directory at /var/aegir/host_master/014 [0.28 sec, 6.48 MB]                        [notice]
Drush bootstrap phase : _drush_bootstrap_drupal_site() [0.28 sec, 6.48 MB]                                   [bootstrap]
Initialized Drupal site a01.realityloop.com at sites/a01.realityloop.com [0.28 sec, 6.48 MB]                    [notice]
Loading drushrc "/var/aegir/host_master/014/sites/a01.realityloop.com/drushrc.php" into "site" scope. [0.28  [bootstrap]
sec, 6.48 MB]
Drush bootstrap phase : _drush_bootstrap_drupal_configuration() [0.3 sec, 6.48 MB]                           [bootstrap]
Drush bootstrap phase : _drush_bootstrap_drupal_database() [0.31 sec, 6.51 MB]                               [bootstrap]
Successfully connected to the Drupal database. [0.31 sec, 6.51 MB]                                           [bootstrap]
Drush bootstrap phase : _drush_bootstrap_drupal_full() [0.31 sec, 6.75 MB]                                   [bootstrap]
Bootstrap to phase 5. [0.5 sec, 12.54 MB]                                                                    [bootstrap]
Found command: hosting-dispatch (commandfile=hosting) [0.5 sec, 12.54 MB]                                    [bootstrap]
Initializing drush commandfile: db [0.5 sec, 12.54 MB]                                                       [bootstrap]
Initializing drush commandfile: dns [0.5 sec, 12.55 MB]                                                      [bootstrap]
Initializing drush commandfile: drush_make [0.5 sec, 12.55 MB]                                               [bootstrap]
Initializing drush commandfile: drush_make_d_o [0.5 sec, 12.55 MB]                                           [bootstrap]
Initializing drush commandfile: example [0.5 sec, 12.55 MB]                                                  [bootstrap]
Initializing drush commandfile: http [0.5 sec, 12.55 MB]                                                     [bootstrap]
Initializing drush commandfile: provision [0.5 sec, 12.55 MB]                                                [bootstrap]
Load alias @server_localhost [0.5 sec, 12.56 MB]                                                                [notice]
Load alias @server_master [0.52 sec, 12.56 MB]                                                                  [notice]
Loading nginx driver for the http service [0.54 sec, 12.63 MB]                                                  [notice]
Loading nginx driver for the cdn service [0.54 sec, 12.65 MB]                                                   [notice]
Loading mysql driver for the db service [0.55 sec, 12.75 MB]                                                    [notice]
Loading nginx driver for the cdn service [0.55 sec, 12.76 MB]                                                   [notice]
Load alias @platform_014 [0.55 sec, 12.76 MB]                                                                   [notice]
Initializing drush commandfile: provision_cdn [0.57 sec, 12.76 MB]                                           [bootstrap]
Including profiles/hostmaster/modules/hosting/dispatch.hosting.inc [0.57 sec, 12.76 MB]                      [bootstrap]
dispatching queues [0.57 sec, 12.81 MB]                                                                         [notice]
found  running tasks, starting 2 out of 2 items [0.58 sec, 13.64 MB]                                            [notice]
Running: /opt/php53/bin/php /opt/tools/drush/4/drush/drush.php --php='/opt/php53/bin/php'  --items='2'         [command]
--invoke --context_type='site' --platform='@platform_014' --server='@server_master'
--db_server='@server_localhost' --uri='a01.realityloop.com' --root='/var/aegir/host_master/014'
--site_path='/var/aegir/host_master/014/sites/a01.realityloop.com' --site_enabled --language='en'
--client_name='admin' --profile='hostmaster' --loaded-config hosting-cron --backend  2>&1 [0.58 sec, 13.65
MB]
Bootstrap to phase 0. [92.16 sec, 14.52 MB]                                                                  [bootstrap]
Drush bootstrap phase : _drush_bootstrap_drush() [92.16 sec, 14.52 MB]                                       [bootstrap]
Bootstrap to phase 5. [92.16 sec, 14.52 MB]                                                                  [bootstrap]
Drush bootstrap phase : _drush_bootstrap_drupal_root() [92.16 sec, 14.52 MB]                                 [bootstrap]
Loading drushrc "/var/aegir/host_master/014/drushrc.php" into "drupal" scope. [92.16 sec, 14.52 MB]          [bootstrap]
Initialized Drupal 6.30 root directory at /var/aegir/host_master/014 [92.16 sec, 14.52 MB]                      [notice]
Drush bootstrap phase : _drush_bootstrap_drupal_site() [92.16 sec, 14.52 MB]                                 [bootstrap]
Initialized Drupal site a01.realityloop.com at sites/a01.realityloop.com [92.16 sec, 14.52 MB]                  [notice]
Loading drushrc "/var/aegir/host_master/014/sites/a01.realityloop.com/drushrc.php" into "site" scope. [92.16 [bootstrap]
sec, 14.52 MB]
Drush bootstrap phase : _drush_bootstrap_drupal_configuration() [92.16 sec, 14.53 MB]                        [bootstrap]
Drush bootstrap phase : _drush_bootstrap_drupal_database() [92.16 sec, 14.53 MB]                             [bootstrap]
Successfully connected to the Drupal database. [92.16 sec, 14.53 MB]                                         [bootstrap]
Drush bootstrap phase : _drush_bootstrap_drupal_full() [92.16 sec, 14.53 MB]                                 [bootstrap]
Bootstrap to phase 6. [92.16 sec, 14.53 MB]                                                                  [bootstrap]
Drush bootstrap phase : _drush_bootstrap_drupal_login() [92.16 sec, 14.53 MB]                                [bootstrap]
Successfully logged into Drupal as Anonymous (uid=0) [92.16 sec, 14.53 MB]                                   [bootstrap]
Found command: hosting-cron (commandfile=hosting) [92.16 sec, 14.53 MB]                                      [bootstrap]
Running: /opt/php53/bin/php /opt/tools/drush/4/drush/drush.php --php='/opt/php53/bin/php'  @www.nsla.org.au    [command]
cron --backend  2>&1 [92.16 sec, 14.53 MB]
Bootstrap to phase 0. [92.16 sec, 14.53 MB]                                                                  [bootstrap]
Drush bootstrap phase : _drush_bootstrap_drush() [92.16 sec, 14.53 MB]                                       [bootstrap]
Load alias @www.nsla.org.au [92.16 sec, 14.53 MB]                                                               [notice]
Loading drushrc "/var/aegir/platforms/nsla-7.0.6/sites/www.nsla.org.au/drushrc.php" into "site" scope. [92.16[bootstrap]
sec, 14.53 MB]
Bootstrap to phase 6. [92.16 sec, 14.53 MB]                                                                  [bootstrap]
Drush bootstrap phase : _drush_bootstrap_drupal_root() [92.16 sec, 14.53 MB]                                 [bootstrap]
Loading drushrc "/var/aegir/platforms/nsla-7.0.6/drushrc.php" into "drupal" scope. [92.16 sec, 14.53 MB]     [bootstrap]
Initialized Drupal 7.26 root directory at /var/aegir/platforms/nsla-7.0.6 [92.16 sec, 14.53 MB]                 [notice]
Drush bootstrap phase : _drush_bootstrap_drupal_site() [92.16 sec, 14.54 MB]                                 [bootstrap]
Initialized Drupal site www.nsla.org.au at sites/www.nsla.org.au [92.16 sec, 14.54 MB]                          [notice]
Loading drushrc "/var/aegir/platforms/nsla-7.0.6/sites/www.nsla.org.au/drushrc.php" into "site" scope. [92.16[bootstrap]
sec, 14.54 MB]
Drush bootstrap phase : _drush_bootstrap_drupal_configuration() [92.16 sec, 14.54 MB]                        [bootstrap]
Drush bootstrap phase : _drush_bootstrap_drupal_database() [92.16 sec, 14.54 MB]                             [bootstrap]
Successfully connected to the Drupal database. [92.16 sec, 14.54 MB]                                         [bootstrap]
Drush bootstrap phase : _drush_bootstrap_drupal_full() [92.16 sec, 14.54 MB]                                 [bootstrap]
Drush bootstrap phase : _drush_bootstrap_drupal_login() [92.16 sec, 14.54 MB]                                [bootstrap]
Successfully logged into Drupal as Anonymous (uid=0) [92.16 sec, 14.54 MB]                                   [bootstrap]
Found command: core-cron (commandfile=core) [92.16 sec, 14.54 MB]                                            [bootstrap]
Initializing drush commandfile: db [92.16 sec, 14.54 MB]                                                     [bootstrap]
Initializing drush commandfile: dns [92.16 sec, 14.54 MB]                                                    [bootstrap]
Initializing drush commandfile: drush_make [92.16 sec, 14.54 MB]                                             [bootstrap]
Initializing drush commandfile: drush_make_d_o [92.16 sec, 14.54 MB]                                         [bootstrap]
Initializing drush commandfile: example [92.16 sec, 14.54 MB]                                                [bootstrap]
Initializing drush commandfile: http [92.16 sec, 14.54 MB]                                                   [bootstrap]
Initializing drush commandfile: provision [92.16 sec, 14.54 MB]                                              [bootstrap]
Load alias @server_localhost [92.16 sec, 14.54 MB]                                                              [notice]
Load alias @server_master [92.16 sec, 14.55 MB]                                                                 [notice]
Loading nginx driver for the http service [92.16 sec, 14.55 MB]                                                 [notice]
Loading nginx driver for the cdn service [92.16 sec, 14.55 MB]                                                  [notice]
Loading mysql driver for the db service [92.16 sec, 14.55 MB]                                                   [notice]
Loading nginx driver for the cdn service [92.16 sec, 14.55 MB]                                                  [notice]
Load alias @platform_nsla706 [92.16 sec, 14.55 MB]                                                              [notice]
Initializing drush commandfile: provision_cdn [92.16 sec, 14.55 MB]                                          [bootstrap]
WD job_scheduler: Finished processing scheduled jobs (0 sec s, 12 total, 0 failed). [92.16 sec, 14.55 MB]       [notice]
WD xmlsitemap: Finished XML sitemap generation in 0 sec. Memory usage: 46.5 MB. [92.16 sec, 14.55 MB]           [notice]
WD cron: Cron run completed. [92.16 sec, 14.55 MB]                                                              [notice]
Cron run successfully. [92.16 sec, 14.55 MB]                                                                 [success]
Created 2 nodes. [92.16 sec, 14.55 MB]                                                                       [status]
There are no new nodes. [92.16 sec, 14.55 MB]                                                                [status]
Created 36 nodes. [92.16 sec, 14.55 MB]                                                                      [status]
There are no new nodes. [92.16 sec, 14.55 MB]                                                                [status]
Command dispatch complete [92.17 sec, 14.55 MB]                                                                 [notice]
Peak memory usage was 50.02 MB [92.17 sec, 14.55 MB]                                                            [memory]
Running: /opt/php53/bin/php /opt/tools/drush/4/drush/drush.php --php='/opt/php53/bin/php'                      [command]
@dee-village.org.au cron --backend  2>&1 [92.17 sec, 14.55 MB]
Bootstrap to phase 0. [92.17 sec, 14.56 MB]                                                                  [bootstrap]
Drush bootstrap phase : _drush_bootstrap_drush() [92.17 sec, 14.56 MB]                                       [bootstrap]
Load alias @dee-village.org.au [92.17 sec, 14.56 MB]                                                            [notice]
Loading drushrc "/var/aegir/platforms/dee-village-2.x/sites/dee-village.org.au/drushrc.php" into "site"      [bootstrap]
scope. [92.17 sec, 14.56 MB]
Bootstrap to phase 6. [92.17 sec, 14.56 MB]                                                                  [bootstrap]
Drush bootstrap phase : _drush_bootstrap_drupal_root() [92.17 sec, 14.56 MB]                                 [bootstrap]
Loading drushrc "/var/aegir/platforms/dee-village-2.x/drushrc.php" into "drupal" scope. [92.17 sec, 14.56 MB][bootstrap]
Initialized Drupal 7.12 root directory at /var/aegir/platforms/dee-village-2.x [92.17 sec, 14.56 MB]            [notice]
Drush bootstrap phase : _drush_bootstrap_drupal_site() [92.17 sec, 14.56 MB]                                 [bootstrap]
Initialized Drupal site dee-village.org.au at sites/dee-village.org.au [92.17 sec, 14.56 MB]                    [notice]
Loading drushrc "/var/aegir/platforms/dee-village-2.x/sites/dee-village.org.au/drushrc.php" into "site"      [bootstrap]
scope. [92.17 sec, 14.56 MB]
Drush bootstrap phase : _drush_bootstrap_drupal_configuration() [92.17 sec, 14.56 MB]                        [bootstrap]
Drush bootstrap phase : _drush_bootstrap_drupal_database() [92.17 sec, 14.56 MB]                             [bootstrap]
Successfully connected to the Drupal database. [92.17 sec, 14.56 MB]                                         [bootstrap]
Drush bootstrap phase : _drush_bootstrap_drupal_full() [92.17 sec, 14.56 MB]                                 [bootstrap]
Drush bootstrap phase : _drush_bootstrap_drupal_login() [92.17 sec, 14.56 MB]                                [bootstrap]
Successfully logged into Drupal as Visitor (uid=0) [92.17 sec, 14.56 MB]                                     [bootstrap]
Found command: core-cron (commandfile=core) [92.17 sec, 14.57 MB]                                            [bootstrap]
Initializing drush commandfile: db [92.17 sec, 14.57 MB]                                                     [bootstrap]
Initializing drush commandfile: dns [92.17 sec, 14.57 MB]                                                    [bootstrap]
Initializing drush commandfile: drush_make [92.17 sec, 14.57 MB]                                             [bootstrap]
Initializing drush commandfile: drush_make_d_o [92.17 sec, 14.57 MB]                                         [bootstrap]
Initializing drush commandfile: example [92.17 sec, 14.57 MB]                                                [bootstrap]
Initializing drush commandfile: http [92.17 sec, 14.57 MB]                                                   [bootstrap]
Initializing drush commandfile: provision [92.17 sec, 14.57 MB]                                              [bootstrap]
Load alias @server_localhost [92.17 sec, 14.57 MB]                                                              [notice]
Load alias @server_master [92.17 sec, 14.57 MB]                                                                 [notice]
Loading nginx driver for the http service [92.17 sec, 14.57 MB]                                                 [notice]
Loading nginx driver for the cdn service [92.17 sec, 14.57 MB]                                                  [notice]
Loading mysql driver for the db service [92.17 sec, 14.57 MB]                                                   [notice]
Loading nginx driver for the cdn service [92.17 sec, 14.57 MB]                                                  [notice]
Load alias @platform_DeeVillage2x [92.17 sec, 14.57 MB]                                                         [notice]
Initializing drush commandfile: provision_cdn [92.17 sec, 14.57 MB]                                          [bootstrap]
WD twitter: exception 'TwitterException' with message 'Not Found' in                                         [error]
/var/aegir/platforms/dee-village-2.x/profiles/deevillage_profile/modules/contrib/twitter/twitter.lib.php:250
Stack trace:
#0
/var/aegir/platforms/dee-village-2.x/profiles/deevillage_profile/modules/contrib/twitter/twitter.lib.php(193):
Twitter->request('http://twitter....', Array, 'GET')
#1
/var/aegir/platforms/dee-village-2.x/profiles/deevillage_profile/modules/contrib/twitter/twitter.lib.php(78):
Twitter->call('statuses/user_t...', Array, 'GET', '0')
#2
/var/aegir/platforms/dee-village-2.x/profiles/deevillage_profile/modules/contrib/twitter/twitter.lib.php(126):
Twitter->get_statuses('statuses/user_t...', Array, '0')
#3 /var/aegir/platforms/dee-village-2.x/profiles/deevillage_profile/modules/contrib/twitter/twitter.inc(128):
Twitter->user_timeline('96704918', Array, '0')
#4
/var/aegir/platforms/dee-village-2.x/profiles/deevillage_profile/modules/contrib/twitter/twitter.module(123):
twitter_fetch_user_timeline('96704918')
#5 [internal function]: twitter_cron()
#6 /var/aegir/platforms/dee-village-2.x/includes/module.inc(795): call_user_func_array('twitter_cron', Array)
#7 /var/aegir/platforms/dee-village-2.x/includes/common.inc(5121): module_invoke('twitter', 'cron')
#8 /opt/tools/drush/4/drush/commands/core/core.drush.inc(520): drupal_cron_run()
#9 [internal function]: drush_core_cron()
#10 /opt/tools/drush/4/drush/includes/command.inc(131): call_user_func_array('drush_core_cron', Array)
#11 /opt/tools/drush/4/drush/includes/command.inc(836): _drush_invoke_args('core-cron', Array, 'core')
#12 [internal function]: drush_command()
#13 /opt/tools/drush/4/drush/includes/command.inc(236): call_user_func_array('drush_command', Array)
#14 /opt/tools/drush/4/drush/drush.php(101): drush_dispatch(Array)
#15 /opt/tools/drush/4/drush/drush.php(41): drush_main()
#16 {main} [92.17 sec, 14.58 MB]
WD cron: Cron run completed. [92.17 sec, 14.58 MB]                                                              [notice]
WD aggregator: There is new syndicated content from Medical News Today. [92.17 sec, 14.58 MB]                   [notice]
WD aggregator: There is new syndicated content from Psych Central News. [92.17 sec, 14.58 MB]                   [notice]
WD aggregator: There is new syndicated content from The Behavioral Medicine Report. [92.17 sec, 14.58 MB]       [notice]
WD aggregator: The feed from Yahoo News- Mental Health seems to be broken, due to an error "Invalid document [warning]
end" on line 2. [92.17 sec, 14.58 MB]
WD aggregator: The feed from SANE Australia- tweets seems to be broken, due to "404 Not Found". [92.17 sec,  [warning]
14.58 MB]
WD aggregator: The feed from Twitter Mental Health News seems to be broken, due to an error                  [warning]
"XML_ERR_NAME_REQUIRED" on line 17. [92.17 sec, 14.58 MB]
WD aggregator: The feed from All in the Mind- ABC News seems to be broken, due to "-1002 missing schema".    [warning]
[92.17 sec, 14.58 MB]
Cron run successfully. [92.17 sec, 14.58 MB]                                                                 [success]
The feed from Yahoo News- Mental Health seems to be broken, because of error "Invalid document end" on line  [error]
2. [92.17 sec, 14.58 MB]
The feed from Twitter Mental Health News seems to be broken, because of error "XML_ERR_NAME_REQUIRED" on line[error]
17. [92.17 sec, 14.58 MB]
There is new syndicated content from Medical News Today. [92.17 sec, 14.58 MB]                               [status]
There is new syndicated content from Psych Central News. [92.17 sec, 14.58 MB]                               [status]
There is new syndicated content from The Behavioral Medicine Report. [92.17 sec, 14.59 MB]                   [status]
The feed from SANE Australia- tweets seems to be broken, because of error "404 Not Found". [92.17 sec, 14.59 [status]
MB]
There is no new syndicated content from SANE Australia- tweets. [92.17 sec, 14.59 MB]                        [status]
There is no new syndicated content from Mental Health and Movies. [92.17 sec, 14.59 MB]                      [status]
The feed from All in the Mind- ABC News seems to be broken, because of error "-1002 missing schema". [92.17  [status]
sec, 14.59 MB]
There is no new syndicated content from All in the Mind- ABC News. [92.17 sec, 14.59 MB]                     [status]
Command dispatch complete [92.17 sec, 14.59 MB]                                                                 [notice]
Peak memory usage was 32.2 MB [92.17 sec, 14.59 MB]                                                             [memory]
Command dispatch complete [92.17 sec, 14.59 MB]                                                                 [notice]
Peak memory usage was 16.59 MB [92.17 sec, 14.59 MB]                                                            [memory]
found 0 running tasks, starting 2 out of 2 items [92.17 sec, 13.72 MB]                                          [notice]
Running: /opt/php53/bin/php /opt/tools/drush/4/drush/drush.php --php='/opt/php53/bin/php'  --items='2'         [command]
--invoke --context_type='site' --platform='@platform_014' --server='@server_master'
--db_server='@server_localhost' --uri='a01.realityloop.com' --root='/var/aegir/host_master/014'
--site_path='/var/aegir/host_master/014/sites/a01.realityloop.com' --site_enabled --language='en'
--client_name='admin' --profile='hostmaster' --loaded-config hosting-tasks --backend  2>&1 [92.17 sec, 13.73
MB]
Bootstrap to phase 0. [92.63 sec, 14.49 MB]                                                                  [bootstrap]
Drush bootstrap phase : _drush_bootstrap_drush() [92.63 sec, 14.49 MB]                                       [bootstrap]
Bootstrap to phase 5. [92.63 sec, 14.49 MB]                                                                  [bootstrap]
Drush bootstrap phase : _drush_bootstrap_drupal_root() [92.63 sec, 14.49 MB]                                 [bootstrap]
Loading drushrc "/var/aegir/host_master/014/drushrc.php" into "drupal" scope. [92.63 sec, 14.49 MB]          [bootstrap]
Initialized Drupal 6.30 root directory at /var/aegir/host_master/014 [92.63 sec, 14.49 MB]                      [notice]
Drush bootstrap phase : _drush_bootstrap_drupal_site() [92.63 sec, 14.49 MB]                                 [bootstrap]
Initialized Drupal site a01.realityloop.com at sites/a01.realityloop.com [92.63 sec, 14.49 MB]                  [notice]
Loading drushrc "/var/aegir/host_master/014/sites/a01.realityloop.com/drushrc.php" into "site" scope. [92.63 [bootstrap]
sec, 14.49 MB]
Drush bootstrap phase : _drush_bootstrap_drupal_configuration() [92.63 sec, 14.49 MB]                        [bootstrap]
Drush bootstrap phase : _drush_bootstrap_drupal_database() [92.63 sec, 14.49 MB]                             [bootstrap]
Successfully connected to the Drupal database. [92.63 sec, 14.49 MB]                                         [bootstrap]
Drush bootstrap phase : _drush_bootstrap_drupal_full() [92.63 sec, 14.5 MB]                                  [bootstrap]
Bootstrap to phase 6. [92.63 sec, 14.5 MB]                                                                   [bootstrap]
Drush bootstrap phase : _drush_bootstrap_drupal_login() [92.63 sec, 14.5 MB]                                 [bootstrap]
Successfully logged into Drupal as Anonymous (uid=0) [92.63 sec, 14.5 MB]                                    [bootstrap]
Found command: hosting-tasks (commandfile=hosting) [92.63 sec, 14.5 MB]                                      [bootstrap]
Running tasks queue [92.63 sec, 14.5 MB]                                                                        [notice]
Command dispatch complete [92.63 sec, 14.5 MB]                                                                  [notice]
Peak memory usage was 13.22 MB [92.63 sec, 14.5 MB]                                                             [memory]
Command dispatch complete [92.63 sec, 13.69 MB]                                                                 [notice]
 Timer  Cum (sec)  Count  Avg (msec) 
 page   92.33      1      92330.4    

Peak memory usage was 14.75 MB [92.64 sec, 13.69 MB]                                                            [memory]
aegir@h01:~$ 

a01 was upgraded via 2.2.0, a02 was straight to 2.2.1

This is a bug report. Tasks queue doesn't work on the Master Instance.

Fixed in HEAD: https://github.com/omega8cc/boa/commit/29974d2070b3bd9c9cc9ed9b85c583f71...

On stable please run the hot fix: https://github.com/omega8cc/boa/blob/master/CHANGELOG.txt#L39

Tasks are processing again, thanks, also.. I did get the following output from the script:

sed: can't read /data/disk/*/.drush/provision/platform/verify.provision.inc: No such file or directory

This error is expected when you didn't install Octopus.

Pardon me for reopening this issue, but I just upgraded BOA-2.2.2 (both Barracuda and Octopus up-stable all) on my Linode Ubuntu 10.04 VPS and my queues are not executing either. (I skipped BOA-2.2.0 and BOA-2.2.1)

I tried running bash /var/xdrago/run-wwm-hosting-admin and got this output:

load is 120 while maxload is 188
... now doing CTL...
Unknown id: wwm-hosting-admin
CTL done

Not sure what to do now.

Thanks for your help in advance!
Steve

P.S. You will see in my barracuda_log.txt file that I ran the update 3 times today. The first time I forgot to change _SSL_FROM_SOURCES to YES, but it upgrade my OpenSSH anyway. But just to be sure I ran it again with SSL_FROM_SOURCES set to YES. I ran it again when with _STRICT_BIN_PERMISSIONS=NO because I could no longer use any commands in the Linode non-root user. See this issue: https://drupal.org/node/2236425

Actually upon further inspection of the last runs on /admin/hosting/queues, it appears that none of the queues (Task, Advanced Cron, Backup, Backup GC, and Task GC) have run since I did the upgrade this morning. Cron Queue is set to run every88 weeks and I have CiviCRM sites installed so that's why they are not included.

Please enable _STRICT_BIN_PERMISSIONS=YES, since you can't really disable this option once it was already used. Then run the upgrade again.

And then maybe read: https://github.com/omega8cc/boa/issues/296

Also, does your Octopus system user is "wwm-hosting-admin" ? The system tells you that there is no such user in the system: "Unknown id: wwm-hosting-admin"

That's what I thought it meant, but I am logged in on my Aegir Hosting System by Octopus as user "wwm-hosting-admin" and this is the user I have been using now for years, so I don't know why it would not be recognized now on the server side. However, for some reason I have 2 files in /var/xdrago/ - run-wwm-hosting-admin and run-wwmhosting-admin.

It's been a very long time since I originally set my BOA up, and I do recall having some issue with my user naming, so this is probably coming back to bite me now.

Also, I did read https://github.com/omega8cc/boa/issues/296, but failed to mention that whitelisting my non-BOA user solved the issue. However, i did not re-enable _STRICT_BIN_PERMISSIONS. I have done that now and I have re-run the barracuda up-stable.

After that I re-ran octopus up-stable all and got message saying it was already up to date, but I could force reinstall of Aegir, Platforms or Both. I decided to force reinstall just Aegir.

Upon doing this I noticed when it loaded the /root/.wwmhosting-admin.octopus.cnf that _USER="wwmhosting-admin".

Also, the queues starting running again.

So I don't know what has gone wrong here, but I guess the question now is, do I have username issue here and if so how do I resolve this issue? I will start another thread if would like me to.

Thanks for your help!
Steve

Just to chime in on this. I finished an upgrade from 2.1.3 > 2.2.2 yesterday and I just checked 1 of my two sites running on an o1 instance. Cron run says it's been 1 day and 5 hours. So I'm assuming, without having checked on the box yet, that cron runs are not happening for me.

I don't know if this is related to my other issue of #2237267: Troubleshooting https for aegir instance but it would seem that tasks have broken on my upgrade too.

@wickwood Having multiple dashes in the important system usernames is a bad practice. We can't really provide any assistance if you have managed to confuse both BOA and yourself to have scripts created for non-existing accounts, sorry.

Please don't re-open this *fixed* issue with incomplete and confusing reports. Instead make sure to debug this on your end and submit proper new issue with all required config files included, and in the correct queue.

My apologies, Omega8cc.

I did not intend to confuse or offend, indeed I was trying to do to the opposite.

Also, I did not know that having multiple dashes in the important system usernames is a bad practice that could cause an issue for BOA. I will open another issue later after I have done some additional debugging on my end.

Sincerely,
Steve

server:~# bash /var/xdrago/run-NAME
load is 184 while maxload is 350
... now doing CTL...
/data/disk/NAME/aegir.sh: line 4: /opt/local/bin/php: No such file or directory
CTL done

Investigation saw that /opt/local/bin/php was infact /opt/local/bin/php-old

I renamed and worked okay. Queues ran.

Chive working again too

Please read and follow docs and don't reopen already fixed issues just because you didn't read the docs or previous comments yet.