My environment: I use drush 2.0 and provision / hosting / hostmaster RC4
LAMP 2gig dual core

Hi,

i just needed to restart a server after adding the dispatch cron. When running dispatch manually with the debug option:

php '/var/spool/www/user/aegir/drush/drush.php' -d hosting dispatch --root='/var/spool/www/user/aegir/drupal-6.13' --uri=aegir.domain.nl

Drush bootstrap phase : _drush_bootstrap_drupal_root() [0.508 sec]   [bootstrap]
Loading drushrc "/var/spool/www/user/aegir/drupal-6.13/drushrc.php"  [bootstrap]
into "drupal" scope. [0.53 sec]
Initialized Drupal 6.13 root directory at                               [notice]
/var/spool/www/user/aegir/drupal-6.13 [0.636 sec]
Drush bootstrap phase : _drush_bootstrap_drupal_site() [0.668 sec]   [bootstrap]
Initialized Drupal site  at sites/aegir.domain.nl [0.708 sec]          [notice]
Loading drushrc                                                      [bootstrap]
"/var/spool/www/user/aegir/drupal-6.13/sites/aegir.domain.nl/drushrc.php"
into "site" scope. [0.708 sec]
Drush bootstrap phase : _drush_bootstrap_drupal_configuration()      [bootstrap]
[0.821 sec]
Drush bootstrap phase : _drush_bootstrap_drupal_database() [0.861    [bootstrap]
sec]
Successfully connected to the Drupal database. [0.861 sec]           [bootstrap]
Drush bootstrap phase : _drush_bootstrap_drupal_full() [0.899 sec]   [bootstrap]
Drush bootstrap phase : _drush_bootstrap_drupal_login() [2.646 sec]  [bootstrap]
Found command: hosting dispatch [2.69 sec]                           [bootstrap]
hosting_dispatch [2.702 sec]                                         [dispatching queues]
Forking : (php /var/spool/www/user/aegir/drush/drush.php  --items='2'   [notice]
--quiet --root='/var/spool/www/user/aegir/drupal-6.13'
--uri='aegir.domain.nl'  'hosting' 'cron' --backend &) > /dev/null
[2.918 sec]
Forking : (php /var/spool/www/user/aegir/drush/drush.php                [notice]
--items='20' --quiet --root='/var/spool/www/user/aegir/drupal-6.13'
--uri='aegir.domain.nl'  'hosting' 'tasks' --backend &) > /dev/null
[2.925 sec]
Command dispatch complete [2.932 sec]                                   [notice]

A couple of minutes later i get:

[aegir@www15 aegir]$ PHP Fatal error:  Allowed memory size of 77594624 bytes exhausted (tried to allocate 19 bytes) in /var/spool/www/user/aegir/drupal-6.13/includes/database.mysql.inc on line 144

regards Volkan

Comments

Anonymous’s picture

Anonymous (not verified) CreditAttribution: Anonymous commented

Were there any tasks in the queue at the time to be executed by the dispatch, and what were they?

Edit: nevermind I just scrolled up to an IRC conversation I missed.

anarcat’s picture

anarcat CreditAttribution: anarcat commented
Status: Active » Postponed (maintainer needs more info)

Maybe you were suffering from #500362: dispatcher locking.. were multiple dispatch processes running simultaneously?

What is the cron schedule?

muschpusch’s picture

muschpusch CreditAttribution: muschpusch commented

I think i found a part of the issue. It's related to this Error generating virtual host config causes generation of 10 000 ServerAlias entries2. I checked the slow query log and found this:
[code]
# User@Host: hostmaster_1[hostmaster_1] @ localhost []
# Query_time: 309 Lock_time: 0 Rows_sent: 706581 Rows_examined: 1413162
SELECT alias FROM hosting_site_alias WHERE vid=156 AND automatic=0 ORDER BY alias ASC;
[/code]

After examining hosting_site_alias i deleted the 700000 identical alias records. This stops the memory limit error but a high load remains when dispatching by cron. I can't find new slow queries but maybe it's #500362: dispatching lock

I'm now running aegir on a busy machine and will migrate it to a newer one in the next couple of days.

EDIT: i have no items in the task queue but 7 in the cron queue (no idea which one... .Where i can i check that?)

anarcat’s picture

anarcat CreditAttribution: anarcat commented

How many dispatch tasks are running in the backend? If there's more than one, then yeah, you may be suffering from the lack of locking.

anarcat’s picture

anarcat CreditAttribution: anarcat commented
Status: Postponed (maintainer needs more info) » Fixed

I have made improvements to the dispatcher code that should alleviate this problem, see #500362: dispatcher locking. Reopen only if the problem still occurs on HEAD.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

  • Commit 3a7731b on 6.x-2.x, 7.x-3.x, dev-ssl-ip-allocation-refactor, dev-sni, dev-helmo-3.x by anarcat: 
    don't start more than N tasks in parallel

this should help with #559006...

  • Commit 3a7731b on 6.x-2.x, 7.x-3.x, dev-ssl-ip-allocation-refactor, dev-sni, dev-helmo-3.x by anarcat: 
    don't start more than N tasks in parallel

this should help with #559006...