Amazon Linux appears to be derived from RHEL, and uses yum/rpm for package management. The particular release in this case is: https://aws.amazon.com/amazon-linux-ami/2013.09-release-notes/

The error in question is:

pcntl_exec(): Error has occured: (errno 13) Permission denied hosting_queued.drush.inc:172                                                [warning]
WD hosting_queued: Could not restart the queue daemon, aborting.                                                                          [error]
Drush command terminated abnormally due to an unrecoverable error. 

The code in question is:

  // We need the PCNTL extension to be able to auto restart.
  if (function_exists('pcntl_exec')) {
    $args = $_ENV['argv'];
    $drush = array_shift($args);

    // Strip sub-array to avoid warning "Array to string conversion"
    unset($_ENV['argv']);

    watchdog('hosting_queued', 'Restarting queue daemon with @drush @args.', array('@drush' => $drush, '@args' => implode(" ", $args)));
    pcntl_exec($drush, $args, $_ENV);
    watchdog('hosting_queued', 'Could not restart the queue daemon, aborting.', array(), WATCHDOG_ERROR);
    /* NOTREACHED */
  }

Watchdog shows that it tried to restart using the following command:

Restarting queue daemon with /usr/share/pear/drush/drush.php --php=/usr/bin/php --php-options= -d magic_quotes_gpc=Off -d magic_quotes_runtime=Off -d magic_quotes_sybase=Off --quiet @hostmaster hosting-queued.
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

ergonlogic’s picture

Issue summary: View changes
ergonlogic’s picture

This appear related to #2077793: Queue daemon collecting quotes. We're still calling drush.php directly, which is not executable on PEAR-based installs.

ergonlogic’s picture

Switching to using drush_build_drush_command():

    $args = $_ENV['argv'];
    //$drush = array_shift($args);
    // Drop 'drush'
    array_shift($args);
    $drush = drush_build_drush_command();

... results in:

pcntl_exec(): Error has occured: (errno 2) No such file or directory hosting_queued.drush.inc:175                                         [warning]
WD hosting_queued: Could not restart the queue daemon, aborting.

... with the watchdog message:

Restarting queue daemon with /usr/bin/php -d magic_quotes_gpc=Off -d magic_quotes_runtime=Off -d magic_quotes_sybase=Off /usr/share/pear/drush/drush.php --php=/usr/bin/php --php-options=' -d magic_quotes_gpc=Off -d magic_quotes_runtime=Off -d magic_quotes_sybase=Off' --php=/usr/bin/php --php-options= -d magic_quotes_gpc=Off -d magic_quotes_runtime=Off -d magic_quotes_sybase=Off --quiet @hostmaster hosting-queued.
ergonlogic’s picture

Status: Active » Needs review
FileSize
1.07 KB

The attached patch is almost certainly the wrong way to do this. It's basically a round-about way to pass '/usr/bin/php' to pcntl_exec(). However, it has the virtue of working. I'm going to to try it on a Debian system, and see how it fares.

ergonlogic’s picture

Status: Needs review » Needs work

This fails on a normal Debian install, as it tries to restart the daemon by running '/usr/share/php/drush/drush.php /usr/bin/drush @hostmaster hosting-queued'

Liam McDermott’s picture

We're still calling drush.php directly, which is not executable on PEAR-based installs.

Hopefully it's not superfluous for me to mention that this is probably the root cause of the bug as I'm seeing it on an Ubuntu installation.

As a workaround I marked /usr/share/php/drush/drush.php executable, haven't seen any errors for a couple of hours and the queue daemon reports it is running.

zhangtaihao’s picture

Based on #2059225: drush.php is not executable with PEAR install, maybe we could adapt the necessary bits of drush_build_drush_command() to obtain PHP binary path (especially given we can't use that function directly for pcntl_exec()).

Or is that not even necessary, i.e. just use drush_get_option('php', PHP_BINDIR . '/php')?

ergonlogic’s picture

Title: Queue daemon fails to restart itself on Amazon Linux » Queue daemon fails to restart itself
Priority: Normal » Major

I'm seeing this on a fresh install on a recent Debian 7. Running drush @hostmaster hosting-queued --debug works fine, but then dies as soon as it tries to restart itself with:

WD hosting_queued: Restarting queue daemon with /usr/share/php/drush/drush.php --php=/usr/bin/php --php-options= -d magic_quotes_gpc=Off     [notice]
-d magic_quotes_runtime=Off -d magic_quotes_sybase=Off @hostmaster hosting-queued --debug. [62.96 sec, 15.78 MB]
pcntl_exec(): Error has occured: (errno 13) Permission denied hosting_queued.drush.inc:172 [62.96 sec, 15.78 MB]                          [warning]
WD hosting_queued: Could not restart the queue daemon, aborting. [62.96 sec, 15.79 MB]                                                    [error]
Drush command terminated abnormally due to an unrecoverable error. [62.96 sec, 15.78 MB]                                                  [error]

Specifying the proper path to Drush in hosting_queued_restart() appears to fix it:

  // We need the PCNTL extension to be able to auto restart.
  if (function_exists('pcntl_exec')) {
    $args = $_ENV['argv'];
    $drush = array_shift($args);
    $drush="/usr/bin/drush";     // Hackily specify the proper path for drush
    // Strip sub-array to avoid warning "Array to string conversion"
    unset($_ENV['argv']);
zhangtaihao’s picture

Yes it does. Unfortunately the path may also be /usr/local/bin/drush (as is the case with the default daemon template). So perhaps a which drush is in order for that case. All of that assumes, of course, that Drush is used and exists as an executable shell script and not a PHP script (e.g. /usr/share/pear/drush/drush.php) executed by /usr/bin/php.

EDIT: And without piping the command through Drush internals we might be missing custom Drush arguments (especially if the daemon has been tailored for the hosting solution).

ergonlogic’s picture

Status: Needs work » Closed (works as designed)

Closing for lack of activity. Consider moving to https://github.com/getvalkyrie/skynet

esolitos’s picture

Status: Closed (works as designed) » Active

If this could be solved it could very useful! Currently I had to use another daemon to check if the hosting-queued is still running, since it always crashes each time it tries to restart.

System: Ubuntu

Wayne Oliver’s picture

like esolitos, I have puppet making sure it's started.
However it dies exactly 1 hour after being started.

ubuntu 14.04
drush 6.2
hostmaster 2.1

Anybody want some more info or want me to test something?

jpwester’s picture

I am having this exact issue. I have limited access to the server, so I need some sort of sustainable way to get this working while keeping the standard PEAR-sourced drush. Any progress?

esolitos’s picture

However it dies exactly 1 hour after being started.

Yes, it's because it is nor able to restart itself, you can change this time to 24h or something, but beware that there will higher memory usage by the daemon, since php leaks some memory with time.

jpwester’s picture

The patch in #7 worked for me.

helmo’s picture

Version: 6.x-2.x-dev » 7.x-3.x-dev
Status: Active » Needs review
FileSize
4.85 KB
470 bytes

Two patches ... one that improves the daemon a bit to survive a restarting DB server. Adding more try {} catch {}

And the other to actually notice that it's lifetime is over .... in the D7 upgrade a time() call was converted to a constant :P

  • helmo committed 4db5002 on 7.x-3.x
    Issue #2150787 by helmo: Prevent tight loop when lock_acquire fails
    
  • helmo committed 98a7a4b on 7.x-3.x
    Issue #2150787 by helmo: Let the queued notice its lifetime if over
    
  • helmo committed f392622 on 7.x-3.x
    Issue #2150787 by helmo, ergonlogic, zhangtaihao: Let Queue daemon catch...
helmo’s picture

Status: Needs review » Fixed

Did another test run with this ... still looks good.

I added one extra commit to prevent a tight loop when lock_acquire fails.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

ergonlogic’s picture

Status: Closed (fixed) » Needs work

Since 3.5, there have been reports on IRC of:

drush[4629]: mysqli_close() expects parameter 1 to be mysqli, null given            [warning]"
drush[4629]: hosting_queued.drush.inc:227

The commits associated with this issue appear to be the only ones relevant. In particular: f392622a.

Steven Jones’s picture

If we're just trying to get a DB connection back because it's been too long since it was used, then I think you can just unset the connection from the global pool by doing this:

Database::closeConnection('default');

Is that what the code is trying to do?

Steven Jones’s picture

Oh weird, so there's a loop that's trying to wait for the DB to come back, using the mysqli functions directly.

helmo’s picture

That code mainly tries to handle interruptions in the db server. Could be a restart... om even a stop ... wait/update... start.

  • helmo committed 92c03b8 on 7.x-3.x
    Issue #2150787 by helmo, ergonlogic, Steven Jones: Queue daemon give...
helmo’s picture

Status: Needs work » Fixed

Just pushed a relatively simple fix for this.

  • helmo committed 4db5002 on 7.x-4.x, aegir4_https
    Issue #2150787 by helmo: Prevent tight loop when lock_acquire fails
    
  • helmo committed 98a7a4b on 7.x-4.x, aegir4_https
    Issue #2150787 by helmo: Let the queued notice its lifetime if over
    
  • helmo committed f392622 on 7.x-4.x, aegir4_https
    Issue #2150787 by helmo, ergonlogic, zhangtaihao: Let Queue daemon catch...
  • helmo committed 92c03b8 on 7.x-4.x, aegir4_https
    Issue #2150787 by helmo, ergonlogic, Steven Jones: Queue daemon give...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.