Cron channel has been running for more than an 3600 secs and is most likely stuck. Last job executed: backup_migrate_cron [#1375096]

I've been using elysia_cron for over a year on all my sites but have persistent problems with backup_migrate_cron which is used to handle scheduled backups. My scheduling is straightforward -- one a day (retain 7) and once a week (retain 1). What appears to be happening is that cron triggers the scheduled backup, backup completes, but doesn't get the signal back correctly from backup_migrate that the process is completed. As such, elysia_cron reports that the cron process is still running and doesn't proceed to the next steps.

The only way to clear things when this happens is to manually update the elysia_cron.running column with '0' values for the :default and backup_migrate_cron rows.

I expect this is one part server configuration issues, one part backup_migrate bug, and one part elysia_cron bug. There needs to be a way to clear execution status from the UI rather than manually having to change table values.

I'll take the backup_migrate_cron issues up with that queue~~, but it appears related to compression of the backup~~. The default backup_migrate schedule profile calls for gzip compression; overriding these settings for no compression resolves the majority of conflicts with elysia_cron.

Comments

Comment #1

mrP commented 18 December 2011 at 18:59

Title:

Cron channel (default) has been running for more than an 3600 secs and is most likely stuck. Last job executed: backup_migrate_c

» Cron channel has been running for more than an 3600 secs and is most likely stuck. Last job executed: backup_migrate_cron

Comment #2

geneatwell commented 16 January 2012 at 23:21

The only thing that happens when I eliminate the compression is that the 3600 secs message remains, but there is not any backup produced. With compression, the 3600 secs message appears, the site drops offline and the backup is mailed to me. In both cases the database values for default and backup_migrate_cron are 0 and I do not need to change them. When the backup is scheduled, it runs when it is supposed to and one of the two results listed above happens.

Drupal 6.22 and Elysia Cron 6.x-2.0. I have not used any other Elysia Cron before this build.

Comment #3

shaisamuel commented 29 January 2012 at 09:38

Same problem with 7.x-2.x-dev (Jan-11-12).

Comment #4

mrP commented 29 January 2012 at 18:17

I've been looking into this quite a bit recently, and in my case elysia_cron wasn't necessarily the culprit and compression was a wild goose chase. I have ~20 sites with this same backup schedule and the event was triggering at the same time for all. These are all single site, independent drupal installations. After rescheduling my cron tasks in a staggered manner (following http://www.drupalcoder.com/blog/spread-your-cron-schedule-on-machines-ru...), my problems with backup_migrate_cron timing out have gone away.

The key takeaway for me is that the elysia_cron default advanced installation recommendation (ie, a request every minute to cron.php for each) is not scalable on a small to moderate server. Further, it is total overkill for small sites.

I'm marking this as documentation rather than a bug based on my experience. Can anyone else with this problem confirm if they have a setup similar to mine?

UPDATED: sorry, pasted the wrong url.

Comment #4.0

mrP commented 29 January 2012 at 18:17

Issue summary:

View changes

updating problem description after further testing.

Comment #5

gotheric commented 29 January 2012 at 17:50

Component:

Miscellaneous

» Documentation

1. each cron job (with elysia_cron or not) should execute and return to caller. If it doesn't do that (for example: it does an "exit" to stop php from execution) the execute chain interrupts, and after a timeout (configured in cron settings) elysia cron detects it and reset channel state.
This is a problem of the job called, and not of elysia_cron (or standard drupal cron).

2. There should be no need to reset running column in db: elysia cron does that after the timeout time (3600 seconds, but you can configure this time)

3. To minimize the problem you should consider putting backup job in its own cron channel (by setting a channel name in cron settings). That way backup job won't prevent other jobs to execute.

4. Elysia cron doesn't change the way each job is called. If the job freezes or don't executes successfully it should not be a problem of Elysia cron (In the past some module hooks that used some weird techniques to execute had some problems with elysia cron. Usually this was a problem fixed in the module, however we should look at the code executed to fully understand... i don't know backup_migrate_cron code...)

pricejn2: i don't understand what you mean about "After rescheduling my cron tasks in a staggered manner" (the article you are linking talks about "www." site prefix...).

About running elysia cron every minutes, it should not be a problem if your machine is well dimensioned. I've a machine with ~50 sites running elysia_cron simultaneously every minute with no problems at all.
However, it depends of jobs your are using: if every sites has a huge job running every 2 minutes maybe THIS IS a problem...
See #831548 about it.