Missed schedule due to congestion [#2311381]

Hello,

i have 10 parallel cron jobs in the running state that say "x minutes behind the schedule" and the log means that "Missed schedule due to congestion". This will hit different cron jobs from different modules. The duration of the jobs are constantly growing and they don't exit the running state. If i manually kill jobs and unlock them the next job will come into this never ending state.

I've tried restarting the server, disabling my custom cron jobs and clearing caches but nothing seams to work. The bad thing is that it has worked until a few days. I've updated one and other module in this time but there are also jobs from untouched modules that become the problem.

How can this be debugged?

Comment	File	Size	Author
#3	bad.png	32.96 KB	fox_01

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Comment #1

fox_01 CreditAttribution: fox_01 commented 29 July 2014 at 05:37

Status:

Active

» Closed (fixed)

After looking arround in the issue queue i found a solution that fixed it.

I have changing my domain from test to productive but a few days before that error happend but didn't had that in mind. I think this can be the only reason because re-determing the service host as described here #1580708-10: Jobs are permanently in "Starting" status seems to fix it for me but only temporarily.

Comment #2

fox_01 CreditAttribution: fox_01 commented 29 July 2014 at 05:34

Status:

Closed (fixed)

» Active

This was a very short success story..... After couple of minutes the jobs are back in the described "problem state".

I think it has something to do with this line (Error 400 - bad request) which repeats very often in my access.log
- - [] "POST /bgp-start/uc-ultimate_cron_plugin_launcher_background_process_legacy_cleanup/8e73461a2b1f673463444e4ccf8ee6bd HTTP/1.0" 400 1258 "" "curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3"

I have also found a log entry from another module badbehavior which tells me the reason (blank referer). See the attached image for more details. I have tried to completly uninstall the badbehavior module to ensure its not the wrong domainname but other requests are not blocked and the log entry shows the correct one.

I can confirm that if the module is disabled that the jobs will get executed normally.

Comment #3

fox_01 CreditAttribution: fox_01 commented 29 July 2014 at 05:35

File	Size
bad.png	32.96 KB

Comment #4

fox_01 CreditAttribution: fox_01 commented 29 July 2014 at 06:03

Component:

User interface

» Documentation

There is a workaround for this by whitelisting my ip-address inside badbehavior

Is there a way from the perspective of ultimate cron to let the requests not look as spam?

Maybe there is another more safe way (in case of shared ip addresses) to whitelist the request (identified by header, url, ip address)?

Related issue from badbehavior queue #2311517: Incompatible with ultimate cron

Comment #5

gielfeldt CreditAttribution: gielfeldt commented 29 July 2014 at 06:04

I'm not familiar with the badbehavior module. But if it intercepts either the outgoing or incoming request somehow, it may interfere with background process. I'll try to take a look at the badbehavior module.

Comment #6

gielfeldt CreditAttribution: gielfeldt commented 29 July 2014 at 07:29

Hmmm. I have a hunch that any POST without a referer will fail. Technically not because of BB, but because Drupal core sets $_SERVER['HTTP_REFERER'] to an empty string. A change that seem to have piggybacked on #302763: Remove time() so that the function referer_uri() could be eliminated?

I'm not sure how to fix this from Background Process short of spoofing a referer. But don't really like the idea of spoofing a request header, in order to make a spamfilter work properly :-)

Comment #7

gielfeldt CreditAttribution: gielfeldt commented 29 July 2014 at 07:34

Actually, it may be possible to whitelist based on User-Agent. But then I should update Background Process to include a token in the User-Agent for reasons of security.

Comment #8

fox_01 CreditAttribution: fox_01 commented 29 July 2014 at 08:20

The token in the user agent sounds like a good idea.

Comment #9

gielfeldt CreditAttribution: gielfeldt commented 30 July 2014 at 09:55

If you update Background Process to the latest dev, you can set a user-agent for background process in your settings.php

<?php
$conf['background_process_user_agent'] = 'My very secret user-agent with t0k3n';
?>

And whitelist that one in BB.

Comment #10

gielfeldt CreditAttribution: gielfeldt commented 2 August 2014 at 18:21

Status:

Active

» Postponed (maintainer needs more info)

Please let me know if this works out for you.

Comment #11

fox_01 CreditAttribution: fox_01 commented 18 August 2014 at 08:01

Status:

Postponed (maintainer needs more info)

» Active

The configured user agent is shown in the logfile of badbehavior. So far so good. Special characters are not supported and dashes are removed. I don't know if this is namingconvention of user agents.

BUT i think you have added a "Header Referer" because its now independently of my configured user agent in settings.php and my whitelist in badbehaviour working. Its just enough to install the new version of background process. I remind you of the error message.
Header 'Referer' present but blank

So it works but not as planned ;)