Hello--- your module is GREAT. Thank you.

I am running background process on a pantheon install-- which uses nginx and a couple of servers.

We have 2 servers in our load balancer, and occasionally these processes fail: this is the support note I received:

"You have to two applications on the Live environment. For modules that require a single application server or connection this can cause issues when the batch jobs as Drupal’s batch will dispatch these and they could either server.

The root of the problem is a majority of modules work on a single server infrastructure. Batches do run, we have some tasks on some of our own sites, but for some reason the way the host is defined in background_process. Hopefully setting one of these variables should get everything to work but there would be some trial and error. The issue queue may have already seen this but if not the maintainers may have some clues."

I am thinking of doing something like this in my settings.php Would this force it to one server?

<?php
$conf['background_process_service_hosts'] = array(
  'mysite' => array(
    'base_url' => 'http://127.0.0.1:13350',
    'http_host' => 'mysite.com',
  ),
);
$conf['background_process_service_groups'] = array(
  'default' => array(
    'hosts' => array('mysite'),
  )
);
?>

Comments

gielfeldt’s picture

gielfeldt commented

It will force it to one server, but not a specific one. It will be the one that the background process was originally issued from.

Say you have to servers with the ip's:
a) 1.2.3.4
b) 5.6.7.8

If a batch job is started from "a", it should stay on "a" throughout the process, as it will re-launch on 127.0.0.1.

To force it to a specific server, you will need to enter the exact ip (or unique hostname) of that server in the base_url option.

acb’s picture

acb commented

So you are saying that I should have two distinct configs in settings.php? one for each server, so that whichever server launches the process will maintain it?

(thanks again!)

gielfeldt’s picture

gielfeldt commented

Well. Hmm. No. Sort of :-). It depends on what you want.

Your example should work (incidentally) as you want.

However, to explicitly configure it to work the way you describe you could do:

<?php
<?php
$conf['background_process_service_hosts'] = array(
  'host1' => array(
    'base_url' => 'http://1.2.3.4:13350',
    'http_host' => 'mysite.com',
  ),
  'host2' => array(
    'base_url' => 'http://5.6.7.8:13350',
    'http_host' => 'mysite.com',
  ),
);
$conf['background_process_service_groups'] = array(
  'default' => array(
    'hosts' => array('host1', 'host2'),
  )
);
?>
acb’s picture

acb commented

Thanks for the above…. Very helpful.

I received this from pantheon support also… which I am not sure is applicable:

>>>>>>
You can get the I.P. address dynamically when this is triggered, it should be part of the $_SERVER variable. From there you would have to set this somehow, possibly setting this when the job is kicked off.

I am not sure of the details about how the module works in much detail, but this may need to persist for the duration of the job, if the next batch is triggered it knows where to run. There may need to be some cleanup of that ID but the module may already do that, in which case setting the I.P. would be enough.
<<<<<<<<<

Does this sound at all plausible?

gielfeldt’s picture

gielfeldt commented

The solution they suggest will most likely work exactly like your own first example (the 127.0.0.1 example).

acb’s picture

acb commented

Hi, I think I am getting close…. could you take a look?

I added this:

$hostname = 'myhost.com';
$dnsr = dns_get_record($hostname, DNS_ALL - DNS_PTR);
foreach($dnsr as $record) {
    if ( $record['type'] == 'A') {
        $ip = $record['ip'];
    }
}
 
$host = $_SERVER['HTTP_HOST'];
$ip = $_SERVER['SERVER_ADDR'];
 
$port = $_SERVER['SERVER_PORT'];
$addr = $ip .":" .$port;
$conf['background_process_service_hosts'] = array(
  'myhost' => array(
    'base_url' => $addr,
    'http_host' => $host,
  ),
);
$conf['background_process_service_groups'] = array(
  'default' => array(
    'hosts' => array('myhost'),
  )
);
variable_set('background_process_service_hosts', $conf['background_process_service_hosts'] );
variable_set('background_process_service_groups', $conf['background_process_service_groups']);

However, now when running a background process, I am getting this in watchdog:

Notice: Undefined property: BackgroundProcess::$callback in BackgroundProcess->dispatch() (line 197 of /srv/bindings/9c6aa76ead80488b88df8d537e146d93/code/sites/all/modules/contrib/background_process/BackgroundProcess.class.php).

-----AND -----

Could not call service bfc2043c520b2b518ef9a173faa842fc for callback : stdClass Object ( [url] => 1.2.3.4:12428/bgp-start/bfc2043c520b2b518ef9a173faa842fc/3c282b5f3929ebb7217ed841bf296c76 [options] => Array ( [method] => POST [headers] => Array ( [User-Agent] => Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_5) AppleWebKit/537.74.9 (KHTML, like Gecko) Version/6.1.2 Safari/537.74.9 [Accept] => text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 [Referer] => http://sprint31-lv-ns.gotpantheon.com/broadcast/5034 [Origin] => http://sprint31-lv-ns.gotpantheon.com [Content-Type] => application/x-www-form-urlencoded [Dnt] => 1 [Accept-Language] => en-us [X-Pantheon-Styx] => 1 [X-Forwarded-For] => 74.101.14.194, ::ffff:127.0.0.1,::ffff:127.0.0.1 [Surrogate-Capability] => styx="ESI/1.0" [Cookie] => has_js=1; SESS6049d2e68cce54c5e5c0e2a92b68aa58=UJfcqpAjtv-KCc9PpuWURWOntLR8qiQmIBV_5SwbUg4; has_js=1; Drupal.tableDrag.showWeight=0; hblid=yGt9hphwf0wQbUzN878pR6JODT1gFgkQ; olfsk=olfsk7364005933050066 [X-Varnish] => 200088023 [X-Pantheon-Client-Ip] => 74.101.14.194 [X-Forwarded-Port] => 59818 [X-Forwarded-Proto] => http [Host] => sprint31-lv-ns.gotpantheon.com [Connection] => close ) ) [code] => -1002 [uri] => Array ( [host] => 1.2.3.4 [port] => 12428 [path] => /bgp-start/bfc2043c520b2b518ef9a173faa842fc/3c282b5f3929ebb7217ed841bf296c76 ) [error] => missing schema [end] => 1397235199.6354 )

when starting a background process. … and nothing fires off.

(I have changed my ip in the above to 1.2.3.4).

Do you have any insight here?

Best, and THANKS for your patience,

ACB

gielfeldt’s picture

gielfeldt commented

"missing schema". You must include the schema (http or https) in the base_url.

However, I don't understand what you are trying to do with the above code. What's the DNS lookup used for?

acb’s picture

acb commented

Pantheon rotates cloud-server IPs on occasion, and changes ports as well… so they suggest we check once a day on cron.

I will try the http:// and report back.

UPDATE:

that seems to have done the trick; sorry for the silly question. However I have not tried it on the production (load-balanced) servers yet. I will let you know if we see any issues.

Thank you so much for your help.

acb’s picture

acb commented

OK.... so I have to revive this thread.

I have been consistently getting this error:

Could not call service df66c0f897d5f362b023fafe5845784d for callback syn_cb_broadcast_email_bg_process: stdClass Object ( [url] => http://192.237.185.114:13350/bgp-start/df66c0f897d5f362b023fafe5845784d/... [options] => Array ( [method] => POST [headers] => Array ( [Cache-Control] => max-age=0 [Accept] => text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8 [Origin] => http://dashboard.mysite.com [User-Agent] => Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36 [Content-Type] => application/x-www-form-urlencoded [Referer] => http://dashboard.mysite.com/broadcast/5333 [Accept-Language] => en-US,en;q=0.8 [X-Pantheon-Styx] => 1 [X-Forwarded-For] => 74.220.236.238, ::ffff:127.0.0.1,::ffff:127.0.0.1 [Surrogate-Capability] => styx="ESI/1.0" [Cookie] => has_js=1; optimizelyEndUserId=oeu1396294282227r0.25475957803428173; is_returning=1; optimizelySegments=%7B%7D; optimizelyBuckets=%7B%7D; __utma=237399197.1673764216.1396294282.1396294282.1396294282.1; __utmz=237399197.1396294282.1.1.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided); __ar_v4=3CQ6BLCTMZAFLINYHUAW2I%3A20140330%3A4%7C7GYIVT5IHVDXLOMT2GKTYS%3A20140330%3A4%7CO3262BOWGFHKRAQKOC5VZU%3A20140330%3A4; _ok=4304-275-10-8078; _okbk=cd5%3Davailable%2Ccd4%3Dtrue%2Cvi5%3D0%2Cvi4%3D1399300104068%2Cvi3%3Dactive%2Cvi2%3Dfalse%2Cvi1%3Dfalse%2Ccd8%3Dchat%2Ccd6%3D0%2Ccd3%3Dfalse%2Ccd2%3D0%2Ccd1%3D0%2C; SESS165637436d0e9626ef7d7b9eb383b955=D9uovfcFARt8quqQzEvA_IShfPetbdi9hzsVQERWkSw; has_js=1; __utma=164989738.1711746805.1399300103.1399300103.1399300103.1; __utmb=164989738.7.10.1399300103; __utmc=164989738; __utmz=164989738.1399300103.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); _oklv=1399300372660%2CrKRuWqCRdJd6yc9n878pR9HeETF014HL; olfsk=olfsk737240900285542; wcsid=rKRuWqCRdJd6yc9n878pR9HeETF014HL; hblid=3STKSBsl4krAYd6Z878pR9HPET1p1F9U [X-Varnish] => 2314514674 [X-Pantheon-Client-Ip] => 74.220.236.238 [X-Forwarded-Port] => 56222 [X-Forwarded-Proto] => http [Host] => dashboard.mysite.com [Connection] => close ) [data] => [max_redirects] => 3 [timeout] => 4 [context] => [blocking] => [postpone] => ) [code] => -111 [uri] => Array ( [scheme] => http [host] => 192.237.185.114 [port] => 13350 [path] => /bgp-start/df66c0f897d5f362b023fafe5845784d/b2dd9d7c4be1a34976183546286af845 ) [socket] => tcp://192.237.185.114:13350 [postponed] => [start] => 1399300400.0293 [data_ready] => 1 [error] => Connection refused [end] => 1399300400.0294 )

As per our earlier discussion this runs on cron:

$pantheon_host = variable_get('pantheon_host', '');
            //only do this if set
            if (strlen($pantheon_host) > 10) {
                $dnsr = dns_get_record($pantheon_host, DNS_ALL - DNS_PTR);
                foreach ($dnsr as $record) {
                    if ( $record['type'] == 'A') {
                        $ip = $record['ip'];
                    }
                }
                $port = $_SERVER['SERVER_PORT'];
                $host = $_SERVER['HTTP_HOST'];
                $addr = 'http://' . $ip .":" .$port;
                $conf['background_process_service_hosts'] = array(
                    'my_bg_host' => array(
                        'base_url' => $addr,
                        'http_host' => $host,
                    ),
                );
                $conf['background_process_service_groups'] = array(
                    'default' => array(
                        'hosts' => array('my_bg_host'),
                    )
                );
                variable_set('background_process_service_hosts', $conf['background_process_service_hosts'] );
                variable_set('background_process_service_groups', $conf['background_process_service_groups']);
                watchdog('bgprocess', 'server ip set as ' . $addr);
            }

We have two web-heads on Pantheon. Sometimes this sets it correctly, but sometimes it gets the IP of one box, and the port of another.. Have I made a simple little mistake here, or is there something in the way they have set up their server environments that is off?

my_bg_host tends to have the right IP/Port in it. and the 'default' when I "determine default host" is 127.0.0.1 and on the same port.

Anyway, any enlightenment you could give would be great. This module (when it's working for me) is incredibly powerful. I just wish I could get it to be consistent.

Thank you.

acb’s picture

acb commented

ok... so now I am doing this, in order to get the correct IP/port Combination. The http_request of robots.txt feels clunky to me. Any thoughts?


            //get pantheon host immediately before start
             $pantheon_host = variable_get('pantheon_host', '');
            //only do this if set
            if (strlen($pantheon_host) > 10) {
                $dnsr = dns_get_record($pantheon_host, DNS_ALL - DNS_PTR);
                $ips = array();
                foreach ($dnsr as $record) {
                    if ( $record['type'] == 'A') {
                        $ips[] = $record['ip'];
                    }
                }
                $port = $_SERVER['SERVER_PORT'];
                $host = $_SERVER['HTTP_HOST'];
                foreach($ips as $ip) {
                    $addr = 'http://' . $ip .":" .$port;
                    $req = drupal_http_request($addr . '/robots.txt' );
                    if($req->code == 200) {
                        $good_addr = $addr;
                    }
                }
                $conf['background_process_service_hosts'] = array(
                    'mysite_bg_host' => array(
                        'base_url' => $good_addr,
                        'http_host' => $host,
                    ),
                );
                $conf['background_process_service_groups'] = array(
                    'default' => array(
                        'hosts' => array('mysite_bg_host'),
                    )
                );
                variable_set('background_process_service_hosts', $conf['background_process_service_hosts'] );
                variable_set('background_process_service_groups', $conf['background_process_service_groups']);
                watchdog('DataGrinder', 'server ip set as ' . $good_addr);
            }
gielfeldt’s picture

gielfeldt commented

I've created a Pantheon account. I'll try to get my hands on this soon.

@acb You say the ports change as well?!? That may be a tough one to crack. Especially if we only have a DNS lookup to get this info, since DNS doesn't tell us anything about ports.

timani’s picture

timani commented

@giefeldt, you will need an account with at least two application servers in production. You can ping me when you are ready and I can get that setup for you.

I have not had much time to test this but the ports are returned in the $_SERVER variable and the full host and port included in the $_SERVER['HTTP_HOST'];

Here is a gist of the sanitized $_SERVER variable from a test script that dumps the var.

https://gist.github.com/timani/30482f6c1290c780ef57

My hunch is that this was missing a check to make sure the I.P. and the port are a match and return a 200 HTTP response code.

I could see this returning 2 I.P.s with the code from https://drupal.org/node/2233843#comment-8748697 but with the update to check for a 200, curious if this works. I tried to step through the logic to see how it would run but feedback would be needed as it appears to be correct. 


// Pseudo code
$dnsr = dns_get_record($pantheon_host, DNS_ALL - DNS_PTR);

// Next I.Ps
$ips = array();
foreach ($dnsr as $record) {
  if ( $record['type'] == 'A') {
    $ips[] = $record['ip'];
  }
}
// $ips [1.1.1.1.1]
// $ips [2.2.2.2.2]

// This request is currently on  2.2.2.2.2 current port 555
$port = $_SERVER['SERVER_PORT']; // 555
$host = $_SERVER['HTTP_HOST']; /// 2.2.2.2.2:555
foreach($ips as $ip) {
  $addr = 'http://' . $ip .":" .$port; // #1 1.1.1.1:555 -- #2 2.2.2.2.2:555
  $req = drupal_http_request($addr . '/robots.txt' ); // #1 1.1.1.1:555/robots.txt -- #2 2.2.2.2.2:555/robots.txt
  // This addition in comment #10 https://drupal.org/node/2233843#comment-8748753 to validate should do the trick and test for a 200
  if($req->code == 200) { 
    $good_addr = $addr; //  2.2.2.2.2:555
  }
}

$conf['background_process_service_hosts'] = array(
  'mysite_bg_host' => array(
  'base_url' => $good_addr,  //  2.2.2.2.2:555
  'http_host' => $host, //  2.2.2.2.2:555
  ),
);
gielfeldt’s picture

gielfeldt commented

Hi

I'm ready. I have a question, perhaps you know the answer :

Does such a setup as yours consists of a pool of known IP's and ports?

timani’s picture

timani commented

Hey @gielfeldt

Does such a setup as yours consists of a pool of known IP's and ports?

Unfortunately not, with the number of servers we have know and we run containers that have access via random ports. The reason we do not have a fixed pool of I.P. addresses is because our provider can not guarantee the IPv4 addresses with be within a certain range anymore.

If you can send me a message I can make your site business level so you will have two application servers in the Live environment.

The limitation that I see for SSL is the I.P. addresses will serve the *.gotpantheon.com wilcard cert and generate an SSL connection exception.

gielfeldt’s picture

gielfeldt commented

Hi

I haven't really had the time to look at this, sorry :-(.

To recap (because it's been a while): Do I understand it correctly, that background batch jobs fail when using 127.0.0.1 as the default service host, IF the server switches IP during a batch run?

acb’s picture

acb commented

We are not using background batch, rather background process (e.g. background_process_start($function, $args).

What seems to be happening somehow is our background_process table seems to fill up with locked jobs. When it is doing so, it generally means that Pantheon has rotated our IP and Port to another server. Currently we are checking to see if it is a valid ip and port before on cron with the following code.

However, this doesn't help us when the port is changed between IP-rotations. However, things don't seem to work when we simply point to "http://mysite.com" in the $conf module.

Is there any way to make this work just using our domain name (e.g. http://mysite.com)?

Thanks SO MUCH for your help. 



        $pantheon_host = ***our pantheon host info***;;
    //get pantheon host immediately before start
    //only do this if set
    if (strlen($pantheon_host) > 10) {
        $dnsr = dns_get_record($pantheon_host, DNS_ALL - DNS_PTR);
        $ips = array();
        foreach ($dnsr as $record) {
            if ( $record['type'] == 'A') {
                $ips[] = $record['ip'];
            }
        }
        $port = $_SERVER['SERVER_PORT'];
        $host = $_SERVER['HTTP_HOST'];
        foreach($ips as $ip) {
            $addr = 'http://' . $ip .":" .$port;
            $req = drupal_http_request($addr . '/bg_check.txt' ); //NOTE THAT THIS FILE HAS BEEN ADDED TO OUR ROOT DIRECTORY (next to robots.txt)
            if($req->code == 200) {
                $good_addr = $addr;
            }
        }

        $conf['background_process_service_hosts'] = array(
            'mysite_bg_host' => array(
                'base_url' => $good_addr,
                'http_host' => $host,
            ),
        );
        $conf['background_process_service_groups'] = array(
            'default' => array(
                'hosts' => array('mysite_bg_host'),
            )
        );
        variable_set('background_process_service_hosts', $conf['background_process_service_hosts'] );
        variable_set('background_process_service_groups', $conf['background_process_service_groups']);
        watchdog('bg', 'Background process server ip set as ' . $good_addr, NULL, WATCHDOG_INFO);
    } else {
        //on someone's local
        if(!strlen($pantheon_host) ) {
            global $base_root ;

            $conf['mysite_process_service_hosts'] = array(
                'mysite_bg_host' => array(
                    'base_url' => $base_root ,
                    'http_host' => $_SERVER['HTTP_HOST'],
                ),
            );
            $conf['background_process_service_groups'] = array(
                'default' => array(
                    'hosts' => array('mysite_bg_host'),
                )
            );
            variable_set('background_process_service_hosts', $conf['background_process_service_hosts'] );
            variable_set('background_process_service_groups', $conf['background_process_service_groups']);
            watchdog('syn_datagrinder', 'Local background process server ip set as ' . $base_root, NULL, WATCHDOG_INFO);
        }
    }
gielfeldt’s picture

gielfeldt commented

How about this:

<?php
function mymodule_background_process_pre_execute($handle, $callback, $args, $token) {
  // Derive proper base url.
  $base_url = 'http://' . $_SERVER['SERVER_ADDR'] . ':' . $_SERVER['SERVER_PORT'];

  // Update default service host if necessary.
  $service_hosts = variable_get('background_process_service_hosts', array());
  if (empty($service_hosts['default']['base_url']) || $service_hosts['default']['base_url'] !== $base_url) {
    $service_hosts['default']['base_url'] = $base_url;
    $service_hosts['default']['http_host'] = $_SERVER['HTTP_HOST'];
    variable_set('background_process_service_hosts', $service_hosts);
  }
}
?>
gielfeldt’s picture

gielfeldt commented
Status: Active » Postponed (maintainer needs more info)
Chris Charlton’s picture

Chris Charlton
Primary language English
Location Los Angeles
commented

+1 on trying to figure this out. My prod environment has more than one web head so I'll give what I find in here a shot.