$url = 'http://www.gumtree.com.au/s-search-results.html?searchView=LIST&pageNum=1&action=find&urgentOnly=false&sortByName=date&adType=OFFER&adPriceType=&posterType=&topAd=false&gpTopAd=false&highlightOnly=false&minPrice=200&maxPrice=1200&categoryId=18626&keywords=cbr250';
$requests = httprl_send_request();

When I try the above, I get this error: Connection refused by destination. Write.

When I perform it with curl though, all is fine.

Something else that may be relevant or not, when I tried it with drupal_http_request, it returned an error of schema missing, as the 301 redirects to a relative url.


mikeytown2’s picture

Status: Active » Postponed (maintainer needs more info)

I can't repo this bug. I currently handle relative redirects.

$url = 'http://www.gumtree.com.au/s-search-results.html?searchView=LIST&pageNum=1&action=find&urgentOnly=false&sortByName=date&adType=OFFER&adPriceType=&posterType=&topAd=false&gpTopAd=false&highlightOnly=false&minPrice=200&maxPrice=1200&categoryId=18626&keywords=cbr250';
$requests = httprl_send_request();

foreach ($requests as $url => &$request) {
  $request->data = '';

echo httprl_pr($requests);

ouputs this

Array (
    [http://www.gumtree.com.au/s-search-results.html?searchView=LIST&pageNum=1&action=find&urgentOnly=false&sortByName=date&adType=OFFER&adPriceType=&posterType=&topAd=false&gpTopAd=false&highlightOnly=false&minPrice=200&maxPrice=1200&categoryId=18626&keywords=cbr250] => stdClass Object (
            [url] => http://www.gumtree.com.au/s-search-results.html?searchView=LIST&pageNum=1&action=find&urgentOnly=false&sortByName=date&adType=OFFER&adPriceType=&posterType=&topAd=false&gpTopAd=false&highlightOnly=false&minPrice=200&maxPrice=1200&categoryId=18626&keywords=cbr250
            [request] => GET /s-motorcycles/cbr250/k0c18626?ad=offering&price=200.00__1200.00 HTTP/1.0
User-Agent: Drupal (+http://drupal.org/)
Connection: close
Host: www.gumtree.com.au

            [status] => Done.
            [data] =>
            [options] => Array (
                    [headers] => Array (
                            [User-Agent] => Drupal (+http://drupal.org/)
                            [Connection] => close
                            [Host] => www.gumtree.com.au
                    [method] => GET
                    [data] =>
                    [max_redirects] => 2
                    [timeout] => 28.66621
                    [context] =>
                    [blocking] => 1
                    [version] => 1.0
                    [referrer] =>
                    [domain_connections] => 8
                    [global_connections] => 128
                    [global_timeout] => 120
                    [chunk_size_read] => 32768
                    [chunk_size_write] => 1024
                    [async_connect] => 1
            [chunk_size] => 32768
            [protocol] => HTTP/1.1
            [status_message] => OK
            [headers] => Array (
                    [date] => Fri, 27 Jul 2012 18:45:36 GMT
                    [server] => Apache
                    [set-cookie] => machId=a1147b67983c9f3babf46d933159bb92a6d4283038f0272ec1b546f6be9dc6f43f120cac720d257992641bfc38ae0e371e6da7c2e3fa47b876cab833abb47fc0; Domain=.gumtree.com.au; Expires=Wed, 14-Aug-2080 21:59:43 GMT; Path=/,bs=%7B%22st%22%3A%7B%7D%7D; Path=/,up=%7B%22ln%22%3A%22199332300%22%2C%22ls%22%3A%22l%3D0%26k%3Dcbr250%26c%3D18626%26r%3D0%26sv%3DLIST%26at%3DOFFER%26map%3D1200.00%26mip%3D200.00%26sf%3Ddate%22%7D; Expires=Wed, 23-Jan-2013 18:45:36 GMT; Path=/
                    [content-language] => en-AU
                    [vary] => Accept-Encoding
                    [connection] => close
                    [content-type] => text/html;charset=UTF-8
            [cookies] => Array (
                    [0] => Array (
                            [name] => machId
                            [value] => a1147b67983c9f3babf46d933159bb92a6d4283038f0272ec1b546f6be9dc6f43f120cac720d257992641bfc38ae0e371e6da7c2e3fa47b876cab833abb47fc0
                            [Domain] => .gumtree.com.au
                            [Expires] => Wed, 14-Aug-2080 21:59:43 GMT
                            [Path] => /
                    [1] => Array (
                            [name] => bs
                            [value] => %7B%22st%22%3A%7B%7D%7D
                            [Path] => /
                    [2] => Array (
                            [name] => up
                            [value] => %7B%22ln%22%3A%22199332300%22%2C%22ls%22%3A%22l%3D0%26k%3Dcbr250%26c%3D18626%26r%3D0%26sv%3DLIST%26at%3DOFFER%26map%3D1200.00%26mip%3D200.00%26sf%3Ddate%22%7D
                            [Expires] => Wed, 23-Jan-2013 18:45:36 GMT
                            [Path] => /
            [code] => 200
            [redirect_code_array] => Array (
                    [0] => 301
            [redirect_code] => 301
            [redirect_url_array] => Array (
                    [0] => http://www.gumtree.com.au/s-motorcycles/cbr250/k0c18626?ad=offering&price=200.00__1200.00
            [redirect_url] => http://www.gumtree.com.au/s-motorcycles/cbr250/k0c18626?ad=offering&price=200.00__1200.00

mikeytown2’s picture

Title: Redirect with relative URL not working (not sure if the relative URL is the problem though) Works with curl. » Connection refused by destination. Write. curl works
Version: 7.x-1.7 » 7.x-1.x-dev
Status: Postponed (maintainer needs more info) » Active

Changing title to match the actual reported issue. Redirect with relative URLs do work in httprl.

I actually recently encountered this issue as well. My problem has to do with running a lot of urls (LOTS) through httprl. After about 60 seconds most web servers will drop the connection if nothing was written to it. httprl opens all connections first and then sends data to a select few; once those connections are done it moves on to the next. I need to change this model so httprl opens up the connection when we are ready to do I/O on it. This involves moving stream_socket_client from httprl_request into httprl_send_request inside of the stream_select loop. A decently large refactor in order to fix the issue.

mikeytown2’s picture

Status: Active » Needs review
26.97 KB

This is the patch that moves the TCP connection inside of the event loop. Big changes in here, will be testing more but it should be backwards compatible for the most part.

mikeytown2’s picture

With the patch non blocking code is a little slower but it's not a terrible thing. Now takes a little under 1ms per non blocking request. Running 100 non blocking requests takes about 70 ms now.

modstore’s picture

Great, thanks mate, I will apply the patch to a dev site and see how it goes.

mikeytown2’s picture

Status: Needs review » Needs work

Still working on the patch... did find some bugs. Will let you know when it is ready; next patch should be in the next 3 hours or so.

mikeytown2’s picture

Status: Needs work » Needs review
34.47 KB

Changes from previous patch:
- Fixes in the readme files examples.
- Put t() around error text.
- Non blocking mode does not use async connect (hopefully this doesn't cause issues).
- Additional error checking for sync connections in httprl_establish_stream_connection().
- Removal of a lot of the static variables inside of httprl_send_request().
- Merge "See if end server has dropped the connection" and timeouts code as they both look the same to PHP.
- Changes to httprl_background_processing().

mikeytown2’s picture

Fixed 3 php notices in this patch. Everything looks good from what I can see on my end.

mikeytown2’s picture

Decided to revert "Non blocking mode does not use async connect". One can do this if they want to make sure the non blocking request was seen by the server.

    // Set the blocking mode.
    $options = array(
      'blocking' => FALSE,
      'async_connect' => FALSE,
mikeytown2’s picture

Status: Needs review » Fixed

#9 has been committed to 6.x & 7.x

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.