When working with web services it is useful to use HTTP proxies (like http://mitmproxy.org/) to help debugging. However HTTP proxies use a special header format, like this:
HTTP/1.1 200 Connection established
Proxy-agent: mitmproxy 0.10
<empty line>
HTTP/1.1 200 OK
Date: Fri, 27 Dec 2013 13:58:25 GMT
Server: Apache/2.2.22 (Ubuntu)
. . . . . . . . . .This format is required by the standards and it is not by mistake. However the module http_client does not recognize this format and is completely confused by it (does not know where the headers end and where the body starts). Fixing it would be useful.
| Comment | File | Size | Author |
|---|---|---|---|
| #7 | http_client-2163459-7-support_for_http_proxies.patch | 3.44 KB | jsst |
| #5 | http_client-support_for_http_proxies-2163459-5-D7.patch | 3.59 KB | jsst |
| #2 | http_client-2163459-2-support_for_http_proxies.patch | 1.31 KB | dashohoxha |
Comments
Comment #1
dashohoxha commentedComment #2
dashohoxha commentedThe attached patch implements this feature and also fixes this existing issue: #2146373: preg_split bug on correctly formatted json by function interpretResponse
Comment #3
dashohoxha commentedComment #4
jsst commentedThis problem needs a better fix: the proposed solution only works for proxies that add the (non-standard) proxy-agent header. The headers of the real response will still end up in the response body when the proxy does not add any headers or adds headers other than proxy-agent.
See: https://curl.haxx.se/mail/lib-2005-10/0023.html
Comment #5
jsst commentedIt is not possible to reliably split the response headers from the response body using CURLOPT_HEADERS=1, that's because we don't know how many header sections are contained in the response (i.e. did we go through a proxy?). The only component that can do that is libcurl and I know of two ways to leverage that information:
I've attached a new patch which can be applied on top of the original patch in this thread - or straight onto the -dev version of this module. The change is backwards-compatible: it introduces a new optional parameter 'header_content' which when omitted (in case someone implemented a custom delegate) triggers the old behaviour of splitting the headers from the body assuming the request didn't go through a proxy server.
Comment #6
jsst commentedComment #7
jsst commentedI've unified patch #2 and #5 into one patch, and fixed a bug where HTTP 100 Continue responses were not properly handled.