I encountered this problem when using aggregator.module to load web feeds. Sometimes the process would return with an message about a "syntax error on line 1." As it turns out, chunked HTTP/1.1 responses have extra content-length encoding in the message body.

drupal_http_request() passes the extra encodings along as part of the message body. This confuses the XML parser.

#8 chunked_1.patch1.06 KBnjivy
#6 chunked_0.patch1.26 KBnjivy
#5 chunked.patch1.25 KBnjivy
Members fund testing for the Drupal project. Drupal Association Learn more


njivy’s picture

The drupal_http_request() behavior hasn't changed since Drupal 4.6, or maybe longer.

If I understand the protocol correctly, we can look for the Transfer-Encoding header and then strip away the extra lines. The problem is, sometimes the content-length encodings appear in the middle of the message body, not just the beginning and end.

njivy’s picture

ChrisKennedy’s picture

Many thanks for the bug report. Drupal actually just switched from HTTP/1.0 to HTTP/1.1 for drupal_http_request() (see http://drupal.org/node/104693), although I think this change was after rc1. This might be helpful in figuring out why gmap times out.

Do you still have this problem with using the latest CVS version of Drupal?

njivy’s picture

Oops, I was wrong about drupal_http_request() not changing. I didn't spot the HTTP/1.0 to HTTP/1.1 difference.

The problem disappears when I revert to HTTP/1.0. Thanks for the tip.

njivy’s picture

Version: 5.0-rc1 » 5.x-dev
Status: Active » Needs review
1.25 KB

This patch attempts to handle "chunked" transfer encodings. My tests indicate it works.

njivy’s picture

1.26 KB

Hrm. This patch uses standard filenames and paths. Sorry 'bout that.

ChrisKennedy’s picture

Status: Needs review » Needs work

It needs minor work to conform to drupal coding standards (see http://drupal.org/node/318) - operator spacing and comment styles both need to be tweaked.

It might be best to hold off until 6.x to add handling for chunked responses.

njivy’s picture

Status: Needs work » Needs review
1.06 KB

I had ported some code from php.net, hence the coding style. This patch cleans it up.

Dries’s picture

Version: 5.x-dev » 6.x-dev

Chris: any chance you can test this with your gmap setup?

I think we need to postpone this to Drupal 6.

What's the advantage of the chunked protocol? I'd document that in the patch.

fgm’s picture

Status: Needs review » Needs work

One advantage of the chunked transfer encoding is a theoretical ability to handle long responses efficiently, by syncing the HTTP level blocks with TCP-level reads, effectively adding minor synchronization points to TCP, which theoretically can make it more efficient: reads can be made to the announced chunk size instead of being to the default block size (see the "fetch response" part in drupal_http_header) and, if the underlying network supports it, will therefore be processed at the optimal speed. In addition, since the content length is known in advance from the chunk length, the end of socket read needn't be done using feof($fp), which relies on the protocol closing sequence and therefore introduces at least one latency iteration, but can close directly when the exact length of data has been read, since it is known in advance. See http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.6.1 for details.

The patch does not seem to take into account the optional final headers which may appear after the data when a Transfer-Encoding is defined as under the trailer BNF production for chunked encoding.

See http://www.w3.org/Protocols/rfc2616/rfc2616-sec19.html#sec19.4.6 for a recommended way of processing such encodings.

FWIW, I just encountered the problem today, where a Wordpress blog would return Transfer-Encoding: chunked to drupal_http_request in D5 and D4.7, although the request is send under HTTP/1.0 and such encodings are not allowed for HTTP/1.0 agents (as per section 3.6: A server MUST NOT send transfer-codings to an HTTP/1.0 client). Maybe someone familiar with WP should create an issue on their site too.

Pancho’s picture

Any progress here?

Steven Jones’s picture

Version: 6.x-dev » 8.x-dev

This is still an issue in Drupal.

mikeytown2’s picture

Helps to know what this looks like: http://en.wikipedia.org/wiki/Chunked_transfer_encoding#Example coming from #1320222: Bring in other drupal patches

Various functions on php.net. IMHO the regex ones seem like a bad idea.
http://php.net/fsockopen#73581 (this one handles compression)

Damien Tournoud’s picture

Status: Needs work » Closed (won't fix)

drupal_http_request() is a HTTP/1.0 client.

If a server replied with chunked encoding to a HTTP/1.0 request, fix the server.

andypost’s picture

Title: drupal_http_request() does not handle 'chunked' responses » drupal_http_request() does not handle 'chunked' responses - Make it support HTTP 1.1
Category: bug » feature
Status: Closed (won't fix) » Needs work

Suppose we should upgrade drupal_http_request() to be a HTTP/1.1 client.

There's a lot of code written so make it migrate easier!

HTTP Kernel is enough to make drupal_http_request() work as wrapper

mikeytown2’s picture

mikeytown2’s picture

Version: 8.x-dev » 7.x-dev

Guzzle is in. Moving this to D7