It would be useful to be able to add/override the header and CURLOPT fields used by the HTTP Fetcher.

For example, I often need to manually override the default useragent value "Drupal (+http://drupal.org/)" since many sites now block it. I'd like to have an easy way to change it without using hacks.

This could be done pretty easily by adding two new configuration fields to the ui and making some minor code tweaks.

The feeds library function http_request_get() would also need to be altered without breaking any other modules.

Currently it is called like this:
http_request_get($url, $username = NULL, $password = NULL, $accept_invalid_cert = FALSE, $timeout = NULL)

I suggest this:
http_request_get($url, $curl_opts = array(), $headers = array(), $accept_invalid_cert = FALSE, $timeout = NULL)
with parameter type checking and mangling for backwards compatibility.

Note that of the three usages documented in Drupal 7, none use all 5 parameters so it simplifies the code in most cases.

I can write the code if there's any interest, although it would be good if someone with more Feeds experience weighed in first.

Comments

berimbolo created an issue. See original summary.

MegaChriz’s picture

MegaChriz CreditAttribution: MegaChriz as a volunteer commented
Related issues: +#2446313: Send cookies in http request headers

Good suggestion!

In the latest release of Feeds http_request_get() is superseded by feeds_http_request(). This function has an 'options' parameter with which it is easier to add extra options, like the one requested in this issue.

Sidenote: when adding a new parameter to an existing function, you should nearly always add it as the last one. While you can adjust the calls to this function in this module, the function may be called as well by other modules and changing the parameter order could then break these modules. You should only rearrange the parameter order when the function is only in a dev release (so it is assumed that no code depends on it yet) or when the function is clearly documented as an internal function (and thus is not expected to be called by other modules).

There was a similar feature request for this as well, so adding that one here for reference: #2446313: Send cookies in http request headers. In there I suggested to add a hook for altering the request. Challenge is that feeds_http_request() does not know which importer initiated the request.