Add proxy support

Thanks, this snippet of information is what I've been after for ages. Wondering if next time the module get's an update this setting gets overridden though.

This is actually an option you can use in your settings.php
Just put in:
$conf['feeds_never_use_curl'] = true;

You can read about it in README.txt.

It would be great if Feeds could recognize and use proxy settings as discussed in http://drupal.org/node/7881, or disable internal cURL-functionality accordingly.

I did some work behind the proxy.
Looks like there are some problems with cURL when behind the proxy.
There is a line of code added to HTTP header which is not catered by CURLINFO_HEADER_SIZE variable used in feeds.

The current patch is harmless and will work once the following patch it applied to core:
http://drupal.org/files/drupal-7881-429-add-proxy-support-for-http-reque...
From comments it seems like it's gonna happen in 7.16

Current patch adds proxy URL and port support to feeds (which you need to setup in settings.php file after above patch is applied).
Username and password support is not too far away.

+++ b/libraries/http_request.incundefined
@@ -160,6 +160,15 @@ function http_request_get($url, $username = NULL, $password = NULL, $accept_inva
+      // Proxy patch: no damage done as it's not fired until the following patch ¶

Need space above the comment. White space issues.

+++ b/libraries/http_request.incundefined
@@ -160,6 +160,15 @@ function http_request_get($url, $username = NULL, $password = NULL, $accept_inva
+        curl_setopt($download, CURLOPT_PROXY, $proxy_server.':'.variable_get('proxy_port', '80'));

Spaces inbetween .

+++ b/libraries/http_request.incundefined
@@ -183,6 +192,16 @@ function http_request_get($url, $username = NULL, $password = NULL, $accept_inva
+      if ($proxy_server && _drupal_http_use_proxy($uri['host'])) {
+        $httpHeaderBreak = "\r\n\r\n";
+        $response = explode($httpHeaderBreak, $data);
+        if (count($response) > 2) {
+          $data = substr($data, strlen($response[0] . $httpHeaderBreak), strlen($data));
+        }
+      }
 
       $header_size = curl_getinfo($download, CURLINFO_HEADER_SIZE);
       $header = substr($data, 0, $header_size - 1);

Variable name should not use camel case.

What is this last part doing? Can you link to the bug report?

Yeah, seems like some space issue. Will resubmit tomorrow.
The last part is actually taking care of extra message that is present in HTTP header only when dealing with proxy.
Here's few examples I found of the same problem.
- http://core.trac.wordpress.org/ticket/17731
- https://github.com/skyzyx/requestcore/issues/1

Syntax updated for patch #5

Patch from #8 works for me - was getting cURL errors due to proxy before applying; able to import feed items with no errors after applying.

Now that proxy support is in latest D7 core, Feeds should really support this too. Let's get this in.

I am reviewing this patch regarding with the proxy settings defined for D7.

I see that the patch "uses" the function parameters that are used for the HTTP authentication attached to the URL (I understood well the function doc.) for the proxy authentication

does it not used the proxy settings of D7 (set in settings.php) instead?

Concerning the proxy authentication method, the patch uses "NTLM" one without the possibility to configure it. Could we not to have it configurable?

So we could have something like that:

      ...
      curl_setopt($download, CURLOPT_ENCODING, '');
      curl_setopt($download, CURLOPT_TIMEOUT, variable_get('http_request_timeout', 30));

      // In case of proxy between the Drupal host server and internet.
      $proxy_server = variable_get('proxy_server', '');
      if (!empty($proxy_server)) {
        curl_setopt($download, CURLOPT_PROXY, $proxy_server);
        // Proxy port
        $proxy_port = variable_get('proxy_port', 8080);
        curl_setopt($download, CURLOPT_PROXYPORT, $proxy_port);
        // Proxy user/password
        $proxy_username = variable_get('proxy_username', '');
        if (!empty($proxy_username)) {
          $proxy_password = variable_get('proxy_password', '');
          $str_usr_pwd = "{$proxy_username}:{$proxy_password}";
          curl_setopt($download, CURLOPT_PROXYUSERPWD, $str_usr_pwd);
          $proxy_auth_method = variable_get('proxy_auth_method', CURLAUTH_BASIC);
          curl_setopt($download, CURLOPT_PROXYAUTH, $proxy_auth_method);
        }
      }
      
      if ($accept_invalid_cert) {
      ...

For the location of the configuration parameter "proxy_auth_method", it could be discussed with people in charge of the issue #7881: Add support to drupal_http_request() for proxy servers (http not https).

What do you think?

This patch needs to be re-rolled. Some different header manipulation for cURL was added.

#10, Correct. This patch should use the username:password found in settings.php, not the ones in the URL. The proxy config should be transparent to the request, not bound to end-user config.

As for the auth method, we should just be able to use CURLAUTH_ANY and be done with it.

We can't bring up the auth method in core because that's a cURL specific setting not found in drupal_http_request().

re-roll of patch including #10.

According to php docs we can't use CURLAUTH_ANY

"The HTTP authentication method(s) to use for the proxy connection. Use the same bitmasks as described in CURLOPT_HTTPAUTH. For proxy authentication, only CURLAUTH_BASIC and CURLAUTH_NTLM are currently supported. " (from http://php.net/manual/en/function.curl-setopt.php)

marking needs review

I've downloaded, applied and tested this patch. It is very straight forward and does exactly what I've been needing!

I've combined #6 and #12 to come up with a workable solution.

http://drupalcode.org/project/feeds.git/commit/ddee3f9

The following workaround in lines 216-221:

216 if ($proxy_server && _drupal_http_use_proxy($uri['host'])) {
217 $http_header_break = "\r\n\r\n";
218 $response = explode($http_header_break, $data);
219 if (count($response) > 2) {
220 $data = substr($data, strlen($response[0] . $http_header_break), strlen($data));
221 }
222 }

Actually breaks the module in the more recent versions of libcurl, where this bug is fixed (http://sourceforge.net/p/curl/bugs/1204/)

Can you open a new issue?

Also, do you know what version of curl/libcurl contains this fix?

Thanks.

For me too. You need to comment line 220 because I think somwhere this bug is patched.