Whitelist download failed - replace file_get_contents with something that sends a User-Agent

I'm facing this issue right now, trying to get it working but no luck, I will keep investigating this issue for now.

killes said that there has been a request filter deployed on drupal.org yesterday which blocks requests without User-Agent. Apparently file_get_contents() does not send a User-Agent, so it is blocked.

Do we fix drupalorg_drush to not use file_get_contents() or do we roll back the request filter on drupal.org?

The request filter let drupal.org's servers stay online, so I think we keep it and patch drupalorg_drush.

Moving to drupalorg_drush.

This patch tries cURL first and adds a user agent.

I get a lot of phpunit test failures with that, need to check if they pass without it.

Restoring title.

cd tests
phpunit --bootstrap=/home/klausi/workspace/drush/tests/drush_testcase.inc drupalorgTest.php

Reports 17 failures before and after the patch, so the test cases seem to be broken anyway.

Seems good to me. We should let somebody else review it too, and then deploy it asap.

patch #4 worked on my local machine

I just tested the patch against the drupal-org.make file of kickstart and It worked.
drush verify-makefile drupal-org.make
Makefile drupal-org.make passed.

#4 looks good to me. Committed in 69e3e69. I'll leave this at RTBC so somebody can deploy the fix.

As I was deploying this, I noticed our deployment system for this was broken a bit. We have a couple Jenkins jobs for this, started by deploy_packaging_drush_drupal.org. Both had the same script in Jenkins, so I moved it to our infrastructure repo, http://drupalcode.org/project/infrastructure.git/blob/036b8b8:/live/depl.... I added the last line, which actually merges in changes if you are updating a branch. The git_id parameter advertises "Can be either a branch name, tag, or even a specific commit hash." I think I might have broken specific commit hashes.

I'd like to see the whole drupalorg_drush_get_remote_file() function replaced with drush_download_file() or something. drupal_http_request() would be ideal since it loads directly to memory, but this drush command is not invoked on a bootstrapped Drupal install.

Cool, thanks everyone!

Re #11:

I'd like to see the whole drupalorg_drush_get_remote_file() function replaced with drush_download_file() or something.

See #1401864: more robust download functionality

Cheers,
-Derek

Still some problems with this. Still a 403 forbidden error on http://drupal.org/packaging-whitelist/json when using fopen method.

From my terminal:

martin@martin-X501A1:~/Downloads/Drupal/scald_galaxy-7.x-1.x-dev/profiles/scald_galaxy$ sudo drush verify-makefile
[sudo] password for martin: 
Starting Drupal.org contrib makefile validation, please wait         [ok]
file_get_contents(http://drupal.org/packaging-whitelist/json): failed[warning]
to open stream: HTTP request failed! HTTP/1.1 403 Forbidden
 drupalorg_drush.drush.inc:818
Whitelist download from http://drupal.org/packaging-whitelist/json   [error]
failed: not found
The Drupal.org validation check failed -- see                        [error]
http://drupal.org/node/1432190 for more information.

Using the latest version:
git clone [user]@git.drupal.org:project/drupalorg_drush.git drupalorg_drush

Any hints on what I'm doing wrong?

Thanks.

EDIT:
Solved by enabling cURL, so it takes that method instead of fopen:
sudo apt-get install php5-curl

Maybe something to look into anyway as it fails when cURL is not enabled. No idea if it would fail also using the wget method.

Still seems to fail if cURL is not enabled. See above.

Thanks.

I wonder if the problem is http vs. https. The http should redirect to https, but maybe that's not happening for your client. What happens if you try to directly curl https://drupal.org/packaging-whitelist/json instead?

Hi,

I got the same problem but the answer does seem working for me. Another idea ?

$ drush verify-makefile
                            [warning]ackaging-whitelist/json): failed to open stream: HTTP request failed! HTTP/1.1 403 Forbidden. You should know better.
 drupalorg_drush.drush.inc:787
Whitelist download from http://drupal.org/packaging-whitelist/json failed: not found                                                                                                   [error]
The drupal.org validation check failed -- see http://drupal.org/node/1432190 for more information.                                                                                     [error]

$ dpkg -l |grep curl
ii  curl                                   7.22.0-3ubuntu4                         Get a file from an HTTP, HTTPS or FTP server
ii  libcurl3                               7.22.0-3ubuntu4                         Multi-protocol file transfer library (OpenSSL)
ii  libcurl3-gnutls                        7.22.0-3ubuntu4                         Multi-protocol file transfer library (GnuTLS)
ii  libcurl3-nss                           7.22.0-3ubuntu4                         Multi-protocol file transfer library (NSS)
ii  php5-curl                              5.3.10-1ubuntu3.8                       CURL module for php5
ii  python-pycurl                          7.19.0-4ubuntu3                         Python bindings to libcurl

$ php -m | grep curl
curl

$ drush status
 PHP executable        :  /usr/bin/php          
 PHP configuration     :  /etc/php5/cli/php.ini 
 PHP OS                :  Linux                 
 Drush version         :  6.1.0                 
 Drush configuration   :                        
 Drush alias files     :                    

Thanks.

Same here, using drush 5.9 on Debian, with php5-curl installed.

Same issue as #19 and #20 whether I use cURL or allow_url_fopen() (only the error message changes).

I read Details on Server SSL Certificates and was able to bypass the issue by using cURL and specifying:

curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);

although I'm not sure that's a good idea. :P