Simpletest broken: CURLOPT_COOKIEJAR cannot be NULL [#1671200]

I installed Drupal 7.14 on a Debian testing box (wheezy by this date), enabled Simpletest module, installed the examples module and when I run the simpletest_example test I got a fail on the testing batch AJAX call. Latter I found out that the fail is produced because the log-in page is printed on the response, after the expected JSON string.

Then I went to the test results and it said that the log-in form could not be read.

After debugging I found that the call to curl_setopt_array on DrupalWebTestCase::curlInitialize() is failing because of the option CURLOPT_COOKIEJAR set to NULL. curl_setopt_array stops processing options when an option fails and since CURLOPT_RETURNTRANSFER is after CURLOPT_COOKIEJAR in the array it was not set and curl_exec is echoing the downloaded page.

PHP's curl_setopt manual page (http://co.php.net/manual/en/function.curl-setopt.php) claims that by setting CURLOPT_COOKIEFILE to an empty string "no cookies are loaded, but cookie handling is still enabled". So I tried by setting CURLOPT_COOKIEFILE to '' but it still fails, also tried by setting both CURLOPT_COOKIEFILE and CURLOPT_COOKIEJAR to '' and then to NULL but it always fails no matter how I mix it. This is not what the manual says so I filed a bug report (http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=680260).

I also tried removing CURLOPT_COOKIEJAR and CURLOPT_COOKIEFILE from the options array but then cookies were disabled and userLogin() failed.

Finally I set $this->cookieFile to a temporary file and now everything is working fine. This is how the method looks now:

protected function curlInitialize() {
    global $base_url;

    if (!isset($this->curlHandle)) {
      $this->curlHandle = curl_init();
      $this->cookieFile = tempnam(NULL, "drupal-simpletest-cookiejar-");
      $curl_options = array(
        CURLOPT_COOKIEJAR => $this->cookieFile,
        CURLOPT_URL => $base_url,
        CURLOPT_FOLLOWLOCATION => FALSE,
        CURLOPT_RETURNTRANSFER => TRUE,
        CURLOPT_SSL_VERIFYPEER => FALSE, // Required to make the tests run on https.
        CURLOPT_SSL_VERIFYHOST => FALSE, // Required to make the tests run on https.
        CURLOPT_HEADERFUNCTION => array(&$this, 'curlHeaderCallback'),
        CURLOPT_USERAGENT => $this->databasePrefix,
      );
      if (isset($this->httpauth_credentials)) {
        $curl_options[CURLOPT_HTTPAUTH] = $this->httpauth_method;
        $curl_options[CURLOPT_USERPWD] = $this->httpauth_credentials;
      }
      curl_setopt_array($this->curlHandle, $this->additionalCurlOptions + $curl_options);

      // By default, the child session name should be the same as the parent.
      $this->session_name = session_name();
    }
    // We set the user agent header on each request so as to use the current
    // time and a new uniqid.
    if (preg_match('/simpletest\d+/', $this->databasePrefix, $matches)) {
      curl_setopt($this->curlHandle, CURLOPT_USERAGENT, drupal_generate_test_ua($matches[0]));
    }
}

Comment	File	Size	Author
#33	drupal.simpletest-curl-cookiejar.33.patch	1.49 KB	sun
#33
#28	1671200_28.patch	1.52 KB	chx
#28
#18	core-simpletest-null-cookiejar-1671200_14.patch	1.56 KB	chx
#18
#14	core-simpletest-null-cookiejar-1671200_14.patch	1.56 KB	jaimealsilva
#14
#10	1671200_10.patch	1.56 KB	chx
#10
#9	1671200_9.patch	1.78 KB	chx
#9
#4	drupal.simpletest-curl-cookiejar.D7-do-not-test.patch	2.19 KB	sun
#4
#3	drupal8.simpletest-curl-cookiejar.3.patch	1.91 KB	sun
#3
	drupal-curlopt_cookiejar-debian-do-not-test.patch	572 bytes	jaimealsilva

	snapshot50.png	45.33 KB	jaimealsilva
	snapshot49.png	148.5 KB	jaimealsilva

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Comment #1

jaimealsilva CreditAttribution: jaimealsilva commented 4 July 2012 at 20:32

Drupal status, in case is needed as reference:

Info	Drupal	7.14
Info	Install profile	Minimal (minimal-7.14)
OK	Access to update.php	Protected
OK	Configuration file	Protected
OK	Cron maintenance tasks	Last run 2 hours 26 min ago
You can run cron manually. To run cron from outside the site, go to ...
OK	Database system	MySQL, MariaDB, or equivalent
OK	Database system version	5.5.24-4
OK	Database updates	Up to date
OK	Drupal core update status	Up to date
OK	File system	Writable (public download method)
OK	Module and theme update status	Up to date
OK	Node Access Permissions	Disabled
If the site is experiencing problems with permissions to content, you may have to rebuild the permissions cache. Rebuilding will remove all privileges to content and replace them with permissions based on the current modules and settings. Rebuilding may take some time if there is a lot of content or complex permission settings. After rebuilding has completed, content will automatically use the new permissions. Rebuild permissions
OK	PHP	5.4.4-2 (more information)
OK	PHP DOMDocument class	Enabled
OK	PHP extensions	Enabled
OK	PHP memory limit	512M
OK	PHP open_basedir restriction	Disabled
OK	PHP register globals	Disabled
Warning	Translation update status	No update data available
No information is available about potential new and updated translations for currently installed modules and themes. To check for updates, you may need to run cron or you can check manually. Please note that checking for available updates can take a long time, so please be patient.
OK	Unicode library	PHP Mbstring Extension
OK	Update notifications	Enabled
OK	Web server	Apache/2.2.22 (Debian)
OK	cURL	Enabled
OK	hash	Enabled

Comment #2

sun

German

Karlsruhe

CreditAttribution: sun commented 18 July 2012 at 16:42

Version:

7.14

» 7.x-dev

Comment #3

sun

German

Karlsruhe

CreditAttribution: sun commented 18 July 2012 at 16:58

Title:	CURLOPT_COOKIEJAR can not be NULL on Debian's php5-curl version 5.4.4-2	» CURLOPT_COOKIEJAR cannot be NULL on Debian's php5-curl version 5.4.4-2
Version:	7.x-dev	» 8.x-dev
Status:	Active	» Needs review
Issue tags:		+Needs backport to D7

File	Size
drupal8.simpletest-curl-cookiejar.3.patch	1.91 KB

The provided patch in #0 has been confirmed to work, also on Fedora.

Attached patch performs this change for D8.

Comment #4

sun

German

Karlsruhe

CreditAttribution: sun commented 18 July 2012 at 17:06

File	Size
drupal.simpletest-curl-cookiejar.D7-do-not-test.patch	2.19 KB

Temporary backport for D7. (proper one is blocked on #1541958: Split setUp() into specific sub-methods)

Comment #5

18 July 2012 at 17:24

Status:

Needs review

» Needs work

The last submitted patch, drupal8.simpletest-curl-cookiejar.3.patch, failed testing.

Comment #6

pfrenssen

Sofia

CreditAttribution: pfrenssen commented 19 July 2012 at 09:38

This also affects the contrib Simpletest module in D7.

Comment #7

pfrenssen

Sofia

CreditAttribution: pfrenssen commented 19 July 2012 at 10:46

Title:

CURLOPT_COOKIEJAR cannot be NULL on Debian's php5-curl version 5.4.4-2

» CURLOPT_COOKIEJAR cannot be NULL on php5-curl version 5.4.4

I rerolled this patch for contrib Simpletest in this issue: #1692836: CURLOPT_COOKIEJAR cannot be NULL on PHP version 5.4.4.

Also changing the title, since this does not only affect Debian.

Comment #8

Anonymous (not verified) CreditAttribution: Anonymous commented 19 July 2012 at 16:55

I can confirm #4 solves my problem on my tests site.

Comment #9

chx CreditAttribution: chx commented 20 July 2012 at 02:21

Title:	CURLOPT_COOKIEJAR cannot be NULL on php5-curl version 5.4.4	» Simpletest broken on 5.4: CURLOPT_COOKIEJAR cannot be NULL on php5-curl version 5.4.4
Priority:	Normal	» Critical
Status:	Needs work	» Reviewed & tested by the community

File	Size
1671200_9.patch	1.78 KB

It's enough to set the jar if it's not already set. The session tests set it to $this->cookieFile = file_stream_wrapper_get_instance_by_scheme('temporary')->getDirectoryPath() . '/cookie.' . $uid . '.txt';. THANKS SO MUCH! I was completely taken aback by curl reporting a 1...

Comment #10

chx CreditAttribution: chx commented 20 July 2012 at 02:25

File	Size
1671200_10.patch	1.56 KB

Moving the COOKIEJAR line in the array was unnecessary (this does not change #9. at all. makes the patch shorter, that's all).

Comment #11

chx CreditAttribution: chx commented 20 July 2012 at 02:26

Also, someone needs to compile a php.net bug report. This is definitely not Debian, I am running Arch, PHP is 5.4.4 and cURL is 7.26.0

Comment #12

pfrenssen

Sofia

CreditAttribution: pfrenssen commented 20 July 2012 at 14:59

This issue is probably caused by this bug which was fixed in PHP 5.4.4: CURLOPT_COOKIEFILE '' raises open_basedir restriction.

Comment #13

chx CreditAttribution: chx commented 20 July 2012 at 15:24

We have a problem with the COOKIEJAR and not the COOKIEFILE and also everyone here reports this being still a problem in 5.4.4. But it's very likely the two are related, yes.

Comment #14

jaimealsilva CreditAttribution: jaimealsilva commented 21 July 2012 at 02:09

File	Size
core-simpletest-null-cookiejar-1671200_14.patch	1.56 KB

I think is best to use "empty" instead of "!isset" since it also checks for zero length strings and FALSE which are values that make curl_setopt fail.

Comment #15

21 July 2012 at 02:11

Status:	Reviewed & tested by the community	» Needs work
Issue tags:	-Needs backport to D7

The last submitted patch, core-simpletest-null-cookiejar-1671200_14.patch, failed testing.

Comment #16

jaimealsilva CreditAttribution: jaimealsilva commented 21 July 2012 at 02:15

Status:

Needs work

» Needs review

#14: core-simpletest-null-cookiejar-1671200_14.patch queued for re-testing.

Comment #17

21 July 2012 at 02:17

Status:	Needs review	» Needs work
Issue tags:		+Needs backport to D7

The last submitted patch, core-simpletest-null-cookiejar-1671200_14.patch, failed testing.

Comment #18

chx CreditAttribution: chx commented 21 July 2012 at 06:20

File	Size
core-simpletest-null-cookiejar-1671200_14.patch	1.56 KB

emtpy => empty.

Comment #19

chx CreditAttribution: chx commented 21 July 2012 at 06:20

Status:

Needs work

» Needs review

Comment #20

jaimealsilva CreditAttribution: jaimealsilva commented 21 July 2012 at 15:36

Ops! that's embarrassing. Sorry. Thanks chx.

Comment #21

sun

German

Karlsruhe

CreditAttribution: sun commented 21 July 2012 at 17:27

Status:

Needs review

» Reviewed & tested by the community

Comment #22

Dries CreditAttribution: Dries commented 21 July 2012 at 19:00

Version:	8.x-dev	» 7.x-dev
Status:	Reviewed & tested by the community	» Patch (to be ported)

Committed to 8.x. Thanks! Moving to 7.x.

Comment #23

Sweetchuck

Hungarian

Budapest

CreditAttribution: Sweetchuck commented 21 July 2012 at 19:13

Version:	7.x-dev	» 8.x-dev
Status:	Patch (to be ported)	» Reviewed & tested by the community

I ran into this issue too.

openSUSE 12.1
PHP 5.3.14
PHP cURL Information 7.22.0
PHP cURL SSL Version OpenSSL/1.0.0e
PHP libSSH Version libssh2/1.2.9
by phpinfo()

The patch #18 works well for me.

RTBC++

Comment #24

Tor Arne Thune CreditAttribution: Tor Arne Thune commented 21 July 2012 at 19:21

Version:	8.x-dev	» 7.x-dev
Status:	Reviewed & tested by the community	» Patch (to be ported)

Comment #25

chx CreditAttribution: chx commented 21 July 2012 at 19:30

If 5.3.14 is broken too then https://bugs.php.net/bug.php?id=61948 likely broke the COOKIEJAR cos that bugfix is in 5.3.14 too.

Comment #26

sun

German

Karlsruhe

CreditAttribution: sun commented 21 July 2012 at 20:00

Status:

Patch (to be ported)

» Postponed

The backport is blocked on #1541958: Split setUp() into specific sub-methods (as outlined in #4)

Comment #27

Anonymous (not verified) CreditAttribution: Anonymous commented 21 July 2012 at 20:35

My PHP version 5.4.4 (in Fedora, package php-5.4.4-4.fc17.x86_64)

Comment #28

chx CreditAttribution: chx commented 26 July 2012 at 20:11

Status:

Postponed

» Needs review

File	Size
1671200_28.patch	1.52 KB

How is that blocked?

Comment #29

26 July 2012 at 20:52

Status:

Needs review

» Needs work

The last submitted patch, 1671200_28.patch, failed testing.

Comment #30

sun

German

Karlsruhe

CreditAttribution: sun commented 26 July 2012 at 22:39

Status:

Needs work

» Postponed

Comment #31

chx CreditAttribution: chx commented 27 July 2012 at 06:00

Status:

Postponed

» Needs work

Thanks for the through review and the explanation why my patch didn't work. This just needs fixing because the split is not going to fix this: the only difference between the D7 and the D8 patch is using variable_get to get the public files directory instead of reading it from a class variable. I am throughly intrigued why is that solved by a property and why can we add new properties to D7 test base classes. If we can do that then without the split we can move the public directory into a class property and have the absolute eqiuivalent to D8 patch -- but I have my doubts that'd pass. I will look at this tomorrow.

Comment #32

sun

German

Karlsruhe

CreditAttribution: sun commented 29 July 2012 at 00:28

Priority:	Critical	» Normal
Issue tags:		+PHP 5.4

Downgrading this to normal, since D8 has been fixed, and people using PHP 5.4 with D7 are most likely running into many other incompatibilities anyway, and this one is merely about Simpletest only.

As I've pointed out in #4, the backport depends on #1541958: Split setUp() into specific sub-methods, because we want to keep the testing framework as aligned/comparable between D7 and D8 as possible. That issue is RTBC and will be committed either before the new D7 release or shortly after. Once that patch is in, we can backport this patch as-is. So there's no reason for purposively letting the code diverge.

Comment #33

sun

German

Karlsruhe

CreditAttribution: sun commented 29 July 2012 at 22:18

Status:

Needs work

» Needs review

File	Size
drupal.simpletest-curl-cookiejar.33.patch	1.49 KB

The dependency landed, so here's the clean backport to D7.

Comment #34

chx CreditAttribution: chx commented 30 July 2012 at 06:53

Status:

Needs review

» Reviewed & tested by the community

Oh, awesome.

Comment #35

djdevin

Philadelphia

CreditAttribution: djdevin commented 9 August 2012 at 13:36

This also applies to D6 running PHP 5.3.14

Patch for D7 worked on D6 with some warnings, but fixed the failures and content getting spit out.

Edit: sorry, wrong issue queue. Meant to post to the contrib thread #1692836: CURLOPT_COOKIEJAR cannot be NULL on PHP version 5.4.4

Comment #36

justageek CreditAttribution: justageek commented 14 August 2012 at 23:26

Hi, I don't see this:

$this->public_files_directory

ever being initialized or being a part of the class definition.

Comment #37

sun

German

Karlsruhe

CreditAttribution: sun commented 14 August 2012 at 23:46

@justageek: http://drupalcode.org/project/drupal.git/blob/refs/heads/7.x:/modules/si...

Comment #38

Xano

he/they

English

Southampton

CreditAttribution: Xano commented 18 August 2012 at 06:43

@justageek: you need Drupal 7.15 for this.

I can confirm this patch works ~~and it fixes the problem as it should.~~ (apparently it doesn't, as this is a PHP bug).

//Edit: some clarification: the patch doesn't fix the bug (as it is in PHP), but it does provide a good workaround as mentioned in #40.

Comment #39

pfrenssen

Sofia

CreditAttribution: pfrenssen commented 17 August 2012 at 10:31

PHP 5.4.6 and PHP 5.3.16 have been released which fix this bug.

CURL - Fixed bug #62499 (curl_setopt($ch, CURLOPT_COOKIEFILE, "") returns false).

references:

Comment #40

pfrenssen

Sofia

CreditAttribution: pfrenssen commented 17 August 2012 at 11:05

@Xano, this issue works around the PHP bug, so that Drupal keeps working on systems that have the affected PHP versions installed.

Comment #41

webchick

she/they

English

Vancouver 🇨🇦

CreditAttribution: webchick commented 18 August 2012 at 11:15

Status:

Reviewed & tested by the community

» Fixed

Committed and pushed to 7.x as well. Thanks!

Comment #42

1 September 2012 at 11:21

Status:

Fixed

» Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.

Comment #43

phl3tch CreditAttribution: phl3tch commented 24 September 2012 at 18:20

FYI, I upgraded to PHP 5.3.17 and still had the issue. Patch #33 works for me though.

Comment #44

alberto56 CreditAttribution: alberto56 commented 26 October 2012 at 14:39

Hi all,

I'm a bit confused:

- Comment #41 states that this was committed to 7.x on August 18, 2012.
- I am running 7.16, which was released long after comment #41 was posted
- Still, I had the problem of Curl returning "1" instead of an actual response
- I applied the patch in comment #33, which applied to my code

Hunk #1 succeeded at 1685 (offset 13 lines).
Hunk #2 succeeded at 1706 (offset 13 lines).

My understanding is that since this was committed to 7.x in august, trying to apply this patch to 7.16 should have resulted in a "detected previously applied patch, revert? (y/n)" message. No?

Thanks,

Albert.

Comment #45

David_Rothstein CreditAttribution: David_Rothstein commented 26 October 2012 at 14:46

@alberto56, Drupal 7.16 was a security release only, no bug fixes (see the release notes at http://drupal.org/node/1815904, and also see http://groups.drupal.org/node/260803).

This patch is in 7.x-dev so it will definitely be in the next bug fix release... (which will probably be Drupal 7.17 assuming no other security releases happen before then).