CI problem with permission when testing 7.x patches [#2837015]

All automated testing of patches at 7.x (both contrib and core) are failing to run, due to a 'permission denied' error. The message is:

23:11:42 Entering setup_fetch().
23:11:42 PHP Warning:  fopen(/var/lib/drupalci/web/jenkins-default-281478/sites/all/modules/scheduler/2794587-3-adding-theme-setting.patch): failed to open stream: Permission denied in /opt/drupalci_testbot/vendor/guzzle/guzzle/src/Guzzle/Http/Message/Request.php on line 438
23:11:42 Write error
23:11:42 An error was encountered while attempting to write https://www.drupal.org/files/issues/2794587-3-adding-theme-setting.patch to /var/lib/drupalci/web/jenkins-default-281478/sites/all/modules/scheduler
23:11:42 Execution Error

The last identified good 7.x test was run on Dec 15 at 11:13 GMT
https://www.drupal.org/pift-ci-job/552804
#1854074: Docblock in field.tpl.php needs clarification, ref theme_field

The first known failure was Dec 16 at 02:23 GMT
https://www.drupal.org/pift-ci-job/553467
#2834022: drupal_sort_weight sort order undefined and possibly inconsistent between PHP 5 and PHP 7

Comment	File	Size	Author
#2	Screen Shot 2016-12-18 at 09.59.37.png	335.64 KB	jonathan1055

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Comment #1

17 December 2016 at 23:19

jonathan1055 created an issue. See original summary.

Comment #2

jonathan1055 CreditAttribution: jonathan1055 as a volunteer commented 18 December 2016 at 10:11

File	Size
Screen Shot 2016-12-18 at 09.59.37.png	335.64 KB

Here's a comparison of the start-up log with failures (right) against a sucessful run from a month ago. There are some differences at the top, in particular the error

[ErrorException]                                                                             
touch(): Unable to create file /home/ubuntu/.composer/config.json because Permission denied

There is also the change about not running as root, but that may be unrelated.

Comment #3

jonathan1055 CreditAttribution: jonathan1055 as a volunteer commented 18 December 2016 at 16:42

I have found other recent issues with this problem
https://www.drupal.org/pift-ci-job/554827 (core 7.x, Dec 18 at 4:04 GMT) from #2043771-41: drupal_mkdir() can't create an absolute path recursively

https://www.drupal.org/pift-ci-job/554128 (views 7.x, Dec 16 at 20:29 GMT) from #2836480-2: Exposed Pager, choosing 0/All results in total rows being set to 0

https://www.drupal.org/pift-ci-job/554124 (Media CK editor 7.x, 16 Dec at 20:10 GMT) from #2833833-4: Extra bullets when widget is in HTML list

Comment #4

jonathan1055 CreditAttribution: jonathan1055 as a volunteer commented 18 December 2016 at 11:39

Title:

CLI Problem with permission on jenkins (since power outage?)

» CI problem with permission when testing 7.x patches (since power outage?)

This only affects tests which try to apply patches. Ongoing testing of the committed codebase run fine. I've only seen it on 7.x so far. My 8.x patch tests run fine.

Comment #5

BR0kEN

Ukrainian

CreditAttribution: BR0kEN as a volunteer and for FFW commented 18 December 2016 at 12:19

Unfortunately this is continue happens. Quite annoying.

Comment #6

jonathan1055 CreditAttribution: jonathan1055 as a volunteer commented 18 December 2016 at 13:27

@BR0kEN, Thanks for the reply.
Is your fail on 7.x or 8.x? Could you post a link to the test page as per #4.
Do you think this is the right queue for this issue? I am surprised nobody from core testing support has responded yet.

When did you have the first failure and when was you most recent successful run before that? All info like this can help to narrow down the cause - it might be due to a change in a test script which can be isolated if we know when the problem first started.

Comment #7

jonathan1055 CreditAttribution: jonathan1055 as a volunteer commented 18 December 2016 at 13:41

Title:

CI problem with permission when testing 7.x patches (since power outage?)

» CI problem with permission when testing 7.x patches

This is nothing to do with the power outage (and possible server restarts) on 18 Dec as it was happening by 16 Dec - see the views failure in #3 above.

Comment #8

drunken monkey

he/him

German

Vienna, Austria

CreditAttribution: drunken monkey as a volunteer commented 18 December 2016 at 20:22

I have two more instances of this happening:
- First one, Dec 16 (#1528806-12: Add 'text_formatted' and 'struct' to allowable field types.)
- Second one (#2632880: Allow bundle setting to be changed on existing indexes)

Thanks for looking into this!

Comment #9

BR0kEN

Ukrainian

CreditAttribution: BR0kEN as a volunteer and for FFW commented 18 December 2016 at 21:57

@jonathan1055, I'm surprised as well. First time I've got this yesterday's morning.

Then I've tried to requeue testing for the patches but results were the same.

I'm suppose that it's a correct place for issue. At least we have it.

Comment #10

jonathan1055 CreditAttribution: jonathan1055 as a volunteer commented 18 December 2016 at 22:42

Thanks drunken monkey and BR0kEN for the links. I am just a contrib maintainer. only collecting information here, in the hope that a Drupal Testbot expert might be able to use it to resolve the problem.

The earliest failure I have found is https://www.drupal.org/pift-ci-job/554124 (Media CK editor 7.x, 16 Dec at 20:10 GMT
The nearest 7.x success before that which I have found is https://www.drupal.org/pift-ci-job/550593 panelizer 7.x, Dec 12 18:03 [I was surpised that so many modules do not have any automated testing! There must be an easier way to find this info other than try to filter the issue board for 'needs review' and looking the the issues manually]

So, between 12 Dec 18:03 and 16 Dec 20:10 there were 27 commits to the DrupalCI testrunner codebase https://www.drupal.org/node/2251767/commits - I am guessing that one of them caused this problem.

Comment #11

jonathan1055 CreditAttribution: jonathan1055 as a volunteer commented 19 December 2016 at 09:47

I have narrowed down the timespan between which the change must have been implemented.

Dec 15 at 11:13 GMT
Successful 7.x core patch test https://www.drupal.org/pift-ci-job/552804
#1854074: Docblock in field.tpl.php needs clarification, ref theme_field

Dec 16 at 02:23 GMT
Failed 7.x core patch test https://www.drupal.org/pift-ci-job/553467
#2834022: drupal_sort_weight sort order undefined and possibly inconsistent between PHP 5 and PHP 7

The comparison of the logs of these two are exactly the same as I showed in #2. Before, at the top of the log we have

Do not run Composer as root/super user! See https://getcomposer.org/root for details

and later there is /usr/local/bin/composer install

For the failures we have instead

[ErrorException]
touch(): Unable to create file /home/ubuntu/.composer/config.json because Permission denied

and composer is run via root/superuser sudo /usr/local/bin/composer install

So that relatively short 14 hour window must contain the commits which caused the 'permission denied' problem. Although the timestamps in the commit log I think are when the commit was made not when it was pushed to live.

Comment #12

MegaChriz CreditAttribution: MegaChriz commented 19 December 2016 at 10:48

Priority:

Normal

» Critical

It happened here as well: #1967240-25: Force IPV4. Sounds like a pretty critical issue if this happens on every patch against D7 code.

Comment #13

DamienMcKenna

NH, USA

CreditAttribution: DamienMcKenna at Mediacurrent commented 19 December 2016 at 13:21

Thanks to jonathan1055 for letting me know about this issue, I ran into the problem with Fieldable Panels Panes' tests.

Comment #14

jonathan1055 CreditAttribution: jonathan1055 as a volunteer commented 19 December 2016 at 14:34

Issue summary:

View changes

Updated summary with info on when the submitted code changed which caused the fault.

Comment #15

MegaChriz CreditAttribution: MegaChriz commented 19 December 2016 at 15:59

In #2828951: Installing 7.x-4.x without Composer I checked if it makes any difference if the module for which a patch is made has a composer.json file (as some of the commits for the testbot from December 16 suggest improvements for contrib projects with a composer.json file, see among more #2597778: Install composer dependencies for contrib projects before running tests), but it appears that there is no difference.

Maybe it is this commit?
http://cgit.drupalcode.org/drupalci_testbot/commit/?id=ab2faf5
In that commit there are a lot of changes related to patch files.

Comment #16

Mixologic

He/Him/His

English

Portland, OR

CreditAttribution: Mixologic at Drupal Association commented 19 December 2016 at 18:06

Assigned:

Unassigned

» Mixologic

Sorry I didnt see this until now. We upgraded jenkins and it changed the order of operations as to when things were being run, and some permissions are out of wack as a result. I had thought I had found them all, but apparently not. Thanks for reporting this, I'll heal this issue ASAP.