The test runner has started to fail for https://www.drupal.org/project/domain. The project has to setup custom subdomains in order to run tests. We had it working on 30 March, but now the same tests fail.

We have not made changes to the test configuration or core module code in that time.

https://www.drupal.org/pift-ci-job/1632512

The failures are all cURL errors related to subdomain handling. For example:

1) Drupal\Tests\domain\Functional\Views\ActiveDomainDefaultArgumentTest::testActiveDomainDefaultArgument
GuzzleHttp\Exception\ConnectException: cURL error 6: Could not resolve host: one.php-apache-jenkins-drupal-contrib-189314 (see http://curl.haxx.se/libcurl/c/libcurl-errors.html)

We handle subdomains using a custom script, which seems to be firing normally as part of the build:

16:38:58 Container command.
16:38:58 /bin/bash -c "cd ${SOURCE_DIR}/modules/contrib/domain && chmod +x ./define_subdomains.sh && ./define_subdomains.sh"

I really have no idea how to debug this issue.

Comments

agentrickard created an issue. See original summary.

drumm’s picture

drumm
he/him
Location NY, US
commented

I’m not sure if either of these ideas will really lead to resolving the issue, but they might help.

I recommend putting set -x after #!/bin/bash in shell scripts, like https://git.drupalcode.org/project/infrastructure/-/blob/master/live/gen.... This will show the commands being run by the script in the console output. (set -uex also exits the script when using a variable that isn’t initialized or if any command has a non-zero exit status.)

A cat /etc/hosts at the end to double check the hosts were written couldn’t hurt too.

example.com is a real domain. Using example.local might provide better errors if the hosts file isn’t properly overriding the host.

agentrickard’s picture

agentrickard
he/him
Primary language English
Location Georgia (US)
commented

OK, I updated the script. 

#!/bin/bash
set -uex

[[ ${PROJECT_DIR} =~ workspace\/(.*?)\/source ]] && echo "${BASH_REMATCH[1]}"
CONTAINER_NAME=${BASH_REMATCH[1]}
CONTAINER_NAME=${CONTAINER_NAME//_/-}
CONTAINER_NAME='php-apache-'$CONTAINER_NAME

declare -a HOSTS=(${CONTAINER_NAME} 'example.local')
declare -a SUBDOMAINS=('one' 'two' 'three' 'four' 'five' 'six' 'seven' 'eight' 'nine' 'ten')

for HOST in ${HOSTS[@]}; do
    echo '127.0.0.1' ${HOST} >> /etc/hosts

    for SUBDOMAIN in ${SUBDOMAINS[@]}; do
      echo '127.0.0.1' ${SUBDOMAIN}.${HOST} >> /etc/hosts
    done
done

cat /etc/hosts
agentrickard’s picture

agentrickard
he/him
Primary language English
Location Georgia (US)
commented

That seems to pinpoint an issue with the parsing of the directory.

10:34:01 ----------------   Starting container_command   ----------------
10:34:01 Directory created at /var/lib/drupalci/workspace/jenkins-drupal8_contrib_patches-33127/ancillary/container_command
10:34:01 Container command.
10:34:01 /bin/bash -c "cd ${SOURCE_DIR}/modules/contrib/domain && chmod +x ./define_subdomains.sh && ./define_subdomains.sh"
10:34:01 + [[ /var/www/html/modules/contrib/domain =~ workspace/(.*?)/source ]]
10:34:01 ./define_subdomains.sh: line 5: BASH_REMATCH[1]: unbound variable

4. [[ ${PROJECT_DIR} =~ workspace\/(.*?)\/source ]] && echo "${BASH_REMATCH[1]}"
5. CONTAINER_NAME=${BASH_REMATCH[1]}

Trying 

echo ${PROJECT_DIR}
[[ ${PROJECT_DIR} =~ workspace\/(.*?)\/ ]] && echo "${BASH_REMATCH[1]}"
CONTAINER_NAME=${BASH_REMATCH[1]}
CONTAINER_NAME=${CONTAINER_NAME//_/-}
CONTAINER_NAME='php-apache-'$CONTAINER_NAME
agentrickard’s picture

agentrickard
he/him
Primary language English
Location Georgia (US)
commented

So the problem is that we are no longer picking up the container name from the ${SOURCE_DIR} variable.

11:05:13 /bin/bash -c "cd ${SOURCE_DIR}/modules/contrib/domain && chmod +x ./define_subdomains.sh && ./define_subdomains.sh"
11:05:14 /var/www/html/modules/contrib/domain
11:05:14 + echo /var/www/html/modules/contrib/domain
11:05:14 + [[ /var/www/html/modules/contrib/domain =~ workspace/(.*?)/ ]]
11:05:14 + CONTAINER_NAME=
11:05:14 + CONTAINER_NAME=
11:05:14 + CONTAINER_NAME=php-apache-
11:05:14 + HOSTS=(${CONTAINER_NAME} 'example.local')
11:05:14 + declare -a HOSTS
11:05:14 + SUBDOMAINS=('one' 'two' 'three' 'four' 'five' 'six' 'seven' 'eight' 'nine' 'ten')
11:05:14 + declare -a SUBDOMAINS
11:05:14 + for HOST in '${HOSTS[@]}'
11:05:14 + echo 127.0.0.1 php-apache-
11:05:14 + for SUBDOMAIN in '${SUBDOMAINS[@]}'
11:05:14 + echo 127.0.0.1 one.php-apache-
11:05:14 + for SUBDOMAIN in '${SUBDOMAINS[@]}'
11:05:14 + echo 127.0.0.1 two.php-apache-
11:05:14 + for SUBDOMAIN in '${SUBDOMAINS[@]}'
11:05:14 + echo 127.0.0.1 three.php-apache-
11:05:14 + for SUBDOMAIN in '${SUBDOMAINS[@]}'
11:05:14 + echo 127.0.0.1 four.php-apache-
11:05:14 + for SUBDOMAIN in '${SUBDOMAINS[@]}'
11:05:14 + echo 127.0.0.1 five.php-apache-
11:05:14 + for SUBDOMAIN in '${SUBDOMAINS[@]}'
11:05:14 + echo 127.0.0.1 six.php-apache-
11:05:14 + for SUBDOMAIN in '${SUBDOMAINS[@]}'
11:05:14 + echo 127.0.0.1 seven.php-apache-
11:05:14 + for SUBDOMAIN in '${SUBDOMAINS[@]}'
11:05:14 + echo 127.0.0.1 eight.php-apache-
11:05:14 + for SUBDOMAIN in '${SUBDOMAINS[@]}'
11:05:14 + echo 127.0.0.1 nine.php-apache-
11:05:14 + for SUBDOMAIN in '${SUBDOMAINS[@]}'
11:05:14 + echo 127.0.0.1 ten.php-apache-
11:05:14 + for HOST in '${HOSTS[@]}'
11:05:14 + echo 127.0.0.1 example.local
11:05:14 + for SUBDOMAIN in '${SUBDOMAINS[@]}'
11:05:14 + echo 127.0.0.1 one.example.local
11:05:14 + for SUBDOMAIN in '${SUBDOMAINS[@]}'
11:05:14 + echo 127.0.0.1 two.example.local
11:05:14 + for SUBDOMAIN in '${SUBDOMAINS[@]}'
11:05:14 + echo 127.0.0.1 three.example.local
11:05:14 + for SUBDOMAIN in '${SUBDOMAINS[@]}'
11:05:14 + echo 127.0.0.1 four.example.local
11:05:14 + for SUBDOMAIN in '${SUBDOMAINS[@]}'
11:05:14 + echo 127.0.0.1 five.example.local
11:05:14 + for SUBDOMAIN in '${SUBDOMAINS[@]}'
11:05:14 + echo 127.0.0.1 six.example.local
11:05:14 + for SUBDOMAIN in '${SUBDOMAINS[@]}'
11:05:14 + echo 127.0.0.1 seven.example.local
11:05:14 + for SUBDOMAIN in '${SUBDOMAINS[@]}'
11:05:14 + echo 127.0.0.1 eight.example.local
11:05:14 + for SUBDOMAIN in '${SUBDOMAINS[@]}'
11:05:14 + echo 127.0.0.1 nine.example.local
11:05:14 + for SUBDOMAIN in '${SUBDOMAINS[@]}'
11:05:14 + echo 127.0.0.1 ten.example.local
11:05:14 + cat /etc/hosts
11:05:14 127.0.0.1	localhost
11:05:14 ::1	localhost ip6-localhost ip6-loopback
11:05:14 fe00::0	ip6-localnet
11:05:14 ff00::0	ip6-mcastprefix
11:05:14 ff02::1	ip6-allnodes
11:05:14 ff02::2	ip6-allrouters
11:05:14 172.18.0.3	97f3f138a539
11:05:14 127.0.0.1 php-apache-
11:05:14 127.0.0.1 one.php-apache-
11:05:14 127.0.0.1 two.php-apache-
11:05:14 127.0.0.1 three.php-apache-
11:05:14 127.0.0.1 four.php-apache-
11:05:14 127.0.0.1 five.php-apache-
11:05:14 127.0.0.1 six.php-apache-
11:05:14 127.0.0.1 seven.php-apache-
11:05:14 127.0.0.1 eight.php-apache-
11:05:14 127.0.0.1 nine.php-apache-
11:05:14 127.0.0.1 ten.php-apache-
11:05:14 127.0.0.1 example.local
11:05:14 127.0.0.1 one.example.local
11:05:14 127.0.0.1 two.example.local
11:05:14 127.0.0.1 three.example.local
11:05:14 127.0.0.1 four.example.local
11:05:14 127.0.0.1 five.example.local
11:05:14 127.0.0.1 six.example.local
11:05:14 127.0.0.1 seven.example.local
11:05:14 127.0.0.1 eight.example.local
11:05:14 127.0.0.1 nine.example.local
11:05:14 127.0.0.1 ten.example.local

What environment variables do we have that can tell us the name of the container?

In this case, for instance, it is jenkins-drupal8-contrib-patches-33134

Perhaps SOURCE_DIR ?

agentrickard’s picture

agentrickard
he/him
Primary language English
Location Georgia (US)
commented

Well, I tried both SOURCE_DIR and PROJECT_DIR, and neither returns the name of the container.

agentrickard’s picture

agentrickard
he/him
Primary language English
Location Georgia (US)
commented

This seems to be related to changes made by #3119432: The variable ${PROJECT_DIR} is not set correctly.

In that patch for drupal_ci_testbot, SOURCE_DIR and PROJECT_DIR are now both mapped to the root path inside the container? I need the path of the container.

SOURCE_DIR is now /var/www/html

PROJECT_DIR is now /var/www/html/modules/contrib/domain

But PROJECT DIR used to be something like var/lib/drupalci/workspace/jenkins-drupal8_contrib_patches-33134/source

The patch commit definitely fits within my timeline of “this was working and now it is not”

agentrickard’s picture

agentrickard
he/him
Primary language English
Location Georgia (US)
commented

We tried using hostname but that does not return the value we want.

https://dispatcher.drupalci.org/job/drupal8_contrib_patches/33173/console

CONTAINER_NAME=$(hostname)
CONTAINER_NAME=${CONTAINER_NAME//_/-}
CONTAINER_NAME='php-apache-'$CONTAINER_NAME

Returns

15:31:53 f2ffb4927e51
15:31:53 ++ hostname
15:31:53 + CONTAINER_NAME=f2ffb4927e51
15:31:53 + CONTAINER_NAME=f2ffb4927e51
15:31:53 + CONTAINER_NAME=php-apache-f2ffb4927e51

The workspace in this case is jenkins-drupal8_contrib_patches-33173

agentrickard’s picture

agentrickard
he/him
Primary language English
Location Georgia (US)
commented
Category: Support request » Bug report

Due to the changes from the other commit, I'm going to mark this as a bug (regression).

I suspect that we may need another system variable -- or someone just needs to document what that variable currently is and how it can be accessed.

Right now we only have PROJECT_DIR and SOURCE_DIR. It looks like we also need CONTAINER_DIR or CONTAINER_NAME.

Mixologic’s picture

Mixologic
He/Him/His
Primary language English
Location Portland, OR
commented
Project: Drupal.org infrastructure » DrupalCI: Test Runner
Component: Continuous Integration » Testrunner Codebase

The PROJECT_DIR was always intended to be the directory of the project _inside_ the container, so that was a bug that we fixed.

When I fixed it, I checked all of the custom drupalci.yml files to see if anybody was using it/relying on that behavior, but I didnt see that there were scripts that use it.

Anyhow, I got drupalci back into an editbable state (some self tests were failing).

I will go ahead and add CONTAINER_NAME as a variable available within the container so you can use it like you were before (and dont have to rely on mining it out of a filesystem path).

Mixologic’s picture

Mixologic
He/Him/His
Primary language English
Location Portland, OR
commented
Status: Active » Fixed

This was deployed, so you can remove all the parts of your script that sets CONTAINER_NAME, and just use it outright and it should work (I tested locally, and all the domain tests pass)

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.