/var/aegir/config/letsencrypt not created on remote server

Thanks for your fast reply!
I didn't mark the other issue as related since I got different output and in the other issue the path to the script was correct.

Since my research got rather lenghty and I wanted to nonetheless provide it here if someone else probably ran into similar problems here is the

TLDR:

Possible improvements for the documentation:

• add a note that the server that hostmaster is running on has to be verified
• add a note that also domain aliases have to be DNS resolveable, similar to the one proposed in https://www.drupal.org/project/hosting_https/issues/2950359
• add somewhere in the documenation the procedure how everything runs. That would be nice to ease troubleshooting by getting a better understanding of the system. Maybe it would be too much for the readme, but a link in README.md to a site on http://docs.aegirproject.org could be useful.

Suggestion: keep a watch on

default file permissions for dehydrated-hooks.sh. Maybe I messed the permissions up myself but I don't think so. Since you set a lot of directory permissions during a server provision-verify maybe that way it could be made sure that all the necessary scripts are executable.

my problem

seems to be finally the same as in the related issue.
Could it be a problem with my network setup?

What I did

I enabled letsencrypt and apache https on the server my hostmaster is hosted.
I verified the server the hostmaster is hosted.
I verified the server the site is hosted.
I verified the site itself.

Excerpt from provision-verify of the site:

HTTPS certificate directory for <em class="placeholder">www.mydomain.at</em> path                                           [success]
/var/aegir/config/letsencrypt.d/www.mydomain.at is writable. [3.79 sec, 43.05 MB]
Generating Let's Encrypt certificates. [3.79 sec, 43.05 MB]                                                                         [notice]
Running: /var/aegir/config/letsencrypt/script --cron --accept-terms --hook /var/aegir/config/letsencrypt/dehydrated-hooks.sh        [notice]
--config /var/aegir/config/letsencrypt/config --out /var/aegir/config/letsencrypt.d --domain www.mydomain.at --domain
mydomain.at [3.79 sec, 43.05 MB]
Executing: /var/aegir/config/letsencrypt/script --cron --accept-terms --hook /var/aegir/config/letsencrypt/dehydrated-hooks.sh --config /var
/aegir/config/letsencrypt/config --out /var/aegir/config/letsencrypt.d --domain www.mydomain.at --domain mydomain.at
  # INFO: Using main config file /var/aegir/config/letsencrypt/config
  Processing www.mydomain.at with alternative names: mydomain.at
   + Signing domains...
   + Generating private key...
   + Generating signing request...
   + Requesting challenge for www.mydomain.at...
   + Requesting challenge for mydomain.at...
  /var/aegir/config/letsencrypt/script: Zeile 569: /var/aegir/config/letsencrypt/dehydrated-hooks.sh: Keine Berechtigung
# INFO: Using main config file /var/aegir/config/letsencrypt/config [11.06 sec, 43.06 MB]                                           [notice]
Processing www.mydomain.at with alternative names: mydomain.at [11.06 sec, 43.06 MB]                                      [notice]
 + Signing domains... [11.06 sec, 43.06 MB]                                                                                         [notice]
 + Generating private key... [11.06 sec, 43.06 MB]                                                                                  [notice]
 + Generating signing request... [11.06 sec, 43.06 MB]                                                                              [notice]
 + Requesting challenge for www.mydomain.at... [11.06 sec, 43.06 MB]                                                           [notice]
 + Requesting challenge for mydomain.at... [11.06 sec, 43.06 MB]                                                               [notice]
/var/aegir/config/letsencrypt/script: Zeile 569: /var/aegir/config/letsencrypt/dehydrated-hooks.sh: Keine Berechtigung [11.06       [notice]
sec, 43.06 MB]
Failed to generate Let's Encrypt certificates. [11.06 sec, 43.06 MB]                                                             [warning]
Injecting Let's Encrypt 'well-known' ACME challenge directory '<em                                                                  [notice]
class="placeholder">/var/aegir/config/letsencrypt.d/well-known/acme-challenge</em>' into Apache vhost entry. [11.06 sec, 43.06
MB]
Template loaded from Provision Config class <em class="placeholder">Provision_Config_Apache_Https_Site</em>: <em                    [notice]
class="placeholder">/var/aegir/hostmaster-7.x-3.140/profiles/hostmaster/modules/aegir/hosting_https/submodules/apache_https/drush/Provision/
Config/Apache/Https/vhost_https.tpl.php</em>
[11.06 sec, 43.08 MB]
Generated config in write(): encrypted virtual host configuration                                                                [success]

It seems the problem with the path to the script has been solved! Thank you!
It seems the problem was that I don't use https on the hostmaster since it is in our internal network and not reachable from the outside I did follow the instructions but didn't think of verifying the hostmaster server. Maybe README.md could be changed to make the necessity more clear.

The new problem still is not the same as in the other remote site issue.
As you see in the output above I get "Keine Berechtigung" notices which means "no permission" when script is calling dehydrated-hooks.sh.

Checking the permissions I found that dehydrated-hooks.sh was 664. When I changed it to 775 I got output which told me that mydomain.at had no DNS resolution.
Since I use www.mydomain.at as primary URL maybe I didn't notice it and it also worked with hosting_le.
Maybe you could change "Ensure that there's a DNS entry for the site that you'd like HTTPS enabled" to "Ensure that there's a DNS entry for the site that you'd like HTTPS enabled and it's aliases" or catch the error in the dehydrated output and add it to the interface.
Of course this is just my bad since I missed to set it up several years ago ;)

So finally I arrived at the same problem the other issue is concerned with.

I get a 403 and a 404 error.
I tried creating a testfile in /var/aegir/config/letsencrypt.d/well-known/acme-challenge (on the remote server) and was able to get to:
http://mydomain.at/.well-known/acme-challenge/test.txt
http://www.mydomain.at/.well-known/acme-challenge/test.txt
from the internet.
In the LAN only http://www.mydomain.at/.well-known/acme-challenge/test.txt is working, because mydomain.at is pointing to our Domaincontroller.

The output now is:

Generating Let's Encrypt certificates. [5.04 sec, 65.49 MB]                                                                         [notice]
Running: /var/aegir/config/letsencrypt/script --cron --accept-terms --hook /var/aegir/config/letsencrypt/dehydrated-hooks.sh        [notice$
--config /var/aegir/config/letsencrypt/config --out /var/aegir/config/letsencrypt.d --domain www.mydomain.at --domain
mydomain.at [5.04 sec, 65.49 MB]
Executing: /var/aegir/config/letsencrypt/script --cron --accept-terms --hook /var/aegir/config/letsencrypt/dehydrated-hooks.sh --config /va$
/aegir/config/letsencrypt/config --out /var/aegir/config/letsencrypt.d --domain www.mydomain.at --domain mydomain.at
  # INFO: Using main config file /var/aegir/config/letsencrypt/config
  Processing www.mydomain.at with alternative names: mydomain.at
   + Signing domains...
   + Generating private key...
   + Generating signing request...
   + Requesting challenge for www.mydomain.at...
   + Already validated!
   + Requesting challenge for mydomain.at...
   + Responding to challenge for mydomain.at...
  ERROR: Challenge is invalid! (returned: invalid) (result: {
    "type": "http-01",
    "status": "invalid",
    "error": {
      "type": "urn:acme:error:unauthorized",
      "detail": "Invalid response from http://mydomain.at/.well-known/acme-challenge/hryviLXP8lVSIePYDwVKzjd54eoxQEgXij5INAF9v00: \"\u$
03c!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\"\u003e\n\u003chtml\u003e\u003chead\u003e\n\u003ctitle\u003e404 Not Found\u003c/title\u$
03e\n\u003c/head\u003e\u003cbody\u003e\n\u003ch1\u003eNot Found\u003c/h1\u003e\n\u003cp\"",
      "status": 403
    },
    "uri": "https://acme-v01.api.letsencrypt.org/acme/challenge/d4VP_VF0rQ96LmuRSzQ9TMhnxvLVNfMelILC-LW9KVA/3811992961",
    "token": "hryviLXP8lVSIePYDwVKzjd54eoxQEgXij5INAF9v00",
    "keyAuthorization": "hryviLXP8lVSIePYDwVKzjd54eoxQEgXij5INAF9v00.qJM4n1t0n5zxq9bnNyt9aALJ-6KMUKJZ-xJjb_apq1A",
    "validationRecord": [
      {
        "url": "http://mydomain.at/.well-known/acme-challenge/hryviLXP8lVSIePYDwVKzjd54eoxQEgXij5INAF9v00",
        "hostname": "mydomain.at",
        "port": "80",
        "addressesResolved": [
          "12.34.56.78"
        ],
        "addressUsed": "12.34.56.78"
      }
    ]
  })
# INFO: Using main config file /var/aegir/config/letsencrypt/config [14.7 sec, 65.49 MB]                                            [notice$
Processing www.mydomain.at with alternative names: mydomain.at [14.7 sec, 65.49 MB]                                       [notice$
 + Signing domains... [14.7 sec, 65.49 MB]                                                                                          [notice$
 + Generating private key... [14.7 sec, 65.49 MB]                                                                                   [notice$
 + Generating signing request... [14.7 sec, 65.49 MB]                                                                               [notice$
 + Requesting challenge for www.mydomain.at... [14.7 sec, 65.49 MB]                                                            [notice$
 + Already validated! [14.7 sec, 65.49 MB]                                                                                          [notice$
 + Requesting challenge for mydomain.at... [14.7 sec, 65.49 MB]                                                                [notice$
 + Responding to challenge for mydomain.at... [14.7 sec, 65.49 MB]                                                             [notice$
ERROR: Challenge is invalid! (returned: invalid) (result: { [14.7 sec, 65.49 MB]                                                    [notice$
  "type": "http-01", [14.7 sec, 65.5 MB]                                                                                            [notice$
  "status": "invalid", [14.7 sec, 65.5 MB]                                                                                          [notice]
  "error": { [14.7 sec, 65.5 MB]                                                                                                    [notice]
    "type": "urn:acme:error:unauthorized", [14.7 sec, 65.5 MB]                                                                      [notice]
    "detail": "Invalid response from                                                                                                [notice]
http://mydomain.at/.well-known/acme-challenge/hryviLXP8lVSIePYDwVKzjd54eoxQEgXij5INAF9v00: \"\u003c!DOCTYPE HTML PUBLIC\"-//IETF//DTD HTML 2.0//EN\"\u003e\n\u003chtml\u003e\u003chead\u003e\n\u003ctitle\u003e404 Not
Found\u003c/title\u003e\n\u003c/head\u003e\u003cbody\u003e\n\u003ch1\u003eNot Found\u003c/h1\u003e\n\u003cp\"", [14.7 sec, 65.5
MB]
    "status": 403 [14.7 sec, 65.5 MB]                                                                                               [notice]
  }, [14.7 sec, 65.5 MB]                                                                                                            [notice]
  "uri": "https://acme-v01.api.letsencrypt.org/acme/challenge/d4VP_VF0rQ96LmuRSzQ9TMhnxvLVNfMelILC-LW9KVA/3811992961", [14.7 sec,   [notice]
65.5 MB]
  "token": "hryviLXP8lVSIePYDwVKzjd54eoxQEgXij5INAF9v00", [14.7 sec, 65.5 MB]                                                       [notice]
  "keyAuthorization": "hryviLXP8lVSIePYDwVKzjd54eoxQEgXij5INAF9v00.qJM4n1t0n5zxq9bnNyt9aALJ-6KMUKJZ-xJjb_apq1A", [14.7 sec, 65.5    [notice]
MB]
  "validationRecord": [ [14.7 sec, 65.5 MB]                                                                                         [notice]
    { [14.7 sec, 65.5 MB]                                                                                                           [notice]
      "url": "http://mydomain.at/.well-known/acme-challenge/hryviLXP8lVSIePYDwVKzjd54eoxQEgXij5INAF9v00", [14.7 sec, 65.5      [notice]
MB]
      "hostname": "mydomain.at", [14.7 sec, 65.5 MB]                                                                           [notice]
      "port": "80", [14.7 sec, 65.5 MB]                                                                                             [notice]
      "addressesResolved": [ [14.7 sec, 65.5 MB]                                                                                    [notice]
        "12.34.56.78" [14.7 sec, 65.5 MB]                                                                                          [notice]
      ], [14.7 sec, 65.5 MB]                                                                                                        [notice]
      "addressUsed": "12.34.56.78" [14.7 sec, 65.5 MB]                                                                             [notice]
    } [14.7 sec, 65.5 MB]                                                                                                           [notice]
  ] [14.7 sec, 65.5 MB]                                                                                                             [notice]
}) [14.7 sec, 65.5 MB]                                                                                                              [notice]
Failed to generate Let's Encrypt certificates. [14.7 sec, 65.5 MB]                                                               [warning]
Injecting Let's Encrypt 'well-known' ACME challenge directory '<em                                                                  [notice]
class="placeholder">/var/aegir/config/letsencrypt.d/well-known/acme-challenge</em>' into Apache vhost entry. [14.7 sec, 65.51 MB]
Template loaded from Provision Config class <em class="placeholder">Provision_Config_Apache_Https_Site</em>: <em                    [notice]
class="placeholder">/var/aegir/hostmaster-7.x-3.140/profiles/hostmaster/modules/aegir/hosting_https/submodules/apache_https/drush/Provision/
Config/Apache/Https/vhost_https.tpl.php</em>
[14.7 sec, 65.52 MB]
Generated config in write(): encrypted virtual host configuration                                                                [success]

What I get from this output is that:
1. www.mydomain.at is already validated (or doesn't need renewal yet since it is issued with hosting_le).
2. the validation for mydomain.at gets a 404 error but it should be working from the internet

Could it be, that my old certificate is interfering?

Do I understand the Workflow of hosting_https correctly?

1. aegir uses dehydrated for letsencrypt webroot domain verification
2. aegir calls dehydrated on the server the hostmaster is running on
3. dehydrated creates a cert and a csr and asks letsencrypt for the challenge token
4. aegir copys the challenge token to /var/aegir/config/letsencrypt.d/well-known/acme-challenge which is mapped in the vhost.conf to http://mydomain.com/.well-known/acme-challenge
5. A Letsencrypt Server tries to read the challenge token and if it is successful it signs the csr and
6. all the chained certificates are created and
7. copied into place by aegir.

I just want to know this to be sure, that our network setup isn't the issue.
In our network both internal (where the hostmaster is running) and the DMZ are connected to the same Firewall which manages all the connections from LAN <=> DMZ, LAN <=> Internet, DMZ <=> Internet.
My hostmaster is not reachable from the internet.
Does dehydrated need to be able to reach http://mydomain.at/.well-known/acme-challenge?

thanks a lot!

Paul

I set up a new hostmaster in a docker container based on https://github.com/aegir-project/dockerfiles and used the remote import module to get my sites there.

Concerning hosting_https I ran into the same issues as before:

• I had to reverify the hostmaster server to get the calling of the script right. Since I ran into this again, would it be possible to add a tast to the queue to reverify the server when hosting_https (or letsencrypt for hosting_https) is turned on?
• I had to fix the permissions as your patch would do.
• An additional problem was that there was no curl in the container, but that is an issue for the aegir - dockerfiles project.

It seems the creation of certificates for www.mydomain.at and test.mydomain.at on a remote server work flawlessly.
As soon as I add mydomain.at as an alias for www.mydomain.at the creation of the let's encrypt certificate fails and I get a warning from drush. It seems before the migration to the new hostmaster it was the same problem. I get the same error messages now as the ones I posted above.
I have no idea why that could be. Since I could narrow the problem down and www.mydomain.at is working it isn't too large of an issue for me, I just wanted to report back and deliver possibly useful information.

Best regards and thanks again for your great work and your help!