Using a clean install of Aegir 3.143 and hosting_https 3.140 under Apache on Ubuntu 16.04.3 LTS, on a publicly available server my.server.com.

Hosting_https module is enabled with Aegir HTTPS Apache, Aegir Let's Encrypt Service and Aegir Self-signed Service

Server is configured to use Let's Encrypt CA with Staging server (but using Production shows same behaviour) and to use Apache HTTPS using standard ports (80 and 443)

When hostmaster site has HTTPS encryption disabled, site verifies fine and works.

I place a simple html file at /var/aegir/config/letsencrypt.d/well-known/acme-challenge/index.html and can access it without problem at http://my.server.com/.well-known/acme-challenge/index.html

I enable HTTPS encryption on the hostmaster site.

Verifying the site then fails with 1 warning :
Failed to generate Let's Encrypt certificates. - warning
and 4 related warnings (fails to copy cert.pem, privkey.pem and fullchain.pem; fails to restart apache as it does not find certificate files) :

copy(/var/aegir/config/letsencrypt.d/my.server.com/cert.pem): failed to open stream: No such file or directory FileSystem.php:19 - warning
copy(/var/aegir/config/letsencrypt.d/my.server.com/privkey.pem): failed to open stream: No such file or directory FileSystem.php:19 - warning
copy(/var/aegir/config/letsencrypt.d/my.server.com/fullchain.pem): failed to open stream: No such file or directory FileSystem.php:19 - warning
apache on my.server.com could not be restarted. Changes might not be available until this has been done. (error: AH00548: NameVirtualHost has no effect and will be removed in the next release /etc/apache2/conf-enabled/aegir.conf:4 AH00526: Syntax error on line 17 of /var/aegir/config/server_master/apache/vhost.d/my.server.com: SSLCertificateFile: file '/var/aegir/config/server_master/ssl.d/my.server.com/openssl.crt' does not exist or is empty Action 'graceful' failed. The Apache error log may have more information.) - warning

Three files have been created in /var/aegir/config/letsencrypt.d/my.server.com :

-rw------- 1 aegir aegir 1683 Feb 16 18:59 cert-1518807541.csr
-rw------- 1 aegir aegir    0 Feb 16 18:59 cert-1518807541.pem
-rw------- 1 aegir aegir 3243 Feb 16 18:59 privkey-1518807541.pem

The CSR file looks fine and shows the right CN :

openssl req -text -noout -verify -in cert-1518807541.csr 
verify OK
Certificate Request:
    Data:
        Version: 0 (0x0)
        Subject: CN=my.server.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)
                Modulus:
etc.

Certificate

The private key looks fine :

openssl rsa -check -in privkey-1518807541.pem 
RSA key ok
writing RSA key
-----BEGIN RSA PRIVATE KEY-----
### etc. ###
-----END RSA PRIVATE KEY-----

The third file (the chain?) shows problems :

openssl x509 -text -noout -in cert-1518807541.pem 
unable to load certificate
140092606260888:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:701:Expecting: TRUSTED CERTIFICATE

I cannot find any logs giving me details about why and how the certificate generation fails. /var/log/aegir is empty. Nothing useful in Apache error logs. Searched for PHP or Dehydrated logs with no success.

If anyone can point me to how to produce and access logs I can research the issue further.

Comments

antoine.guichard created an issue. See original summary.

agaq’s picture

agaq CreditAttribution: agaq as a volunteer commented

Could anyone give me a hint on how I can produce and access more detailed logs so I can try to understand why letsencrypt certificate generation fails?

I can see in https://www.drupal.org/project/hosting_https/issues/2935425 that one can have more detailed logs but I cannot figure how..

Many thanks.

agaq’s picture

agaq CreditAttribution: agaq as a volunteer commented

Any hint on how I can try figure out why certificate generation fails?
Any help will be greatly appreciated.

colan’s picture

colan
he/him
Primary language English
Location Toronto 🇨🇦
CreditAttribution: colan at Colan Schwartz Consulting commented

I just upgraded myself, and noticed that too. The full Let's Encrypt output doesn't show up in the task log anymore.

I doubt that was intentional, as these are essential for debugging. Let's make it available again.

I ran into this too, but it seemed to work after disabling HTTPS on the site, verifying it, and then enabling it again.

In any case, we need that log output back.

agaq’s picture

agaq CreditAttribution: agaq as a volunteer commented

Disabling HTTPS on the site, verifying it, and then enabling it again did not work for me.
Tried also to revert server to Certificate=None then back to Certificate=LetsEncrypt, but certificate generation still fails.

I will resume testing and researching once we have log output back.

SocialNicheGuru’s picture

SocialNicheGuru CreditAttribution: SocialNicheGuru commented

I have had this issue with providing ssl to my aegir system and to a remote site

Here is what I get when I run verify from the command line
(drush hosting-task --debug --force verify-task-id)

HTTPS certificate directory for <em class="placeholder">mysite.com</em> path [success]
/var/aegir/config/letsencrypt.d/mysite.com is writable. [12.62 sec, 32.41 MB]
Generating Let's Encrypt certificates. [12.62 sec, 32.41 MB]                                        [notice]
Running: /var/aegir/config/letsencrypt/script --cron --accept-terms --config                        [notice]
/var/aegir/config/letsencrypt/config --out /var/aegir/config/letsencrypt.d --domain
mysite.com --domain www.mysite.com [12.62 sec, 32.41 MB]
Executing: /var/aegir/config/letsencrypt/script --cron --accept-terms --config /var/aegir/config/letsencrypt/config --out /var/aegir/config/letsencrypt.d --domain mysite.com --domain www.mysite.com
  # INFO: Using main config file /var/aegir/config/letsencrypt/config
  Processing mysite.com with alternative names: www.mysite.com
   + Signing domains...
   + Generating private key...
   + Generating signing request...
   + Requesting challenge for mysite.com...
   + Requesting challenge for www.mysite.com...
   + Responding to challenge for mysite.com...
   + Responding to challenge for www.mysite.com...
  ERROR: Challenge is invalid! (returned: invalid) (result: {
    "type": "http-01",
    "status": "invalid",
    "error": {
      "type": "urn:acme:error:unauthorized",
      "detail": "Invalid response from http://www.mysite.com/.well-known/acme-challenge/fRzg_CzrK5DHAsWqBSnSwgOg57Iwq8cNeEeL-5pmg2g: \"\u003c!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\"\u003e\n\u003chtml\u003e\u003chead\u003e\n\u003ctitle\u003e404 Not Found\u003c/title\u003e\n\u003c/head\u003e\u003cbody\u003e\n\u003ch1\u003eNot Found\u003c/h1\u003e\n\u003cp\"",
      "status": 403
    },
    "uri": "https://acme-v01.api.letsencrypt.org/acme/challenge/FngEg1o2GOKK71ka-wMT6ILNqW-9FBBh7P7No37J26g/3699015301",
    "token": "fRzg_CzrK5DHAsWqBSnSwgOg57Iwq8cNeEeL-5pmg2g",
    "keyAuthorization": "fRzg_CzrK5DHAsWqBSnSwgOg57Iwq8cNeEeL-5pmg2g.jyRB7cmpB502yvt9ZZvunlzBIUNuWJT_klSumXpr4Rg",
    "validationRecord": [
      {
        "url": "http://www.mysite.com/.well-known/acme-challenge/fRzg_CzrK5DHAsWqBSnSwgOg57Iwq8cNeEeL-5pmg2g",
        "hostname": "www.mysite.com",
        "port": "80",
        "addressesResolved": [
          "my-ip-address"
        ],
        "addressUsed": "my-ip-address"
      }
    ]
  })
# INFO: Using main config file /var/aegir/config/letsencrypt/config [18.46 sec, 32.41 MB]           [notice]
Processing mysite.com with alternative names:                                   [notice]
www.mysite.com [18.46 sec, 32.41 MB]
 + Signing domains... [18.46 sec, 32.41 MB]                                                         [notice]
 + Generating private key... [18.46 sec, 32.41 MB]                                                  [notice]
 + Generating signing request... [18.46 sec, 32.41 MB]                                              [notice]
 + Requesting challenge for mysite.com... [18.46 sec, 32.41 MB]                 [notice]
 + Requesting challenge for www.mysite.com... [18.46 sec, 32.41 MB]             [notice]
 + Responding to challenge for mysite.com... [18.46 sec, 32.41 MB]              [notice]
 + Responding to challenge for www.mysite.com... [18.46 sec, 32.41 MB]          [notice]
ERROR: Challenge is invalid! (returned: invalid) (result: { [18.46 sec, 32.41 MB]                   [notice]
  "type": "http-01", [18.46 sec, 32.41 MB]                                                          [notice]
  "status": "invalid", [18.46 sec, 32.41 MB]                                                        [notice]
  "error": { [18.46 sec, 32.41 MB]                                                                  [notice]
    "type": "urn:acme:error:unauthorized", [18.46 sec, 32.41 MB]                                    [notice]
    "detail": "Invalid response from                                                                [notice]
http://www.mysite.com/.well-known/acme-challenge/fRzg_CzrK5DHAsWqBSnSwgOg57Iwq8cNeEeL-5pmg2g:
\"\u003c!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML
2.0//EN\"\u003e\n\u003chtml\u003e\u003chead\u003e\n\u003ctitle\u003e404 Not
Found\u003c/title\u003e\n\u003c/head\u003e\u003cbody\u003e\n\u003ch1\u003eNot
Found\u003c/h1\u003e\n\u003cp\"", [18.46 sec, 32.41 MB]
    "status": 403 [18.46 sec, 32.41 MB]                                                             [notice]
  }, [18.46 sec, 32.41 MB]                                                                          [notice]
  "uri":                                                                                            [notice]
"https://acme-v01.api.letsencrypt.org/acme/challenge/FngEg1o2GOKK71ka-wMT6ILNqW-9FBBh7P7No37J26g/3699015301",
[18.46 sec, 32.41 MB]
  "token": "fRzg_CzrK5DHAsWqBSnSwgOg57Iwq8cNeEeL-5pmg2g", [18.46 sec, 32.41 MB]                     [notice]
  "keyAuthorization":                                                                               [notice]
"fRzg_CzrK5DHAsWqBSnSwgOg57Iwq8cNeEeL-5pmg2g.jyRB7cmpB502yvt9ZZvunlzBIUNuWJT_klSumXpr4Rg", [18.46
sec, 32.41 MB]
  "validationRecord": [ [18.46 sec, 32.41 MB]                                                       [notice]
    { [18.46 sec, 32.41 MB]                                                                         [notice]
      "url":                                                                                        [notice]
"http://www.mysite.com/.well-known/acme-challenge/fRzg_CzrK5DHAsWqBSnSwgOg57Iwq8cNeEeL-5pmg2g",
[18.46 sec, 32.41 MB]
      "hostname": "www.mysite.com", [18.46 sec, 32.41 MB]                       [notice]
      "port": "80", [18.46 sec, 32.41 MB]                                                           [notice]
      "addressesResolved": [ [18.46 sec, 32.42 MB]                                                  [notice]
        "my-ip-address" [18.46 sec, 32.42 MB]                                                       [notice]
      ], [18.46 sec, 32.42 MB]                                                                      [notice]
      "addressUsed": "my-ip-address" [18.46 sec, 32.42 MB]                                          [notice]
    } [18.46 sec, 32.42 MB]                                                                         [notice]
  ] [18.46 sec, 32.42 MB]                                                                           [notice]
}) [18.46 sec, 32.42 MB]                                                                            [notice]
Failed to generate Let's Encrypt certificates. [18.46 sec, 32.42 MB]                             [warning]
Injecting Let's Encrypt 'well-known' ACME challenge directory '<em                                  [notice]
class="placeholder">/var/aegir/config/letsencrypt.d/well-known/acme-challenge</em>' into Apache
vhost entry. [18.46 sec, 32.42 MB]
Template loaded from Provision Config class <em                                                     [notice]
class="placeholder">Provision_Config_Apache_Https_Site</em>: <em
class="placeholder">/var/aegir/hostmaster-7.x-3.144/profiles/hostmaster/modules/aegir/hosting_https/submodules/apache_https/drush/Provision/Config/Apache/Https/vhost_https_disabled.tpl.php</em>
[18.46 sec, 32.43 MB]
Generated config in write(): encrypted virtual host configuration                                [success]
(/var/aegir/config/server_master/apache/vhost.d/mysite.com) [18.46 sec, 32.43
MB]
HTTPS Certificate directory for <em class="placeholder">mysite.com</em> on   [success]
<em class="placeholder">insitehost.socialnicheguru.com</em> path
/var/aegir/config/server_master/ssl.d/mysite.com exists. [18.46 sec, 32.43
MB]
HTTPS Certificate directory for <em class="placeholder">mysite.com</em> on   [success]
<em class="placeholder">insitehost.socialnicheguru.com</em> ownership of
/var/aegir/config/server_master/ssl.d/mysite.com has been changed to aegir.
[18.46 sec, 32.44 MB]
HTTPS Certificate directory for <em class="placeholder">mysite.com</em> on   [success]
<em class="placeholder">insitehost.socialnicheguru.com</em> permissions of
/var/aegir/config/server_master/ssl.d/mysite.com have been changed to 700.
[18.46 sec, 32.44 MB]
HTTPS Certificate directory for <em class="placeholder">mysite.com</em> on   [success]
<em class="placeholder">insitehost.socialnicheguru.com</em> path
/var/aegir/config/server_master/ssl.d/mysite.com is writable. [18.46 sec,
32.44 MB]
copy(/var/aegir/config/letsencrypt.d/mysite.com/cert.pem): failed to open    [warning]
stream: No such file or directory FileSystem.php:19 [18.46 sec, 32.44 MB]
copy(/var/aegir/config/letsencrypt.d/mysite.com/privkey.pem): failed to open [warning]
stream: No such file or directory FileSystem.php:19 [18.46 sec, 32.44 MB]
copy(/var/aegir/config/letsencrypt.d/mysite.com/fullchain.pem): failed to    [warning]
open stream: No such file or directory FileSystem.php:19 [18.46 sec, 32.44 MB]
Executing: sudo /usr/sbin/apache2ctl graceful
agaq’s picture

agaq CreditAttribution: agaq as a volunteer commented

Many thanks @SocialNicheGuru for the info on how to re-run a task on the command line in a verbose manner. :-)

I did that and it pointed immediately to my problem: I had an alternative name 'my.alternate.name' for the site but the DNS record for ' my.alternate.name' was a CName instead of an A record:

   + Requesting challenge for my.alternate.name...
   + Responding to challenge for my.alternate.name...
  ERROR: Challenge is invalid! (returned: invalid) (result: {
    "type": "http-01",
    "status": "invalid",
    "error": {
      "type": "urn:acme:error:connection",
      "detail": "DNS problem: NXDOMAIN looking up A for my.alternate.name",
      "status": 400
    },
    "uri": "https://acme-v01.api.letsencrypt.org/acme/challenge/####################",
    "token": "####################",
    "keyAuthorization": "####################",
    "validationRecord": [
      {
        "url": "http://my.alternate.name/.well-known/acme-challenge/####################",
        "hostname": "my.alternate.name",
        "port": "80"
      }
    ]
  })

When I remove the domain alias or change it from a CName to an A Record, then all works fine.

I have found mentions of the need for an A record in the LetsEncrypt documentation, but not overly prominent and evident.

I will suggest in a separate ticket some changes in the documentation to help clarify the need for an A Record.

agaq’s picture

agaq CreditAttribution: agaq as a volunteer commented

@SocialNicheGuru:

Your log shows a URI for your certificate at https://acme-v01.api.letsencrypt.org/acme/challenge/FngEg1o2GOKK71ka-wMT...

This URI shows that the ACME challenge it needs to access is expected to be at http://www.insiteapps.socialnicheguru.com/.well-known/acme-challenge/fRz...

But there is nothing at this address

Not Found

The requested URL /.well-known/acme-challenge/fRzg_CzrK5DHAsWqBSnSwgOg57Iwq8cNeEeL-5pmg2g was not found on this server.
Apache/2.4.29 (Ubuntu) Server at www.insiteapps.socialnicheguru.com Port 80

Have you removed it or was it never there?

Have you checked that files in your /.well-known/acme-challenge/ are accessible? - see
See in https://matteobrusa.github.io/md-styler/?url=cgit.drupalcode.org/hosting...

Troubleshooting

If you notice that the certificate generation fails you can check the Aegir 'Verify' task logs for details.
Test the challenge directory

Create a file e.g. called index.html in /var/aegir/config/letsencrypt.d/well-known/acme-challenge/ and test if you can access it over http via http://www.example.com/.well-known/acme-challenge/index.html

If your request is redirected to a https url then that could pose a problem when the certificate there is either invalid or expired. Try to remove the redirects.
SocialNicheGuru’s picture

SocialNicheGuru CreditAttribution: SocialNicheGuru commented

I am creating a site on a remote server where both the apache2 and mysql servers for the site are.
There is no config directory on that machine.

There is no .well-known directory on the remote server.
There are config and .well-known directories on the master server

Is there a problem with deploying letsencyrpt on a remote server?

Jon Pugh’s picture

Jon Pugh
he/him
Primary language English
Location New Windsor NY
CreditAttribution: Jon Pugh at ThinkDrop commented
Category: Support request » Bug report

I received the same warning after installing this, with no further information.

"Failed to generate Let's Encrypt certificates." needs to be more descriptive, and help users figure out what's wrong.

Might want to add a pre_hosting_task to check if the library is available.

colan’s picture

colan
he/him
Primary language English
Location Toronto 🇨🇦
CreditAttribution: colan at Colan Schwartz Consulting commented
Status: Active » Postponed

We need to get #2952411: Restore Let's Encrypt log output in Verify tasks fixed first.

g33kg1rl’s picture

g33kg1rl CreditAttribution: g33kg1rl commented

-

colan’s picture

colan
he/him
Primary language English
Location Toronto 🇨🇦
CreditAttribution: colan at Colan Schwartz Consulting commented
Status: Postponed » Active

The prerequisite is fixed. Can someone post their logs (as an attachment if very long) for this after re-enabling full logging? That will give us the much needed information.