[meta] Let's encrypt support

Hi there,

This would be a great feature and I'll help if I can.

I'm not sure I understand what the problem is with IP mapping in the first place? Seems like it should work just as well for letsencrypt as it does elsewhere. Is this part of a bigger issue?

Another problem that I see is that the main letsencrypt client "requires" port 80: https://community.letsencrypt.org/t/le-client-needs-to-bind-to-port-80-w...

So I'm not sure if we can use another client or if we should make have handlers/drivers or whatnot (most other letsencrypt OSS clients don't do this as far as I can tell). Maybe someone can clarify this.

Hi folks,

glad there's an official place to discuss this. I was going to start working on a POC module but it seems that there is some discussion to be done. A couple of points:

• The letsencrypt-auto command line client is fairly slow as it re-verifies all supporting libs each time, but if you run the letsencrypt client directly, it's more than quick enough, I think.
• SNI appears to be a non-issue in my testing - I created 2 separate vhosts on an aegir server configured without an IP, copied certificates generated with letsencrypt over the self-signed certificates generated by aegir, and both work properly.

Here's the way I was thinking of attacking the problem, not arguing that the above doesn't seem more complete, but I think the following could be done in a contrib module without modifying provision:

• Use the letsencrypt command-line client, with its webroot functionality - requires over-riding default apache config & .htaccess to allow serving the .well-known directory, but should be doable in aegir hooks.
• Use the post verify hook for sites to kick off a task that will run the letsencrypt client and copy the certificates to where aegir expects them.
• have a cron task that runs every few days, inspects the certificates for each site with letsencrypt enabled, and re-generates them if they expire in less than 30 days.

I'm happy to work on a sandbox using the approach above, but would like some indication that it makes sense and the code will get used before starting.

@valkum: can you expand on why you wish to use something other than the official client? It will be easily installable via official package repositories and seems like the path of least resistance for most admins, rather than installing a third-party client by hand and writing extra code to interact with it.

@valkum are you referring to running the client directly, or using letsencrypt-auto? I was able to create a certificate for a new domain using the webroot plugin in under 13 seconds, including user input. This doesn't feel like a long time for a hosting task to complete, installing a site with a complex installation profile can easily take an order of magnitude more time.

I'm running ~/.local/share/letsencrypt/bin/letsencrypt directly. From the docs: https://letsencrypt.readthedocs.org/en/latest/using.html#installation:

letsencrypt-auto is the recommended method of running the Let’s Encrypt client beta releases on systems that don’t have a packaged version.

As letsencrypt matures, it's my assumption that the preferred option would be to use system packages.

Note that there's a pure PHP ACME client available: https://github.com/kelunik/acme

I'm not thrilled with the idea of Aegir and some other process trying to manage the same vhosts. That sounds like a recipe for things being broken very badly.

Yeah, I know. I've been a let's encrypt user for a while now (was in the closed beta). We don't need a lot of what the official client does. We just need something that speaks ACME. If it turns out we can use the official client for that, that's great. Just wanted to throw out an alternative approach :)

Check out this library, it has way less dependencies.
https://github.com/analogic/lescript

I just installed from source, as the Debian Jessie packages is not ready yet.

git clone https://github.com/letsencrypt/letsencrypt

# ./letsencrypt-auto

./letsencrypt-auto --apache -d certonly

./letsencrypt-auto --apache -d install

To get the install working I had to copy the vhost config to the apache/sites-enabled dir to avoid an "Unsupported filesystem layout. sites-available/enabled expected" error"

It copied the vhost to a port 443 version and added these lines on the bottom.
```
SSLCertificateFile /etc/letsencrypt/live/s2.dev.voorbeeld-site.nl/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/s2.dev.voorbeeld-site.nl/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
```

It prefers sudo (also to check it's dependencies), but when de jessie Deb is ready we should see if we can use an option to let it store it's stuff in ~aegir/config ... although having the keys just readable for root has it's advantages.

Issues:
- We sync config and ssl certs to slave servers... would we be able to still generate on the master? Or would letsencrypt have to run on the slave? Then we might have to look at smart slaves again... #2037993: Make remote servers "smart"

Pseudo code:

give $key_name a 'letsencrypt_' prefix (Might even become the default if available)

in the code http/Provision/Service/http/ssl.php get_certificates() detect this prefix, call letsEncrypt, and return the full paths.

In http/Provision/Config/Http/Ssl/Site.php we also preg_match on /letsencrypt/
and skip our code copying it to the server specific ssl dir. Using a filename like /etc/letsencrypt/live/example.com/privkey.pem instead

add sudo privs to let provision-verify call letsencrypt.

I tested it manually by copying the generated key and fullchain.pem to our ssl.d dir and running a verify task.

LE supports SNI, so why are we bringing up IPs here? I'm of the opinion that we should drop all IP handling in Aegir, as it's buggy, difficult to manage (e.g., no UI) and doesn't appear to provide any real value (XP support, really?).

Splitting out the self-signed certificate generation seems like a good start. I'd suggest experimenting with promoting certificate generation to a stand-along service. Assuming that was workable, we could then have a default implementation (self-signed) and allow for alternatives (letsencrypt, manual upload via UI, etc.).

Talked about this in the Aegir scrum today. Follow ups here:

* New installations should use SNI by default and should have to specifically opt-in to IP allocation
* Certificate handling will be abstracted out to a separate service with a default implementation of generating self-signed certificates.
* Let's Encrypt can then be implemented as another backend to the certificate service.

Whether or not Let's Encrypt will be included in core or if it will need to be contrib or not was not discussed, but it doesn't much matter. After the service is written, it will be easy to move it.

Still some open questions from helmo above as well. I *think* we will be able to generate certificates on master and sync out to the slaves, but there may be some complications there.

I'm moving this issue under the hostmaster project, as there will need to be both front-end and back-end changes required to accomplish the current plan.

See #2685931: Move SSL certificate creation to a separate service.

Linking the core issue blocking verification...

@helmo we don't need to wait for that issue to get in I don't think, since we control the vhost for the site anyway, we could do what we like with the .well-known directory. For example, I have some custom Puppet code that maps the .well-known/acme-challenge directory to a totally different webroot and then I have a process that manages that.
It's then not-specific to Drupal or relying on some setup in the hosted application.

I'll need LetsEncrypt support on some 800 sites in the next couple of months, so will pick up development of this if no one else does.

The hosting_le module should work now for both Apache and Nginx -- please check its improved readme for details.

We have added a minor tweaks to Apache config templates in Provision to make them compatible with hosting_le:

http://cgit.drupalcode.org/provision/commit/?id=ccce1ca
http://cgit.drupalcode.org/provision/commit/?id=85002c9
http://cgit.drupalcode.org/provision/commit/?id=a163785

The only remaining tasks to fully support it with Apache are:

1. Silence or remove TLS SNI warning: https://github.com/omega8cc/hosting/commit/8ce9922c7f4156556399f52327c62...

2. Hide SSL options on Add Site: https://github.com/omega8cc/hosting/commit/f09ee2e216bf6cb0e16b44f7ee862...

3. Set default value for SSL key on site Edit: https://github.com/omega8cc/hosting/commit/15192e4ec9e7d0f4480a0434ab4ec...

4. Hide all confusing and no longer needed settings (optional to improve UX): https://github.com/omega8cc/eldir/commit/293fc395757e93c0c6366fc929846c1...

5. Use wildcard instead of IP address in listen directive for SSL vhosts -- we did this for Nginx for now: http://cgit.drupalcode.org/provision/commit/?id=f7e2d8d

The commits are problematic, see #2731605: Hardcoded hosting_le code limits behavior with regards to the .well-known directory.

It had to be hardcoded because we don't have proper frontend integration in hosting_le, so it was not possible to make the path configurable.

This can be solved via proper implementation, like hosting_certificate

Once it is configurable (either in hosting_certificate and/or in hosting_le), we can safely replace the hardcoded paths with something like d('@server_master')->http_le_path, etc.

It is not set in stone, after all, and it was a quick shot to make the first integration option working.

That said, since I'm switching my activity elsewhere, so I will not be able to work on Aegir personally anymore, except for some emergency situations/fixes, any further improvements will be submitted as patches for review by Adam. Privately he is my husband, who already runs Omega8.cc since November 2015 and will continue to support Aegir as well, so any prior commitments like Aegir CI server remain unchanged. I may chime in from time to time, though, in case I may be able to help in something. Cheers!

I described the current state of this work following Monday's sprint in IRC: https://hefring.mig5.net/bot/log/aegir/2016-05-25#T622007

Note that we have reverted all hosting_le specific commits and instead implemented this via proper hooks.

We (@colan & I) have made a lot of progress towards getting Let's Encrypt running, but have been fighting with hosting_ssl's self-signed certificate generation. LE requires the use of a chain certificate, so simply overwriting the existing (self-signed) certificates with the LE ones wasn't sufficient. We looked at adding an alter hook in Provision_Service_http_ssl::get_certificates to allow us to re-write the paths that would be injected into the vhosts, but that meant generating our certificates earlier, in a hook_pre_provision_verify(). But this approach ended up with hosting_ssl overwriting our LE certs with the self-signed ones. We considered touching the default chain cert, to allow the web server to restart, or overriding the https vhost templates.

At that point we figured we were piling hacks on top hacks, and after much consideration decided to take the opportunity to clean up hosting_ssl. We're working on this over here: https://gitlab.com/aegir/hosting_https/issues/2

Wish us luck!

Added a link to hosting_le in the summary ...

@ergonlogic where did you get to with your refactoring? What can I do to help?

I've been tracking progress mostly in this catch-all issue: https://gitlab.com/aegir/hosting_https/issues/2.

I'm basically testing Replace all "ssl" instances with "https". at the moment, or last week, anyway; when I last touched this. Haven't committed that bit yet, since it requires re-installing the suite of modules, as db table and field names, context variables and other stuff changed. I can dump that into a commit to a separate branch, if you'd like to debug and such.

The Let's Encrypt logic largely works, and mostly just needs to be moved into get_certificates()/generate_certificates() methods on the service. I suspect that a fair bit can be moved from the self-signed implementation into the Certificate (service-type) class, and called as parent::get_certificates(), as we commonly do in Provision Services.

On a broader note, I think we may want to move back to using a standard location for certificates (rather than letsencrypt.d/), since we are no longer having to try to overwrite/override the self-signed ones.

A huge amount of code was pared away that was solely present to allow for certificate re-use. I think it'll be important to introduce a 'manual' implementation that can allow pasting certificates into the front-end, as there isn't currently a good way to use CA-signed (e.g., EV) certificates.

I would love to test this module and provide feedback. Is there a quick start guide I can use to get Let's Encrypt working with hosting_le?

@g33kg1rl -- We have added a Quick Start guide.

Is this closable now? https://github.com/omega8cc/hosting_le

@realityloop: hosting_le wasn't meant to be a long-term solution.

On that note, I just released the first alpha for Aegir HTTPS. So it's now available for testing. I've been working exclusively with Nginx so feedback on Apache would be useful. Please direct all feedback to the project's issue queue as this ticket is exclusively for meta discussion.

Merge requests (especially those that fix bugs) will happily be reviewed for inclusion.

Please pay attention to the renewal queue as that could use more eyeballs.

Edit: Make sure you're running the latest Hosting development release (until 3.9) or apply the patch from #2824329: Remove 'node_access' check from default hosting_get_servers() calls. See hosting_certificate_prevent_orphaned_services() causing recursive/loop cache rebuild for details.

I have a use case that I would love your help on.

I am using Aegir with Let's Encrypt.
Let's Encrypt only allows ssl on 443.

I am also deploying nodejs on the same server that the aegir site is on.
I need nodejs to be able to connect to any port that I specifcy in my nodejs.config.js file per site.
For example I might need 9243 as my port of choice.

The url for nodejs requests would be https://myurl:9243/socket.io/socket.io.js

However I cannot get this to work with Let's Encrypt.

Is this use case supported by Let's Encrypt? Is it supported by this module?

@SocialNicheGuru if you point nodejs to use the same ssl key files as apache/nginx use then it should 'just' work.

I have hosting_https on a live server now.... seems to work fine. I've also seen the first extra verify tasks from the added 'renew queue'.

So I think we can move to a beta and maybe target the next Aegir release for inclusion. I don't see it upgrade from hosting_le or the other ssl modules automatically though just yet. https://gitlab.com/aegir/hosting_https/issues/21

https://gitlab.com/aegir/hosting_https/issues/29

Here are the draft release blockers. Feel free to update, and/or help: https://gitlab.com/aegir/hosting_https/issues?label_name=Release+blocker

Updated summary.

If you use pound, there is an issue there dealing with reading certs on a mutlisite setup where there is one IP with multiple sites and multiple certs.

https://www.howtoforge.com/community/threads/pound-how-to-implement-open...

However, it seems that only the last certificate specified in pound configuration file is taken in account... The first certificate is shown to the client, if it is related to the domain name, this is fine, else, no more certificate is tried and a warning is shown on the client browser.

I write this because I thought it was an issue with this module.

We finally released an rc1 version of this module.

It's been in production use for ages now ... so please give it a spin.

Hosting_https now has a stable release and should be included in Aegir 3.14 ... #2936056: First stable release