As Let's encrypt is now stable we should use and promote it.

We have a short term solution in hosting_le module, but the supported long term solution is hosting_https.

Here are the current release blockers. Feel free to update, and/or help: https://gitlab.com/aegir/hosting_https/issues?label_name=Release+blocker It's usable today!

Original content

As Let's encrypt is now in public beta i think it would be nice to, at least, collect some information on how to implement support with Aegir. At the end of this i hope to get let's encrypt signed certificates for my sites in Aegir.

Marking this as meta as i don't know how much code change/rework is needed. If there are open issues for fixing some points of my liste below I will add a reference to that points.

After digging a little bit through the provision 7.x-3.2 code here are some points that I think are important before implementing the actual letsencrypt support.

  1. Decouple the openssl code from the default Provision_Service_http_ssl
  2. Introduce something like Provision_Service_http_ssl_selfsigned for selfsigned Certificates
  3. fixing SAN/SNI problems
  4. Finally: add something like Provision_Service_http_ssl_letsencrypt

Short term

For the short term there is the hosting_le module. Which is usable today.

Details

Decouple the openssl code from the default Provision_Service_http_ssl


ATM the only way for ssl support is Provision_Service_http_ssl. Here is checked if a certificate for a given ssl_key (the variable ssl_key is IMHO wrong named. Should be named cert_id or so) is present if not nevertheless generate a self signed key.
So for easy encryption of customers or others needs access to the ssl.d dir to put signed certs in there.
Decoupling the generating process (maybe the deletion process too) from this class could ease up the creation of other creation modes (like lets encrypt and maybe key and cert text input for customers.)

Introduce something like Provision_Service_http_ssl_selfsigned for selfsigned Certificates


As we removed this from Provision_Service_http_ssl bring back selfsigned-certificate generation.
But in addition it would be nice to only generate only new certificates on action (Maybe a new task in hosting)

Some additional thoughts for Hosting.
On Site creation/edit the following should happen: After crawling all certificates (parsing them) hosting should give a list of certs that are suitable for this domain and the manager of the site can choose which cert should be used. Anyway show generate option. (Supporting various generators such as the selfsigned-cert generator or letsencrypt).


fixing SAN/SNI problems - Should be a new issue


ATM the vhost_ssl template forces the ip to *. I'm not quite sure how much the IP per certificate issue is fixed right now.
But here is a sample configuration that should be doable IMHO.
Given a server with 3 IPs.
IP1 used with SNI. The Webserver listens on IP1:80 and IP1:443.
IP2 used for stand alone cert. The Webserver additionally listens on IP2:80 and IP2:443. (Clients that want windows XP support)
IP3 is used for non-SSL sites only. The Webserver additionally listens only on IP3:80 not on IP3:443. (Imagine a site has no cert but is hosted on IP1, and someone requesting https://site/)

I have no finished solution for this. I think atm for the scenario above i would create 3 servers in Aegir on the same local machine, each its own IP. (I did not test this)
Maybe someone has such a setup and can report here how he solved this.
I tested this and looked some code up. So it seems we have to change the mapping of ports to servers to ports to ips to servers in hosting (see class hostingService). As mysql doesn't support mulitple IPs and various ports on these IPs. Apache does and i think we should support this as well.

Finally: add something like Provision_Service_http_ssl_letsencrypt for Let's encrypt support

Implemente the letsencrypt part.
There are some question on how to do this.
1. Use a letsencrypt client (ACME client) not written in php and call it via drush_shell_exec or use a php implementation of an ACME client.
I tested the letsencrypt client from letsencrypt (python) on my server and the start was pretty slow. (updates and so on) So maybe another client implementation would be better.
For authentication i would suggest some .htaccess magic (nginx support would be nice too) for the .acme/ dir under the domain root. (For SAN certificates )

2. How to renew the certs.
As the Let's Encrypt certs are short lived (90 days) certificates at least once in 3 month a SSL renew Task should be triggered automatically.
For this, using the letsencrypt reference client, running the same command as for creation is enough. Maybe different on other ACME clients.

Oh that was a wall of text. I hope some people around here are interested in such a letsencrypt support or further away such a rewrite. If i missed anything, i'm open for discussion.

Comments

valkum created an issue. See original summary.

valkum’s picture

Issue summary: View changes

Tested my possible solution for the scenario I presented. Seems like it doesn't work that way.

gboudrias’s picture

Hi there,

This would be a great feature and I'll help if I can.

I'm not sure I understand what the problem is with IP mapping in the first place? Seems like it should work just as well for letsencrypt as it does elsewhere. Is this part of a bigger issue?

Another problem that I see is that the main letsencrypt client "requires" port 80: https://community.letsencrypt.org/t/le-client-needs-to-bind-to-port-80-w...

So I'm not sure if we can use another client or if we should make have handlers/drivers or whatnot (most other letsencrypt OSS clients don't do this as far as I can tell). Maybe someone can clarify this.

acrollet’s picture

Hi folks,

glad there's an official place to discuss this. I was going to start working on a POC module but it seems that there is some discussion to be done. A couple of points:

  • The letsencrypt-auto command line client is fairly slow as it re-verifies all supporting libs each time, but if you run the letsencrypt client directly, it's more than quick enough, I think.
  • SNI appears to be a non-issue in my testing - I created 2 separate vhosts on an aegir server configured without an IP, copied certificates generated with letsencrypt over the self-signed certificates generated by aegir, and both work properly.

Here's the way I was thinking of attacking the problem, not arguing that the above doesn't seem more complete, but I think the following could be done in a contrib module without modifying provision:

  • Use the letsencrypt command-line client, with its webroot functionality - requires over-riding default apache config & .htaccess to allow serving the .well-known directory, but should be doable in aegir hooks.
  • Use the post verify hook for sites to kick off a task that will run the letsencrypt client and copy the certificates to where aegir expects them.
  • have a cron task that runs every few days, inspects the certificates for each site with letsencrypt enabled, and re-generates them if they expire in less than 30 days.

I'm happy to work on a sandbox using the approach above, but would like some indication that it makes sense and the code will get used before starting.

valkum’s picture

@gboudrias: Yeah, maybe the IP Stuff is better handled in another issue. SNI is working atm. Issues with the implementation atm should also then handled an the other issue.

The letsencrypt client only needs port 80 for one of his auth types. @as acrolet stated we can use the auth type using the webroot. The client will put the acme challenge into the sites files dir. The only problem here is with SAN (if we want to support SAN), because we need to tell the client multiple webroots (I'm not quite shure if this is working atm in the official client, simp_le supports this (-d example.net:/var/www/other_html) ).

@acrollet: Yeah. maybe we should consider using one of the other clients.
simp_le only saves the cert in his dir without any domain specific file names, so this is maybe a bad solution.
There are two other python clients: https://github.com/diafygi/acme-tiny and https://github.com/mail-in-a-box/free_tls_certificates
And there is a go client.
https://github.com/letsencrypt/letsencrypt/wiki/Links

the most tiny client (acme-tiny) sounds great, but then we need to create our key and csr by ourselfs and the current csr generation only works for one domain. (so only www.example.net or example.net)

free_tls_certificates and go are more libraries than clients but we can write wrappers for our purpose.

In general this seems like a good start for lets encrypt support.

For the start sys admins needs to create the acme account and the email should be fixed in aegir. Maybe we can add aegir client <->acme account mappings later. Same for the TOS agreement.

valkum’s picture

Issue summary: View changes
valkum’s picture

Issue summary: View changes
acrollet’s picture

@valkum: can you expand on why you wish to use something other than the official client? It will be easily installable via official package repositories and seems like the path of least resistance for most admins, rather than installing a third-party client by hand and writing extra code to interact with it.

valkum’s picture

because it tries to upload the client on every call. i don't know if this is also the case if you install it via a package manager. so the startup is rather slow.

acrollet’s picture

@valkum are you referring to running the client directly, or using letsencrypt-auto? I was able to create a certificate for a new domain using the webroot plugin in under 13 seconds, including user input. This doesn't feel like a long time for a hosting task to complete, installing a site with a complex installation profile can easily take an order of magnitude more time.

valkum’s picture

Are you running ~/.local/share/letsencrypt/bin/letsencrypt directly? or letsencrypt-auto?
The Docs of Let's Encrypt say to run the letsencrypt-auto bin. ("Throughout the documentation, whenever you see references to letsencrypt script/binary, you can substitute in letsencrypt-auto"

13 seconds sounds nice.

acrollet’s picture

I'm running ~/.local/share/letsencrypt/bin/letsencrypt directly. From the docs: https://letsencrypt.readthedocs.org/en/latest/using.html#installation:

letsencrypt-auto is the recommended method of running the Let’s Encrypt client beta releases on systems that don’t have a packaged version.

As letsencrypt matures, it's my assumption that the preferred option would be to use system packages.

realityloop’s picture

Issue summary: View changes
cweagans’s picture

Note that there's a pure PHP ACME client available: https://github.com/kelunik/acme

I'm not thrilled with the idea of Aegir and some other process trying to manage the same vhosts. That sounds like a recipe for things being broken very badly.

valkum’s picture

@cweagans: the official letsencrypt client has different plugins. One standalone plugin which requires root as it will start a local webserver to verify you are the owner of this domain. one for apache and one for nginx, which will setup a vhost for you with the given domain. A manual mode which will print the ACME challange to stdout for further processing (e.g. putting it into a file) and a webroot plugin which will put the ACME challange file into the given webroot.

I tried to install the acme php library, but with my composer setup a dependency of the lib was deprecated. It relies on rdlowrey/nbsock which is abandoned. A pure php solution would be the best option in my eyes, but the kelunik/acme is unsing some php async magic where i don't know how well it will work with drush.

cweagans’s picture

Yeah, I know. I've been a let's encrypt user for a while now (was in the closed beta). We don't need a lot of what the official client does. We just need something that speaks ACME. If it turns out we can use the official client for that, that's great. Just wanted to throw out an alternative approach :)

thtas’s picture

Check out this library, it has way less dependencies.
https://github.com/analogic/lescript

helmo’s picture

I just installed from source, as the Debian Jessie packages is not ready yet.

git clone https://github.com/letsencrypt/letsencrypt

# ./letsencrypt-auto

./letsencrypt-auto --apache -d certonly

./letsencrypt-auto --apache -d install

To get the install working I had to copy the vhost config to the apache/sites-enabled dir to avoid an "Unsupported filesystem layout. sites-available/enabled expected" error"

It copied the vhost to a port 443 version and added these lines on the bottom.
```
SSLCertificateFile /etc/letsencrypt/live/s2.dev.voorbeeld-site.nl/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/s2.dev.voorbeeld-site.nl/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
```

It prefers sudo (also to check it's dependencies), but when de jessie Deb is ready we should see if we can use an option to let it store it's stuff in ~aegir/config ... although having the keys just readable for root has it's advantages.

Issues:
- We sync config and ssl certs to slave servers... would we be able to still generate on the master? Or would letsencrypt have to run on the slave? Then we might have to look at smart slaves again... #2037993: Make remote servers "smart"

Pseudo code:

give $key_name a 'letsencrypt_' prefix (Might even become the default if available)

in the code http/Provision/Service/http/ssl.php get_certificates() detect this prefix, call letsEncrypt, and return the full paths.

In http/Provision/Config/Http/Ssl/Site.php we also preg_match on /letsencrypt/
and skip our code copying it to the server specific ssl dir. Using a filename like /etc/letsencrypt/live/example.com/privkey.pem instead

add sudo privs to let provision-verify call letsencrypt.

I tested it manually by copying the generated key and fullchain.pem to our ssl.d dir and running a verify task.

ergonlogic’s picture

LE supports SNI, so why are we bringing up IPs here? I'm of the opinion that we should drop all IP handling in Aegir, as it's buggy, difficult to manage (e.g., no UI) and doesn't appear to provide any real value (XP support, really?).

Splitting out the self-signed certificate generation seems like a good start. I'd suggest experimenting with promoting certificate generation to a stand-along service. Assuming that was workable, we could then have a default implementation (self-signed) and allow for alternatives (letsencrypt, manual upload via UI, etc.).

cweagans’s picture

Talked about this in the Aegir scrum today. Follow ups here:

* New installations should use SNI by default and should have to specifically opt-in to IP allocation
* Certificate handling will be abstracted out to a separate service with a default implementation of generating self-signed certificates.
* Let's Encrypt can then be implemented as another backend to the certificate service.

Whether or not Let's Encrypt will be included in core or if it will need to be contrib or not was not discussed, but it doesn't much matter. After the service is written, it will be easy to move it.

Still some open questions from helmo above as well. I *think* we will be able to generate certificates on master and sync out to the slaves, but there may be some complications there.

ergonlogic’s picture

Project: Provision » Hostmaster (Aegir)
Component: HTTP Service » API

I'm moving this issue under the hostmaster project, as there will need to be both front-end and back-end changes required to accomplish the current plan.

ergonlogic’s picture

helmo’s picture

Linking the core issue blocking verification...

Steven Jones’s picture

@helmo we don't need to wait for that issue to get in I don't think, since we control the vhost for the site anyway, we could do what we like with the .well-known directory. For example, I have some custom Puppet code that maps the .well-known/acme-challenge directory to a totally different webroot and then I have a process that manages that.
It's then not-specific to Drupal or relying on some setup in the hosted application.

I'll need LetsEncrypt support on some 800 sites in the next couple of months, so will pick up development of this if no one else does.

omega8cc’s picture

The hosting_le module should work now for both Apache and Nginx -- please check its improved readme for details.

We have added a minor tweaks to Apache config templates in Provision to make them compatible with hosting_le:

http://cgit.drupalcode.org/provision/commit/?id=ccce1ca
http://cgit.drupalcode.org/provision/commit/?id=85002c9
http://cgit.drupalcode.org/provision/commit/?id=a163785

The only remaining tasks to fully support it with Apache are:

1. Silence or remove TLS SNI warning: https://github.com/omega8cc/hosting/commit/8ce9922c7f4156556399f52327c62...

2. Hide SSL options on Add Site: https://github.com/omega8cc/hosting/commit/f09ee2e216bf6cb0e16b44f7ee862...

3. Set default value for SSL key on site Edit: https://github.com/omega8cc/hosting/commit/15192e4ec9e7d0f4480a0434ab4ec...

4. Hide all confusing and no longer needed settings (optional to improve UX): https://github.com/omega8cc/eldir/commit/293fc395757e93c0c6366fc929846c1...

5. Use wildcard instead of IP address in listen directive for SSL vhosts -- we did this for Nginx for now: http://cgit.drupalcode.org/provision/commit/?id=f7e2d8d

gboudrias’s picture

omega8cc’s picture

It had to be hardcoded because we don't have proper frontend integration in hosting_le, so it was not possible to make the path configurable.

This can be solved via proper implementation, like hosting_certificate

Once it is configurable (either in hosting_certificate and/or in hosting_le), we can safely replace the hardcoded paths with something like d('@server_master')->http_le_path, etc.

It is not set in stone, after all, and it was a quick shot to make the first integration option working.

omega8cc’s picture

That said, since I'm switching my activity elsewhere, so I will not be able to work on Aegir personally anymore, except for some emergency situations/fixes, any further improvements will be submitted as patches for review by Adam. Privately he is my husband, who already runs Omega8.cc since November 2015 and will continue to support Aegir as well, so any prior commitments like Aegir CI server remain unchanged. I may chime in from time to time, though, in case I may be able to help in something. Cheers!

ergonlogic’s picture

I described the current state of this work following Monday's sprint in IRC: https://hefring.mig5.net/bot/log/aegir/2016-05-25#T622007

omega8cc’s picture

ergonlogic’s picture

We (@colan & I) have made a lot of progress towards getting Let's Encrypt running, but have been fighting with hosting_ssl's self-signed certificate generation. LE requires the use of a chain certificate, so simply overwriting the existing (self-signed) certificates with the LE ones wasn't sufficient. We looked at adding an alter hook in Provision_Service_http_ssl::get_certificates to allow us to re-write the paths that would be injected into the vhosts, but that meant generating our certificates earlier, in a hook_pre_provision_verify(). But this approach ended up with hosting_ssl overwriting our LE certs with the self-signed ones. We considered touching the default chain cert, to allow the web server to restart, or overriding the https vhost templates.

At that point we figured we were piling hacks on top hacks, and after much consideration decided to take the opportunity to clean up hosting_ssl. We're working on this over here: https://gitlab.com/aegir/hosting_https/issues/2

Wish us luck!

ergonlogic’s picture

Issue tags: +DrupalNorth2016
helmo’s picture

Issue summary: View changes

Added a link to hosting_le in the summary ...

Steven Jones’s picture

@ergonlogic where did you get to with your refactoring? What can I do to help?

ergonlogic’s picture

I've been tracking progress mostly in this catch-all issue: https://gitlab.com/aegir/hosting_https/issues/2.

I'm basically testing Replace all "ssl" instances with "https". at the moment, or last week, anyway; when I last touched this. Haven't committed that bit yet, since it requires re-installing the suite of modules, as db table and field names, context variables and other stuff changed. I can dump that into a commit to a separate branch, if you'd like to debug and such.

The Let's Encrypt logic largely works, and mostly just needs to be moved into get_certificates()/generate_certificates() methods on the service. I suspect that a fair bit can be moved from the self-signed implementation into the Certificate (service-type) class, and called as parent::get_certificates(), as we commonly do in Provision Services.

On a broader note, I think we may want to move back to using a standard location for certificates (rather than letsencrypt.d/), since we are no longer having to try to overwrite/override the self-signed ones.

A huge amount of code was pared away that was solely present to allow for certificate re-use. I think it'll be important to introduce a 'manual' implementation that can allow pasting certificates into the front-end, as there isn't currently a good way to use CA-signed (e.g., EV) certificates.

g33kg1rl’s picture

I would love to test this module and provide feedback. Is there a quick start guide I can use to get Let's Encrypt working with hosting_le?

memtkmcc’s picture

@g33kg1rl -- We have added a Quick Start guide.

realityloop’s picture

colan’s picture

Status: Active » Needs review

@realityloop: hosting_le wasn't meant to be a long-term solution.

On that note, I just released the first alpha for Aegir HTTPS. So it's now available for testing. I've been working exclusively with Nginx so feedback on Apache would be useful. Please direct all feedback to the project's issue queue as this ticket is exclusively for meta discussion.

Merge requests (especially those that fix bugs) will happily be reviewed for inclusion.

Please pay attention to the renewal queue as that could use more eyeballs.

Edit: Make sure you're running the latest Hosting development release (until 3.9) or apply the patch from #2824329: Remove 'node_access' check from default hosting_get_servers() calls. See hosting_certificate_prevent_orphaned_services() causing recursive/loop cache rebuild for details.

SocialNicheGuru’s picture

I have a use case that I would love your help on.

I am using Aegir with Let's Encrypt.
Let's Encrypt only allows ssl on 443.

I am also deploying nodejs on the same server that the aegir site is on.
I need nodejs to be able to connect to any port that I specifcy in my nodejs.config.js file per site.
For example I might need 9243 as my port of choice.

The url for nodejs requests would be https://myurl:9243/socket.io/socket.io.js

However I cannot get this to work with Let's Encrypt.

Is this use case supported by Let's Encrypt? Is it supported by this module?

helmo’s picture

@SocialNicheGuru if you point nodejs to use the same ssl key files as apache/nginx use then it should 'just' work.

helmo’s picture

I have hosting_https on a live server now.... seems to work fine. I've also seen the first extra verify tasks from the added 'renew queue'.

So I think we can move to a beta and maybe target the next Aegir release for inclusion. I don't see it upgrade from hosting_le or the other ssl modules automatically though just yet. https://gitlab.com/aegir/hosting_https/issues/21

https://gitlab.com/aegir/hosting_https/issues/29

colan’s picture

Here are the draft release blockers. Feel free to update, and/or help: https://gitlab.com/aegir/hosting_https/issues?label_name=Release+blocker

helmo’s picture

Issue summary: View changes

Updated summary.

SocialNicheGuru’s picture

If you use pound, there is an issue there dealing with reading certs on a mutlisite setup where there is one IP with multiple sites and multiple certs.

https://www.howtoforge.com/community/threads/pound-how-to-implement-open...

However, it seems that only the last certificate specified in pound configuration file is taken in account... The first certificate is shown to the client, if it is related to the domain name, this is fine, else, no more certificate is tried and a warning is shown on the client browser.

I write this because I thought it was an issue with this module.