Traefik/LetsEncrypt SSL Certificate Revocation (Jan 2022)

Latest News
1

Hi everyone,

It has come to our attention that an error in the service Traefik uses to generate SSL certificates (that validate encrypted traffic to LAMP), LetsEncrypt, has caused a subset of certificates to not be compliant with their standards. These certificates are invalid and have been revoked as of Jan. 28th.

This issue may affect any LAMP users who have set up their own instance of LAMP - it does not apply to anyone working directly with BIDMC’s instance of LAMP. Basically, if you don’t fill in your own api address on the dashboard log-in page, this won’t affect you.

In addition to @lido 's excellent solution below, LAMP users who use Portainer to manage their LAMP instance can follow these steps to solve the problem through Portainer:

  1. In portainer, navigate to the container currently running your traefik service. It will probably be called something like router_traefik. ​The easiest way I have found to do this is through the container listings accessible through the sidebar.
  2. Use the ‘exec’/‘console’ page (the button should look like >_) to directly interact with the container (note: you CANNOT use /bin/bash/ as your shell command and must use /bin/ash or /bin/sh)
  3. Enter cd /data/ to navigate to the data directory, then run ls - you should see a file named acme.json
    3.5 (OPTIONAL) Run cp acme.json acme_backup_2022.json to create a backup json file.
  4. Exit the console, then scale the traefik service down to 0 and up to 1 (or update the service) using Portainer. This will refresh the SSL certificate. It may take up to 5 minutes.

Please make a post on the Deployment and Development boards [1] if you need our help resolving this issue!

Thanks,

The LAMP team

  1. Deployment & Development - LAMP Consortium ↩︎

1 Like
2

If you host your own instance of LAMP and believe you are impacted by this issue visit the LE Community Post “Questions about Renewing before TLS-ALPN-01 Revocations - Help - Let's Encrypt Community Support” for details.

You should have received an auto-generated email to the certificatesResolvers.default.acme.email address specified in your traefik.yml. The email was sent from Let’s Encrypt noreply@letsencrypt.org prior to Jan 27, 2022 with the subject “[Urgent] Let’s Encrypt revocations affecting your TLS certificates” and body containing one or more impacted ACME registration (account) ID(s).

Let’s Encrypt has also provided a downloadable list (linked in the LE community post above) that you can use to verify if your domain is included on the list of impacted domains:

https://tls-alpn-01-data.letsencrypt.org/tls-alpn-01-affected-certs-by-regID.csv.gz

To remediate the issue I took the following steps:

Use docker inspect router_traefik-ssl to find the mountpoint of your traefik container’s volume:

root@devlamp# docker inspect router_traefik-ssl
[
    {
        "CreatedAt": "2022-01-27T11:38:05-05:00",
        "Driver": "local",
        "Labels": {
            "com.docker.stack.namespace": "router"
        },
        "Mountpoint": "/var/lib/docker/volumes/router_traefik-ssl/_data",
        "Name": "router_traefik-ssl",
        "Options": null,
        "Scope": "local"
    }
]

Make a backup copy of the the acme.json file:

cp /var/lib/docker/volumes/router_traefik-ssl/_data/acme.json /home/devuser/

Remove the traefik docker stack named “router”:

docker stack rm router

Remove the current acme.json file:

rm /var/lib/docker/volumes/router_traefik-ssl/_data/acme.json

Re-deploy the docker stack for traefik. This will regenerate a new acme.json:

docker stack deploy --compose-file traefik.yml router

Now you can comapre the backup copy of acme.json with the newer version that will be automatically generated once the traefik container is initialized. You should take note that the “certificate” and “key” will differ between the two files:

diff /home/devuser/acme.json /var/lib/docker/volumes/router_traefik-ssl/_data/acme.json

Finally, navigating to your production URI “https://api...” you can view the cert details in your browser to verify it is valid and the certificate issue date has been updated.

Hope this helps!

2 Likes
3

@lido That’s a fantastic outline of how to resolve this issue! Thank you so much for sharing!

EDIT: Additionally, here’s the relevant GitHub issue to track this from the technical/development side.