Automating Wildcard HTTPS Renewal Through LetsEncrypt
Posted: Tue Jun 25, 2024 4:15 pm
serynn.ca and bbs.serynn.ca (as well as EQAtlas, ECAtlas, etc.) rely on TLS/SSL (over HTTPS) to ensure that communication between the server and the viewer is encrypted. Since the release of a free Certificate Authority in 2015 (LetsEncrypt), it's now become pretty standard practice for all web services to run on HTTPS by default (instead of basic HTTP).
The caveat of using LetsEncrypt is it only offers certificates that last 3 months - which means they have to be renewed and reinstalled every three months or the service will stop working.
I've been doing this manually for a long time, with the goal of automating it "one day"... well with a new certificate renewal right around the corner, now is the time I want to automate this!
Below are the steps I've taken to help achieve this goal.
This is a lot longer than it needs to be, because I go through the kind of process I do for most things I'm learning on-the-fly (problem/goal => research => attempt => testing => troubleshooting)
Problem / Goal
LetsEncrypt certificates expire every 3 months... I don't want to do this manually for all my personal & clients' domains.
Automatically renew my HTTPS Encryption Certificates for Wildcard Certs from LetsEncrypt, through DNS Challenge Verification!
Research
I found what looks like a good tutorial that will work out for me : https://kevingoedecke.com/2023/05/17/ho ... ernal-dns/
This tutorial does have information about the authentication requirements I'll need to setup, and plugins I will need to install, which will be helpful.
One issue is that this doesn't explicitly mention wildcard certificates, so I might run into an issue there with my setup (or more specifically the command I need to use to generate the certificate).
The tutorial also outlines the general process:
1. A LetsEncrypt plugin (certbot-dns-route53 aka dns_route53) is used to manage Route53
* This plugin will create temporary TXT DNS records to verify the ownership of the domain
2. A user account must be created with enough permissions to use the AWS APIs for the DNS management (used by the plugin in step 1)
3. (Legacy) A Cronjob could be setup to execute the process at pre-defined schedules
Setup the necessary role / permissions
Reference: https://certbot-dns-route53.readthedocs.io/en/stable/
Install the dns_route53 plugin
Testing & Troubleshooting the Plugin / Connections
After adding the AWS config access key and secret to the server, I executed a test - which failed:
Needs further investigation as to why the config is not being used successfully.
When testing the automatic renewal (dry run) process:
This configuration ( /etc/letsencrypt/renewal/serynn.ca.conf ) was created automatically in previous domain setup, so it looks like I need to modify this for the new process.
Under [renewalparams] I modified the "authenticator" from 'manual' to 'dns-route53':
Re-Ran the dry run and we're one step closer, but still a new error that now clarifies it IS using the correct AWS Key and Secret, but there seems to be a policy error denying the permissions:
After retesting the dry-run multiple times, I noticed that the specified resource ( arn:aws:route53:::change/<AWSHOSTEDZONEID> ) changed EVERY time I retried it... this led me to believe it was specifically creating a new resource. Now that I think about this, it makes a lot of sense - the DNS challenge requires creating new DNS TXT records as part of the verification process, and the permissions need allow the plugin to confirm that the new records were actually created.
My IAM permissions were in fact too restrictive - I had to open up the permissions more. The following permission worked (but I should research more if I can lock this down further - you should only grant just enough permissions for the bot to be able to achieve its task, and nothing more!):
And the successful Dry Run execution / output:
Success!
(Legacy/Optional) Define the cronjob
I was researching the best recommendations for the cron, but it seems like it's no longer required - certbot will install a system.d timer, and I confirmed my version (1.21) does have the timer installed.
Here's the link to where I found this information: https://serverfault.com/questions/79077 ... wer-924695
I need to check back within 24 hours to confirm that this is working automatically, now that I have the successful dry-run tests confirmed.
Even if it works automatically, I'll probably still need to automate the reload of my web server (nginx) configuration every day though so that it uses the latest certificate that's automatically installed.
Final Notes
This information will be filled out as its completed.
The caveat of using LetsEncrypt is it only offers certificates that last 3 months - which means they have to be renewed and reinstalled every three months or the service will stop working.
I've been doing this manually for a long time, with the goal of automating it "one day"... well with a new certificate renewal right around the corner, now is the time I want to automate this!
Below are the steps I've taken to help achieve this goal.
This is a lot longer than it needs to be, because I go through the kind of process I do for most things I'm learning on-the-fly (problem/goal => research => attempt => testing => troubleshooting)
Problem / Goal
LetsEncrypt certificates expire every 3 months... I don't want to do this manually for all my personal & clients' domains.
Automatically renew my HTTPS Encryption Certificates for Wildcard Certs from LetsEncrypt, through DNS Challenge Verification!
Research
I found what looks like a good tutorial that will work out for me : https://kevingoedecke.com/2023/05/17/ho ... ernal-dns/
This tutorial does have information about the authentication requirements I'll need to setup, and plugins I will need to install, which will be helpful.
One issue is that this doesn't explicitly mention wildcard certificates, so I might run into an issue there with my setup (or more specifically the command I need to use to generate the certificate).
The tutorial also outlines the general process:
1. A LetsEncrypt plugin (certbot-dns-route53 aka dns_route53) is used to manage Route53
* This plugin will create temporary TXT DNS records to verify the ownership of the domain
2. A user account must be created with enough permissions to use the AWS APIs for the DNS management (used by the plugin in step 1)
3. (Legacy) A Cronjob could be setup to execute the process at pre-defined schedules
Setup the necessary role / permissions
Reference: https://certbot-dns-route53.readthedocs.io/en/stable/
1. First I had to create the IAM policy through the Visual Editor, restricting ARNs to those hosted zones I want to allow automatic renewals forUse of this plugin requires a configuration file containing Amazon Web Sevices API credentials for an account with the following permissions:
Code: Select all
route53:ListHostedZones route53:GetChange route53:ChangeResourceRecordSets
- certbot-dns-route53
- Console Sign-In disabled for security
Install the dns_route53 plugin
Code: Select all
sudo apt-get install certbot python3-certbot-dns-route53After adding the AWS config access key and secret to the server, I executed a test - which failed:
Code: Select all
$ sudo certbot certonly --dns-route53 -d "*.serynn.ca"
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Requesting a certificate for *.serynn.ca
Unable to locate credentials
To use certbot-dns-route53, configure credentials as described at https://boto3.readthedocs.io/en/latest/guide/configuration.html#best-practices-for-configuring-credentials and add the necessary permissions for Route53 access.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.When testing the automatic renewal (dry run) process:
Code: Select all
$ sudo certbot renew --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/serynn.ca.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Failed to renew certificate serynn.ca with error: The manual plugin is not working; there may be problems with your existing configuration.
The error was: PluginError('An authentication script must be provided with --manual-auth-hook when using the manual plugin non-interactively.')
Under [renewalparams] I modified the "authenticator" from 'manual' to 'dns-route53':
Code: Select all
# renew_before_expiry = 30 days
version = 1.21.0
...
# Options used in the renewal process
[renewalparams]
account = <removed>
pref_challs = dns-01,
server = https://acme-v02.api.letsencrypt.org/directory
authenticator = dns-route53
Code: Select all
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/serynn.ca.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Simulating renewal of an existing certificate for *.serynn.ca and serynn.ca
Failed to renew certificate serynn.ca with error: An error occurred (AccessDenied) when calling the GetChange operation: User: arn:aws:iam::<AWSACCOUNTID>:user/<USERNAME> is not authorized to perform: route53:GetChange on resource: arn:aws:route53:::change/<AWSHOSTEDZONEID> because no identity-based policy allows the route53:GetChange action
To use certbot-dns-route53, configure credentials as described at https://boto3.readthedocs.io/en/latest/guide/configuration.html#best-practices-for-configuring-credentials and add the necessary permissions for Route53 access.
My IAM permissions were in fact too restrictive - I had to open up the permissions more. The following permission worked (but I should research more if I can lock this down further - you should only grant just enough permissions for the bot to be able to achieve its task, and nothing more!):
Code: Select all
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"route53:GetChange",
"route53:ChangeResourceRecordSets"
],
"Resource": "*"
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": "route53:ListHostedZones",
"Resource": "*"
}
]
}
Code: Select all
$ sudo certbot renew --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/serynn.ca.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Simulating renewal of an existing certificate for *.serynn.ca and serynn.ca
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
The following simulated renewals succeeded:
/etc/letsencrypt/live/serynn.ca/fullchain.pem (success)
(Legacy/Optional) Define the cronjob
I was researching the best recommendations for the cron, but it seems like it's no longer required - certbot will install a system.d timer, and I confirmed my version (1.21) does have the timer installed.
Here's the link to where I found this information: https://serverfault.com/questions/79077 ... wer-924695
I need to check back within 24 hours to confirm that this is working automatically, now that I have the successful dry-run tests confirmed.
Even if it works automatically, I'll probably still need to automate the reload of my web server (nginx) configuration every day though so that it uses the latest certificate that's automatically installed.
Final Notes
This information will be filled out as its completed.