Certificate Lifecycle Management with SaltStack
Author: Nicholas M. Hughes
Digital certificates are the backbone of modern Internet communication. They provide the mechanism by which users of a network service can verify the legitimacy of an endpoint. Malicious actors can very easily mimic the interfaces of your banking institution or favorite online store. What’s to stop them from tricking you into surrendering your sensitive information? Trust.
Start with Why
A chain of trust is built between you (your browser) and the network endpoint by virtue of trusting a relative few companies who are issuers of digital certificates. Basically, you trust those companies implicitly and you also trust them to verify that any certificates they issue are for the organization that you’d expect. This means that I can’t run out and get a certificate for your banking website, because I have no way to convince a certificate issuer that I’m your bank.
So, we’ve established that certificates are important and organizations should have them. What next? Can we just get them and keep them forever? Nope. Much like the milk in my refrigerator, certificates have expiration dates. Certificate expiration allows for a built-in way to limit the timeframe that a malicious party can use a certificate if the private key is compromised. The most popular validity timeframes are either 1 or 2 years. This means that a bad actor can impersonate a legitimate endpoint for up to one to two years if they are able to gain access to the private key. That’s a long time! However, the other side of the coin is the maintenance tail associated with certificates. If you have 1,000 instances in your environment with 1 year certificate validity, you could be performing certificate lifecycle operations for up to 3 of your instances every day of the year… and that’s only if you’re able to spread it out. Who deploys systems one at a time spread out over the year? It’s more likely that you’ll end up with times during the year where you’re working to renew dozens of certificates. Shorter validity periods are arguably more secure, but the overhead associated with generating a Certificate Signing Request (CSR), requesting a certificate, and deploying it to an instance is non-trivial.
Enter Certificate Lifecycle Management software. The goal of this software is to handle the inventory, request, renewal, and deployment of certificates in your organization. It’s extremely important to stay on top of certificate operations, because expired certificates can cause outages in both end user and backend systems. There are quite a few software vendors and certificate authorities out there who make software specifically for this purpose. Some of the features are certainly helpful and necessary, but I’m a big proponent of toolset consolidation. If I have a need for automating a process, I want to use something that’s already in my organization or bring in new software which can take over multiple tasks and phase out legacy software. That’s why I love SaltStack… it can do ALL THE THINGS.
How Now?
SaltStack has a lot of the native capabilities we’ll need in order to pull this off. Using Salt’s Beacon and Reactor systems, we can react to arbitrary events in the infrastructure. It also integrates with my certificate provider, DigiCert, who can handle my needs for both internal (private) and external (public) certificates.
Workflow
Beacon: On an interval, the beacon checks a list of certificate files on the system. If a given certificate is expiring within the notification threshold, the certificate information is added to the beacon return.
Reactor: Salt’s Reactor system “sees” the beacon tag on the Salt event bus and fires an orchestration runner to order a new certificate.
Orchestration: The orchestration leverages the DigiCert runner module to generate a private key (if necessary), generate a CSR, and order the certificate.
Reactor: Salt’s Reactor system “sees” the DigiCert runner return tag on the Salt event bus and fires an orchestration runner to deploy the certificate.
Orchestration: This orchestration step takes return information from the DigiCert runner module and uses it to drop a replacement certificate file on the filesystem.
Beacon: Now that the expiring certificate has been replaced, the beacon will not fire an event for that file on the next run.
Setup
1. This automation requires the cert_info beacon, which I recently committed to Salt’s open source GitHub repository. Check out the pull request to see what version it finally pops into (maybe Sodium?), but in the meantime… you can download the cert_info.py from my fork or Salt’s develop branch once it’s merged, pop it into a _beacons directory in the Salt file root then run saltutil.sync_beacons against your systems to push it out.
2. Configure the DigiCert runner in the Salt master configuration. At a minimum, you’ll require an API key with permissions to order certificates in the DigiCert API. The salt-master service will need to be restarted after adding this configuration, but we can hold off on that since we’ll be adding another master configuration later on.
digicert:
api_key: <insert API key here>3. Create a new file named /srv/salt/digicert/process_beacons.sls. This will be called anytime a cert_info beacon hits the event bus (reactor configuration to follow). We’re targeting the Salt master since the DigiCert code is a runner (master-only) module. Then, we’re passing in the entire information set regarding the existing certificate and also the associated minion ID. That information will be used in the subsequent file.
{%- set minion_id = data['id'] %}
{%- for cert in data['certificates'] %}
trigger_certificate_order:
local.state.apply:
- tgt: {{ grains['id'].replace('_master', '') }}
- arg:
- digicert.order
- kwarg:
pillar:
cert_data: {{ cert | json }}
minion_id: '{{ minion_id }}'
{% endfor -%}4. Create a new file name /srv/salt/digicert/order.sls. This will be used to order the actual certificate from DigiCert and will rely upon the DigiCert API key we added in Step #2. Note that the DigiCert runner has more options than I’m using in this example, so you might have to insert more logic to handle different organizations, validity periods, subject alternative names (SANs), and other items based upon your needs. This is the minimal configuration you’ll need to have it work.
{%- set cert_data = pillar.get('cert_data', {}) %}
{%- set minion_id = pillar.get('minion_id') %}
{%- if minion_id and cert_data %}
order_certificate:
salt.runner:
- name: digicert.order_certificate
- minion_id: {{ minion_id }}
- common_name: {{ cert_data['subject_dict']['CN'] }}
- organization_id: 663786
- validity_years: 2
{% endif -%}5. Create a new file named /srv/salt/digicert/process_returns.sls. This will be called when ANY return hits the event bus (reactor configuration to follow). This is probably not a great way to implement if you have a large amount of minions. However, it’ll suit us just fine for our tiny demonstration. When a return hits the event bus, we check for returns of the order_certificate function of the DigiCert runner. We then gather up some information from the event and pass it into a state we’re using to deploy the certificate. Additionally, we’re grabbing the associated private key for deployment as well. The DigiCert runner will create a private key for a given common name and cache it on the master, so this is our way of getting out of the cache and making sure it’s passed to the minion. I had to read the runner code to figure out where that private key was going, so now I’m passing it along to you.
{%- if data.get('fun') == 'runner.digicert.order_certificate' %}
{%- set minion_id = data['fun_args'][0].get('minion_id') %}
{%- set common_name = data['fun_args'][0].get('common_name') %}
{%- set cert_order = data['return'].get('order') %}
{%- if cert_order %}
deploy_certificate:
local.state.apply:
- tgt: {{ minion_id }}
- arg:
- digicert.deploy
- kwarg:
pillar:
common_name: '{{ common_name }}'
cert_order: {{ cert_order | json }}
rsa_key: |
{{ salt['saltutil.runner']('digicert.show_rsa', arg=[minion_id, common_name]) | indent(14) }}
{% endif -%}
{%- endif %}6. Create a new file named /srv/salt/digicert/deploy.sls. This is taking the incoming data from the DigiCert runner return, which contains the CA chain and certificate we just ordered, and dropping them in files on the filesystem. I have static values set for the file locations, but you could insert some logic to determine dynamic locations fairly easily as well.
{%- set cert_order = pillar.get('cert_order') %}
{%- set common_name = pillar.get('common_name') %}
{%- set rsa_key = pillar.get('rsa_key') %}
{%- set certificate = [] %}
{%- set ca_chain = [] %}
{%- if cert_order %}
{%- for cert in cert_order.get('certificate_chain', []) | reverse %}
{%- if cert.get('subject_common_name') == common_name %}
{%- do certificate.append(cert['pem']) %}
{%- else %}
{%- do ca_chain.append(cert['pem']) %}
{%- endif %}
{%- endfor %}
drop_certificate:
file.managed:
- name: /etc/ssl/certs/ssl-cert-snakeoil.pem
- mode: '0644'
- contents: |
{{ certificate | join('') | indent(8) }}
drop_ca_chain:
file.managed:
- name: /etc/ssl/certs/cacerts.pem
- mode: '0644'
- contents: |
{{ ca_chain | join('') | indent(8) }}
drop_rsa_key:
file.managed:
- name: /etc/ssl/private/ssl-cert-snakeoil.key
- mode: '0644'
- contents: |
{{ rsa_key | indent(8) }}
{% endif -%}7. Configure the reactors shown below in the Salt master configuration. The salt-master service will need to be restarted after adding this configuration.
reactor:
- 'salt/run/*/ret':
- /srv/salt/digicert/process_returns.sls
- 'salt/beacon/*/cert_info/*':
- /srv/salt/digicert/process_beacons.sls8. The only thing left to do is enable the beacon to monitor our certificate file. I’m a big fan of enabling beacons via Pillar, so target your test system with the Pillar configuration shown below. It will cause the beacon to run every day (86,400 seconds) and trigger a dump of the certificate information to the event bus on the last day of certificate validity. I’m working with 2 day certificates for demonstration purposes, so you might want to increase that threshold based upon your own environment.
beacons:
cert_info:
- files:
- /etc/ssl/certs/ssl-cert-snakeoil.pem
- notify_days: 1
- interval: 86400Event Sample
Well, that’s all of the configuration! Once the cert_info beacon fires, you’ll get an event like the one shown below. You can use that information for anything your little heart desires… certificate renewal workflows, certificate inventory databases, the sky’s the limit!
salt/beacon/u18-certmgmt-01.example.local/cert_info/ {
"_stamp": "2019-08-26T14:36:04.481734",
"certificates": [
{
"cert_path": "/etc/ssl/certs/ssl-cert-snakeoil.pem",
"extensions": [
{
"ext_data": "keyid:0F:80:61:1C:82:31:61:D5:2F:28:E7:8D:46:38:B4:2C:E1:C6:D9:E2\n",
"ext_name": "authorityKeyIdentifier"
},
{
"ext_data": "7F:03:E8:CC:4F:CF:3D:1F:C2:88:AE:57:5B:E6:F4:B6:A4:96:2B:31",
"ext_name": "subjectKeyIdentifier"
},
{
"ext_data": "DNS:certauto.eitr.tech",
"ext_name": "subjectAltName"
},
{
"ext_data": "Digital Signature, Key Encipherment",
"ext_name": "keyUsage"
},
{
"ext_data": "TLS Web Server Authentication, TLS Web Client Authentication",
"ext_name": "extendedKeyUsage"
},
{
"ext_data": "\nFull Name:\n URI:http://crl3.digicert.com/ssca-sha2-g6.crl\n\nFull Name:\n URI:http://crl4.digicert.com/ssca-sha2-g6.crl\n",
"ext_name": "crlDistributionPoints"
},
{
"ext_data": "Policy: 2.16.840.1.114412.1.1\n CPS: https://www.digicert.com/CPS\nPolicy: 2.23.140.1.2.2\n",
"ext_name": "certificatePolicies"
},
{
"ext_data": "OCSP - URI:http://ocsp.digicert.com\nCA Issuers - URI:http://cacerts.digicert.com/DigiCertSHA2SecureServerCA.crt\n",
"ext_name": "authorityInfoAccess"
},
{
"ext_data": "CA:FALSE",
"ext_name": "basicConstraints"
}
],
"has_expired": true,
"issuer": "C=\"US\",O=\"DigiCert Inc\",CN=\"DigiCert SHA2 Secure Server CA\"",
"issuer_dict": {
"C": "US",
"CN": "DigiCert SHA2 Secure Server CA",
"O": "DigiCert Inc"
},
"issuer_raw": [
[
"C",
"US"
],
[
"O",
"DigiCert Inc"
],
[
"CN",
"DigiCert SHA2 Secure Server CA"
]
],
"notAfter": "2019-08-26 12:00:00Z",
"notAfter_raw": "20190826120000Z",
"notBefore": "2019-08-24 12:00:00Z",
"notBefore_raw": "20190824000000Z",
"serial_number": "19055980287126395460104448178509444947",
"signature_algorithm": "sha256WithRSAEncryption",
"subject": "C=\"US\",ST=\"md\",L=\"Sykesville\",O=\"EITR Technologies, LLC\",CN=\"certauto.eitr.tech\"",
"subject_dict": {
"C": "US",
"CN": "certauto.eitr.tech",
"L": "Sykesville",
"O": "EITR Technologies, LLC",
"ST": "md"
},
"subject_raw": [
[
"C",
"US"
],
[
"ST",
"md"
],
[
"L",
"Sykesville"
],
[
"O",
"EITR Technologies, LLC"
],
[
"CN",
"certauto.eitr.tech"
]
],
"version": 2
}
],
"id": "u18-certmgmt-01.example.local"
}Wut?
Digital certificates are essential to secure network communication for both end user traffic such as web browsing and backend system traffic like when your application stack needs to talk to a database. The workflow above shows one scenario to automatically renew expiring certificates within your infrastructure, but it could easily be changed or extended in order to interface with a different public or private certificate authority, send the certificate information to an inventory system, or send notifications of expiring certificates to a Slack channel. Certificate Lifecycle Management implementations such as this (or really any automation whatsoever) free up operations staff to do more important work and leave the mundane tasks to the machines. So, what are you waiting for? Get automating!
Nicholas Hughes is a founding partner and CEO of EITR Technologies LLC. As part of his daily duties, he’s responsible for all of those super awesome elements of the CEO job that you read about as a kid, like setting the strategic direction of the company and modeling corporate values. Additionally, Nick still performs technical consulting work with specializations in Automation & Orchestration, Cloud Infrastructure, Cloud Security, and Systems Architecture. He has over 15 years of experience in a wide breadth of roles within Information Technology, which is invaluable to clients seeking comprehensive technical solutions to business problems. Nick highly values pragmatism, logical thinking, and integrity in both his business and personal life… which is a decidedly boring set of core values that reap great results when applied to the task at hand. He also has a wonderful wife and two boys who keep him on his toes.
