SSL Certificate Expiry — Check Before the Outage Post Hits HN
Site fine Thursday. Friday 17:00 cert expires because auto-renew failed silently since March. Monitoring didn’t alert because HTTP health check hit a CDN edge that still served cached content. HN post writes itself. r/sysadmin comments split between “just use cert-manager” and “someone forgot the staging ACME account.”
- “Cloudflare = I don’t need to worry” — Full strict mode still needs valid origin cert. Orange-cloud doesn’t fix expired cert on origin if clients connect directly.
- Checking only the leaf cert — Chain incomplete → works in Chrome, fails in older Android or Java trust stores.
- Renewal ≠ installation — New cert on disk but nginx not reloaded is the same outage.
- Wildcard covers everything — Wrong SAN still fails on
api.example.comif only*.example.comand path rules differ.
Quick manual checks
openssl (authoritative for what clients see on that host:port):
echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null \
| openssl x509 -noout -dates -subject -issuer
Look for notAfter= — that’s expiry UTC.
Browser: Padlock → connection secure → certificate → Valid until.
curl:
curl -vI https://example.com 2>&1 | grep -i expire
What to monitor
| Signal | Alert when |
|---|---|
| Days to expiry | < 30 days (14 for LE) |
| Renewal job success | Any failure |
| TLS handshake from external probe | Fails outside your VPC |
| cert-manager Certificate Ready | False for > 1h |
Internal-only checks miss DNS mispointing — use external synthetic monitors (Pingdom, Better Stack, etc.) hitting public hostname.
Let’s Encrypt failure modes
- HTTP-01 blocked — Firewall, CDN misconfig,
.well-known/acme-challengenot routed - DNS-01 stale creds — API token rotated, cert-manager secret not updated
- Rate limits — Too many failed orders; fix staging first with Let’s Encrypt staging CA
- IPv6 only broken — LE validates AAAA; A record works, AAAA broken → random failures
Certbot --dry-run before production renew saves weekends.
Enterprise / multi-host inventory
Spreadsheets rot. Maintain machine-readable inventory:
- Primary domain + all SANs
- Issuer (Let’s Encrypt, DigiCert, …)
- Auto vs manual renew
- Where cert is installed (LB, nginx, Cloudflare origin)
Run weekly script exporting notAfter for all production hostnames — diff in git.
Incident runbook (short)
- Confirm expiry vs misconfiguration (wrong cert served on SNI).
- Renew or re-issue immediately (LE) or contact CA (paid certs).
- Deploy + reload all terminators (nginx -s reload, ALB listener update).
- Verify from external vantage + mobile network.
- Postmortem: why monitor silent? add expiry + renew job alerts.
Use ssl-check tool for quick handshake + expiry read without memorizing openssl flags.
Related: mTLS and internal certs
Internal service mesh certs rotate faster — same discipline, different tooling (Istio, step-ca). Expired internal cert takes down east-west traffic while public site looks fine.
Certificate Transparency logs
Public CAs log issued certs to CT logs. Tools like crt.sh help find shadow certificates you forgot — old staging subdomains still valid for 90 days are a common pen-test finding. Quarterly CT review catches certs installed on decommissioned load balancers.
HSTS preload reminder
If you’re on the HSTS preload list, a botched renew hurts harder — browsers won’t offer easy bypass. Keep a break-glass procedure (status page on separate domain, comms template) before expiry day, not during the outage.
Tool: expiry at a glance
The ssl-check tool checks certificate validity and expiry from your browser — good for spot checks before deploys.
TL;DR
SSL expiry is predictable; outages are process failures. Monitor days-to-expiry and renewal job success externally. openssl notAfter, automate LE, reload after install, verify from outside your network.