The delicate art of cryptographic maintenance: When small certs break big chains

If there is one universal law in Internet infrastructure, it is that no good intention goes unpunished by the practical realities of legacy software maintenance.

Recently, around mid-2026, for those keeping score, we began receiving a trickle of polite, if frustrated, user complaints concerning certificate validation failures on a few of our sites, most notably the CIDR Report. The error messages were classic Public Key Infrastructure (PKI) friction — valid site, valid dates, but a client throwing up its hands in disgust at its inability to validate the chain of trust.

As is typical with the web’s PKI, the failure wasn’t a single catastrophic bug, but rather the slow-motion collision of a well-meaning optimization on one side of the Internet, and a slumbering distribution ecosystem on the other.

The recipe for disruption

How do you break standard Transport Layer Security (TLS) validation for some fraction of the Internet? It apparently takes three distinct ingredients:

Let’s Encrypt swaps defaults: Late in 2025, Let’s Encrypt, the largest Certificate Authority (CA) on the Internet, made a sensible shift. To keep handshake overhead lean and signatures compact, they shifted their preference toward issuing Elliptic Curve Cryptography (ECDSA) certificates rather than the bulkier, computationally heavier RSA variants. Naturally, issuing ECDSA certificates at scale implied an update to their operational intermediate certificates and root trust paths.
certbot follows suit: If you run the certbot, a certificate management client, without explicit key-type flags, it does what any obedient tool should do — it requests whatever the CA currently prefers. In the case of Let’s Encrypt, this implied that suddenly, certbot-driven standard automation setups across the web began silently renewing using smaller ECDSA keys signed by these newer intermediates.
But certbot drops the intermediate certificate: certbot's behaviour also changed with this change in cryptographic algorithm, and in the returned certificate set, the fullchain certificate bundle no longer included the intermediate certificate that ‘bound’ the Let’s Encrypt root certificate (YE) to the domain name certificate for www.cidr-report.org. The so-called YE1 certificate, issued by the YE root certificate, was missing from the fullchain bundle in this instance.

While Apple’s platforms and Microsoft’s platforms keep their system CA trust stores refreshed with clockwork regularity, other distribution release cycles tell a different story. Debian, for example, hadn’t updated its default ca-certificates package since early 2025 — a gap that persisted even into recent builds of the recently released Debian 13.

Consequently, any client operating on an unpatched or conservatively maintained distribution attempts to validate the new intermediate signature against a local trust store that simply hasn’t heard of it yet. The result? Validation fails, connection aborts, and the web access fails.

The remedy: Dual-stack certificates

The web protocol suite, in its infinite wisdom, accounts for this. Most modern HTTP servers (including Apache) happily support loading dual certificates. They’ll serve an ECDSA certificate to modern clients that support the new chain, and fall back to an RSA certificate signed by the ubiquitous, battle-tested ISRG Root X1 chain for older or lagging trust stores.

Achieving this with certbot requires two separate issuance calls and a precise set of parameters.

Step 1: Issue the standard ECDSA certificate

First, let certbot fetch the modern, default ECDSA certificate as usual:

certbot -v certonly \
  --force-renewal \
  --manual \
  --manual-auth-hook /usr/local/letsencrypt/acme-dns-auth.py \
  --preferred-challenges dns \
  --debug-challenges \
  --key-type ecdsa \
  -d *.cidr-report.org

Step 2: Issue the legacy RSA fallback certificate

To obtain the legacy RSA certificate alongside the primary one, you must explicitly request the key type, pin the preferred chain, and — crucially — assign a distinct --cert-name. If you forget --cert-name, certbot will helpfully overwrite your newly minted ECDSA cert.

certbot -v certonly \
  --force-renewal \
  --manual \
  --manual-auth-hook /usr/local/letsencrypt/acme-dns-auth.py \
  --preferred-challenges dns \
  --debug-challenges \
  --key-type rsa \
  --preferred-chain "ISRG Root X1" \
  --cert-name cidr-report-rsa \
  -d *.cidr-report.org

Note: Pay close attention to two flags here: --preferred-chain "ISRG Root X1" forces the ACME server to hand back the older path, and --cert-name cidr-report-rsa keeps the local file structures completely isolated.

Web server integration

Once issued, you will need to configure Apache to point to both pairs of certs and keys within the VirtualHost block:

<VirtualHost *:443>
    ServerName cidr-report.org
    # Modern ECDSA Certificate & Key
    SSLCertificateFile    /etc/letsencrypt/live/cidr-report.org/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/cidr-report.org/privkey.pem
    # Legacy RSA Fallback Certificate & Key
    SSLCertificateFile    /etc/letsencrypt/live/cidr-report-rsa/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/cidr-report-rsa/privkey.pem
</VirtualHost>

When a client initiates a TLS handshake, Apache will negotiate the best cipher suite and present the appropriate certificate chain based on the client’s advertised capabilities.

A plague on all your houses!

Once Apache is restarted and tests pass on whatever devices you can lay your hands on, you are free to curse the CA ecosystem, operating system packagers, and the inherent fragility of global PKI, at your leisure.

It won’t update Debian’s certificate trust store any faster, but it is an essential part of the modern network administrator’s ritual.

The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.