PKI: Certificate Authorities, Trust Chains and a Private CA

TLDR: A certificate is trusted because it chains to a root CA in the client's trust store, through one or more intermediates. Keep root keys offline, issue from intermediates, and limit what an intermediate can sign with name constraints. Revocation is moving from OCSP to CRLs and short lifetimes. For internal services and mTLS, run a private CA (step-ca, Vault, cert-manager or a cloud private CA) and automate issuance with short-lived certificates. Distribute your root to every trust store that needs it: the OS, language runtimes and container images.

Public Key Infrastructure (PKI) answers one question: why should I believe that this public key belongs to orders.example.com? Anyone can generate a key pair and write any name into a certificate. PKI makes the name believable by having a certificate authority (CA) that the client already trusts vouch for it, with a signature.

In this part you read a real public chain, then build a private PKI for a lab domain: a root CA, an intermediate restricted to that domain, server and client certificates, a revocation list, an nginx server that requires client certificates, and a similar hierarchy in three commands with the step CLI. You finish with code signing, where the same ideas apply to commits and container images.

Prerequisites

• The lab tools from the guide overview: OpenSSL 3.5 or later, nginx, curl, Git, the step CLI and cosign.
• Part 3, for how certificates are used in the TLS handshake.
• sudo on a test machine for the trust store step (optional).

The outputs are real, from Ubuntu 26.04 LTS (OpenSSL 3.5.5, nginx 1.28.3, step 0.31.0, Git 2.53.0, cosign 3.1.4), recorded in October 2026. Keys, serial numbers and dates will differ on your machine.

How a Chain of Trust Works

Component	Role	Example
Root CA	Trust anchor, shipped in trust stores; signs intermediates only	ISRG Root X1, ISRG Root X2
Intermediate CA	Signs end-entity certificates day to day; can be replaced	Let's Encrypt YE2, YR2
End-entity (leaf)	The server's or client's certificate	CN=letsencrypt.org
CRL	A signed list of revoked serial numbers, published by each CA	http://ye2.c.lencr.org/96.crl
Certificate Transparency (CT) logs	Public, append-only logs of every issued public certificate	Checked by browsers through SCTs in the certificate

The client verifies each signature with the public key of the certificate above it, until it reaches a root it already trusts. It also checks that every certificate is inside its validity period, that each CA certificate is allowed to sign (basicConstraints CA:TRUE, keyUsage keyCertSign), that the leaf is allowed for TLS servers (extendedKeyUsage serverAuth), and that the hostname is in the leaf's SAN.

Intermediates exist so the root key can stay offline. If an intermediate is compromised or must be retired, the CA revokes it and issues a new one, and the root in billions of devices does not change.

A Real Chain, With Cross-Signing

Look at what letsencrypt.org sends:

The server sends four certificates. The leaf is signed by the intermediate YE2, which is signed by Root YE, one of the new roots Let's Encrypt created in 2025. Root YE is not yet in most trust stores, so the server also sends a cross-signed copy of Root YE (issued by ISRG Root X2) and a cross-signed X2 (issued by ISRG Root X1). A client that trusts any of these roots can build a path. This is how CAs introduce new roots without breaking older devices: the same key gets a second certificate signed by an established root.

Two more details in the leaf show how the Web PKI works in 2026:

The certificate has a CRL distribution point but no OCSP URL, and it carries Signed Certificate Timestamps (SCTs): signed promises from CT logs to publish it within a fixed delay. Browsers reject public certificates without enough SCTs, and the logs are audited, so every publicly trusted certificate for your domain ends up in public logs. That lets you watch for certificates you did not request, with services such as crt.sh or Cert Spotter.

Trust Stores

A trust store is the set of root certificates a client trusts. Different software uses different stores, which explains many "works in the browser, fails in my app" problems.

Client	Where its roots come from
Debian, Ubuntu	/usr/local/share/ca-certificates/*.crt, then update-ca-certificates (bundle in /etc/ssl/certs/ca-certificates.crt)
RHEL, Fedora, Amazon Linux	/etc/pki/ca-trust/source/anchors/, then update-ca-trust
macOS	Keychain (security add-trusted-cert)
Windows	Certificate stores (certlm.msc, Import-Certificate)
curl, OpenSSL, Go on Linux	The OS bundle
Python requests	pip installs use the certifi bundle, not the OS store (set REQUESTS_CA_BUNDLE, or use the truststore package); distribution packages of certifi often point to the OS bundle. python3 -m certifi shows which file is used
Node.js	Its own copy of Mozilla's roots; add yours with NODE_EXTRA_CA_CERTS, or use the OS store with --use-system-ca
Java	$JAVA_HOME/lib/security/cacerts (keytool -importcert -cacerts)
Containers	The image's OS bundle: add your root in the Dockerfile and run update-ca-certificates

Never fix a trust problem by turning verification off. Add the right root to the store the client actually uses. You will do that for the lab root below.

Build a Private CA

Public CAs issue certificates only for names and IP addresses they can validate on the public internet, never for private names such as orders.lab.internal, and public TLS CAs are phasing out client certificates. For internal services, service-to-service mTLS, VPNs and devices, you run your own CA. The lab uses orders.lab.internal; .internal is reserved for private use and never resolves on the public internet.

The Root CA

The root signs only intermediates and CRLs. pathlen:1 allows one level of intermediates below it.

Subject and issuer are the same: a root is self-signed. In production the root key never sits unencrypted on a networked machine. Generate it on an offline machine or an HSM, back it up encrypted in two places, and bring it out only to sign a new intermediate or a CRL. Cloud private CAs (AWS Private CA, Google CA Service) and HSM-backed Vault keep it in hardware for you.

The Intermediate CA, Limited by Name Constraints

The intermediate gets pathlen:0 (it cannot create further CAs) and name constraints: DNS names only under lab.internal, URIs (such as the SPIFFE IDs used below) only with the host lab.internal, and no IP addresses at all. If its key leaks, the attacker still cannot mint a trusted certificate for your bank or your public domains. Constraints only cover the name types they list, which is why the IP exclusion is there, for IPv4 (0.0.0.0/0.0.0.0) and IPv6 (::/::); if you do issue for IP addresses, list your ranges with permitted;IP: instead.

Issue Certificates with openssl ca

openssl ca keeps a small database of everything it issued, which you need for revocation. Its configuration defines the profiles: what a server certificate and a client certificate may contain. Save it as intermediate.cnf in ~/crypto-lab/part4/pki: Note copy_extensions = none: the CA decides the extensions, never the requester's CSR. The subject alternative name comes from the SAN environment variable at signing time.

# Default for $ENV::SAN, so commands that issue nothing (revoke, gencrl) can
# read this file. Issuing without SAN set in the environment fails.
SAN =

[ ca ]
default_ca = lab_intermediate

[ lab_intermediate ]
dir              = .
database         = $dir/db/index.txt
serial           = $dir/db/serial
crlnumber        = $dir/db/crlnumber
new_certs_dir    = $dir/issued
certificate      = $dir/intermediate-ca.crt
private_key      = $dir/intermediate-ca.key
default_md       = sha256
default_days     = 30
default_crl_days = 7
policy           = policy_cn_only
copy_extensions  = none
unique_subject   = no

[ policy_cn_only ]
commonName = supplied

[ server_cert ]
basicConstraints       = critical,CA:FALSE
keyUsage               = critical,digitalSignature
extendedKeyUsage       = serverAuth
subjectKeyIdentifier   = hash
authorityKeyIdentifier = keyid
subjectAltName         = $ENV::SAN
crlDistributionPoints  = URI:http://pki.lab.internal/intermediate.crl

[ client_cert ]
basicConstraints       = critical,CA:FALSE
keyUsage               = critical,digitalSignature
extendedKeyUsage       = clientAuth
subjectKeyIdentifier   = hash
authorityKeyIdentifier = keyid
subjectAltName         = $ENV::SAN
crlDistributionPoints  = URI:http://pki.lab.internal/intermediate.crl

Each service generates its own key and sends only a certificate signing request (CSR); the private key never leaves the service. Then the CA signs:

Verify, and See Each Check Fail

openssl verify checks the chain the way a TLS client does; add -purpose sslserver and -verify_hostname for the purpose and name checks, which it skips by default. Each command after the first breaks one rule:

The name constraint did its job: the intermediate signed www.example.com (a CA signs whatever its software lets through), but no client that enforces constraints will accept it. Keep these error messages in mind; they are the same ones you get from curl, nginx and application logs.

Mutual TLS with the Lab CA

Issue a client certificate for a worker service. The SAN carries a SPIFFE-style URI, the identity format service meshes use, which the server can authorize on:

The nginx server from Part 3 now uses the lab certificate and requires a client certificate signed by the lab root. If the Part 3 server still runs, stop it first; both use port 8443. Create the directories:

Then save this as nginx/conf/nginx.conf:

# Lab only: the paths below make nginx run as a normal user
worker_processes 1;
pid logs/nginx.pid;
error_log logs/error.log;
events {}

http {
    access_log off;
    client_body_temp_path tmp/body;
    proxy_temp_path tmp/proxy;
    fastcgi_temp_path tmp/fastcgi;
    uwsgi_temp_path tmp/uwsgi;
    scgi_temp_path tmp/scgi;

    server {
        listen 127.0.0.1:8443 ssl;
        server_name orders.lab.internal;

        ssl_certificate ../../pki/orders-fullchain.crt;
        ssl_certificate_key ../../pki/orders.key;
        ssl_protocols TLSv1.2 TLSv1.3;
        ssl_ecdh_curve X25519MLKEM768:X25519:prime256v1:secp384r1;
        ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305;
        ssl_prefer_server_ciphers off;

        ssl_client_certificate ../../pki/root-ca.crt; # trust anchor for client certificates
        ssl_verify_client on;
        ssl_verify_depth 2;                           # client cert + intermediate

        location / {
            return 200 "hello $ssl_client_s_dn\n";
        }
    }
}

The server checks the client's chain against the lab root, and the application gets the verified identity in $ssl_client_s_dn (or the SAN). Authorize on that identity, for example "only spiffe://lab.internal/billing-worker may call /charges", instead of accepting any certificate from the CA.

Revocation

When a private key leaks, the CA revokes the certificate: it adds the serial number to its next CRL, and clients that check revocation refuse it.

A CRL is valid only until its nextUpdate time (7 days here), so the CA must publish a fresh one before then, even if nothing changed, at the URL in the certificates' CRL distribution point. Serve the DER file there (clients fetching it over HTTP expect DER); OpenSSL and nginx read the PEM file locally. In nginx, ssl_crl enables CRL checks for client certificates; nginx then checks every CA in the chain, so the file must contain a CRL from the root as well as from the intermediate.

Revocation on the public web changed in the last two years:

• OCSP is ending. Let's Encrypt removed OCSP URLs in May 2025 and shut down its responders in August 2025, citing the privacy cost (every lookup told the CA which site you visited) and reliability. Browsers use CRLs that they collect and compress centrally (Firefox's CRLite, Chrome's CRLSets).
• Short lifetimes replace revocation. A certificate that expires in six days (Let's Encrypt's shortlived profile) or 24 hours (common for internal mTLS) needs revocation far less, because a stolen key stops working soon anyway.

Distribute Your Root

Clients only trust the lab certificates if the lab root is in their trust store. On Debian and Ubuntu:

To remove it again, delete the file and run sudo update-ca-certificates --fresh. For other clients, use the table in the trust stores section. In Kubernetes, trust-manager (from the cert-manager project) distributes CA bundles to every namespace as ConfigMaps, so pods can mount them.

The Same CA with the step CLI

Hand-written OpenSSL configuration is good for learning and for unusual profiles. For daily work, step creates a similar hierarchy with sensible defaults in three commands. Its default profiles add no name constraints; use a certificate template for those.

--no-password --insecure leaves the keys unencrypted, which is fine for a lab only. step certificate inspect is also a readable alternative to openssl x509 -text for any certificate.

To issue certificates automatically, run a CA service: step-ca (with an ACME endpoint, so certbot and cert-manager work against it), HashiCorp Vault's PKI secrets engine, cert-manager's CA issuer inside Kubernetes, or a cloud private CA. With Vault, for example, once a PKI mount pki_int holds a signed intermediate (example commands, not run here):

vault write pki_int/roles/orders allowed_domains=lab.internal allow_subdomains=true max_ttl=72h
vault write pki_int/issue/orders common_name=orders.lab.internal ttl=24h

The Certificate Lifecycle

Every certificate goes through the same steps: generate a key, request, validate, issue, deploy, monitor, renew (or revoke). The only sustainable way to run that loop is to automate all of it, and the industry is forcing the issue: the maximum lifetime of public TLS certificates is shrinking in steps under CA/Browser Forum ballot SC-081.

Let's Encrypt moves faster than the rules: 64-day certificates by default from 10 February 2027 and 45 days from 2028. In Kubernetes, cert-manager runs the loop for you. Here orders-lab-intermediate is a ClusterIssuer of type ca whose Secret holds the intermediate's certificate and key:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: orders
  namespace: orders
spec:
  secretName: orders-tls
  dnsNames:
    - orders.lab.internal
  duration: 720h # 30 days
  renewBefore: 240h # renew 10 days before expiry
  privateKey:
    algorithm: ECDSA
    size: 256
    rotationPolicy: Always # new key on every renewal
  issuerRef:
    name: orders-lab-intermediate
    kind: ClusterIssuer

What to monitor, whatever tool issues the certificates:

• Days until expiry of what each endpoint actually serves (the Part 3 script or the Prometheus blackbox exporter).
• Renewal failures in your ACME client or cert-manager. A certificate stays Ready while its replacement keeps failing, so alert on cert-manager's certmanager_certificate_expiration_timestamp_seconds and certmanager_certificate_renewal_timestamp_seconds metrics (expiry close and renewal time passed), and check failed CertificateRequests with kubectl describe certificate.
• CT logs for your public domains, to catch certificates you did not request.
• For private CAs: CRL freshness, intermediate expiry, and who can call the issue endpoint.

Signing: The Same Ideas for Code

Signatures with certificates and trust anchors are not only for TLS. Two cases every DevOps team meets:

Signed Git Commits with an SSH Key

Git can sign commits with an SSH key, so you do not need GPG. An "allowed signers" file plays the role of the trust store:

To use this for real, sign with the email address of your account instead of the lab address, and upload the same public key to GitHub or GitLab as a signing key; they show a commit as verified only when its email matches a verified address on the account. Branch protection rules can then require signed commits.

Signed Container Images with Sigstore

Sigstore's cosign signs images with short-lived certificates from its Fulcio CA, issued to an OIDC identity (a person's account or a CI workflow), and logs every signature in a transparency log, just as CT does for TLS certificates. In CI it is one command, cosign sign <image>@<digest>, with the workflow's OIDC token. Verifying pins the expected identity, as in Part 2. Kubernetes signs its own release images this way:

In a cluster, an admission policy (Sigstore's policy-controller, Kyverno or Connaisseur) runs the same check before any pod starts.

Common PKI Problems

Error	Usual cause	Fix
unable to get local issuer certificate	Server sends no intermediate, or the client lacks the root	Serve the full chain; add the root to the client's trust store
self-signed certificate / self-signed certificate in certificate chain	A private root that the client does not trust	Add the root to the store that client uses (see the table above)
certificate has expired	Renewal failed, or the clock is wrong	Fix renewal; check timedatectl and NTP
hostname mismatch	The name is not in the SAN	Reissue with the right SAN; connect with the right name and SNI
unsupported certificate purpose	Wrong extended key usage, such as a server certificate used for client auth	Issue the certificate with the right profile
permitted subtree violation	A name outside the CA's name constraints	Use a CA that may issue for that name
certificate revoked	The serial is on the CA's CRL	Issue a new key and certificate

Summary

• Trust comes from a chain: leaf, intermediates, and a root in the client's trust store. Servers must send the intermediates.
• CAs introduce new roots with cross-signed certificates, as Let's Encrypt does with Root YE and YR.
• Keep root keys offline, constrain intermediates (pathlen, name constraints, EKU), and let the CA, not the CSR, decide extensions.
• Revocation is moving to CRLs and to short lifetimes; public OCSP is ending.
• For internal TLS and mTLS, run a private CA with automated, short-lived issuance, and distribute its root to every trust store that needs it.
• The same tools sign code: SSH-signed commits and Sigstore-signed images, verified against pinned identities.

This is the end of the guide. To keep practicing, step through the SSL/TLS Handshake simulator, or go back to the guide overview.

Keep Learning

• Zero Trust Architecture
• Let's Encrypt Goes to 64-Day Certificates: When Does Your ACME Client Actually Renew?
• Kubernetes Security Checklist
• Interview questions: Security Architecture and DevSecOps
• HashiCorp Vault Secrets Management Quiz