When the Clock Is Wrong: TLS, JWT and Signed-Request Failures From Clock Skew

A certificate can be freshly renewed and still be "not yet valid." A JWT can be correctly signed and still fail iat or nbf. An AWS request can have the right key and receive RequestTimeTooSkewed. When those symptoms arrive together, stop rotating credentials and check time first. A clock is an input to each decision, and a host that is only slightly early or late can turn an infrastructure problem into an authentication-looking incident.
This post maps the time check behind each failure, shows safe reproductions without changing a machine clock, and gives a Linux response path that makes loss of sync visible before users find it.
TL;DR
- TLS validation compares the verifier's current time with
notBeforeandnotAfter. It has no protocol-level grace period. - JWT leeway is a verifier policy. PyJWT accepts a
leewayargument, and its default is zero when you do not pass one. - S3 documents a 15-minute request timestamp limit. MIT Kerberos defaults to 300 seconds. TOTP recommends 30-second steps and at most one step for network delay.
- On Linux, first check
timedatectl, then usechronyc trackingandchronyc sources -vwhen chrony is your time daemon. - Correct the host clock, not a container clock. Alert on sync state and offset, then preserve UTC timestamps and offsets in incident logs.
Prerequisites
- Shell access to a Linux host or VM.
- Permission to inspect the active time service. Correcting time needs an approved change path because a clock step can surprise applications.
chronyinstalled if you use the chrony commands and configuration below.- OpenSSL, Python with
cryptographyfor the TLS fixture, and PyJWT for the JWT demo.
The time window behind the error
The same skew produces different errors because each protocol asks a different question. RFC 5280 requires certificate path validation to include the current time in the certificate validity period. RFC 7519 defines nbf, exp, and iat as NumericDate-based claims. SigV4 includes a UTC request timestamp in the signature input. TOTP derives its moving value from time.
| Check | Time comparison | Documented tolerance | Typical symptom |
|---|---|---|---|
| TLS certificate | Local verifier time versus notBefore and notAfter |
None in the certificate validity check | certificate is not yet valid or expired |
| JWT | Verifier time versus nbf, exp, and possibly iat |
Library policy. PyJWT 2.15.1 defaults to 0; RFC 7519 permits small leeway, usually no more than a few minutes | ImmatureSignatureError, expired token |
| AWS SigV4 | Signed request timestamp versus AWS time | S3 rejects a difference greater than 15 minutes | RequestTimeTooSkewed; other signature paths can report signature mismatch or expiry |
| MIT Kerberos | Client or service time versus KDC time | clockskew default: 300 seconds |
Clock skew too great |
| TOTP | Prover time step versus validator time step | RFC 6238 recommends a 30-second step and at most one step for network delay | A correct-looking code is rejected |
The S3 limit and error name come from AWS's request authentication documentation. AWS also documents SignatureDoesNotMatch and the Signature expired diagnostic for SigV4 troubleshooting in its IAM guide. MIT documents its 300-second default in krb5.conf. RFC 6238 calls for a 30-second default step and at most one step of delay. Those numbers are useful triage boundaries, not a reason to let time drift until the next boundary.
Prove TLS time failures without changing the host clock
Do not set a workstation's clock backward to test certificate behavior. OpenSSL's -attime verification option supplies the verification time instead. That isolates the test from package updates, log ordering, token expiry, and every other process on the machine.
This run creates a local CA that began 30 days earlier and two leaf certificates issued at the run time. One leaf starts at issuance. The other has notBefore one hour earlier. The verifier clock is then simulated 10 minutes behind, and then 8 days ahead. The fixtures come from this short script, which uses the Python cryptography package so it can set notBefore freely (python3 mkcerts.py writes ca.pem, leaf-now.pem and leaf-backdated.pem):
# Build a demo CA (valid since 30 days ago) and two leaf certs issued "now":
# leaf-now.pem has notBefore = issuance time, leaf-backdated.pem has notBefore = issuance time - 1 hour.
import datetime as dt
from cryptography import x509
from cryptography.x509.oid import NameOID
from cryptography.hazmat.primitives import hashes, serialization
from cryptography.hazmat.primitives.asymmetric import ec
now = dt.datetime.now(dt.timezone.utc).replace(microsecond=0)
def name(cn): return x509.Name([x509.NameAttribute(NameOID.COMMON_NAME, cn)])
ca_key = ec.generate_private_key(ec.SECP256R1())
ca = (x509.CertificateBuilder().subject_name(name("Demo CA")).issuer_name(name("Demo CA"))
.public_key(ca_key.public_key()).serial_number(x509.random_serial_number())
.not_valid_before(now - dt.timedelta(days=30)).not_valid_after(now + dt.timedelta(days=365))
.add_extension(x509.BasicConstraints(ca=True, path_length=None), critical=True)
.sign(ca_key, hashes.SHA256()))
open("ca.pem", "wb").write(ca.public_bytes(serialization.Encoding.PEM))
for fname, back in (("leaf-now.pem", 0), ("leaf-backdated.pem", 3600)):
k = ec.generate_private_key(ec.SECP256R1())
c = (x509.CertificateBuilder().subject_name(name("api.example.com")).issuer_name(ca.subject)
.public_key(k.public_key()).serial_number(x509.random_serial_number())
.not_valid_before(now - dt.timedelta(seconds=back)).not_valid_after(now + dt.timedelta(days=7))
.add_extension(x509.SubjectAlternativeName([x509.DNSName("api.example.com")]), critical=False)
.sign(ca_key, hashes.SHA256()))
open(fname, "wb").write(c.public_bytes(serialization.Encoding.PEM))
Then verify with OpenSSL 3.0, passing -attime as a Unix timestamp:
The result is not a claim about every public CA's issuance policy. It shows the direct consequence of the two validity timestamps: a verifier behind the issuing clock rejects the first leaf, while the locally backdated leaf passes. OpenSSL documents that it reports not-yet-valid or expired when the verification time is outside the validity window.
Shorter certificate lifetimes mean renewal happens more often. That increases the frequency of encounters with newly issued certificates, not the size of the clock error needed to fail. Check your renewal automation as described in our 64-day certificate renewal check, and use the SSL/TLS Handshake Simulator to review where certificate validation belongs in the handshake.
JWTs: a small leeway is a policy, not a clock fix
RFC 7519 says an implementation may provide small leeway, usually no more than a few minutes, for clock skew. It does not choose a value for you. PyJWT's decode() examples show that leeway can be seconds or a datetime.timedelta; its 2.15.1 source sets the default to zero.
Here is the complete local demonstration. The issuer clock is 90 seconds ahead of the verifier. The token has iat, nbf, and exp, so a validator that checks issued-at time rejects it before it is valid.
# Clock skew demo for JWTs: the issuer's clock runs 90 seconds ahead of the API that validates the token.
import time, jwt
SECRET = "demo-secret-not-for-production-use-only"
skew = 90 # issuer clock minus verifier clock, in seconds
issuer_now = int(time.time()) + skew
token = jwt.encode({"sub": "svc-billing", "iat": issuer_now, "nbf": issuer_now, "exp": issuer_now + 300}, SECRET, algorithm="HS256")
for leeway in (0, 30, 120):
try:
jwt.decode(token, SECRET, algorithms=["HS256"], leeway=leeway)
print(f"leeway={leeway:>3}s accepted")
except jwt.PyJWTError as e:
print(f"leeway={leeway:>3}s rejected: {type(e).__name__}: {e}")
$ python jwt_skew.py # PyJWT 2.15.1, issuer clock 90 s ahead of the verifier
leeway= 0s rejected: ImmatureSignatureError: The token is not yet valid (iat)
leeway= 30s rejected: ImmatureSignatureError: The token is not yet valid (iat)
leeway=120s accepted
The 120-second value makes the test pass. It does not make a 90-second clock difference healthy. For a service, use a deliberately small policy allowance, such as 30 to 60 seconds if the threat model permits it, alert when it is consumed, and fix time sync. Never use a client-supplied timestamp as the authority for a security decision. Signed webhook schemes have the same replay-window concern, so keep the sender timestamp, receive time, and verifier decision in the delivery record. See what production webhook delivery needs for the surrounding signature and retry design.
Inspect the Linux time path before restarting auth services
Start with the service-independent view:
timedatectl
Inspect System clock synchronized and NTP service. On a host using systemd-timesyncd, timedatectl timesync-status reports the selected server, leap state, offset, delay, jitter, and poll interval. The systemd timedatectl manual documents those fields.
On a chrony host, inspect both the remaining system correction and the sources:
chronyc tracking
chronyc sources -v
In tracking, System time is the remaining offset between chronyd's NTP clock and the system clock. Leap status: Not synchronised is a bad state. In sources -v, Reach is an eight-bit octal reachability register, so 377 means a valid response on each of the last eight polls. A ? source is not selectable, including when it is unreachable. Chrony's chronyc(1) manual defines these fields and their selection markers.
The common failure modes are mundane:
- A device without a working RTC can boot at an old time and needs a usable time source before time-sensitive work starts. systemd-timesyncd records a synchronized time to help advance a machine without a battery-backed RTC on a later boot, but does not hold all units until accurate synchronization by default. See the systemd-timesyncd manual.
- A paused or restored VM can resume with stale time. Chrony's FAQ calls out a virtual machine that is suspended and resumed with an incorrect time as a case that may need stepping on any clock update, not only at boot.
- A firewall rule can block NTP's UDP port 123. When
chronyc sourcesshows no valid responses from a server, the chrony FAQ points first at a firewall blocking requests to UDP port 123 or the replies. - A Linux container shares the host kernel, so the container cannot independently repair the host's system clock. Fix the node or VM that owns the kernel. Docker's container explanation describes that shared-kernel model.
Correct time safely, then make loss of sync noisy
Chrony normally slews a correction, which avoids a sudden jump but can take a long time for a large offset. Its makestep threshold limit directive controls when it may jump instead. A useful boot-time policy is:
# Add to chronyd's active configuration file
# Step offsets over 1 second during the first three clock updates.
makestep 1.0 3
# Keep the kernel's synchronization flag (and the RTC) updated.
# node_exporter's node_timex_sync_status depends on it.
rtcsync
# Replace these with separate, approved NTS-capable sources.
server ntp-a.example.net iburst nts
server ntp-b.example.net iburst nts
When the sources support NTS and chronyd was built with NTS support, the nts option enables Network Time Security; support was added in chrony 4.0. If a host is currently wrong and the operational impact of a jump has been assessed, chronyc makestep immediately applies the remaining correction. Chrony's manual warns that time jumps can seriously affect some software. Confirm the source state first, make the correction in the approved window, then re-run chronyc tracking.
For Prometheus, node_exporter's timex collector exposes node_timex_offset_seconds and node_timex_sync_status. Its source defines the former as offset between the local and reference clocks and the latter as 1 for synchronized and 0 for unsynchronized. Start with a rule that catches loss of sync or a sustained offset, then tune the threshold to the most time-sensitive system you operate:
groups:
- name: clock-health
rules:
- alert: NodeClockNotSynchronized
expr: node_timex_sync_status == 0
for: 5m
labels:
severity: warning
annotations:
summary: 'Node clock is not synchronized'
- alert: NodeClockOffsetHigh
expr: abs(node_timex_offset_seconds) > 0.05
for: 5m
labels:
severity: warning
annotations:
summary: 'Node clock offset exceeds 50 ms'
node_timex_sync_status depends on the kernel time synchronization flag. Node exporter's time notes say every chrony version needs the rtcsync option to maintain that flag, so on chrony hosts the sync alert is only meaningful with rtcsync set. Pair the alert with the daemon's own status and chronyc tracking during investigation.
Incident checklist
- Record the failing host's UTC time, offset, synchronization state, source reachability, and the exact verifier error before changing anything.
- Compare issuer, verifier, and identity-provider times. Do not assume that a successful
datecommand on one pod says anything about the node behind another service. - Repair the source or network path, then use a safe step policy when the offset is too large to slew promptly.
- Retest one TLS verification with
-attime, one JWT with its configured leeway, and one signed request after the host reports synchronized. - Keep logs in UTC, include an offset where available, and return the server time in safe error responses. This makes cross-host event ordering possible without exposing sensitive details.
Time is shared infrastructure. Treat it like DNS or certificate trust: monitor it, make dependencies wait where they need accurate time, and investigate divergence before widening every authentication window.
Try it hands-on
Run the commands from this article in the browser. Nothing to install.
Learn Linux - Interactive Tutorial
Master essential Linux commands through interactive lessons. Practice pwd, ls, cd, cat, grep, chmod, pipes, and more in a simulated terminal environment with 6 progressive lessons.
SSL/TLS Handshake Simulator
Visualize how SSL/TLS handshakes work with certificate verification, key exchange, and cipher negotiation. Compare TLS 1.2 and 1.3 protocols.
We earn commissions when you shop through the links below.
Bitrise
Managed runners for GitHub Actions
Bitrise Build Hub runs your GitHub Actions jobs on M4 Macs and AMD Linux machines. Change runs-on and keep your workflows, secrets and logs on GitHub.
Atomsized
AWS platform engineering and GitOps
Design and automation for reliable AWS and Kubernetes platforms, safer delivery workflows, and preview and UAT environments your engineers can understand and own.
DigitalOcean
Cloud infrastructure for developers
Simple, reliable cloud computing designed for developers
DevDojo
Developer community & tools
Join a community of developers sharing knowledge and tools
SMTPfast
Developer-first email API
Send transactional and marketing email through a clean REST API. Detailed logs, webhooks, and embeddable signup forms in one dashboard.
QuizAPI
Developer-first quiz platform
Build, generate, and embed quizzes with a powerful REST API. AI-powered question generation and live multiplayer.
Want to support DevOps Daily and reach thousands of developers?
Become a SponsorTags
Found an issue?
Related Posts
Also worth your time on this topic
Disk Full but du Disagrees: Finding Deleted Files That Still Use Space
When df reports a full filesystem but du cannot find the bytes, an unlinked log held open by a process is often the missing space. Find the holder, release space safely, and prevent the next midnight disk-full incident.
Linux Performance Tuning and Troubleshooting
Master essential Linux performance analysis and optimization techniques to diagnose and resolve system bottlenecks.
80 minutes
Using grep for Text Search
How do you use grep to search for text patterns in files? What are some useful flags?
junior