Skip to main content
10 min read

When the Clock Is Wrong: TLS, JWT and Signed-Request Failures From Clock Skew

When the Clock Is Wrong: TLS, JWT and Signed-Request Failures From Clock Skew

A certificate can be freshly renewed and still be "not yet valid." A JWT can be correctly signed and still fail iat or nbf. An AWS request can have the right key and receive RequestTimeTooSkewed. When those symptoms arrive together, stop rotating credentials and check time first. A clock is an input to each decision, and a host that is only slightly early or late can turn an infrastructure problem into an authentication-looking incident.

This post maps the time check behind each failure, shows safe reproductions without changing a machine clock, and gives a Linux response path that makes loss of sync visible before users find it.

TL;DR

  • TLS validation compares the verifier's current time with notBefore and notAfter. It has no protocol-level grace period.
  • JWT leeway is a verifier policy. PyJWT accepts a leeway argument, and its default is zero when you do not pass one.
  • S3 documents a 15-minute request timestamp limit. MIT Kerberos defaults to 300 seconds. TOTP recommends 30-second steps and at most one step for network delay.
  • On Linux, first check timedatectl, then use chronyc tracking and chronyc sources -v when chrony is your time daemon.
  • Correct the host clock, not a container clock. Alert on sync state and offset, then preserve UTC timestamps and offsets in incident logs.
Press play to watch a server's clock fall 1, 6 and then 20 minutes behind: a fresh JWT fails first, then a certificate issued 2 minutes ago and Kerberos (5 minutes allowed), and finally an S3 signed request (15 minutes allowed).

Prerequisites

  • Shell access to a Linux host or VM.
  • Permission to inspect the active time service. Correcting time needs an approved change path because a clock step can surprise applications.
  • chrony installed if you use the chrony commands and configuration below.
  • OpenSSL, Python with cryptography for the TLS fixture, and PyJWT for the JWT demo.

The time window behind the error

The same skew produces different errors because each protocol asks a different question. RFC 5280 requires certificate path validation to include the current time in the certificate validity period. RFC 7519 defines nbf, exp, and iat as NumericDate-based claims. SigV4 includes a UTC request timestamp in the signature input. TOTP derives its moving value from time.

Check Time comparison Documented tolerance Typical symptom
TLS certificate Local verifier time versus notBefore and notAfter None in the certificate validity check certificate is not yet valid or expired
JWT Verifier time versus nbf, exp, and possibly iat Library policy. PyJWT 2.15.1 defaults to 0; RFC 7519 permits small leeway, usually no more than a few minutes ImmatureSignatureError, expired token
AWS SigV4 Signed request timestamp versus AWS time S3 rejects a difference greater than 15 minutes RequestTimeTooSkewed; other signature paths can report signature mismatch or expiry
MIT Kerberos Client or service time versus KDC time clockskew default: 300 seconds Clock skew too great
TOTP Prover time step versus validator time step RFC 6238 recommends a 30-second step and at most one step for network delay A correct-looking code is rejected

The S3 limit and error name come from AWS's request authentication documentation. AWS also documents SignatureDoesNotMatch and the Signature expired diagnostic for SigV4 troubleshooting in its IAM guide. MIT documents its 300-second default in krb5.conf. RFC 6238 calls for a 30-second default step and at most one step of delay. Those numbers are useful triage boundaries, not a reason to let time drift until the next boundary.

Prove TLS time failures without changing the host clock

Do not set a workstation's clock backward to test certificate behavior. OpenSSL's -attime verification option supplies the verification time instead. That isolates the test from package updates, log ordering, token expiry, and every other process on the machine.

This run creates a local CA that began 30 days earlier and two leaf certificates issued at the run time. One leaf starts at issuance. The other has notBefore one hour earlier. The verifier clock is then simulated 10 minutes behind, and then 8 days ahead. The fixtures come from this short script, which uses the Python cryptography package so it can set notBefore freely (python3 mkcerts.py writes ca.pem, leaf-now.pem and leaf-backdated.pem):

# Build a demo CA (valid since 30 days ago) and two leaf certs issued "now":
# leaf-now.pem has notBefore = issuance time, leaf-backdated.pem has notBefore = issuance time - 1 hour.
import datetime as dt
from cryptography import x509
from cryptography.x509.oid import NameOID
from cryptography.hazmat.primitives import hashes, serialization
from cryptography.hazmat.primitives.asymmetric import ec
now = dt.datetime.now(dt.timezone.utc).replace(microsecond=0)
def name(cn): return x509.Name([x509.NameAttribute(NameOID.COMMON_NAME, cn)])
ca_key = ec.generate_private_key(ec.SECP256R1())
ca = (x509.CertificateBuilder().subject_name(name("Demo CA")).issuer_name(name("Demo CA"))
      .public_key(ca_key.public_key()).serial_number(x509.random_serial_number())
      .not_valid_before(now - dt.timedelta(days=30)).not_valid_after(now + dt.timedelta(days=365))
      .add_extension(x509.BasicConstraints(ca=True, path_length=None), critical=True)
      .sign(ca_key, hashes.SHA256()))
open("ca.pem", "wb").write(ca.public_bytes(serialization.Encoding.PEM))
for fname, back in (("leaf-now.pem", 0), ("leaf-backdated.pem", 3600)):
    k = ec.generate_private_key(ec.SECP256R1())
    c = (x509.CertificateBuilder().subject_name(name("api.example.com")).issuer_name(ca.subject)
         .public_key(k.public_key()).serial_number(x509.random_serial_number())
         .not_valid_before(now - dt.timedelta(seconds=back)).not_valid_after(now + dt.timedelta(days=7))
         .add_extension(x509.SubjectAlternativeName([x509.DNSName("api.example.com")]), critical=False)
         .sign(ca_key, hashes.SHA256()))
    open(fname, "wb").write(c.public_bytes(serialization.Encoding.PEM))

Then verify with OpenSSL 3.0, passing -attime as a Unix timestamp:

The result is not a claim about every public CA's issuance policy. It shows the direct consequence of the two validity timestamps: a verifier behind the issuing clock rejects the first leaf, while the locally backdated leaf passes. OpenSSL documents that it reports not-yet-valid or expired when the verification time is outside the validity window.

Shorter certificate lifetimes mean renewal happens more often. That increases the frequency of encounters with newly issued certificates, not the size of the clock error needed to fail. Check your renewal automation as described in our 64-day certificate renewal check, and use the SSL/TLS Handshake Simulator to review where certificate validation belongs in the handshake.

JWTs: a small leeway is a policy, not a clock fix

RFC 7519 says an implementation may provide small leeway, usually no more than a few minutes, for clock skew. It does not choose a value for you. PyJWT's decode() examples show that leeway can be seconds or a datetime.timedelta; its 2.15.1 source sets the default to zero.

Here is the complete local demonstration. The issuer clock is 90 seconds ahead of the verifier. The token has iat, nbf, and exp, so a validator that checks issued-at time rejects it before it is valid.

# Clock skew demo for JWTs: the issuer's clock runs 90 seconds ahead of the API that validates the token.
import time, jwt

SECRET = "demo-secret-not-for-production-use-only"
skew = 90  # issuer clock minus verifier clock, in seconds
issuer_now = int(time.time()) + skew
token = jwt.encode({"sub": "svc-billing", "iat": issuer_now, "nbf": issuer_now, "exp": issuer_now + 300}, SECRET, algorithm="HS256")

for leeway in (0, 30, 120):
    try:
        jwt.decode(token, SECRET, algorithms=["HS256"], leeway=leeway)
        print(f"leeway={leeway:>3}s  accepted")
    except jwt.PyJWTError as e:
        print(f"leeway={leeway:>3}s  rejected: {type(e).__name__}: {e}")
$ python jwt_skew.py   # PyJWT 2.15.1, issuer clock 90 s ahead of the verifier
leeway=  0s  rejected: ImmatureSignatureError: The token is not yet valid (iat)
leeway= 30s  rejected: ImmatureSignatureError: The token is not yet valid (iat)
leeway=120s  accepted

The 120-second value makes the test pass. It does not make a 90-second clock difference healthy. For a service, use a deliberately small policy allowance, such as 30 to 60 seconds if the threat model permits it, alert when it is consumed, and fix time sync. Never use a client-supplied timestamp as the authority for a security decision. Signed webhook schemes have the same replay-window concern, so keep the sender timestamp, receive time, and verifier decision in the delivery record. See what production webhook delivery needs for the surrounding signature and retry design.

Inspect the Linux time path before restarting auth services

Start with the service-independent view:

timedatectl

Inspect System clock synchronized and NTP service. On a host using systemd-timesyncd, timedatectl timesync-status reports the selected server, leap state, offset, delay, jitter, and poll interval. The systemd timedatectl manual documents those fields.

On a chrony host, inspect both the remaining system correction and the sources:

chronyc tracking
chronyc sources -v

In tracking, System time is the remaining offset between chronyd's NTP clock and the system clock. Leap status: Not synchronised is a bad state. In sources -v, Reach is an eight-bit octal reachability register, so 377 means a valid response on each of the last eight polls. A ? source is not selectable, including when it is unreachable. Chrony's chronyc(1) manual defines these fields and their selection markers.

The common failure modes are mundane:

  • A device without a working RTC can boot at an old time and needs a usable time source before time-sensitive work starts. systemd-timesyncd records a synchronized time to help advance a machine without a battery-backed RTC on a later boot, but does not hold all units until accurate synchronization by default. See the systemd-timesyncd manual.
  • A paused or restored VM can resume with stale time. Chrony's FAQ calls out a virtual machine that is suspended and resumed with an incorrect time as a case that may need stepping on any clock update, not only at boot.
  • A firewall rule can block NTP's UDP port 123. When chronyc sources shows no valid responses from a server, the chrony FAQ points first at a firewall blocking requests to UDP port 123 or the replies.
  • A Linux container shares the host kernel, so the container cannot independently repair the host's system clock. Fix the node or VM that owns the kernel. Docker's container explanation describes that shared-kernel model.

Correct time safely, then make loss of sync noisy

Chrony normally slews a correction, which avoids a sudden jump but can take a long time for a large offset. Its makestep threshold limit directive controls when it may jump instead. A useful boot-time policy is:

# Add to chronyd's active configuration file
# Step offsets over 1 second during the first three clock updates.
makestep 1.0 3

# Keep the kernel's synchronization flag (and the RTC) updated.
# node_exporter's node_timex_sync_status depends on it.
rtcsync

# Replace these with separate, approved NTS-capable sources.
server ntp-a.example.net iburst nts
server ntp-b.example.net iburst nts

When the sources support NTS and chronyd was built with NTS support, the nts option enables Network Time Security; support was added in chrony 4.0. If a host is currently wrong and the operational impact of a jump has been assessed, chronyc makestep immediately applies the remaining correction. Chrony's manual warns that time jumps can seriously affect some software. Confirm the source state first, make the correction in the approved window, then re-run chronyc tracking.

For Prometheus, node_exporter's timex collector exposes node_timex_offset_seconds and node_timex_sync_status. Its source defines the former as offset between the local and reference clocks and the latter as 1 for synchronized and 0 for unsynchronized. Start with a rule that catches loss of sync or a sustained offset, then tune the threshold to the most time-sensitive system you operate:

groups:
  - name: clock-health
    rules:
      - alert: NodeClockNotSynchronized
        expr: node_timex_sync_status == 0
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'Node clock is not synchronized'
      - alert: NodeClockOffsetHigh
        expr: abs(node_timex_offset_seconds) > 0.05
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: 'Node clock offset exceeds 50 ms'

node_timex_sync_status depends on the kernel time synchronization flag. Node exporter's time notes say every chrony version needs the rtcsync option to maintain that flag, so on chrony hosts the sync alert is only meaningful with rtcsync set. Pair the alert with the daemon's own status and chronyc tracking during investigation.

Incident checklist

  1. Record the failing host's UTC time, offset, synchronization state, source reachability, and the exact verifier error before changing anything.
  2. Compare issuer, verifier, and identity-provider times. Do not assume that a successful date command on one pod says anything about the node behind another service.
  3. Repair the source or network path, then use a safe step policy when the offset is too large to slew promptly.
  4. Retest one TLS verification with -attime, one JWT with its configured leeway, and one signed request after the host reports synchronized.
  5. Keep logs in UTC, include an offset where available, and return the server time in safe error responses. This makes cross-host event ordering possible without exposing sensitive details.

Time is shared infrastructure. Treat it like DNS or certificate trust: monitor it, make dependencies wait where they need accurate time, and investigate divergence before widening every authentication window.

Run the commands from this article in the browser. Nothing to install.

Published: 2026-11-18|Last updated: 2026-11-18T07:00:00Z

Found an issue?

Also worth your time on this topic