Kubernetes Security Best Practices

Secure your Kubernetes clusters and workloads with comprehensive security measures

Security is a critical aspect of running Kubernetes in production environments. As Kubernetes adoption grows, it has become an increasingly attractive target for attackers. In this section, we'll explore comprehensive security measures to protect your clusters, workloads, and data.

Security Fundamentals in Kubernetes

Kubernetes security should be approached from multiple angles:

  1. Cluster security: Securing the Kubernetes control plane and nodes
  2. Container security: Securing the container images and runtime
  3. Application security: Securing the workloads running in your cluster
  4. Network security: Controlling traffic within and outside the cluster
  5. Data security: Protecting sensitive information

Let's explore each of these dimensions.

Securing the Kubernetes Cluster

Control Plane Security

The Kubernetes control plane is a critical security component. Protect it with these measures:

  1. Use TLS for all API communications:

    apiVersion: v1
kind: Pod
metadata:
  name: kube-apiserver
spec:
  containers:
    - name: kube-apiserver
      command:
        - kube-apiserver
        - --secure-port=6443
        - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt
        - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key
        - --client-ca-file=/etc/kubernetes/pki/ca.crt

  2. Enable API server admission controllers:

    apiVersion: v1
kind: Pod
metadata:
  name: kube-apiserver
spec:
  containers:
    - name: kube-apiserver
      command:
        - kube-apiserver
        - --enable-admission-plugins=NodeRestriction,PodSecurityPolicy,AlwaysPullImages

  3. Restrict access to etcd:

    • Use TLS for etcd client and peer communication
    • Firewall etcd ports from unnecessary access
    • Run etcd on dedicated nodes when possible

  4. Implement network policies for the control plane:

    apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: control-plane-protection
  namespace: kube-system
spec:
  podSelector:
    matchLabels:
      tier: control-plane
  policyTypes:
    - Ingress
  ingress:
    - from:
        - ipBlock:
            cidr: 10.0.0.0/24 # Admin workstations
      ports:
        - protocol: TCP
          port: 6443 # API server

Node Security

Secure your worker nodes with these practices:

  1. Keep nodes updated:

    • Apply security patches promptly
    • Use minimal, hardened OS images
    • Consider immutable infrastructure patterns

  2. Secure kubelet configuration:

    # /var/lib/kubelet/config.yaml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
authentication:
  anonymous:
    enabled: false
  webhook:
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
  mode: Webhook
readOnlyPort: 0
protectKernelDefaults: true

  3. Node isolation and segmentation:

    • Use separate node pools for different workload types
    • Isolate nodes for sensitive workloads
    • Apply appropriate node taints and labels

Authentication and Authorization

Implement proper access controls:

  1. Use strong authentication methods:

    • X.509 client certificates
    • OpenID Connect (OIDC) with an identity provider
    • Service accounts for applications

  2. Role-Based Access Control (RBAC):

    Define roles with specific permissions:

    apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: pod-reader
rules:
  - apiGroups: ['']
    resources: ['pods']
    verbs: ['get', 'watch', 'list']

    Bind roles to users or groups:

    apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods
  namespace: default
subjects:
  - kind: User
    name: jane
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

  3. Follow the principle of least privilege:

    • Grant minimal permissions needed for each role
    • Regularly audit and review RBAC permissions
    • Use namespaces to enforce separation

Container and Image Security

Image Security

Secure your container images:

  1. Use minimal base images:

    • Alpine Linux or distroless images
    • Remove unnecessary packages and tools

  2. Scan images for vulnerabilities:

    # Using Trivy
trivy image nginx:1.21

# Or in CI/CD pipeline
trivy image --exit-code 1 --severity HIGH,CRITICAL myapp:latest

  3. Implement a secure image registry:

    • Use private registries
    • Enable authentication and authorization
    • Enforce image signing and verification with Docker Content Trust or Cosign:
    # Sign an image with Cosign
cosign sign --key cosign.key registry.example.com/myapp:latest

# Verify an image
cosign verify --key cosign.pub registry.example.com/myapp:latest

  4. Create and enforce image policies:

    # Using OPA Gatekeeper
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sAllowedRepos
metadata:
  name: require-trusted-repos
spec:
  match:
    kinds:
      - apiGroups: ['']
        kinds: ['Pod']
  parameters:
    repos:
      - 'registry.example.com/'
      - 'docker.io/bitnami/'

Container Runtime Security

Secure the container runtime:

  1. Use container-optimized operating systems:

    • Container-Optimized OS (Google)
    • Bottlerocket (AWS)
    • RancherOS or K3OS

  2. Configure seccomp profiles:

    apiVersion: v1
kind: Pod
metadata:
  name: secure-pod
spec:
  securityContext:
    seccompProfile:
      type: Localhost
      localhostProfile: profiles/restricted.json
  containers:
    - name: app
      image: myapp:1.0

  3. Use container runtime sandboxes:

    • gVisor
    • Kata Containers
    • Firecracker

  4. Implement runtime security monitoring:

    • Falco for runtime threat detection
    • Sysdig for container visibility
    • Aqua Security or Twistlock for comprehensive protection

Workload Security

Pod Security

Apply security measures to your pods:

  1. Use Pod Security Standards:

    Since Kubernetes 1.25, Pod Security Admission provides built-in enforcement of the Pod Security Standards, replacing the deprecated Pod Security Policies:

    # Namespace-level enforcement
apiVersion: v1
kind: Namespace
metadata:
  name: secure-namespace
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/audit: restricted

  2. Configure security contexts:

    apiVersion: v1
kind: Pod
metadata:
  name: secure-pod
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    runAsGroup: 3000
    fsGroup: 2000
    seccompProfile:
      type: RuntimeDefault
  containers:
    - name: app
      image: myapp:1.0
      securityContext:
        allowPrivilegeEscalation: false
        capabilities:
          drop:
            - ALL
        readOnlyRootFilesystem: true

  3. Limit container capabilities:

    • Drop all capabilities by default
    • Add only required capabilities

  4. Configure resource limits:

    resources:
  requests:
    memory: '64Mi'
    cpu: '250m'
  limits:
    memory: '128Mi'
    cpu: '500m'

Service Account Security

Secure your service accounts:

  1. Disable automounting of service account tokens:

    apiVersion: v1
kind: ServiceAccount
metadata:
  name: restricted-sa
automountServiceAccountToken: false

  2. Create and use minimal service accounts:

    apiVersion: v1
kind: ServiceAccount
metadata:
  name: app-sa
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: app-role
rules:
  - apiGroups: ['']
    resources: ['configmaps']
    verbs: ['get']
    resourceNames: ['app-config']
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: app-rb
subjects:
  - kind: ServiceAccount
    name: app-sa
roleRef:
  kind: Role
  name: app-role
  apiGroup: rbac.authorization.k8s.io

  3. Use projected service account tokens:

    apiVersion: v1
kind: Pod
metadata:
  name: token-projected-pod
spec:
  containers:
    - name: app
      image: myapp:1.0
      volumeMounts:
        - name: token
          mountPath: /var/run/secrets/tokens
  volumes:
    - name: token
      projected:
        sources:
          - serviceAccountToken:
              path: token
              expirationSeconds: 3600
              audience: api

Secret Management

Properly manage sensitive information:

  1. Use external secret management systems:

    • HashiCorp Vault
    • AWS Secrets Manager
    • Google Secret Manager
    • Azure Key Vault

    Using External Secrets Operator:

    apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
spec:
  refreshInterval: '15m'
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: database-credentials
  data:
    - secretKey: username
      remoteRef:
        key: databases/mysql/credentials
        property: username
    - secretKey: password
      remoteRef:
        key: databases/mysql/credentials
        property: password

  2. Encrypt etcd data:

    apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: <base64-encoded-key>
      - identity: {}

  3. Use sealed secrets for GitOps workflows:

    # Encrypt a secret
kubectl create secret generic db-creds \
  --from-literal=username=admin \
  --from-literal=password=secretpassword \
  --dry-run=client -o yaml | \
  kubeseal --format yaml > sealed-db-creds.yaml

Network Security

Network Policies

Implement network segmentation:

  1. Default deny policy:

    apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

  2. Allow specific communication:

    apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-allow
spec:
  podSelector:
    matchLabels:
      app: api
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: frontend
      ports:
        - protocol: TCP
          port: 8080
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: database
      ports:
        - protocol: TCP
          port: 5432

  3. Namespace isolation:

    apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: namespace-isolation
  namespace: production
spec:
  podSelector: {}
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              purpose: production

Secure Ingress

Protect external access:

  1. TLS termination:

    apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: secure-ingress
spec:
  tls:
    - hosts:
        - api.example.com
      secretName: api-tls
  rules:
    - host: api.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: api-service
                port:
                  number: 80

  2. Authentication with OAuth2 Proxy:

    apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: oauth-ingress
  annotations:
    nginx.ingress.kubernetes.io/auth-url: 'https://$host/oauth2/auth'
    nginx.ingress.kubernetes.io/auth-signin: 'https://$host/oauth2/start'
spec:
  # ...

  3. Web Application Firewall (WAF):

    apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: waf-protected-ingress
  annotations:
    nginx.ingress.kubernetes.io/enable-modsecurity: 'true'
    nginx.ingress.kubernetes.io/modsecurity-transaction-id: '$request_id'
spec:
  # ...

Service Mesh Security

Implement a service mesh for enhanced security:

  1. Istio with mTLS:

    apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system
spec:
  mtls:
    mode: STRICT

  2. Authentication and authorization policies:

    apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: frontend-to-backend
  namespace: default
spec:
  selector:
    matchLabels:
      app: backend
  rules:
    - from:
        - source:
            principals: ['cluster.local/ns/default/sa/frontend-sa']
      to:
        - operation:
            methods: ['GET']
            paths: ['/api/v1/*']

Compliance and Auditing

Audit Logging

Enable and configure audit logging:

apiVersion: v1
kind: Pod
metadata:
  name: kube-apiserver
spec:
  containers:
    - name: kube-apiserver
      command:
        - kube-apiserver
        - --audit-log-path=/var/log/kubernetes/audit.log
        - --audit-log-maxage=30
        - --audit-log-maxbackup=10
        - --audit-log-maxsize=100
        - --audit-policy-file=/etc/kubernetes/audit-policy.yaml

With a policy like:

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
  # Log pod changes at RequestResponse level
  - level: RequestResponse
    resources:
      - group: ''
        resources: ['pods']
  # Log configmaps and secrets at Metadata level
  - level: Metadata
    resources:
      - group: ''
        resources: ['secrets', 'configmaps']
  # Log nothing else
  - level: None

Security Scanning and Benchmarking

Regularly scan your cluster:

  1. CIS Kubernetes Benchmark:

    # Using kube-bench
kubectl run kube-bench --image=aquasec/kube-bench:latest

  2. Kubernetes Security Posture Management (KSPM):

    • Kubescape
    • Falco
    • Prisma Cloud

  3. Configuration scanning:

    # Using Trivy for configuration scanning
trivy config --severity HIGH,CRITICAL ./kubernetes/

Compliance Frameworks

Align with industry standards:

  1. Map controls to standards:

    • PCI DSS
    • HIPAA
    • SOC 2
    • ISO 27001

  2. Implement compliance as code:

    # OPA Gatekeeper constraint for PCI requirement 2.2
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: required-compliance-labels
spec:
  match:
    kinds:
      - apiGroups: ['']
        kinds: ['Pod']
  parameters:
    labels:
      - key: 'compliance'
        allowedRegex: 'pci|hipaa|sox'

Security Response

Incident Response

Prepare for security incidents:

  1. Create an incident response plan:

    • Detection procedures
    • Containment strategies
    • Eradication processes
    • Recovery steps

  2. Implement detection mechanisms:

    • Runtime security monitoring
    • Anomaly detection
    • Alert correlation

  3. Practice containment strategies:

    # Isolate a compromised node
kubectl cordon <node-name>
kubectl drain <node-name> --ignore-daemonsets

# Isolate a namespace with network policies
kubectl apply -f quarantine-policy.yaml

Automated Remediation

Set up automated responses:

  1. Pod restart on suspicious activity:

    # Falco response using falco-sidekick
spec:
  alertOutput:
    webhooks:
      - url: 'http://falco-sidekick:2801'

  2. Network quarantine:

    apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: quarantine
spec:
  podSelector:
    matchLabels:
      security.kubernetes.io/quarantine: 'true'
  policyTypes:
    - Ingress
    - Egress

Security Hardening in DigitalOcean Kubernetes

DigitalOcean Kubernetes Service (DOKS) provides several security features:

  1. Control plane security:

    • Managed and secured by DigitalOcean
    • Automatic security patching
    • Private networking for nodes

  2. Node security:

    • Regular security updates
    • Minimal images

  3. Network security:

    • Support for NetworkPolicies
    • Private networking
    • VPC integration

Sign up with DigitalOcean to get $200 in free credits and deploy your applications on a secure Kubernetes platform.

Securing the CI/CD Pipeline

Protect your deployment pipeline:

  1. Secure the build environment:

    • Use isolated build environments
    • Implement least privilege for CI systems
    • Scan dependencies during builds

  2. Implement GitOps security:

    • Require signed commits
    • Enforce code reviews
    • Use protected branches

  3. Secure deployment credentials:

    • Use short-lived credentials
    • Implement specific deployment service accounts
    • Restrict deployment to specific namespaces

Example secure CI/CD workflow:

# GitHub Actions workflow
name: Secure Kubernetes Deployment

on:
  push:
    branches: [main]

jobs:
  scan-and-deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0

      - name: Verify commit signature
        run: git verify-commit HEAD

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2

      - name: Login to registry
        uses: docker/login-action@v2
        with:
          registry: ${{ secrets.REGISTRY }}
          username: ${{ secrets.REGISTRY_USERNAME }}
          password: ${{ secrets.REGISTRY_PASSWORD }}

      - name: Build and push
        uses: docker/build-push-action@v4
        with:
          context: .
          push: true
          tags: ${{ secrets.REGISTRY }}/app:${{ github.sha }}

      - name: Scan image
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: ${{ secrets.REGISTRY }}/app:${{ github.sha }}
          format: 'sarif'
          output: 'trivy-results.sarif'
          exit-code: '1'
          severity: 'CRITICAL'

      - name: Deploy to Kubernetes
        uses: Azure/k8s-deploy@v1
        with:
          manifests: |
            kubernetes/deployment.yaml
            kubernetes/service.yaml
          images: |
            ${{ secrets.REGISTRY }}/app:${{ github.sha }}
          namespace: production

Security Best Practices Checklist

Use this checklist to ensure comprehensive security:

Cluster Security

  1. TLS encryption for all control plane communication
  2. RBAC properly configured
  3. API server admission controllers enabled
  4. etcd encrypted and secured
  5. Nodes regularly updated and hardened
  6. Kubelet configured securely

Container Security

  1. Images scanned for vulnerabilities
  2. Base images minimized
  3. Images signed and verified
  4. Container registry secured
  5. Runtime security monitoring implemented
  6. Security contexts configured

Workload Security

  1. Pod Security Standards enforced
  2. Service accounts properly managed
  3. Secrets stored securely
  4. Resource limits defined
  5. Non-root users configured
  6. ReadOnly file systems where possible

Network Security

  1. Network policies implemented
  2. Default deny policies applied
  3. Ingress traffic secured with TLS
  4. Service mesh with mTLS (if applicable)
  5. Egress traffic controlled

Monitoring and Audit

  1. Audit logging enabled
  2. Security scanning automated
  3. Compliance checks implemented
  4. Monitoring for suspicious activity
  5. Incident response plan documented

Conclusion

Kubernetes security requires a defense-in-depth approach, addressing multiple layers of the stack. By implementing the practices outlined in this guide, you can significantly improve your cluster's security posture.

Remember that security is not a one-time task but an ongoing process. Stay updated with security best practices and vulnerabilities that affect Kubernetes and its ecosystem.

For production deployments, consider using a managed Kubernetes service like DigitalOcean Kubernetes, which handles many security aspects for you while giving you the flexibility to implement additional security measures for your specific needs.

In this comprehensive guide, we've covered the fundamentals of Kubernetes, from architecture to security. You now have the knowledge to deploy, manage, and secure Kubernetes applications in various environments. Continue exploring and experimenting to deepen your expertise in this powerful container orchestration platform.

Updated: 5/17/2025

Tags

KubernetesContainersDevOpsOrchestrationCloud Native

Found an issue?