Kubernetes Policy, Security, and Multi-Tenancy

Most sections below end with a quick knowledge check — try to answer before revealing.

Admission Controllers — The Policy Enforcement Gate

Every kubectl apply request passes through the API server pipeline. Admission controllers are the last gate before the object is written to etcd.

flowchart LR
    REQ["kubectl apply"] --> AUTHN["Authentication<br/>who are you?"]
    AUTHN --> AUTHZ["Authorization<br/>RBAC: are you allowed?"]
    AUTHZ --> MUT["Mutating Webhooks<br/>OPA/Kyverno: inject defaults<br/>add labels, set limits"]
    MUT --> VAL_SCHEMA["Schema Validation<br/>OpenAPI: is the YAML valid?"]
    VAL_SCHEMA --> VAL_WH["Validating Webhooks<br/>OPA/Kyverno: policy checks<br/>reject if non-compliant"]
    VAL_WH --> ETCD["etcd<br/>object stored"]
    MUT -->|"failurePolicy:Fail + webhook down"| DENY["❌ Request denied"]
    VAL_WH -->|"policy violation"| DENY2["❌ Request denied with message"]

Two tools dominate: OPA/Gatekeeper (declarative Rego policies) and Kyverno (K8s-native YAML policies). Both work as ValidatingWebhookConfiguration + MutatingWebhookConfiguration.

The order matters — mutation happens before validation, not after. Step through what a single kubectl apply goes through:

OPA Gatekeeper

OPA (Open Policy Agent) + Gatekeeper implements K8s policy as code using the Rego language.

Install

kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/release-3.14/deploy/gatekeeper.yaml

ConstraintTemplate — defines a policy type

# Define a new policy type: "must have required labels"
apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: requirelabels
spec:
  crd:
    spec:
      names:
        kind: RequireLabels
      validation:
        openAPIV3Schema:
          type: object
          properties:
            labels:
              type: array
              items:
                type: string

  targets:
  - target: admission.k8s.gatekeeper.sh
    rego: |
      package requirelabels

      violation[{"msg": msg}] {
        provided := {label | input.review.object.metadata.labels[label]}
        required := {label | label := input.parameters.labels[_]}
        missing := required - provided
        count(missing) > 0
        msg := sprintf("Missing required labels: %v", [missing])
      }

Constraint — applies the policy to resources

# Enforce that all Pods have "app" and "team" labels
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: RequireLabels
metadata:
  name: require-pod-labels
spec:
  match:
    kinds:
    - apiGroups: [""]
      kinds: ["Pod"]
    namespaces: ["production", "staging"]   # only enforce in these namespaces
  parameters:
    labels: ["app", "team"]

# Test: create a pod without labels → should be denied
kubectl run nginx --image=nginx -n production
# Error: [require-pod-labels] Missing required labels: {"app", "team"}

# Audit: find existing violations
kubectl get requirelabels.constraints.gatekeeper.sh -o yaml
# status.violations lists all existing objects that violate

Kyverno — K8s-Native Policies (No Rego)

Kyverno uses pure YAML — no new language to learn. Policies are Kubernetes resources.

Install

helm install kyverno kyverno/kyverno -n kyverno --create-namespace

Validate — reject non-compliant resources

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-labels
spec:
  validationFailureAction: Enforce   # Enforce=block, Audit=warn only
  rules:
  - name: check-team-label
    match:
      any:
      - resources:
          kinds: ["Pod"]
          namespaces: ["production"]
    validate:
      message: "Pod must have 'team' label"
      pattern:
        metadata:
          labels:
            team: "?*"   # must exist and be non-empty

Mutate — auto-inject fields

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-default-labels
spec:
  rules:
  - name: inject-team-label
    match:
      any:
      - resources:
          kinds: ["Pod"]
    mutate:
      patchStrategicMerge:
        metadata:
          labels:
            +(managed-by): kyverno   # + prefix = only add if missing

Generate — create resources automatically

# Auto-create a NetworkPolicy when a new Namespace is created
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: default-deny-networkpolicy
spec:
  rules:
  - name: default-deny
    match:
      any:
      - resources:
          kinds: ["Namespace"]
    generate:
      apiVersion: networking.k8s.io/v1
      kind: NetworkPolicy
      name: default-deny-all
      namespace: "{{request.object.metadata.name}}"
      data:
        spec:
          podSelector: {}
          policyTypes: ["Ingress", "Egress"]

Three rule types, one policy engine — flip between them to see what each one actually does to an object:

ValidatingAdmissionPolicy

Both OPA/Gatekeeper and Kyverno work by standing up a webhook server that the API server calls out to over the network on every matching request. ValidatingAdmissionPolicy (VAP), GA since K8s 1.30, skips that entirely: policies are written in CEL (Common Expression Language) — a simple, side-effect-free expression language — and evaluated directly by the API server against fields of the object under review. No external process, no webhook pod to run or scale, no network hop in the admission path.

VAP follows the same separation-of-concerns pattern this repo already covers elsewhere — Role vs. RoleBinding, or PriorityLevelConfiguration vs. FlowSchema — split into two objects instead of one:

• ValidatingAdmissionPolicy — the CEL rule itself, plus matchConstraints defining which resources, namespaces, and operations (CREATE, UPDATE, DELETE) it applies to.
• ValidatingAdmissionPolicyBinding — binds a policy to specific resources, namespaces, or parameters, so the same policy definition can be reused across different scopes without rewriting the CEL expression each time.

Inside a policy's validations, a CEL expression has access to:

• object — the incoming resource, as it exists after any mutating webhooks have run.
• oldObject — the resource's previous state, relevant only for UPDATE operations (null on CREATE).
• request — admission request metadata, like the operation type and the calling user's info.

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: replica-limit
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
    - apiGroups: ["apps"]
      apiVersions: ["v1"]
      operations: ["CREATE", "UPDATE"]
      resources: ["deployments"]
  validations:
  - expression: "object.spec.replicas <= 10"
    message: "Deployments cannot request more than 10 replicas"
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: replica-limit-binding
spec:
  policyName: replica-limit
  validationActions: ["Deny"]   # or "Warn", "Audit"
  matchResources:
    namespaceSelector:
      matchLabels:
        environment: production

This isn't a strict upgrade over OPA/Kyverno, though. CEL's tooling and ecosystem are considerably less mature than Rego's or Kyverno's native YAML — fewer libraries, less community policy content, thinner debugging support. VAP is also validating-only: there's no mutating equivalent in mainline, GA Kubernetes, so it can't inject defaults or rewrite fields the way a Kyverno mutate rule or an OPA mutating webhook can. In practice these approaches aren't mutually exclusive — a real cluster might lean on VAP for simple structural CEL rules (replica caps, required fields, image registry checks) precisely because it needs no extra infrastructure, while keeping Kyverno or OPA around for anything that needs to mutate objects or that's too complex to express cleanly as a CEL one-liner.

OPA/Gatekeeper vs Kyverno vs ValidatingAdmissionPolicy

Multi-Tenancy — Namespace Isolation

K8s multi-tenancy means multiple teams share one cluster safely. Each team gets namespaces with enforced isolation.

graph TD
    subgraph "Cluster"
        subgraph "team-payments NS"
            P_QUOTA["ResourceQuota:<br/>cpu: 8 / mem: 16Gi / pods: 50"]
            P_LR["LimitRange:<br/>default: 200m/256Mi"]
            P_NP["NetworkPolicy:<br/>default deny all<br/>allow only from same NS"]
            P_RBAC["RoleBinding:<br/>payments-team → developer role"]
        end
        subgraph "team-platform NS"
            PL_QUOTA["ResourceQuota: higher limits"]
            PL_NP["NetworkPolicy: allow cross-NS for monitoring"]
        end
    end

ResourceQuota per namespace

apiVersion: v1
kind: ResourceQuota
metadata:
  name: payments-quota
  namespace: team-payments
spec:
  hard:
    requests.cpu: "8"
    requests.memory: 16Gi
    limits.cpu: "16"
    limits.memory: 32Gi
    pods: "50"
    services: "10"
    persistentvolumeclaims: "5"
    count/deployments.apps: "20"

Try It Yourself: Live ResourceQuota Admission

A ResourceQuota's hard limits aren't just a display number — every pod creation gets evaluated against the namespace's running total, the same admission chain covered above (this check happens in-tree in the API server, no webhook involved). Set a quota below, then try creating pods with different cpu/memory requests. A creation that would push any single dimension over its hard limit gets rejected outright — nothing partially applies, and the running totals don't move. Delete a pod to free its share back up, then retry a request that was previously rejected.

LimitRange — defaults for pods without requests

apiVersion: v1
kind: LimitRange
metadata:
  name: default-limits
  namespace: team-payments
spec:
  limits:
  - type: Container
    default:          # applied if no limits set
      cpu: "500m"
      memory: "256Mi"
    defaultRequest:   # applied if no requests set
      cpu: "100m"
      memory: "128Mi"
    max:              # hard ceiling per container
      cpu: "4"
      memory: "4Gi"

Network isolation per team

# Default deny all — add to every namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: team-payments
spec:
  podSelector: {}
  policyTypes: ["Ingress", "Egress"]
---
# Allow intra-namespace traffic
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-same-namespace
  namespace: team-payments
spec:
  podSelector: {}
  ingress:
  - from:
    - podSelector: {}    # any pod in THIS namespace
  egress:
  - to:
    - podSelector: {}
---
# Allow egress to DNS (CoreDNS in kube-system)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: team-payments
spec:
  podSelector: {}
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
    ports:
    - port: 53
      protocol: UDP

Pod Security — PSA, seccomp, AppArmor

Pod Security Admission (PSA) — built-in since K8s 1.25

Replaces the deprecated PodSecurityPolicy. Three levels applied at namespace level.

# Label a namespace to enforce Pod Security Standards
apiVersion: v1
kind: Namespace
metadata:
  name: team-payments
  labels:
    pod-security.kubernetes.io/enforce: restricted    # block violations
    pod-security.kubernetes.io/warn: restricted       # warn on violations
    pod-security.kubernetes.io/audit: restricted      # log violations

Each level is a strict superset of the one below it — flip through what actually changes:

seccomp — restrict syscalls

spec:
  securityContext:
    seccompProfile:
      type: RuntimeDefault   # use container runtime's default profile
      # or: type: Localhost, localhostProfile: profiles/my-profile.json
  containers:
  - name: app
    securityContext:
      allowPrivilegeEscalation: false
      runAsNonRoot: true
      runAsUser: 65534
      readOnlyRootFilesystem: true
      capabilities:
        drop: ["ALL"]    # drop ALL Linux capabilities
        add: ["NET_BIND_SERVICE"]  # add back only what's needed

AppArmor — restrict file/network access

# Apply AppArmor profile to a container
metadata:
  annotations:
    container.apparmor.security.beta.kubernetes.io/app: localhost/my-profile
    # or: runtime/default  (use container runtime's default)
    # or: unconfined       (no AppArmor — avoid in production)

Full Security Checklist per Namespace

# 1. Apply PSA restricted label
kubectl label namespace team-payments \
  pod-security.kubernetes.io/enforce=restricted

# 2. Create ResourceQuota
kubectl apply -f quota.yaml -n team-payments

# 3. Create LimitRange (so pods without requests get defaults)
kubectl apply -f limitrange.yaml -n team-payments

# 4. Create default NetworkPolicies (deny-all + allow-same-ns + allow-dns)
kubectl apply -f network-policies.yaml -n team-payments

# 5. Create Kyverno/Gatekeeper policies (require labels, block latest tag)
kubectl apply -f policies.yaml

# 6. RBAC: bind team to Role (not ClusterRole)
kubectl create rolebinding payments-dev \
  --role=developer \
  --group=payments-team \
  -n team-payments

# Verify: check what a team member can do
kubectl auth can-i create deployments \
  --namespace team-payments \
  --as-group payments-team \
  --as bob@company.com

The order isn't arbitrary — each step assumes the one before it is already in place. Walk through why: