Phantom — Technical Architecture

Phantom Architecture

How It Works

Phantom is a Kubernetes operator that injects a sidecar via mutating webhook. The sidecar fetches secrets from an EU-hosted OpenBao/Vault instance directly into process memory — secrets never touch etcd, never enter Kubernetes Secrets, and the cloud provider never holds the keys.

Key Strengths

• Secrets never touch etcd. Eliminates an entire class of attacks (etcd dump, backup exfiltration, KMS compulsion). The correct approach for managed Kubernetes where you have zero control over the control plane.
• Three-tier caching is well-designed. Hot cache → sealed local cache → grace period progression is operationally sound. Sealed cache key derivation from SA token + cluster HMAC is reasonable.
• Circuit breaker on the webhook is the right pattern for fail-closed security products. The override escape hatch (namespace label) is correctly positioned as an auditable last resort.
• Canary injection via namespace labels is operationally mature thinking for a product that modifies every pod in the cluster.

Known Concerns

• gVisor as “optional lightweight sandbox” is undersold. Without it, a root-level attacker on the node can read process memory via /proc/[pid]/mem. The “optionally” qualifier weakens the story.
• The sidecar is a single point of failure per pod. If phantom-proxy crashes and the sealed cache is expired, the application loses access to all secrets. Consider a direct (attested) fallback path to OpenBao.
• Env var patching requires applications to use environment variables or a specific socket protocol. Applications that read secrets from files need a different mechanism — solvable but not addressed.

Trust Model

Security Flows

1. Initial Bootstrap — Trust Establishment

Before any secrets flow, the cluster must be registered with EU OpenBao. This is a one-time setup per cluster.

sequenceDiagram
    participant Admin as Platform Admin
    participant OB as EU OpenBao
    participant K8s as K8s Cluster
    participant PH as Phantom Operator

    rect rgb(30, 40, 55)
    Note over Admin,OB: One-time setup (EU side)
    Admin->>OB: Enable Kubernetes auth method
    Admin->>OB: Register cluster (API server URL + CA cert)
    Admin->>OB: Create policies (namespace → secret paths)
    Admin->>OB: Generate bootstrap token (time-limited)
    end

    rect rgb(30, 45, 40)
    Note over Admin,PH: One-time setup (cluster side)
    Admin->>K8s: helm install phantom --set openbao.addr=... --set openbao.token=...
    PH->>PH: Create ServiceAccount, MutatingWebhookConfig, CRDs
    PH->>OB: Authenticate with bootstrap token (mTLS)
    OB->>K8s: Validate cluster identity via TokenReview API
    OB-->>PH: Confirm trust. Issue renewable accessor token
    PH->>PH: Discard bootstrap token. Use accessor token for renewals
    end

    Note over K8s,OB: Trust established. Bootstrap token is now useless.
      

2. Pod Startup — Secret Injection

What happens every time a labeled pod is created.

sequenceDiagram
    participant Dev as Developer
    participant API as K8s API Server
    participant WH as Phantom Webhook
    participant Pod as App Container
    participant SC as Phantom Sidecar
    participant OB as EU OpenBao

    Dev->>API: kubectl apply (Deployment)
    API->>WH: AdmissionReview (pod spec)
    WH->>WH: Check namespace labels, compatibility
    WH-->>API: Mutated spec (sidecar + init container injected)
    API->>Pod: Schedule pod

    rect rgb(30, 45, 40)
    Note over Pod,OB: Secret injection (happens before app starts)
    SC->>SC: Init container copies wrapper binary to shared volume
    SC->>OB: Auth with pod ServiceAccount token (mTLS)
    OB->>OB: Validate SA token via TokenReview
    OB->>OB: Check policies (namespace + SA → allowed secret paths)
    OB-->>SC: Return secrets (encrypted in transit)
    SC->>SC: Store in hot cache (in-memory) + sealed cache (tmpfs)
    SC->>Pod: Inject as env vars / tmpfs files / Unix socket
    Pod->>Pod: App starts with secrets in process memory
    end

    Note over Pod: Secrets never in etcd. Never on disk. Never in K8s API.

    loop Every 4 minutes
    SC->>OB: Renew lease + check for rotation
    OB-->>SC: Updated secrets (if rotated)
    SC->>Pod: Hot-reload updated secrets
    end
      

3. CLOUD Act Subpoena

What happens when a US legal order compels the cloud provider to hand over data.

sequenceDiagram
    participant USG as US Government
    participant CP as Cloud Provider
    participant K8s as K8s Cluster
    participant OB as EU OpenBao

    USG->>CP: CLOUD Act subpoena: produce all customer data

    rect rgb(55, 30, 30)
    Note over CP,K8s: Provider complies (they must)
    CP->>K8s: Dump etcd
    K8s-->>CP: etcd contents (pods, deployments, configmaps...)
    Note over CP: No secrets found in etcd
    CP->>K8s: Snapshot VM memory
    K8s-->>CP: Memory dump (encrypted if TEE, otherwise readable)
    CP->>K8s: Copy persistent volumes
    K8s-->>CP: Volume data (no secret material)
    end

    CP-->>USG: Deliver: etcd dump + memory + volumes

    rect rgb(30, 40, 55)
    Note over USG,OB: Cannot reach EU OpenBao
    USG->>OB: Request secrets/keys?
    OB-->>USG: EU jurisdiction. Requires EU court order.
    Note over OB: US legal process has no authority here
    end

    Note over USG: Without keys from OpenBao, extracted data is incomplete.
    Note over USG: Memory contents (if no TEE) contain only short-lived tokens that have expired.
      

4. eBPF Memory Access Detection

How the eBPF DaemonSet detects attempts to read protected process memory.

sequenceDiagram
    participant Att as Attacker (node access)
    participant Kernel as Linux Kernel
    participant eBPF as eBPF DaemonSet
    participant SC as Phantom Sidecar
    participant Alert as Alert Pipeline

    Att->>Kernel: ptrace(PTRACE_ATTACH, pid)
    Kernel->>eBPF: sys_ptrace hook fires
    eBPF->>eBPF: Check target PID against protected pod list
    eBPF-->>Alert: ALERT: ptrace on protected pod (pid, namespace, caller)

    Att->>Kernel: open("/proc/{pid}/mem")
    Kernel->>eBPF: sys_openat hook fires
    eBPF->>eBPF: Path matches /proc/*/mem for protected PID
    eBPF-->>Alert: ALERT: /proc/mem read attempt

    Att->>Kernel: process_vm_readv(pid, ...)
    Kernel->>eBPF: sys_process_vm_readv hook fires
    eBPF-->>Alert: ALERT: cross-process memory read

    Note over Alert: Alerts → SaaS dashboard + SIEM + PagerDuty
    Note over SC: Meanwhile: secrets are short-lived tokens (15-min TTL)
      

5. Bootstrap Token — Where Does the First Secret Come From?

The bootstrap token is the one secret that cannot come from OpenBao (because you need it to connect to OpenBao). It must be communicated out-of-band:

1. Admin generates a time-limited token in OpenBao CLI: openbao token create -ttl=10m -use-limit=1 -policy=phantom-bootstrap
2. Token is passed directly to Helm: helm install phantom --set openbao.bootstrapToken=hvs.xxx
3. Phantom operator uses it once to register via Kubernetes auth method
4. Token expires (10 min) and is never persisted in K8s

6. Key Transfer Flows

6a. Initial Provisioning

sequenceDiagram
    participant Pod as New Pod
    participant SC as Phantom Sidecar
    participant Cache as Sealed Cache (tmpfs)
    participant OB as EU OpenBao

    Pod->>SC: Container starts
    SC->>SC: Check hot cache (empty - first run)
    SC->>SC: Check sealed cache (empty - first run)
    SC->>OB: Auth with SA token + request secrets (mTLS)
    OB->>OB: Validate SA token via TokenReview
    OB->>OB: Check policy: namespace/SA → allowed paths
    OB-->>SC: Secrets + lease ID + TTL
    SC->>SC: Store in hot cache (in-memory, 5 min TTL)
    SC->>Cache: Encrypt with HKDF(cluster_key, pod_uid) → sealed cache
    SC->>Pod: Inject secrets (env vars / tmpfs / socket)
    Note over Pod: App starts. Secrets in process memory only.
      

6b. Secret Rotation

sequenceDiagram
    participant Admin as Admin / CI
    participant OB as EU OpenBao
    participant SC as Phantom Sidecar
    participant Pod as App Process

    Admin->>OB: Rotate secret (new version)
    Note over SC: Renewal loop runs every 4 min
    SC->>OB: Renew lease + check version
    OB-->>SC: New secret value + new lease
    SC->>SC: Update hot cache
    SC->>SC: Update sealed cache (re-encrypt)
    SC->>Pod: Signal secret change (SIGHUP or socket notification)
    Pod->>Pod: Reload config with new secret
    Note over Pod: Zero downtime. Old secret zeroed from memory.
      

6c. Node Restart / Pod Reschedule

sequenceDiagram
    participant K8s as K8s Scheduler
    participant Pod as Rescheduled Pod
    participant SC as Phantom Sidecar
    participant Cache as Sealed Cache (tmpfs)
    participant OB as EU OpenBao

    K8s->>Pod: Schedule pod on new node
    SC->>SC: Check hot cache (empty - new pod)
    SC->>Cache: Check sealed cache (empty - new pod, new tmpfs)
    SC->>OB: Auth with SA token (mTLS)
    alt OpenBao reachable
        OB-->>SC: Fresh secrets + new lease
        SC->>Pod: Inject secrets. App starts normally.
    else OpenBao unreachable (outage)
        SC->>SC: No cache, no OpenBao
        SC->>Pod: Block startup. Clear error: "Cannot reach OpenBao"
        Note over Pod: Pod stays in Init. No silent failure.
        Note over Pod: Existing pods on other nodes still serve from cache.
    end
      

7. Break Glass — Webhook Disabled or Bypassed

What happens when the MutatingWebhookConfiguration is deleted, modified, or bypassed.

sequenceDiagram
    participant Att as Attacker / Admin
    participant API as K8s API Server
    participant eBPF as eBPF DaemonSet
    participant Op as Phantom Operator
    participant Alert as Alert Pipeline
    participant Pods as Existing Pods

    Att->>API: Delete MutatingWebhookConfiguration
    API->>API: Webhook removed

    par Detection (immediate)
        eBPF->>eBPF: Watch: webhook config changed
        eBPF-->>Alert: CRITICAL: Webhook deleted (who, when, kubectl context)
        Op->>Op: Reconciliation loop detects missing webhook
        Op->>API: Re-create MutatingWebhookConfiguration
        Note over Op: Webhook restored within seconds
    end

    Note over Pods: Existing pods UNAFFECTED (secrets already in memory)
    Note over API: New pods created during gap: deployed WITHOUT sidecar
    Note over API: After restore: new pods get sidecar again

    rect rgb(55, 30, 30)
    Note over Att,API: What the attacker gains
    Note right of Att: ❌ Cannot extract secrets from running pods (not in etcd)
    Note right of Att: ❌ Cannot access OpenBao (no valid SA token from outside)
    Note right of Att: ⚠️ New pods during gap run without protection
    Note right of Att: ⚠️ If also has node access: can read unprotected pod memory
    end
      

8. MITM Attack Surfaces

Two critical network paths where man-in-the-middle attacks could compromise the system.

8a. EU OpenBao ↔ US Cluster (Cross-Jurisdiction)

sequenceDiagram
    participant SC as Phantom Sidecar (US)
    participant Net as Network Path
    participant MITM as Potential MITM
    participant OB as EU OpenBao

    rect rgb(55, 30, 30)
    Note over Net,MITM: Attack surface: internet/VPN between jurisdictions
    end

    SC->>Net: TLS ClientHello
    Note over SC,OB: Protection: mTLS with pinned certificates
    SC->>OB: Client cert (signed by Phantom CA) + SA token
    OB->>OB: Verify client cert chain
    OB->>OB: Verify SA token via TokenReview
    OB-->>SC: Secrets (encrypted in TLS tunnel)

    rect rgb(55, 30, 30)
    Note over MITM: MITM sees: encrypted traffic only
    Note over MITM: Cannot forge client cert (needs Phantom CA private key)
    Note over MITM: Cannot forge server cert (pinned in sidecar config)
    Note over MITM: Can: block traffic (DoS) → triggers grace period
    Note over MITM: Can: traffic analysis (volume, timing, frequency)
    end
      

8b. DaemonSet ↔ Sidecar (Intra-Cluster)

sequenceDiagram
    participant DS as eBPF DaemonSet
    participant Node as Node Kernel
    participant SC as Phantom Sidecar
    participant Pod as App Process

    Note over DS,Node: DaemonSet operates at kernel level (eBPF programs)
    Note over DS,SC: No network communication needed

    rect rgb(30, 45, 40)
    Note over DS,Pod: eBPF hooks are kernel-space, not network-based
    DS->>Node: Attach eBPF programs to syscall tracepoints
    Node->>DS: Events: ptrace, /proc/mem reads, process_vm_readv
    Note over DS: No MITM possible — eBPF is in-kernel, not over network
    end

    rect rgb(30, 45, 40)
    Note over SC,Pod: Sidecar ↔ App is localhost (same pod network namespace)
    SC->>Pod: Secrets via env vars (set before process start)
    SC->>Pod: Or: secrets via Unix domain socket (filesystem, not network)
    SC->>Pod: Or: secrets via tmpfs mount (shared volume)
    Note over SC,Pod: No MITM possible — same pod, no network traversal
    end

    rect rgb(55, 30, 30)
    Note over Node: Remaining risk: compromised node kernel
    Note over Node: If attacker has root on node: can intercept eBPF, read tmpfs
    Note over Node: Mitigation: TEE (optional) or eBPF tamper detection
    end
      

Technical Feasibility

Phantom Complexity Breakdown

• Mutating webhook: well-understood pattern, excellent Go libraries. 2-3 weeks for a senior Go engineer.
• OpenBao integration (secret fetch, caching, renewal): 3-4 weeks. Three-tier cache adds complexity but is well-scoped.
• Sidecar injection with mesh awareness: 4-6 weeks. Compatibility matrix (Istio, Linkerd, OTel, Dapr, GKE FUSE) is the time sink.
• Circuit breaker + operator lifecycle: 2-3 weeks.
• Cross-provider testing matrix: 4-6 weeks. The hidden cost — testing on GKE Standard, GKE Autopilot, EKS (EC2 + Fargate), and AKS.
• Attestation (SEV-SNP/TDX): 6-8 weeks. Requires specialized knowledge.
• Total: ~5-7 months for production-ready Phantom with attestation.

What Needs More Research

1. Nitro Enclaves integration — fundamentally different from SEV-SNP/TDX. Needs PoC before committing.
2. eBPF memory-access monitoring — what can eBPF detect that’s actionable? Detect-and-alert vs. detect-and-block?

Key Technical Limitations

What Phantom Cannot Do

1. Cannot protect data processed in cleartext. Once the app decrypts a secret, data exists in cleartext in application memory. TEE mitigates this but isn’t universal.
2. Cannot protect against a compromised application. If the application is malicious (supply chain attack), it has legitimate access to decrypted secrets.
3. Cannot protect Kubernetes metadata. Pod names, labels, annotations, network policies — all visible to the cloud provider.
4. Cannot protect against hardware-level attacks on TEEs. AMD SEV-SNP and Intel TDX have had side-channel vulnerabilities (CacheWarp, speculative execution).
5. Cannot enforce key sovereignty after key release. Once a secret is released into the sidecar’s memory, it’s in the cloud provider’s infrastructure.
6. Cannot protect against legal coercion of the customer. This product protects against US extraterritorial reach, not all legal compulsion.

Protection Model Breakdown

Attacks NOT Defended Against

• Supply chain attacks on application container images
• Side-channel attacks on TEE implementations (timing, power analysis, cache-based)
• Social engineering of personnel with access to OpenBao
• Insider threats from the customer’s own team
• Network-level DDoS preventing connectivity to OpenBao
• Container escape followed by host memory access (without TEE)
• Coerced firmware updates on TEE hardware by cloud provider at government request

Cross-Provider Compatibility

Phantom Cross-Provider Status

Phantom’s core architecture (webhook + sidecar + external secrets) works across all three providers with provider-specific code paths for networking, identity, and attestation.

Undocumented Issues to Address

1. GKE Workload Identity Federation — default SA token behavior changes may affect sealed cache key derivation.
2. EKS Pod Identity — sidecar’s OpenBao auth must support both traditional K8s auth and provider-specific identity federation.
3. AKS Node Auto-Provisioning (Karpenter) — eBPF programs must handle different kernel versions within the same cluster.
4. GKE Gateway API migration — network policies may need to understand Gateway API resources.
5. EKS Access Entries — changes who can interact with the API server and bypass webhooks.
6. Multi-tenant GKE clusters (GKE Enterprise) — fleet-level policies can override per-cluster webhook configurations.
7. ARM / Graviton nodes — eBPF programs, sidecar images, and crypto must be multi-arch.
8. Windows node pools — current architecture is Linux-only. Should be explicitly documented as unsupported.
9. Spot / preemptible node eviction — sidecar must handle SIGTERM gracefully and clean up sealed cache.
10. Network policies (Calico/Cilium) — sidecar needs explicit NetworkPolicy rules to reach OpenBao.

Scalability Analysis

Pod Scale Assessment

OpenBao — Biggest Scalability Risk

Missing Mitigations

• Request coalescing — if 50 pods request the same secret simultaneously, OpenBao should be hit once, not 50 times.
• Batch secret fetch — 3 secrets in one API call instead of 3 sequential calls reduces connection overhead 3x.
• Staggered renewal — add jitter to TTL to spread renewal load.

eBPF Overhead at Scale

At 100 pods per node, a memory-access tracepoint on sys_read/sys_write could fire millions of times per second. Even incrementing a counter adds 50-200ns per syscall.

Sidecar Resource Overhead

Tech Stack

Go — Right Choice for Phantom

OpenBao vs Alternatives

eBPF vs Alternatives for Monitoring

MVP Scope — “Secrets That Never Touch etcd”

Phantom Core Components

1. Mutating admission webhook that injects phantom-proxy sidecar into labeled pods
2. Sidecar that fetches secrets from external OpenBao and exposes them via: environment variables, Unix domain socket, and mounted tmpfs file
3. In-memory cache with TTL-based renewal (skip sealed local cache for MVP)
4. Pre-flight connectivity check (Job-based, writes to ConfigMap)
5. Helm chart designed for EKS add-on constraints (no hooks, no lookup)
6. Single-provider launch: GKE Standard (simplest webhook behavior)

What to Cut from v1

Critical Path to First Deployable Version

Comparison to Alternatives

Alt 1: Full Confidential Computing (Just Use TEEs)

Trade-off: Full CC is simpler but more expensive and less available. Phantom works on standard VMs and adds CC as optional enhancement — correct positioning for reaching the broadest market.

Alt 2: Sovereign Cloud (Use EU Providers)

Alt 3: Client-Side Encryption Libraries

Alt 4: VPN to On-Premises HSM

Technically works but adds significant operational complexity (VPN management, on-premises infrastructure, latency). Phantom’s managed OpenBao is operationally simpler. However, for customers with existing on-prem HSMs (banks, defense), this should be a supported deployment mode.

Technical Risks

High-Impact Risks

Dependency Risks

Architecture Improvements

Concrete changes that raise the architecture score to 8.5/10.

┌──────────────────────────────────────┐
│  Node                                      │
│  [Pod A] [Pod B] [Pod C] [Pod D]           │
│     │       │       │       │              │
│     └───────┴───┬───────┘  UDS         │
│                │                           │
│       [Phantom DaemonSet, ~80MB]             │
│       [    Shared Cache         ]             │
│                │ mTLS                      │
└────────────────┴─────────────────────┘
                 │
          [  OpenBao EU  ]

type SecretProvider interface {
    GetSecret(ctx context.Context, path string, identity PodIdentity) (*Secret, error)
    WatchSecret(ctx context.Context, path string) (<-chan SecretEvent, error)
    RevokeLeases(ctx context.Context, identity PodIdentity) error
    HealthCheck(ctx context.Context) error
}

# compatibility-db/charts/bitnami/postgresql/16.4.0.yaml
chart:
  repository: bitnami
  name: postgresql
  versions_tested: ["16.4.0", "16.3.x", "15.x"]

injection:
  status: "compatible"    # compatible | partial | incompatible | untested
  mode: "sidecar"         # sidecar | daemonset | both

testing:
  method: "automated"
  platform: "gke-standard"
  k8s_versions: ["1.29", "1.30", "1.31"]

# Cache key derivation for StatefulSets
cache_key = HKDF-SHA256(
    ikm:  service_account_token,
    salt: cluster_hmac_key,
    info: "phantom-statefulset:" + statefulset_name + ":" + pod_ordinal
)

Score Impact Summary

Verdict

Recommendations

1. Ship Phantom alone. Nothing else in v1. Phantom on GKE Standard is the MVP. Add EKS in v1.1, AKS in v1.2.
2. Add a DaemonSet mode as alternative to per-pod sidecars for customers with 1,000+ pods.
3. Implement request coalescing and batch secret fetching to mitigate OpenBao bottleneck risk.
4. Abstract the OpenBao dependency behind a SecretProvider interface from day one.
5. Invest in a testing strategy proportional to security claims: fuzzing, property-based testing, integration tests on all providers.
6. Be explicit about what’s not protected. Build honest security documentation that CISOs will trust.