Customer data moves through places most teams never modeled. A support agent pulls a ticket, sends it to a model, calls internal APIs, and stores the trace. The ticket may contain an address or account number. The model call may leave your network. The trace may land in a logging system with different access rules. Kubernetes is one part of that path, but it is where you enforce boundaries, rotate credentials, and prove who touched what.
The goal is not to bolt on a security product. It is to decide where data lives, who can reach it, and how long it stays. Those decisions affect security questionnaires, deletion commitments, and your cloud bill.
Before policies, list every hop in each workflow: ingress, agent runtime, retrieval store, tool APIs, model provider, cache, logs, traces, and backups. For each, record the data class, who can read it, retention, encryption, region, and deletion method.
The exercise can surface two or three problems a checklist would miss: prompts copied into a debug bucket, a vector index without a tenant filter, a shared API key used by six services. Let those findings set priorities. The map also supports sales conversations and audits.
Kubernetes gives you controls at several layers. None is sufficient alone. The useful question is which layer still enforces a decision when application code has a bug.
A Kubernetes Secret is a convenient way to inject configuration, not a secure vault. By default, Secret data is stored unencrypted in etcd, and base64 encoding is not encryption. Anyone who can read the object through the API can read the value, as the Secret documentation explains.
Enable encryption at rest with a KMS provider so etcd and its backups are protected. Keep high-value credentials out of Secrets: workload identity can issue short-lived cloud credentials, and an external secret manager can sync only what a workload needs. Scope that controller to one namespace at a time.
Turn off automatic service account token mounting for pods that do not call the Kubernetes API. If a pod needs access, use a dedicated ServiceAccount and a narrow Role; the default ServiceAccount is shared by pods that do not specify one. KMS encryption adds key-management cost and a service dependency; budget for it.
An agent often needs to read customer records, write tickets, and call a payment API. One broad credential is easier to build and harder to defend. Split tools into separate services with their own identities, and let the agent call them through a narrow interface. If a tool reads one table, give it a role for that table.
RBAC is the cluster-side half. A RoleBinding that grants get or list on Secrets is a data-access grant. Avoid cluster-admin, and check what a ServiceAccount can do with kubectl auth can-i --list. The RBAC documentation explains why Secret read access is sensitive.
Human approval is a control, not a UX detail. Tools that move money, delete data, change permissions, or send data outside the company should require approval or a hard limit. Prompt injection makes this concrete: untrusted content can redirect tool use. OWASP’s LLM guidance lists prompt injection as a leading risk; the durable mitigation is architectural. Limit what the agent can do, not just what it is told to do.
Default-deny NetworkPolicy is one of the highest-value controls in a multi-tenant agent platform. Once a pod is selected for a direction, traffic is denied unless a rule allows it; policies are additive. The NetworkPolicy documentation explains the model. Enforcement depends on the CNI plugin; a policy nothing enforces is documentation, not control. Test it with a connection that should fail.
Start by denying ingress and egress in the agent namespace, then allow DNS and specific destinations.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
namespace: agent
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns
namespace: agent
spec:
podSelector: {}
policyTypes:
- Egress
egress:
- to:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: kube-system
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
NetworkPolicy works at Layers 3 and 4, so it cannot express “only this hostname” or “only this path.” For domain-level control, route traffic through a proxy or gateway. That adds latency and cost, but gives you one place to log and block destinations. For high-risk tenants, a separate cluster or VPC is stronger but more expensive.
A service mesh can add mTLS, but it does not stop an authorized agent from sending data to an unauthorized destination. Treat mTLS as one layer, not a data-loss prevention system.
Agent systems are hard to debug without traces. The temptation is to record the full prompt, retrieved documents, tool arguments, and response. That creates a second copy of customer data, often with broader access and longer retention than the primary system. Decide what you need for debugging, what you need for audit, and what should never be recorded.
Redact at the emitter, not in the dashboard. Drop sensitive fields before they leave the pod, or hash them with a key you control. Sample successful traces aggressively, keep full detail for errors, and time-box it. Set retention by data class and restrict who can query the backend; a read-only observability account can still cause a breach if the index holds everything.
This is where cost and security meet. Trace storage grows with traffic, and retaining everything is expensive. Sampling and tiering reduce both risk and spend, at the cost of forensic detail.
Sending a prompt to an external model API is a data transfer. Before you do it, answer four questions in writing: what data leaves, where it is processed, whether it is retained or used for training, and how deletion works. Get contract terms that match your promises; if you cannot, use a self-hosted model or remove the data before the call.
Apply the same discipline to vector databases, caches, and evaluation tools. A retrieval index built from customer documents is customer data. Enforce tenant filters in the data layer, not only in the application query. If one index serves several customers, a missing filter becomes a cross-customer leak. Separate indexes per tenant cost more but shrink the blast radius of a query bug. Shared infrastructure with enforced row filters may suit smaller customers. Document the choice and test the filter.
A deletion request tests your data map. Remove the record from the primary store, retrieval index, caches, logs, traces, and backups within the promised window. Backups are easy to overlook: production deletion can succeed while snapshots retain the record. Define a maximum backup retention and a restore procedure that reapplies deletion records. If you cannot meet the commitment, change it before a customer finds the gap.
Audit logs can record Secret reads if you configure an audit policy and ship logs somewhere durable. That costs money, but it separates investigation from guessing.
Controls nobody owns become shelfware, and cost cuts that remove a needed boundary are worse. Fund the highest-risk path and simplify the rest.
Can we store model API keys in Kubernetes Secrets?
You can, but that is a starting point. Enable encryption at rest, restrict Secret reads, and prefer short-lived credentials or an external manager for high-value keys. Base64 encoding provides no protection.
Do NetworkPolicies stop an agent from sending data to the wrong place?
They filter at Layers 3 and 4, not by hostname. That blocks lateral movement and unexpected destinations. Add an egress proxy for domain rules and scoped tool permissions for application-level exfiltration.
Should we log full prompts and responses?
Only if you have a specific need and the access, retention, and region match your commitments. Full prompt logging creates a second copy of customer data. Redact before storage, sample successes, and keep detail briefly.
Is one namespace per customer enough isolation?
Sometimes. Namespaces plus RBAC, quotas, NetworkPolicies, and per-tenant filters can be a reasonable middle ground. Separate clusters reduce blast radius further and cost more. Choose based on data sensitivity and your tenant checks.
What should we fix first?
The credential path. Long-lived, broad keys are the first target. Then enable Secret encryption, restrict Secret reads, and add default-deny policies. Those changes cover several realistic failures without a large project.
If you want a clearer view of the data paths, access model, and cost profile in your agent workflows, talk to Kubeify. A short review can produce a plan that fits your customers and budget.
Kubeify's team decrease the time it takes to adopt open source technology while enabling consistent application environments across deployments... letting our developers focus on application code while improving speed and quality of our releases.
– Yaron Oren, Founder Maverick.ai (acquired by OutboundWorks)
Let us know what you are working on?
We would help you to build a
fault tolerant, secure and scalable system over kubernetes.