Kubernetes Secrets Management: Architecture, Tools, and Best Practices

Good secrets management is never easy, but Kubernetes makes it even more difficult. Any secrets management strategy requires answering a few questions:

• Where do credentials live?
• Where do credentials need to end up?
• How do secrets get delivered there?

This serves two main goals: First, to keep secrets out of plaintext so they can’t leak. And second, to centralize credential storage and ensure the right secrets end up in the right places. The latter increases engineering velocity because nobody needs to manually handle secrets.

This is simple when infrastructure is permanent. Let’s say a VM needs a sensitive database string. The string lives in a secrets manager (please don’t use .env) the workload authenticates to with a machine identity. Its configuration stores the secret’s location and pulls the secret into its environment at runtime. The VM pulls the secret and can read and/or write the database. Life is simple. Bliss.

Now the party pooper: Kubernetes. Its workloads are constantly spawned, killed, and replaced, and deployments are opinionated about how to pull secrets.

Same scenario: a microservice needs to authenticate to a database, and that sensitive database string lives in a secrets manager. Let’s see how complicated Kubernetes can make it:

• Workloads typically consume secrets via native Secret resources, which are base64-encoded rather than encrypted, so they are plaintext for all practical purposes. That means the secret exists in two places: the Secret resource and a pod’s env vars or volume mount.
• A cluster needs every secret the full app needs, but no single pod needs all of them. So you configure every secret in the cluster while keeping each pod scoped to only the ones it uses.
• Pods frequently need credentials because they’re constantly killed and restarted, which means you need to avoid hitting rate limits.
• Secret delivery requires ensuring the app incorporates updated secrets and that plaintext secrets don’t end up in crash dump or config files.

And that’s just the machine part: Any developer who can create a pod in a namespace can read every Secret in that namespace, so the new hire might suddenly see prod database credentials.

Managing secrets in Kubernetes isn’t easy, but becomes much easier once you understand its fundamentals.

What is a Kubernetes Secret?

A Kubernetes Secret is an API object that holds sensitive data, which the API server stores in etcd. The kubelet fetches it and projects it into the container, either as a file in a mounted volume or as an environment variable set at container start. For a closer look at the object itself, see what Kubernetes Secrets are and what they do not protect.

You can create one from literal values or from a file:

kubectl create secret generic db-creds \
  --from-literal=username=app \
  --from-literal=password=hunter2 \
  --namespace=payments

kubectl create secret generic tls-cert \
  --from-file=cert=/path/to/cert.pem

Or as a manifest, for GitOps workflows:

apiVersion: v1
kind: Secret
metadata:
  name: db-creds
  namespace: payments
type: Opaque
stringData:
  username: app
  password: hunter2

Most secrets management techniques eventually create Secret objects because the kubelet can work with it natively. Kubernetes secrets have a few peculiarities that can increase the complexity:

• Kubernetes secrets aren’t encrypted by default. Secret objects sit unencrypted in etcd by default, and the Base64 encoding of the data field is not encryption, so anyone holding the manifest holds the secret.
• Pod creation is effectively secret access. RBAC governs the Secret API, but anyone who can create a pod in a namespace can mount any secret in it and read the value, and anyone who can list secrets can print them directly.
• Rotated secrets only reach running pods via volume mounts, which the kubelet updates in place. Secrets read from environment variables resolve at container start and the process holds the value until it dies. One exception catches people: a volume mounted with subPath is never updated, so it behaves like an environment variable no matter how the secret rotates.

This leads us to a few rules we should follow. Whatever tooling or architecture you eventually choose, these will be good guides to building secure, efficient secrets management.

Kubernetes secrets management best practices

Some of the principles of Kubernetes secrets management overlap with managing secrets anywhere while others are more specific:

• Encrypt at rest. Kubernetes stores Secrets base64-encoded in etcd, which isn't encryption. To ensure plaintext secrets can’t leak, configure an EncryptionConfiguration.
• Scope access tightly. Grant get, list, and watch only to accounts that need them, per namespace and ideally per resource name. Ensure you control for overrides.
• Keep plaintext out of Git (even if you use GitOps). Values in manifests end up in version control, CI logs, and Helm history.
• Use one source of truth. Secret copies scattered across manifests, CI variables, .env files, and Slack messages are insecure and messy. Centralize secret storage in an external store, then pull from there.
• Automate rotation or use dynamic secrets. Long-standing credentials can be dangerous.
• Give every workload its own identity. Shared credentials mean a compromise anywhere is a compromise everywhere, with no way to attribute the leak.
• Log Secret access. Turn on API server audit logging to know where things go wrong and be audit-ready anytime.
• Scan for secrets everywhere, not just commits. Secrets end up in commit history, container images, crash dumps, error logs, etc.
• Know your revocation path. If you can't say how fast you can invalidate a credential and what breaks when you do, that's the first gap to close.

There are different methods to accomplish all of this, but which one to choose depends on your situation.

How to choose Kubernetes secrets management architecture

Kubernetes secrets management may be more complex than managing credentials for persistent infrastructure, but remains about answering a few basic questions.

• Where do credentials live?
• Where do credentials need to end up?
• How do secrets get delivered there?

The first question is the most consequential because it determines what workflows are available for the latter two.

Where to store Kubernetes secrets

There are three methods to store secrets Kubernetes needs which offer varying levels of security and convenience.

The first is treating the cluster as the source of truth and keep the value wherever somebody last typed it.

Native plaintext secrets

Pro:

• Simple to manage
• Easiest way to test and get started

Con:

• Keeps secrets in plaintext so anyone who sees the manifest has all your secrets
• No way to collaborate and assign differing levels of access, every collaborator has every secret
• No way to manage secrets centrally

This is fine for local experimentation, but indefensible as soon as you’re working with a growing team or want to ship a secure app to customers.

Encrypted secrets

The second option is to store secrets in Git as ciphertext. You encrypt the value, commit it, and decrypt it at runtime. Two tools dominate this approach and they differ in where the decryption key lives.

Sealed Secrets runs a controller inside the cluster that holds a private key. You encrypt with kubeseal against the cluster's public key, commit the resulting SealedSecret, and the controller decrypts it into a normal Secret. Because the key is cluster-bound, a SealedSecret encrypted for staging cannot be decrypted by production, and by default it is scoped to the namespace and name you encrypted it for. Moving a secret between clusters means re-encrypting it. If you are already on Sealed Secrets and outgrowing it, we have a step-by-step migration guide.

SOPS encrypts the file itself against an external key, usually AWS KMS, GCP KMS, age, or PGP. Nothing is cluster-bound, so the same encrypted file works across clusters, but decryption has to happen somewhere: in CI before apply, or through an operator or Helm plugin at deploy time. That is a KMS dependency and a key-access policy you now own.

Pro:

• Keeps secrets out of plaintext
• Doesn’t add another tool to your stack

Con:

• Requires manual decryption and encryption processes
• Requires additional key management processes and a KMS dependency
• Doesn’t centralize secrets management in one place
• Rotations are a pain because they need to be updated manually

This method is fine for a while, but is manual and doesn’t scale well. Many teams report that encrypting and committing secrets turns the DevOps/platform team into a bottleneck because they need to do the de- and encryption for every PR and manually update rotated secrets.

External secret store

The final option is an external secrets manager like Infisical. A secrets manager is a separate tool hosted by yourself or in the cloud that stores your secrets and lets workloads pull them into environments.

Pro:

• Works beyond Kubernetes (centralizes all secrets management in one place)
• Automates rotations and other workflows
• Lets anyone do their work without exposing plaintext secrets
• Native Kubernetes operators automate delivery
• Avoids downstream manual work to keep DevOps from being a bottleneck

Con:

• Adds another tool to your stack

Mature engineering organizations typically choose an external secrets manager to centralize secrets management in one place (beyond Kubernetes) and automate what would otherwise be manual workflows. Startups often choose this option to avoid costly migrations later.

An external secrets manager is also the only solution that scales. Larger clusters often need hundreds of secrets, and nobody wants to manually keep track of those. Coactive AI runs this setup across 42 projects, and HeyGen uses the operator to auto-redeploy workloads on EKS when a secret changes.

Each of these “work” in the sense that correct secrets reach pods, but not all of them offer the same developer ergonomics.

How Kubernetes Secrets reach pods

Pods can receive secrets in two ways: via native Secret objects the kubelet projects or mounted at runtime by a Container Storage Interface (CSI) provider or an injected agent, which creates no Secret object and sends nothing to etcd.

Native Secret object

The native Secret object is simple because the kubelet can read secrets directly from the Kubernetes API and projects secrets into pods. You only need to figure out how to create the correct Secret. Any method that relies on copy-pasting secrets or ciphertext has this built in.

An external secrets manager delivers into this path through an operator that watches the manager and reconciles values into Secret objects, which makes the manager the source of truth while pods keep consuming ordinary Kubernetes secrets.

External Secrets Operator (ESO) is the generic option. It supports a long list of backends through a provider interface, which is the right call if you are pulling from several managers at once. The tradeoff is setup: a SecretStore or ClusterSecretStore per backend, an ExternalSecret per secret, and provider-specific auth wiring for each.

The Infisical Kubernetes Operator is purpose-built for one backend, so the setup collapses to a single InfisicalSecret resource. It also handles the part generic operators leave to you: it can automatically restart the deployments consuming a secret when that secret changes, so a rotation reaches running pods without anyone triggering a redeploy.

Pro:

• Once the Secret is in the cluster, pods run with common environment variables
• Secrets are pulled at runtime and secret storage isn’t duplicated

Con:

• Requires one-time setup
• Rotations don’t automatically propagate into running pods
• Secrets still end up in etcd

The other way to get a secret into a pod is to create a runtime mount, which sidesteps the Secret object completely and creates nothing in etcd.

Runtime mount (CSI injection)

The Container Storage Interface (CSI) is how third-party storage plugs into pod volumes. This enables a tool to write to pods directly instead of having the kubelet project secrets, the Kubernetes API never sees the secret value. The secret is written directly into a mounted volume.

It’s a bit more complex because it requires installing a driver, provider, and defining a separate class. If you’re willing to deal with a bit of extra complexity, this enables you to pull the right secrets into Kubernetes pods without them ending up in etcd.

Pro:

• Nothing ends up in Secret objects or etcd
• Rotated secrets get synced into running pods

Con:

• Added engineering complexity
• More difficult to debug because there’s no secret object to inspect

This creates a matrix of outcomes:

The most versatile option that creates the least downstream work is to use a centralized third-party secrets manager with an operator that reconciles secrets into native Secret objects.

How to sync secrets from Infisical into Kubernetes with the Infisical Kubernetes Operator

Infisical is an open-source secrets manager that can connect to Kubernetes. It offers multiple ways to connect to Kubernetes, including the Infisical Agent Injector and CSI provider, which can directly mount secrets as pod files.

Let’s say you want to automate Kubernetes secrets management with Infisical. The Operator is the most common way to get started and the best way to demonstrate the principle. All you need is:

• Your secrets in Infisical
• A machine identity for the operator to authenticate to Infisical, typically using Kubernetes Auth, which authenticates the operator through a Kubernetes service account.

The basic setup is that Infisical is the source of truth for secrets and three CRDs aid in turning those into native Secret objects. Your existing workloads consume it as if you hand-wrote the object. This example will be relatively simple, but Infisical enables different authentication flows as well.

Establishing connection with the Infisical Operator

After installing the Helm charts and verifying the operator is installed, the core secrets management workflow consists of defining three CRDs from the operator's v1beta1 API:

1. InfisicalAuth, which authenticates to Infisical.
2. InfisicalConnection, which specifies the instance address and any optional TLS configuration you want to add.
3. InfisicalStaticSecret, which defines which secret to pull from where.

InfisicalAuth is a shared cache that holds the token for 70% of its TTL across every InfisicalStaticSecret, which defines each secret itself. InfisicalConnection is a reusable definition.

As an example, here’s how you might configure the operator’s connection:

apiVersion: secrets.infisical.com/v1beta1
kind: InfisicalConnection
metadata:
  name: infisical-connection
  namespace: payments
spec:
  address: https://app.infisical.com

---

apiVersion: secrets.infisical.com/v1beta1
kind: InfisicalAuth
metadata:
  name: infisical-auth
  namespace: payments
spec:
  infisicalConnectionRef:
    name: infisical-connection
    namespace: payments
  method: kubernetes
  kubernetes:
    identityIdRef:
      name: kubernetes-credentials
      namespace: payments
      key: identityId
    serviceAccountRef:
      name: infisical-service-account
      namespace: payments

This lets the operator authenticate to Infisical and know where to pull secrets from.

Pulling secrets with the Infisical Kubernetes Operator

InfisicalStaticSecret names the sources to read and the targets to write:

apiVersion: secrets.infisical.com/v1beta1
kind: InfisicalStaticSecret
metadata:
  name: payments-db-creds
  namespace: payments
spec:
  infisicalAuthRef:
    name: infisical-auth
    namespace: payments
  syncOptions:
    refreshInterval: 1m
    instantUpdates: false
  sources:
    - projectSlug: your-project-slug
      environmentSlug: prod
      secretPath: "/payments"
  targets:
    - name: db-creds
      namespace: payments
      kind: Secret
      creationPolicy: Owner

A few important things to know:

• Each source sets exactly one of projectSlug or projectId, and setting both or neither fails to reconcile.
• refreshInterval is required, takes at least five seconds, and is your polling floor.
• creationPolicy: Owner ties the managed Secret's lifecycle to the CRD so deleting the CRD deletes the Secret, which helps with hygiene and other tools like Argo CD's pruning.
• Set targets[].name and the keys in Infisical to match what your pod specs already reference. That is what makes this a drop-in replacement for hand-created Secret objects.

This is enough for the Operator to pull secrets from Infisical as configured, reconcile them into native Secret objects and let the kubelet do its work.

Verifying the sync works

kubectl apply succeeding tells you that the manifest parse, which is a great first step, but only the status column tells you the secret arrived:

kubectl get infisicalstaticsecret payments-db-creds -n payments
kubectl get secret db-creds -n payments

You want True in the SYNCED column. If it is not there after a few seconds, the reason is in the resource's status conditions:

kubectl describe infisicalstaticsecret payments-db-creds -n payments

Most failures stem from one of four things:

• The service account name or namespace does not match what Kubernetes Auth allows,
• The Kubernetes host URL is not reachable from Infisical
• The environment slug is not what you assumed
• The identity has no access to the project.

If it all checks out, you can specify the environment variable in the pods.

Consume the secret in pods

The secret pulled from Infisical is now a normal Kubernetes Secret, which means you can either consume it as a volume mount or project it as environment variables.

Environment variables need the operator to restart the workload, because a running process never re-reads its environment. You can, however, trigger a restart every time a secret changes in Infisical:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-api
  namespace: payments
  annotations:
    secrets.infisical.com/auto-reload: "true"
spec:
  template:
    spec:
      containers:
        - name: api
          image: your-registry/payments-api:1.4.2
          envFrom:
            - secretRef:
                name: db-creds

This automates Kubernetes secrets management and requires zero copy-pasting. Secrets end up in the right place automatically and live in the same secret store as any other application or developer secret.

Frequently asked questions

Do I still need Kubernetes Secrets if I use an external secrets manager?

Usually yes. Most external managers deliver through an operator that creates a normal Secret object, because that is what the kubelet knows how to project into a pod. The manager becomes the source of truth, but the Secret still exists and still lands in etcd. The exception is a runtime CSI mount, which writes into the pod's volume directly and creates no Secret object at all.

Are Kubernetes Secrets encrypted?

Not by default. The data field is base64-encoded, which is an encoding, not encryption, and anyone holding the manifest holds the secret. Encryption at rest in etcd requires configuring an EncryptionConfiguration on the API server, and even then the value is decrypted whenever it is served through the API.

What happens to a running pod when a secret rotates?

It depends on how the secret reached the pod. Environment variables resolve once at container start and the process holds that value until it dies, so a rotation never reaches it without a restart. Volume mounts are updated in place by the kubelet, so a rotated value arrives, but only takes effect if the application re-reads the file rather than caching it at startup.

Can one Secret be shared across namespaces?

No. A Secret is namespaced and a pod can only reference one in its own namespace. Copying the same value into several namespaces is what most teams do first, and it is exactly the duplication an external manager removes, since the operator can reconcile one upstream value into as many namespaces as need it.

Start using Infisical for Kubernetes secrets

Infisical is an open-source secrets management platform built to be the external manager in this architecture, with a Kubernetes Operator that syncs into native Secret objects, a CSI provider and agent injector for runtime mounts, and workload identity so the cluster never holds a long-lived credential.

Infisical also supports almost any other type of infrastructure, including self-hosted deployments and multi-cloud. It also offers certificate management and privileged access management within the same platform and secures AI agents’ credentials with a proxy architecture that ensures LLMs never read plaintext secrets.

You can get started for free or talk to an expert about what the migration looks like on your clusters.