External Secrets Operator Explained (ESO) | Kubernetes Secrets Made Simple

Hey everyone. In this video, we're going to discuss Kubernetes secrets and the external secrets operator. Specifically, I'm going to show you how the external secrets operator or ESO for short works in Kubernetes, why we use it, and then we'll wire it up to Infisical so we can sync our secrets into the cluster automatically. Now, before we dive in, it's important to state the problem.

Native Kubernetes secrets and config maps aren't really a secret manager. They're just B64 encoded blobs sitting in etcd. So the moment you want GitOps, multiple clusters or real rotation, things get messy fast. ESO fixes that by letting you store secrets in an external secret store like Infisical as the source of truth and then it automatically materializes them into Kubernetes secrets when your app needs them.

If you've watched our earlier video, we covered the other Kubernetes secrets patterns at a high level. Today we're going deep on ESO. Let's walk through the mental model, then I'll demo the whole flow.

All right, let's first talk about native Kubernetes secrets. As mentioned out of the box, Kubernetes gives us secrets that are B64 encoded and stored in etcd. I can create one with kubectl create secret generic. I can also define it in a YAML file with the kind secret the name ESO demo secret.

But Kubernetes native secrets are really just a storage primitive, not a secrets management system. The moment you have multiple clusters, multiple environments or multiple teams, the problems start.

First, managing and rotating secrets becomes difficult. When your database password needs to rotate, you find yourself manually updating the secret across every cluster. Miss one cluster and your app breaks. Second, the issue of observability and control. Who has access to which secrets? Who changed what and when. Native secrets give you basic RBAC, but no real audit trail or centralized access management.

This is why we use external secrets management solutions like Infisical, AWS Secrets Manager or Azure Key Vault. Centralized management, built-in rotation, proper audit logs, and granular access controls. An external secrets manager is how we connect that external secrets store to Kubernetes automatically.

So, what does ESO actually do? ESO is a Kubernetes operator, software that runs in your cluster and automates tasks by looking for specific resources. It extends Kubernetes with custom resources, new resource types beyond built-in ones like pods and services.

Now, there are three pieces. First, secret store. This defines how you connect to your external secret store, authentication, API endpoint, and which project and environment. Second, external secret. This defines what to fetch, which secrets to pull, and what to name the resulting Kubernetes secret.

Here's how they work together. ESO watches for external secret resources. It uses the secret store config to authenticate to your external secrets management solution. It fetches the values and then materializes those as normal Kubernetes secrets. Your manifests only contain references, never actual secret values. Secrets stay centralized in your external secrets manager and Kubernetes just materializes them when needed.

So to set this up end to end, there are four steps and we'll walk through all of this with a live demo in a moment.

First, you install the external secrets operator in your cluster.

Second, we set up authentication between ESO and our external secrets manager. ESO needs to prove its identity to your external provider whether that's AWS Secrets Manager, Azure Key Vault, HashiCorp Vault or Infisical. ESO supports multiple authentication methods, static credentials, workload identity federation, native Kubernetes auth and more. The best choice depends on your security requirements and environment.

For this demo, we'll use Kubernetes auth with Infisical as our backing secrets provider. This lets Infisical verify that ESO itself is making the request. ESO proves its identity by using its Kubernetes service account and Infisical verifies that directly with the Kubernetes cluster's API.

Third, we're going to set up a secret store. Again, this resource tells ESO how to connect to the provider, the authentication method, endpoints, and which secrets to access. Each provider has a slightly different configuration here.

And fourth, you create an external secret. This is provider agnostic. It references your secret store and specifies which keys to pull and what to name the resulting Kubernetes secret object.

Once those four pieces exist, ESO handles the rest. It authenticates to your provider, pulls the values, creates a Kubernetes secret, and keeps it in sync based off of your refresh interval. So, the workflow is the same across providers. Install ESO, configure authentication, define the connection in a secret store, and map secrets with external secrets.

Now, let's see this in action with a full demo. I've got a simple Flask app running in EKS. It's configured to read five environment variables, some database credentials, and an API key, and then display them on a page. Right now, every variable shows not set. Nothing is providing these values to our pod yet. We need a way to get secrets from our external provider Infisical into Kubernetes automatically and securely. That's exactly what external secrets operator does. Let's set this up end to end.

First, we add the external secrets operator Helm chart repository. Next, we install ESO into its own namespace. And this install CRDs flag registers the custom resource definitions that ESO needs. Things like secret store and external secret. Quick note, if you're running multiple ESOs in the cluster, you only want to set install CRDs to true once.

These are cluster level resources, so they can only exist once. For this demo, we're just installing one operator, so we're good. Now, if we run kubectl get pods in the namespace external secrets, we should see three pods. The operator, a webhook, and a cert controller. Once all three of these are running, ESO is ready.

And you'll also see ESO registered several custom resource definitions. The two we'll use today are Secret Store and external secrets. The others handle advanced workflows. And now that we have ESO installed, the next question is how does ESO authenticate with our external secrets manager? In our case, Infisical to fetch secrets back into the cluster.

Now there are different ways to handle this authentication. You could use universal auth, which uses a client ID and secret, but this means storing long-lived credentials in your cluster. We're going to use something called Kubernetes auth instead, which is more secure because it leverages your cluster's built-in identity verification.

Here's how it works. On the left, we have our Kubernetes cluster and our external secrets operator pod running. On the right, we have Infisical, our external secret store. When ESO needs to authenticate with Infisical, here's what happens. First, ESO sends an authentication request to Infisical along with its service account token. This is a JWT that Kubernetes automatically provides to the pod. Step two, Infisical doesn't just blindly trust this token. Instead, it calls back to the Kubernetes API's token review endpoint and it asks, "Hey, is this token valid? Who does it belong to?"

Step three, the Kubernetes API verifies the token cryptographically and it responds, "Yes, this token is valid. It belongs to the external secrets service account in the external secrets namespace." Step four, now that Infisical has verified identity, it issues a temporary access token back to ESO. This token is short-lived. It expires after a set period. Now lastly, ESO can use this temporary token to fetch back the secrets that it needs.

The beauty of this approach is that there are no static credentials ever stored in the cluster. ESO proves its identity using tokens that Kubernetes manages automatically and Infisical verifies that identity through the Kubernetes API and it's all cryptographic verification. There are no passwords or static credentials sitting in YAML files.

Now that we understand how Kubernetes auth works, let's set it up. For Infisical to verify those service account tokens, it needs permission to call the token review API on our cluster. That's what we're configuring now.

First, a service account. This is a dedicated identity that Infisical will use when it calls back into our cluster. This is not for our app. It exists solely for Infisical to validate tokens.

Then we have a cluster role binding. This grants the service account the built-in system auth delegator role. That role does exactly one thing. It allows calling the token review API. That API takes tokens and answers the question is this valid and who does it belong to? It can't read secrets, create pods or anything else. Minimal permissions.

And then finally a long lived token for that service account. Kubernetes will automatically populate this secret with a JWT. This is the token Infisical will use every time it needs to call back to the cluster to verify a pod's identity. And this creates a non-expiring token. Though in production, you'd want to rotate these periodically. Now, let's apply that.

Now, we can use this command to retrieve the JWT from that token secret. Copy this. We'll paste it back into Infisical when we're creating our machine identity. Again, this is what allows Infisical to call back into our cluster's token review API to verify that incoming requests are coming from legitimate pods.

Now in Infisical my external secrets manager I've created a project named Kubernetes secrets demo. The first thing that I'm going to do is I'm going to go to settings and I'm going to note this project slug as I'm going to need it. This is autogenerated. It might not match your project name exactly and you need this for the secret store manifest.

Now you'll note in the dev environment I've created these four dummy secrets. DB host, password, port, and username. These are the source of truth. We never define these in YAML files or pass them around manually. Now, inside our project, we go to access control and we create a machine identity. We're going to call this ESO demo. And this is what represents our Kubernetes cluster. Think of it as Infisical's way of knowing who's asking for secrets. Now, before we set up the authentication method, we're going to create a custom role scoped to exactly the secrets that the ESO needs. We'll go here to access control roles.

Now I've created this ESO read only role that gives read access in the dev environment. We don't need write delete access to other environments. Least privilege. We go back and assign this role to our machine identity and it's locked down. And as mentioned earlier we will not be using universal auth to store static credentials in our cluster. So I'm going to remove that and rather we'll be using Kubernetes auth on this identity. This is what lets Infisical verify that requests are actually coming from our cluster.

For the Kubernetes host, we need our cluster's API server URL and we can grab that with kubectl cluster info. Copy the control plane address and plug it in. Next, for the token reviewer JWT, we paste in the token we received back in step three.

Now, you'll note I didn't show you my token because that's a sensitive value. And this is what gives Infisical permission to call back into our cluster's token review API and verify that incoming tokens are legitimate. So, we've put that in. For allowed namespaces, we're going to put external secrets. And then the same thing for allowed service account names, external secrets. These two fields lock it down so that only the external secrets pod in the external secrets namespace can authenticate using this machine identity. Any other pod in any other cluster trying to use it gets rejected.

Next, we're going to go to advanced the CA certificate. This is the certificate authority that signed your cluster's TLS certificates. Infisical needs this to verify that it's actually talking to your cluster's API server when it calls the token review endpoint, not some impostor. We extract it from the cluster using this command and we copy it and paste it in.

Now we save that and trust is established. Infisical now knows how to reach our cluster, has permission to verify tokens and knows exactly which service account to trust.

After saving, we're going to copy the identity ID from this machine identity page. This is how ESO will identify itself when authenticating with Infisical. We need to store the machine identity ID somewhere that ESO can reference it. We put it in a Kubernetes secret in the default namespace. This is not a sensitive credential. It's just a reference ID. You can safely keep this file in source control. And this identity ID is the one that we copied from the machine identity in Infisical. And we'll apply that. And we could verify.

We see that the secret was created. Now our secret store YAML. The secret store is the ESO custom resource that defines the connection to Infisical. It tells ESO how to authenticate and what scope to use. We specify Infisical as the provider and the API host. If you were self-hosting Infisical, you'd point this to your own instance. Or if you were in the EU, you'd point it to this instance.

And for authentication, we're using Kubernetes auth, not a client ID and secret like we would with universal auth. And this references the identity ID that we just stored. And we also have the scope, which project environment and path within that environment. Make sure the project slug here matches exactly the one that you had in Infisical earlier. It's autogenerated and it might not match your project name exactly. Now this store can only access secrets in this project in this environment. So go ahead and apply that.

We'll check the status as well. The status field should show valid and it should show ready true. This means that ESO successfully connected to Infisical. Now onto our external secret. The external secret is the bridge. The secret store defines how to connect. The external secret defines what to fetch and where to put it. So we have a refresh interval. We have this set so that ESO resyncs from Infisical every 60 seconds. If someone rotates a password in Infisical, the Kubernetes secret updates within the minute. Now this is on the shorter side, which is great for a demo or for frequent rotations. In production, you choose this based on how quickly you need secrets updated versus how much polling load you want. Shorter intervals mean faster propagation with more load. Longer intervals are going to have less total load but slower propagation.

And we reference our secret store. That's how ESO knows which provider and auth method to use. And then the target ESO will automatically create a Kubernetes secret called ESO demo secret. Creation policy owner means that ESO owns the life cycle. It creates the secret. It updates it when values change in Infisical and it cleans it up if you delete external secret. We never create or edit this secret by hand.

And then we have the data mappings. Each entry maps a key in Infisical the remote ref to a key in the Kubernetes secret. We're pulling our four database secrets for now. And notice this file has key names but no secret values. Nothing sensitive ever touches git. So we'll apply this and we can verify the sync.

Note the status column shows secret synced true. ESO connected to Infisical through Kubernetes auth, fetched our secrets and created the Kubernetes secret. The Kubernetes secret exists now, but our pod was already running without it. Since we're using environment variables, the pod needs a restart to pick up the new values. Okay, so we've rolled out and restarted.

We'll go ahead and refresh. And there it is. Four of our five secrets are now showing values pulled directly from Infisical through ESO. All of this has synced. Now API key is still not set because we haven't told ESO to fetch it yet. So let's see the sync in action. I'm going to add an API key secret in Infisical and we put it in the development environment. Now we'll add the new key mapping and then we'll reapply.

ESO immediately picks up this change to our external secret resource and fetches the new secret from Infisical. Okay, we've rolled out and restarted once again and there it is. API key is synced live from Infisical. Infisical is the single source of truth. Kubernetes auth means that our cluster has to prove its identity cryptographically and ESO keeps everything in sync automatically.

Now, we had to restart the deployment because environment variables only load at startup. There's also a tool called Reloader that can automate that for you if you need it. It watches for secret changes and restarts deployments automatically. I'll link that in the description as well.

A few quick production tips worth noting. Use a cluster secret store if multiple namespaces need the same provider. This saves duplicating config just like we did. Scope machine identities with least privilege in mind. One per app or per team, not one for everything. Monitor external secret status and alert on sync failures before your app notices. And remember, there are patterns other than ESO like provider specific Kubernetes operators or CSI providers that can update secrets without pod restarts. Pick the pattern that fits your rotation and availability needs.

So that is ESO deep dive with Infisical end to end. Infisical is your source of truth. ESO syncs secrets into Kubernetes automatically. Your app reads normal Kubernetes secrets and Kubernetes auth proves identity cryptographically. And we covered the core pattern today. In future videos, we'll dive into cluster secret stores for multi-namespace setups, secret rotation strategies, and how to handle this across multiple clusters. If you found this helpful, hit subscribe. We're covering the full Kubernetes secret landscape. Thanks for watching and we'll see you in the next one.