Automated Secret Rotation Is Chaos Engineering for Your Identity Plane

I’ve always seen parallels between reliability and security. At their core, both require us to understand risks and diligently mitigate them. However, advances in reliability engineering have not always translated to resilient security practices. We’ve celebrated Netflix’s Chaos Monkey for randomly terminating servers to prove resilience, yet many organizations still panic at the thought of cycling an API key and fear they’ll take something down.

The problem is that we often treat secrets like permanent configuration. Even though they don’t live in the code, an application will load them once and then act as though they will never change. We focus on storing the secret safely, but we don’t consider whether the application knows what to do when the secret changes.

That unaccounted behavior is a dangerous blind spot, and it reduces confidence in being able to replace the secret without disruption. If the first time you test that process is during an incident, you could end up with downtime or delayed remediation of a leaked credential.

That’s where automated secret rotation comes in. By making credential changes a routine part of the application’s operation, you expose hidden dependencies and flawed assumptions before an emergency. Automated secret rotation forces you to confront the things that can go wrong in secret rotation.

Just as chaos engineering regularly tests system failures, regularly rotating secrets tests whether your applications can survive changes to identities and access.

The case for automating secret rotation

Chaos engineering shifted mindsets from “hoping your application doesn’t fail” to “planning for your application to fail”. Automated secret rotation applies the same principle by “planning for the credential to be compromised”. With that framing, secret rotation is less of a security ritual and more so proving that applications continue working when they receive new credentials and the old ones are revoked.

Automating secret rotation gives you operational confidence, but there is a major security argument to be made too. Credentials end up in internal code, documentation, tickets, and chats without anyone noticing. Those credentials remain a risk as long as they are still valid, waiting for a hacker to find them. Rotating a credential will invalidate its original value. But if you’re manually rotating secrets, those internal leaks will only get rotated if they are discovered. You can invest time and money in uncovering all the places you’ve leaked secrets, but automating rotation will reduce a lot of that risk by design.

Whether you want to be proactive or just prepared for the inevitable, rotating secrets without downtime takes practice and preparation. You need to understand the assumptions you are making about your secrets that could fail and cause an outage.

Failure modes in secret rotation

Shared secrets

One of the most frustrating things to cause an outage during secret rotation is finding out that the credential was shared with another system. Picture this: you’re rotating a secret in your staging environment because it was leaked. You’ve planned diligently and the redeployment with a new secret is successful. You finally go to revoke the leaked secret, and… suddenly prod goes down.

Sometimes secrets are shared between clients or different environments, and it’s difficult to know that before you go to revoke a credential. Because shared secrets are behavioral, it’s a tough ban to enforce.

Stalled rollout

Regardless of how well your team follows best practices, some secret rotation failures have less to do with people and are more subtle. When a scalable service allows the old and new credentials to overlap, a stalled rollout can be surprisingly quiet. New instances may pick up the replacement while others continue using the original, and requests keep succeeding. But healthy traffic doesn’t tell you whether the application can survive revocation. Once you retire the old credential, any instance still depending on it now has a problem.

There are many more things that could go wrong beyond shared secrets and stalled rollouts. Reliable secret rotation requires designing for credential replacement and continuously testing your assumptions. Those assumptions have to hold up not just once, but continuously as your application changes over time.

Secret rotation in practice

You can continuously validate secret rotation manually, but doing so comes with significant operational overhead. Manual rotations will create a backlog of tickets and slow down development. As the number of credentials grows, repeatedly exercising the rotation process becomes increasingly impractical without automation.

Automating secret rotation makes continuous validation scalable without requiring engineers to coordinate every credential change. Modern secret managers can handle the heavy lifting for you, but reliable rotation still takes some coordination between the system issuing the credential, the secret store distributing it, and the applications using it.

Coordinating secret rotation

Before you can reliably automate secret rotation, you need a way to distribute updated credentials to every application that depends on them. Manually generating replacement credentials, copying them into application configurations, and revoking the originals doesn't scale. The more applications and environments you manage, the harder it becomes to ensure every credential is updated without disrupting something.

Centralized secret managers solve this problem by providing a single place to manage credentials and distribute them to applications. When a credential changes, its replacement is stored in the secret manager, where applications can retrieve the updated value through an SDK, API, or environment variable injection. Rather than relying on a human to update credentials across multiple systems, you store them centrally and establish a programmatic way for applications to receive changes.

With credential distribution centralized, the next challenge is automating the rotation itself: generating replacement credentials, making sure applications receive them, and safely revoking the originals. For example, a JavaScript application can retrieve automatically rotating PostgreSQL credentials from Infisical, a secrets management platform.

On the secret manager side, rotated secrets require a backend connection to Postgres that can manage credentials. This connection will actually manage two accounts. Account A starts as the “active” consumable credential, and Account B will take its place when it’s almost time to rotate Account A’s password. This alternation continues forever, with Infisical swapping in the “active” account and rotating the other’s password behind the scenes.

On the code side, the application needs a way to retrieve updated credentials. Rather than redeploy the whole app every time a credential updates, we should pull it directly from Infisical using the Infisical Node SDK while our app is running. Since Infisical generates separate username and password secrets for PostgreSQL, our credential supplier function needs to retrieve both:

let cached;
let refreshAt = 0;

async function rotatedCredentials(forceRefresh = false) {
  if (forceRefresh || Date.now() >= refreshAt) {
    const [user, password] = await Promise.all([
      infisical.secrets().getSecret({
        projectId: PROJECT_ID,
        environment: "prod",
        secretName: "DATABASE_USER",
      }),
      infisical.secrets().getSecret({
        projectId: PROJECT_ID,
        environment: "prod",
        secretName: "DATABASE_PASSWORD",
      }),
    ]);

    cached = {
      user: user.secretValue,
      password: password.secretValue,
    };
    refreshAt = Date.now() + 3_600_000;
  }

  return cached;
}

When we fetch the credentials, we cache them for one hour to reduce repeated calls to Infisical. Although one hour is well within the transition window when both credentials are active, we still need to be able to force a refresh when an emergency rotation happens. We won’t know that a secret was immediately revoked until a connection fails, so we need to handle that in our outer openDatabase function that calls the credential supplier function.

async function openDatabase(credentials) {
  const value = await credentials();

  try {
    return await connect(value);
  } catch (error) {
    if (!["28P01", "28000"].includes(error.code)) throw error;

    // Bypass the cache. A second failure reaches the caller.
    return connect(await credentials(true));
  }
}

const db = await openDatabase(rotatedCredentials);

28P01 is PostgreSQL’s invalid-password error, and 28000 indicates an invalid authorization or a role that no longer exists. Either error indicates that the app may be attempting to use outdated credentials. This triggers a forced refresh of the rotated credential from Infisical, retrying the connection.

On Kubernetes, you can avoid changing application code by using the Infisical Kubernetes Operator and have it restart your app when secrets change to ensure each pod gets the latest version of a secret.

By making changes to the way your code handles database connections, you can move away from “secrets are static configuration” and adopt a resilient architecture that automatically handles secret rotation at any time.

Automating credential rotation helps you reliably replace secrets without disrupting applications, but it doesn’t minimize the window in which those secrets could be abused. Consider a credential that rotates every 30 days. While that schedule can help reduce the risk of internal leaks eventually being discovered, it may not help you during an active breach or public leak. During active compromise, secrets are typically abused in under an hour.

If you want to minimize the chance that your app’s credential can be abused by attackers, you need to use dynamic secrets.

Dynamic secrets

Dynamic secrets are credentials that are generated upon request, renewed as needed, and destroyed when you are done. Not only are dynamic secrets better for security, they also give you more control over the secret lifecycle than standard rotated secrets.

Rather than alternating between two accounts, you get a dedicated ephemeral account provisioned each time you request a dynamic secret. This allows you to have isolated credentials for each client, ensuring no two workers or users ever use the same secret.

To utilize dynamic secrets in our example app, we first need to set up a Dynamic Secret for PostgreSQL in Infisical. Then we need to create a new credential supplier function:

let current;

async function dynamicCredentials(forceRefresh = false) {
  return leaseLock.runExclusive(async () => {
    const expiresAt = current
      ? Date.parse(current.lease.expireAt)
      : 0;

    if (forceRefresh || Date.now() >= expiresAt) {
      current = await issueLease();
    } else if (Date.now() >= expiresAt - 60_000) {
      try {
        const { lease } = await renewLease(current.lease.id);
        current = { ...current, lease }; // Keep the same credentials.
      } catch {
        current = await issueLease(); // Acquire replacement credentials.
      }
    }

    if (Date.now() >= Date.parse(current.lease.expireAt)) {
      throw new Error("Database lease has expired");
    }

    return {
      user: current.data.DB_USERNAME,
      password: current.data.DB_PASSWORD,
    };
  });
}

// Renew proactively, including when the application is idle.
setInterval(() => {
  dynamicCredentials().catch(reportLeaseFailure);
}, 5_000);

const db = await openDatabase(dynamicCredentials);

Our application will “lease” credentials for a limited amount of time. A whole new account gets generated in Postgres each time a new credential lease is issued. Leases have short TTL (time-to-live, typically 1 hour long), at which point it will be automatically revoked.

In our code, we can renew the lease if we reconnect to the database during the TTL window, or we can even proactively keep renewing the credential as long as our app instance is running. If the lease does expire, we simply create a new one and use that. Lastly, we provide a fail-safe way to force credential refresh from the outer calling code.

Rotated credentials force you to think about acquisition and replacement, but dynamic credentials go even further and give you access to the full secret lifecycle within your app. Each client manages its own short-lived identity, and attackers have a tiny window of opportunity to exploit a credential. Dynamic secrets introduce additional lease management overhead, but you gain greater control over secret validity and ultimately uptime.

The messy reality

While dynamic secrets offer good security and control, they are not even the best option for mitigating compromised secrets. Workload identity federation can remove the need to manage long-lived application secrets by letting your application use an identity trusted by another system. However, there is a shared challenge that workload identity, dynamic secrets, and rotated secrets all have: provider support.

Not all systems support multiple credentials at the same time, and few support workload identity federation protocols like OIDC (OpenID Connect). To adopt an architecture designed for automated secret rotation, you need to figure out which approach works with the technology you have.

The table below shows which secret management method to use based on what your target system supports (first match gets preference).

Just like chaos engineering, there's no single solution you can turn on to make secret rotation reliable. Different systems support different approaches, and each comes with its own challenges. For a database, you might use dynamic secrets to give each application instance its own short-lived credentials. For a SaaS platform, you might alternate between two API keys to allow applications to transition without downtime. Where supported, workload identity federation can eliminate the need to manage long-lived credentials altogether. And for systems that only support one active credential, you may still need a carefully coordinated maintenance window.

Ultimately, the method matters less than the outcome. Just as chaos engineering tests whether applications can withstand infrastructure failures, regularly exercising credential replacement tests whether your systems can survive changes to their identities and access. The goal is to know that replacement credentials will reach the applications that need them, that old credentials can be revoked, and that the process won't cause unexpected outages.

Building that confidence takes more than adopting a secret manager or setting a rotation schedule. It requires designing applications around credential changes, testing those assumptions continuously, and ensuring that when a credential is compromised, replacing it is a routine operation rather than an emergency.