What to do when cloud credentials leak

Credentials get committed to public repositories, pasted into tickets, left in container images and logged by accident. It is common enough that automated scanners continuously watch public sources specifically to find them, and the time between exposure and first use is frequently measured in minutes.

If you discover exposed credentials, the order of what you do next matters more than the speed of any individual step.

Do this first, in this order

1. Revoke, do not just rotate. Disable or delete the exposed credential immediately. Creating a replacement first feels safer and wastes the minutes that matter. Revoking may break something; that is the correct trade against an attacker holding valid credentials.

2. Assume it was used. Do not wait to confirm compromise before responding. The default assumption after exposure is that the credential was used, and the investigation proceeds from there.

3. Check for new identities. The most common first action after a compromise is establishing persistence: a new user, a new access key on an existing user, a new role with a trust policy allowing an external account, an added SSH key, a new OAuth application. Attackers do this immediately because they expect the original credential to be revoked.

Find and remove anything created that you cannot account for. If you skip this step, revoking the original key achieves nothing, because access continues through the identity it created.

4. Look for expensive compute. Cryptomining is the most common monetisation. Check every region, not just the ones you use, because unused regions are exactly where this is placed. Look for instances, container tasks, batch jobs and spot fleets.

5. Check what data was reached. Review access logs for reads of storage buckets, database snapshots created or shared with external accounts, and anything exported. Snapshot sharing is a quiet exfiltration route that does not look like a data transfer.

6. Review persistence beyond identity. Modified security groups, altered network routing, new scheduled tasks, changed logging configuration. Attackers frequently disable or reduce logging early, so a gap in your audit trail is itself a finding.

7. Preserve the evidence. Export the relevant logs before retention windows close, and before anyone starts changing things to clean up. If this becomes a regulatory or legal matter, the logs are the record.

Then work out how it happened

Fixing the exposure without fixing the cause guarantees repetition.

Common paths, roughly in order of frequency: committed to a repository, including in history after being removed from the current version; embedded in a container image layer; hardcoded in client-side code; pasted into a ticket, chat message or document; written to logs by a debug statement; left in a snapshot or backup that was shared.

The repository case deserves emphasis. Removing a secret in a later commit does not remove it from history, and public repository history is indexed. Rewriting history is required, and the credential must be revoked regardless because it was public in the interim.

Preventing the next one

Eliminate long-lived credentials. This is the structural fix and it removes most of the problem class. Workloads on cloud infrastructure should assume roles and receive short-lived credentials automatically. External systems should use workload identity federation. There is then nothing to leak.

Scan before commit and in CI. Pre-commit hooks catch secrets on the developer’s machine; repository scanning catches what gets through. Both are quick to add.

Enable provider-side exposure detection. Major providers monitor public sources for their own credential formats and will notify you, sometimes applying a restrictive policy automatically. Make sure those notifications reach someone.

Use a secrets manager rather than environment files distributed by hand, so that rotation is a routine operation instead of a project.

Scope credentials narrowly so that an exposure is bounded. A key that can only read one bucket is a much smaller event than one with administrative rights.

Set budget anomaly alerts. For many organisations the first sign of compromise is the bill, and an alert on a sharp daily increase shortens discovery from weeks to hours.

The mistake to avoid

The instinct on discovering exposed credentials is to quietly delete the commit and hope. It is understandable and it is the worst option available.

The credential was public. Assume it was collected. Revoke it, investigate properly, and treat the embarrassment as much cheaper than the alternative. Organisations that handle this badly almost always did so by underestimating the first hour.

If you are weighing providers rather than techniques, our cloud account catalogue lists what each one is strongest at, alongside the configurations available.

For the vendor’s own reference on the services involved here, see the AWS documentation.

Putting any of this into practice needs an account to work in. We stock AWS accounts in sixteen configurations, and the cloud account catalogue sets them beside nine other providers.

Questions people ask