Stolen Platform Secrets: A Rotation and Detection Runbook

Why This Matters Now

When Northflank disclosed its incident on October 30, the message to customers was clear: assume all environment variables have been decrypted, and rotate everything. Wiz's alert reached Northflank at 11:43 UTC, and engineers reviewed it a minute later. The vendor acted quickly. Your exposure window, though, depends on how fast you can respond to such notices.

This pattern is more significant than any single vendor. Wiz's report, "Black October for AI Developer Platforms," identified at least nine incidents in one month, including Northflank, Railway, OpenCode, Warp, and Lightning AI. Attackers target environment variables because that's where AI API keys reside. Stolen keys lead to LLMJacking: using another organization's credentials to run models, generating charges or reselling access. In two cases Wiz investigated, compromised enterprise credentials resulted in roughly 2.5 million Amazon Bedrock API calls and about $100,000 in usage.

This runbook provides your incident response team with a repeatable procedure for the next "assume it's all stolen" notice.

Preparation Before an Incident

Get these in place before an incident, not during one.

• Secrets Inventory: Map every platform, repo, CI system, and workload that holds a credential to an owner. If you can't list where a key lives, you can't rotate it.
• Provider Console Access: Ensure access for each AI service (OpenAI, Anthropic, AWS Bedrock, etc.) with permission to revoke keys and read usage and billing data.
• Log Access: Use cloud audit logs (like AWS CloudTrail), AI provider usage exports, and any audit logs your hosting platform offers. Northflank enabled audit logs for all customers during its response; don't rely on a vendor to do that for you.
• Billing Alerts and Hard Quotas: Configure these per key or project.
• Named Incident Commander: Decide whether your Breach Coach and insurer need First Notice of Loss. Check if your policy requires Insurer Consent before engaging outside forensic vendors.

Step-by-Step Implementation

Step 1: Scope the Exposure (First 30 Minutes)

List every secret stored on the affected platform: workload environment variables, secret groups, global secrets, and database credentials. Treat this list as the rotation backlog. Don't wait to learn if your specific secrets were taken. Assume they were, as advised.

Step 2: Rotate in Priority Order

Order by blast radius and cost-to-abuse:

1. AI Provider API Keys: These carry direct financial exposure and are the confirmed target.
2. Cloud Credentials: Prefer replacing long-lived keys with short-lived role-based access.
3. Database Credentials: Use platform-native rotation where available.
4. Tokens and Webhooks: Include CI tokens, signing secrets, OAuth client secrets.

For each key: create the new one, deploy it, confirm the workload works, then revoke the old one. Never revoke first on production services unless you're actively seeing abuse. In that case, accept the outage and revoke.

AWS example for IAM user keys:

aws iam create-access-key --user-name svc-app
aws iam update-access-key --user-name svc-app \
  --access-key-id AKIAOLDKEYID --status Inactive
# after validation:
aws iam delete-access-key --user-name svc-app \
  --access-key-id AKIAOLDKEYID

Deactivating before deleting allows a quick rollback.

Step 3: Close Side Doors

Disable public access on managed add-ons like databases and caches. Restrict AI keys by IP range, project, or model allow-list where supported. Set per-key spend limits so a leaked key hits a ceiling.

Step 4: Hunt for Abuse

Pull usage and billing logs from each AI provider and look for:

• Calls from unfamiliar IPs, regions, or user agents
• Models you never use, especially the most expensive ones
• Usage spikes outside your normal hours
• Requests with no matching application trace

On AWS, query CloudTrail for Bedrock activity:

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventSource,AttributeValue=bedrock.amazonaws.com \
  --start-time 2025-10-30T00:00:00Z

Adjust the time window to start before the vendor's disclosure, since compromise may predate it. Report anything suspicious to both the AI provider and your hosting vendor, as Northflank requested.

Validation: How to Verify It Works

Rotation is only complete when you can prove the old credentials are dead.

• Test the Old Keys: Attempt an authenticated call with each revoked key and confirm a rejection. Log the result as evidence.
• Confirm No Stragglers: Search code repositories, CI variables, and config management for old key prefixes or IDs.
• Check for Dependent Breakage: Review application error rates for authentication failures that signal a service still using a retired secret.
• Verify Alerting: Generate a controlled test call against a low-limit key and confirm your spend or anomaly alert fires.
• Document the Timeline: Record notice received, scoping complete, each rotation, and abuse review conclusions. This record supports any claim under Cyber Extortion Coverage or other applicable coverage, and it answers regulator and customer questions later.

Maintenance and Ongoing Tasks

• Rotate on a Schedule: Not only after incidents. Quarterly is a workable start for static keys; automate it where possible.
• Shrink What Lives in Environment Variables: Move high-value secrets to a dedicated secrets manager with access logging and short-lived retrieval.
• Scope Keys Narrowly: One key per application and environment, so a single leak doesn't expose everything and abuse is attributable.
• Subscribe to Threat Intelligence: Covering your hosting and developer platforms. Wiz's alert is what started this response chain.
• Run a Tabletop: Use this runbook against a "vendor says assume all secrets stolen" scenario. Time how long full rotation takes. That number is your real exposure window.
• Review Vendor Risk: Add developer and hosting platforms to your Contingent Business Interruption and third-party risk assessments, and confirm your Underwriting Questionnaire answers on secrets management still match what you actually do. An inaccurate answer can create Application Fraud Warranty problems at claim time.

The next notice will arrive without warning. A current inventory, practiced rotation, and spend limits on every AI key turn it from a crisis into a checklist.