A CI/CD Compromise Runbook You Can Copy Today

Purpose of This Template

The 'wfx-loot' campaign exposed vulnerabilities in CI/CD pipeline security, using stolen credentials from 4,837 GitHub accounts to access 61,764 repositories. Attackers opened pull requests titled "docs: dependency audit notes," and 92% of these remained open, unnoticed.

This runbook is designed to help your team detect and contain a pipeline compromise, decide what to rotate, and communicate with your insurer. It's tailored for Business Continuity Managers who need a document that an engineer can execute at 2 a.m. without prior meetings.

The runbook includes a pre-incident hardening checklist, a response script for suspected compromise, and an evidence log for your Breach Coach and insurance carrier.

Prerequisites

Before using this runbook, ensure you have:

• A named owner for each repository's workflow files.
• Organization-level admin access to GitHub and read access to audit logs.
• An inventory of cloud roles, OIDC trust relationships, and secrets accessible by each workflow.
• Your insurer's First Notice of Loss contact and Insurer Consent Requirement language in your policy.
• A pre-agreed decision-maker authorized to freeze deployments.

If you lack the inventory, create it first. Rotation without it is guesswork.

The Template

Copy the following into your runbook system and replace bracketed fields.

SECTION A: Standing Controls (Review Quarterly)

1. Default GITHUB_TOKEN permission is read-only at the organization level. Workflows must request write scopes explicitly.
2. Branch protection on [default branches] requires at least [1] reviewer and blocks direct pushes.
3. CODEOWNERS file covers .github/workflows/ and package.json install hooks, requiring review from [security-owner group].
4. Pull requests from first-time contributors can't trigger workflows with secret access without manual approval.
5. Cloud OIDC trust policies are scoped to specific repositories, branches, and environments. No wildcard subjects.
6. Replace long-lived cloud keys with short-lived OIDC sessions where feasible. List remaining static secrets in [secrets register] with an owner and rotation date.
7. Pin third-party actions to full commit SHAs, not tags.
8. Enable audit log streaming to [SIEM/log store]. Alert on: new workflow files, workflow edits by non-owners, new branches matching [wfx-docs], or branches created and deleted within [60 minutes].
9. Review stale pull requests older than [14 days] from unfamiliar accounts weekly.

SECTION B: Response Script (Suspected Compromise)

Step 1, Declare (Owner: [incident lead]). Record the time and the trigger. Open the evidence log (Section C).

Step 2, Contain (Target: [30 minutes]).

• Disable GitHub Actions for affected repositories, or org-wide if scope is unclear.
• Pause automated publishing to package registries. In this campaign, automated publishing carried the payload into downstream packages, turning your pipeline into a distribution channel.
• Suspend or reset credentials for any account that opened or merged suspicious changes.

Step 3, Hunt. Search for:

• A branch named wfx-docs.
• A pull request titled "docs: dependency audit notes."
• Gaps in workflow run numbers (deleted runs leave holes).
• Unexpected edits to existing workflows, new workflow files on short-lived branches, or new npm install hooks.

Step 4, Decide on Rotation. If payload execution is confirmed for a workflow, treat every secret and cloud role reachable from that workflow as compromised. Rotate all of them. Include the GitHub token, OIDC-derived cloud sessions, registry tokens, SSH and signing keys, and AI service keys. Review cloud identity policies for tampering, as persistence can be created by modifying trust policies or minting long-lived IAM credentials.

Step 5, Check Downstream. List every package or artifact published from affected pipelines since the earliest suspicious activity. Yank or deprecate affected versions and notify consumers.

Step 6, Notify. Contact your Breach Coach and submit First Notice of Loss per your policy's timing language. Assess any Breach Notification Requirement and, where applicable, Network and Information Security Directive (NIS2) or CIRCIA reporting duties.

Step 7, Restore. Re-enable Actions only after workflow files are reviewed against a known-good commit and secrets are rotated.

SECTION C: Evidence Log

Export audit logs before making changes. The attacker in this campaign deleted workflow runs and logs quickly, so your external log copy may be the only reliable record.

Customization Tips

Scale Section A to Risk, Not Repository Count. Start with repositories that publish packages or hold cloud deploy roles, as they have the highest blast radius.

Tune Your Containment Trade-Off. Disabling Actions org-wide stops the bleeding but halts releases. Decide in advance who can authorize this and what your tolerance is. Tie this to your Business Interruption Coverage waiting period so finance understands the exposure.

Map Rotation to Your Register. The hardest part of Step 4 is knowing what's reachable. If your secrets register lists consuming workflows, rotation becomes a query instead of an investigation.

Align with Your Policy. Check whether forensic spend and vendor engagement require carrier approval under an Insurer Consent Requirement. Also, check how your Underwriting Questionnaire answers describe CI/CD controls; an Application Fraud Warranty makes accuracy a coverage issue.

Map to NIST CSF Core Functions. Section A supports Protect, alerting rules support Detect, Section B covers Respond, and Step 7 covers Recover.

Validation Steps

A runbook nobody has tested is just hope. Validate it:

1. Tabletop It. Give your team a scenario: a stale pull request exists and one workflow ran. Time how long it takes to produce the list of reachable secrets.
2. Test Your Detection. In a sandbox repository, create a branch and workflow edit that should trigger your alerts. Confirm someone receives them.
3. Verify Log Durability. Confirm audit logs exist outside GitHub and that deleting workflow runs doesn't erase your evidence.
4. Rehearse One Rotation. Rotate a single low-risk secret end to end and record how long it takes. Multiply thoughtfully.
5. Confirm Contacts. Call the carrier notice line and Breach Coach number listed in the runbook. Dead numbers are a common finding.
6. Review Quarterly. Re-run the stale-PR sweep and update the secrets register.

If your tabletop reveals you can't answer "what could this workflow reach?" within an hour, that's your first remediation item, ahead of any new tooling.