Runbooks

The step-by-step procedures live in the engineering handbook, next to the code, so they change in the same commit as the behaviour they describe. This page says what each one is for, who follows it and where its commands are explained in these docs. Follow the runbook itself when acting; this page is a map.

Situation	Start with
Something looks wrong (an alert, a user report, a failed check)	Incident response
A key or token must change, or may have leaked	Keys and tokens
Reads fail or a document cannot be fetched	RPC providers and pinning
An admin action on the contracts	Admin keys or Timelock
A new contract version	Adding a contract set
A partner asks a question or reports a problem	Partner support
Someone reports a vulnerability	Vulnerability reports
A new institution wants to issue	Onboarding playbook and Governance

Incident response

docs/process/runbooks/incident.md, for whoever is on call (today one maintainer).

• Turning the monitor on (section 0): CRON_SECRET, optional webhook and key expiries, a scheduler calling /api/ops/check. See Monitoring.
• Severity (section 2): level 1 when wrong answers could be accepted or keys may be stolen (start within the hour); level 2 when service is down but answers are right (same working day); level 3 when a risk is coming (within the week).
• First ten minutes (section 3): write down the time; for level 1 a guardian pauses first and investigates after; rotate a possibly stolen key before looking for the cause; then RPC or pinning fixes; then tell people.
• Do not: send from a key you cannot account for, paste secrets anywhere, delete pins or contracts, or unpause before the cause is understood.
• Record (section 6): within two working days, a dated incident note with a time line, cause, what the runbook got wrong and follow-ups. Security incident records stay private.

Keys and tokens

docs/process/runbooks/keys.md, for the person who holds the keys.

• An inventory of every key and token: the variable it lives in, whether it may hold funds (none should, apart from the testnet deployer), and what breaks if it is lost.
• Rotation steps per kind. The status list publisher and the API signer are pinned by outsiders, so their rotation is announced first and the key and its public address change together in one deploy (otherwise the signers check fails, by design).
• A leak check (npm run check:no-dev-keys and a history search) and the OPS_KEY_EXPIRIES format.

The variables are listed in Configuration; key custody is discussed in Key management.

RPC providers and pinning

docs/process/runbooks/rpc-and-pinning.md, for whoever runs the deployment.

• How reads fail over between endpoints (8 seconds per try, a failed endpoint skipped for 30 seconds) and the known limit that a lagging backup can answer an authorisation read with slightly old data.
• Adding, replacing or retiring a provider; what to do when the primary is down; event scans need an endpoint that allows wide eth_getLogs ranges.
• How the second pin copy and the monitor's repair work, what each pins message means (different bytes from a gateway is level 1), and a manual repair procedure. The second pin service is not chosen yet, so this part is not active.

Admin keys, guardian and the Safe

docs/process/runbooks/admin-keys.md, for Safe owners and the guardian.

• Who can do what; deploying and nominating the Safe; accepting after the delay; everyday admin actions through the Safe's Transaction Builder.
• Pause (section 6): the guardian pauses on /admin, suspends a compromised issuer, and two Safe owners restart. If the guardian key may be stolen, revoking the guardian, unpausing and lifting suspensions go in one Safe batch.
• Rotating a Safe signer, what losing signers means, moving to a new admin on purpose, and honest limits.

Background: Governance, Admin, Guardian.

The admin timelock

docs/process/runbooks/timelock.md, for Safe signers.

• Deploying the timelock and handing every registry over to it through three Safe batches (begin, schedule, execute), including a set that joins later.
• Everyday admin actions: timelock:propose:sepolia writes schedule, execute and cancel batch files; the Safe signs the schedule; anyone executes after the delay.
• Watching the queue, emergencies (the guardian pauses at once; unpausing waits for the delay, except on 0.5.0 sets where the guardian may unpause), honest limits.
• Institution key procedures (section 8): registering a successor and retiring an old key, and the order of steps for a stolen key. See Issuer key retirement.

Out of date header

The runbook's status line still says the timelock is not yet deployed on Sepolia. It was deployed and rehearsed on 2026-10-17 (ADR-0092); see Deployments.

Adding a contract set

docs/process/runbooks/contract-sets.md, for the developer or operator shipping new contracts.

• Deploy on a local chain first, then on Sepolia with ADD_SET=1 (each public deployment needs a go-ahead), verify, hand over admin.
• Making a set active (section 3a): set:ready:sepolia, then set:activate:sepolia, commit the manifest, test on a preview deployment, rebuild.
• Announcing the set, moving holders forward by re-issuing, what several sets cannot do, and retiring a set.

Background: Contract set upgrade.

Partner support

docs/process/runbooks/support.md, for whoever answers institutions and verifiers.

• Channels named per partner at onboarding (the addresses are placeholders as of 2026-10-24); security reports never go through them.
• Every report logged with its first reply time, which objective S6 measures. The log names partners, so it is not committed.
• Response targets: severity 1 within 4 hours, severity 2 within 1 working day, severity 3 within 3 working days, with escalation after twice that.
• How to tell partners about an incident, common questions, and the monthly report.

Vulnerability reports

docs/process/runbooks/vulnerability-reports.md, for whoever triages GitHub private vulnerability reports. The public promises are in the repository's SECURITY.md.

• Acknowledge within 5 working days, first assessment within 15, a 90 day coordinated disclosure window.
• Reproduce on a local chain or the test network only; rate in four bands (Critical, High, Medium, Low).
• Fix on an unpushed branch with a regression test written first; a contract fix ships as a new contract set.
• Rewards open only after the external audit fixes are released. A dry run with a mock report comes before the first real one.

Institution onboarding playbook

docs/onboarding/, written against contract version 0.5.0 and the test network. Six stages, each with a checklist in docs/onboarding/checklists/ and a sign-off sheet at the end:

Stage	Chapter	Ends when
1	Fit and review	The institution decides in writing to proceed
2	Key ceremony	The issuer address exists with two named custodians and a tested recovery plan
3	Registration	The address is registered and visible on the explorer
4	Staff training	Each role has walked its procedures on the test network
5	First credentials	A rehearsal and a small first batch were checked by a verifier
6	Operating and leaving	The routine is in the institution's calendar

Ground rules: nobody ever shares an issuer key or seed phrase with the operator; nothing is issued to a real person before stage 5 passes on the test network; anything on chain is public and permanent. Chapter 2 states that a contract wallet cannot issue through the site today. See Issuer.

Consortium governance

docs/governance/: a draft charter (version 0.1) for when the Safe may admit, suspend or remove an institution, an accreditor sign-off template, a public log format (one dated file per decision) and a rehearsal procedure on the test network.

• The charter does not change who holds admin: the Safe acting through the timelock.
• An admission's timelock salt is the hash of the accreditor's sign-off file (SIGNOFF_FILE=<path> npm run timelock:propose:sepolia), so anyone can match the sign-off to the scheduled operation with admission:check:sepolia.
• npm run governance:log:check validates every log entry.
• The charter still needs review by counsel and a partner, and the rehearsal needs a go-ahead.

See Governance.

Source files

• docs/process/runbooks/admin-keys.md, contract-sets.md, incident.md, keys.md, rpc-and-pinning.md, support.md, timelock.md, vulnerability-reports.md
• docs/onboarding/README.md, docs/onboarding/01-fit-and-review.md to 06-operating-and-leaving.md, docs/onboarding/checklists/
• docs/governance/README.md, charter.md, signoff-template.md, rehearsal.md, log/README.md
• SECURITY.md

Tests

• blockchain/test/v03/Admission.test.ts, blockchain/test/v03/GovernanceLog.test.ts, blockchain/test/v03/TimelockProposal.test.ts (the tooling these runbooks call)
• src/lib/server/ops/*.test.ts (the monitor the incident runbook relies on)