Monitoring endpoints

Two routes let an external scheduler watch the deployment:

Route	Job
GET /api/ops/check	Run every check, remember each check's state, and send one alert when a state changes
GET /api/ops/report	Build the monthly service report from the stored history, and optionally post it once

Both are read only. They never send a transaction, never sign anything, and their answers name states and contract addresses only, never a person's wallet, a document, a key or an endpoint address.

The operating procedure (what to do when an alert arrives) is in Runbooks and Monitoring flow.

Off until configured, as of 2026-10-24

The monitor is built but not running on the Sepolia deployment. It needs CRON_SECRET, a shared Redis store, an alert channel (OPS_WEBHOOK_URL) and a schedule. The repository holds no scheduler configuration (no vercel.json cron entry), so the operator sets up the scheduler. The probe check also needs a canary credential and the OPS_PROBE_* names.

Calling the routes

GET /api/ops/check HTTP/1.1
Authorization: Bearer <CRON_SECRET>

Step	Failure
Method is GET	405 BAD_REQUEST "Use GET." with Allow: GET
CRON_SECRET is set and at least 32 characters	401 UNAUTHORIZED "Not allowed." and a server log line
The bearer value equals the secret (constant-time comparison)	401 "Not allowed."
A store is available	500 CONFIG_MISSING "The monitor is not configured."
At most 6 calls per 60 seconds	429 with Retry-After
The run completes	500 INTERNAL_ERROR "The monitor could not run."

The secret is checked before anything else, so a caller without it learns nothing about the setup and cannot use up the limit. The limit stops a leaked secret from being used to flood the RPC providers. Both routes run on Node.js with maxDuration = 60 seconds.

A successful check run answers:

{
  "data": {
    "results": [
      { "name": "rpc", "state": "ok", "detail": "2 of 2 endpoints answer on chain 11155111." },
      { "name": "governance", "state": "ok", "detail": "6 registries have the same admin and no pending transfer." }
    ],
    "alertSent": null
  }
}

Each check writes a fixed phrase with counts and contract labels; the numbers above are an example. alertSent is null when nothing changed, true when an alert went out, and false when one was due but there is no channel or sending failed.

The checks

buildOpsChecks (src/lib/server/ops/wiring.ts) builds the list from the settings:

Check	What it looks at	Warn	Fail
rpc	Every configured RPC endpoint, 5 seconds each	An endpoint is more than OPS_MAX_LAG_BLOCKS (default 20) behind the others, or did not answer while another did	An endpoint is on another chain, or none answered
chain-events	Admin events over the last OPS_LOOKBACK_BLOCKS (default 600, max 50,000) in every listed set and the timelock	Any watched event happened (up to 5 listed)	The logs could not be read
governance	The admin of every registry and any pending admin transfer	A transfer is pending, the registries have different admins, or a registry is not administered by the manifest's timelock (when one is recorded)	The admins could not be read
pins	Retries failed second pins, then reads back the 10 newest pins through each gateway and compares their SHA-256	A gateway could not serve a pin, or copies are still waiting for repair	A gateway returned different bytes, or no gateway could serve a pin
signers	The addresses of STATUS_LIST_SIGNER_KEY and API_SIGNER_KEY against their published addresses	A key without a published address, or the reverse	A key does not match its published address
probe	Fetches the canary credential through the public API, checks the signed envelope against the pinned signer, and times it	Verified but took 2 seconds or more	No answer, a status other than 200, not signed by the pinned signer, outside its window, nonce not echoed, or the canary not in good standing
keys	Expiry dates in OPS_KEY_EXPIRIES	Within OPS_WARN_DAYS (default 14)	Past
config	Present only when a setting is malformed or the chain settings are missing		Always

pins exists only when a second pin provider is configured, and probe only when the OPS_PROBE_* names are set. A check that throws becomes a fail with "The check could not run." A check that cannot be built becomes "The <name> check cannot be set up. See the server log."

The watched events are:

Contract	Events
IssuerRegistry	Paused, Unpaused, IssuerAdded, IssuerRemoved, IssuerRetired, RoleGranted, RoleRevoked, DefaultAdminTransferScheduled, DefaultAdminTransferCanceled, DefaultAdminDelayChangeScheduled
Timelock	CallScheduled, CallExecuted, Cancelled, MinDelayChange, RoleGranted, RoleRevoked
StatusRegistry	IssuerSuspensionChanged, RoleGranted, RoleRevoked, DefaultAdminTransferScheduled, DefaultAdminTransferCanceled

An event is not necessarily a problem: a planned issuer registration also shows. The warning makes sure somebody looks.

The probe in detail

The probe (checks/probe.ts) calls GET {OPS_PROBE_API_URL}/api/v1/credentials/{OPS_PROBE_CANARY_ID} with a fresh nonce, optionally with OPS_PROBE_API_KEY, and a 10 second timeout. It then:

• checks the envelope with checkEnvelope against NEXT_PUBLIC_API_SIGNER, allowing 60 seconds of clock difference;
• requires the canary to be found, active, not expired and from a registered, unsuspended issuer;
• records the time taken, and how far behind the chain the answer could be: the gap between generatedAt and the timestamp of the block it read, plus dataMaxAgeSeconds.

These numbers feed objectives S1 to S3 below.

Alerts

src/lib/server/ops/monitor.ts keeps each check's last state in the store (ops:state:<name>, 30 days). After a run it compares:

• A change to warn or fail, or a recovery from them to ok, is put in one alert listing every change.
• ok after ok, or the first ok ever, sends nothing.
• A new state is remembered only after the alert was sent, so a failed send is retried on the next run.

src/lib/server/ops/alerts.ts posts { "text": ..., "content": ... } (the same text for Slack-style and Discord-style webhooks) to OPS_WEBHOOK_URL. The address must be https; redirects are refused; the timeout is 5 seconds; each detail is cut to 200 characters and the whole text to 1,400.

History and the monthly report

Each run adds to a daily record in the store (slo:<day>, kept 400 days): per-check counts of ok, warn and fail, and probe counts (runs, good, fast, measured staleness, within 5 minutes). Alerts of the month are kept under slo:alerts:<month>. Without a shared store (Redis) there is no durable history.

GET /api/ops/report takes:

Query	Meaning
month=YYYY-MM	This month or one of the twelve before it; default the previous month. Other values: 400 "The month must be YYYY-MM, this month or one of the twelve before it."
post=1	Post the report to the webhook once per month. Not for an unfinished month. A failed post answers 502 "The report could not be posted. Try again."

The report measures six objectives (src/lib/server/ops/slo/report.ts):

Id	Objective	Target	Measured from
S1	Verification answers	99.5% of probes	Probes that returned a good, signed answer
S2	Revocation visible	95% within 5 minutes	Probes whose answer was at most 300 seconds behind the chain
S3	Answer speed	95% under 2 seconds	Good probes faster than 2,000 ms
S4	Documents retrievable	99% of runs	pins runs that did not fail
S5	Chain reads	99.5% of runs	rpc runs that did not fail
S6	Support response	none	Reported from the support log, not by the monitor

Percentages are rounded down. A warning counts as available, because it means one of several providers had a problem. The report states what is not subtracted from the figures: the chain itself halting, a partner's own wallet or network, and maintenance announced three working days ahead.

Targets are proposals

The targets are proposals for pilot partners (P5-S05) and have not yet been agreed or measured over a month.

Configuration

Name	Meaning
CRON_SECRET	Bearer secret for the scheduler, at least 32 characters
OPS_WEBHOOK_URL	https webhook for alerts and reports
OPS_MAX_LAG_BLOCKS, OPS_LOOKBACK_BLOCKS, OPS_WARN_DAYS	Thresholds (defaults 20, 600, 14)
OPS_KEY_EXPIRIES	JSON object of key names to YYYY-MM-DD dates
OPS_PROBE_API_URL, OPS_PROBE_CANARY_ID, OPS_PROBE_API_KEY	The probe; setting only some of url and canary is a configuration error
NEXT_PUBLIC_API_SIGNER	The signer the probe pins
NEXT_PUBLIC_NETWORK_ID, RPC names	The chain
Redis names	Store for state, history and limits

Source files

• src/app/api/ops/check/route.ts, src/app/api/ops/report/route.ts
• src/lib/server/ops/ (config.ts, wiring.ts, instance.ts, handler.ts, monitor.ts, alerts.ts, state-store.ts, types.ts)
• src/lib/server/ops/checks/ (rpc-health.ts, chain-events.ts, watched.ts, governance.ts, signer-config.ts, probe.ts, probe-fetch.ts, key-expiry.ts)
• src/lib/server/ops/slo/ (history.ts, aggregate.ts, report.ts, report-handler.ts, report-instance.ts)
• src/lib/server/pinning/check.ts
• docs/process/runbooks/ (incident, keys, RPC and pinning, support)

Tests

• src/lib/server/ops/*.test.ts, src/lib/server/ops/checks/*.test.ts, src/lib/server/ops/slo/*.test.ts
• src/lib/server/pinning/repair-and-check.test.ts
• E2E: e2e/journeys/service-probe.journey.ts