Monitoring and service reports

Certichain has no servers of its own to watch apart from the Next.js routes, but it depends on things that can fail or be misused: RPC providers, the chain's governance state, signer keys, the verification API and the pinned documents. A monitor checks these on a schedule and tells an operator when something changes. A monthly report turns the monitor's history into figures for pilot partners.

Both are read-only server routes protected by one secret. The route details are in Monitoring endpoints. The operator's role is in Operator.

Off until configured

The monitor is off unless CRON_SECRET (at least 32 characters), NEXT_PUBLIC_NETWORK_ID and a server RPC address (RPC_URL or RPC_URLS) are set, and a scheduler calls the route. Alerts need OPS_WEBHOOK_URL; history and the monthly report need the shared Redis store; the canary probe needs OPS_PROBE_* and a canary credential. As of 2026-10-24 the handbook lists these as not yet set on the hosted site.

Who takes part

Actor	Part
Scheduler	Calls GET /api/ops/check (and once a month GET /api/ops/report?post=1) with Authorization: Bearer <CRON_SECRET>. Vercel Cron sends that header itself when CRON_SECRET is set; any scheduler that can send it works.
Monitor	src/lib/server/ops/monitor.ts: runs the checks, compares with the stored state, sends alerts
State store	Upstash Redis when configured, else memory of one server instance (src/lib/server/ops/state-store.ts)
Alert channel	One incoming webhook (Slack or Discord style), https only
Operator	Reads alerts, follows the incident runbook

The checks

buildOpsChecks in src/lib/server/ops/wiring.ts builds the checks that apply. A check that cannot be set up shows as a failure instead of stopping the monitor.

Check	Looks at	Warn	Fail
rpc	Every server RPC endpoint: chain id, head block, answer time (5 second timeout)	One endpoint down, behind by more than OPS_MAX_LAG_BLOCKS (default 20) or slow	No endpoint answers, or one serves another chain
chain-events	Administrative events of every listed set and the timelock in the last OPS_LOOKBACK_BLOCKS (default 600, max 50,000)	Pause, unpause, issuer added or removed, role change, admin transfer, issuer suspension, timelock operations	The chain cannot be read
governance	Admin and pending admin of every watched registry	A transfer is pending, registries have different admins, or a registry is not administered by the recorded timelock	The chain cannot be read
signers	STATUS_LIST_SIGNER_KEY against NEXT_PUBLIC_STATUS_LIST_PUBLISHER, API_SIGNER_KEY against NEXT_PUBLIC_API_SIGNER	A key with no published address, or the reverse	The key in use is not the published address
keys	OPS_KEY_EXPIRIES (names to dates)	Expiry within OPS_WARN_DAYS (default 14)	Expired
probe	A canary credential through the signed verification API	Verified but slow (2 seconds or more)	No answer, an error, a wrong signer, a stale answer, or the canary not in good standing
pins	A sample of 10 recent pins through each read gateway, and repair of copies waiting for the second pin service	A gateway cannot serve a recent pin, or a copy waits for repair	No gateway serves a pin, or a gateway returns other bytes
config	Appears only when a monitor setting is malformed or the chain or RPC is missing		Names the setting, never its value

The probe and pins checks exist only when their settings are present. Details name checks, registries and states; they never carry an address, a key or personal data.

What the governance check says on Sepolia

The Sepolia manifest records a timelock, and the 0.3.0 registries were still administered by the Safe with the timelock nominated as of the last record. With the monitor on, governance would therefore warn on every run until the handover of every set is finished. That is intended: a set left under another admin stays visible. See Governance.

One run

How alerts behave:

• A message is sent when a check's state or wording changes. An ok after an ok or after nothing stored is not a change.
• The new state is stored only after the alert went out, so a failed alert is tried again next run.
• Stored states expire after 30 days, so a lasting problem is reminded about once a month.
• Without a webhook the monitor still runs and returns states; nobody is told (alertSent: false).

Route answers

Status	When
200	{ results, alertSent } with Cache-Control: no-store
401 UNAUTHORIZED	Missing or wrong secret, or the secret is not configured (same answer, so strangers learn nothing)
405	Not a GET
429	More than 6 runs in 60 seconds
500 CONFIG_MISSING or INTERNAL_ERROR	No rate-limit store, or the monitor failed

The canary probe

The probe treats the verification API like a relying party would:

1. Ask GET /api/v1/credentials/{OPS_PROBE_CANARY_ID} on OPS_PROBE_API_URL with a fresh nonce, optionally with its own key OPS_PROBE_API_KEY so it does not use the anonymous budget.
2. Check the EIP-712 signature against the pinned NEXT_PUBLIC_API_SIGNER (not a signer the API names), the nonce echo and the freshness window (60 seconds of clock skew allowed).
3. Work out how far the answer's chain data can trail the chain: the cache age the server states plus the age of its block.
4. Report the time taken and whether the canary is in good standing.

The canary is a real credential the operator issues for this purpose and never revokes.

History and the monthly report

When the store is Redis, every run adds daily counts per check and records the month's alerts (src/lib/server/ops/slo/history.ts). GET /api/ops/report (same secret, same 6-per-minute limit) builds the report for the previous month, or ?month=YYYY-MM for this month or one of the twelve before. With ?post=1 it posts the text once to the webhook and remembers the month for 400 days, so a monthly scheduler never posts twice. Posts are cut at 1,900 characters.

Objective	Target (proposal for pilot partners)	Measured from
S1 Verification answers	99.5% of probes	probe runs that verified
S2 Revocation visible	95% within 5 minutes	Probe runs whose data age was within the bound
S3 Answer speed	95% under 2 seconds	Probe durations
S4 Documents retrievable	99% of runs	pins available
S5 Chain reads	99.5% of runs	rpc available
S6 Support response	See the support log	Not measured by the monitor

The targets are proposals in the P5-S05 spec, to be agreed with partners. A month with no runs for an objective shows "not measured this month". S2 is a continuous bound on data age, not a monthly revoke-and-watch test (DEV-006). Without a shared store the report says the history is not durable.

Turning it on

1. Create CRON_SECRET (32 or more random characters; Sensitive in Vercel).
2. Set OPS_WEBHOOK_URL (https). Optional: OPS_MAX_LAG_BLOCKS, OPS_LOOKBACK_BLOCKS (keep it longer than the schedule interval, or events between runs are missed), OPS_WARN_DAYS, OPS_KEY_EXPIRIES.
3. Set UPSTASH_REDIS_REST_URL and UPSTASH_REDIS_REST_TOKEN for history and shared limits.
4. For the probe: issue a canary credential, set OPS_PROBE_API_URL, OPS_PROBE_CANARY_ID and NEXT_PUBLIC_API_SIGNER (the API must be on), and optionally OPS_PROBE_API_KEY.
5. Add a schedule that calls /api/ops/check, and a monthly one for /api/ops/report?post=1. The Vercel Hobby plan allows one cron run a day; Pro allows every minute.

See Configuration for every name.

Limits

• The monitor sees only what the server can read. It is not an uptime service for the site itself.
• An event older than the look-back window is missed if a run is skipped; the governance check catches lasting admin changes regardless.
• One webhook, no escalation. The support runbook's contact and channels are placeholders as of 2026-10-24.
• The pins check is built only when SECOND_PIN_PROVIDER names a pin service adapter. The adapter table in src/lib/server/pinning/providers.ts is empty as of 2026-10-24 (the service is still to be chosen), so the check does not run and S4 is not measured; setting the variable today makes the pins check fail with "The pins check cannot be set up. See the server log." See Runbooks.

Source files

• src/app/api/ops/check/route.ts, src/app/api/ops/report/route.ts
• src/lib/server/ops/handler.ts, config.ts, wiring.ts, monitor.ts, alerts.ts, state-store.ts, instance.ts
• src/lib/server/ops/checks/rpc-health.ts, chain-events.ts, governance.ts, signer-config.ts, key-expiry.ts, probe.ts, probe-fetch.ts, watched.ts
• src/lib/server/ops/slo/history.ts, aggregate.ts, report.ts, report-handler.ts
• Handbook: docs/process/runbooks/incident.md, docs/process/runbooks/support.md, docs/process/runbooks/rpc-and-pinning.md

Tests

• src/lib/server/ops/*.test.ts (handler, config, monitor, alerts, state store and its Redis form, wiring)
• src/lib/server/ops/checks/*.test.ts, src/lib/server/ops/slo/*.test.ts
• E2E: e2e/journeys/service-probe.journey.ts