Investigate an incident

This walkthrough follows a real-looking incident in the icLens demo: version 0.4.1 of payments-api ships, and payments start failing. Open the demo beside this page and follow along. Nothing you do in the demo is real.

1. Spot it in the metrics. In Telemetry › Metrics, chart app.error_ratio for payments-api, split by method. While 0.4.1 runs, the ratio on icrc2_transfer_from climbs to 2.6%.

2. Jump to an example. Chart instructions_per_call: it peaks at 2.3 billion. The dots on the chart are exemplars, real requests the store kept with their trace. Select one to open it.

3. Read the trace. The waterfall shows where the time went: payments-api waits 11 seconds on the ckUSDC ledger (which sends no spans, so it reads not reported), then decode_transfer traps. Open this trace in the demo.

4. Read the logs. In the trace’s span panel, the Logs tab shows the lines the code wrote for that span, ending with trapped: heap out of bounds in decode_transfer.

5. See what users felt. In Telemetry › Frontends, open app.acme.io and check its requests to payments-api and its JavaScript errors over the same hours.

6. Check it from outside. In Monitor › Availability, payments-api still answers its check, a read-only query, but slowly. The failures are in the payment path, not the whole canister.

Fix it

Once you know the cause, fix it with an ordinary reviewed change: deploy the previous release from Deploy code, or restore the snapshot taken before the upgrade. Deploy code · Snapshots.

Good to know

• Things happening at the same time aren’t proof of a cause. The trace and the log line are.
• On your own fleet, traces and code logs need a few lines in your canister. Send telemetry from your code.
• Ask the AI assistant to explain a chart. It cites the evidence it used. Ask the AI assistant.