> ## Documentation Index
> Fetch the complete documentation index at: https://docs.clavion.xyz/llms.txt
> Use this file to discover all available pages before exploring further.

# Incident Runbook

> Symptom-indexed guide for diagnosing operational issues

## How to use this runbook

<Steps>
  <Step title="Find the symptom">
    Find the symptom below that matches what you are observing.
  </Step>

  <Step title="Follow the diagnosis">
    Follow the diagnosis steps in order.
  </Step>

  <Step title="Apply the resolution">
    Apply the recommended resolution.
  </Step>

  <Step title="Verify via audit trail">
    Check the [Audit Trail](/operations/audit-trail) for root cause analysis.
  </Step>
</Steps>

All SQL queries reference the SQLite audit database (default: `~/.iscl/audit.sqlite`).

***

## Symptoms

<Accordion title="Unauthorized Transaction Executed" icon="circle-exclamation">
  **Severity:** Critical
  **Description:** Funds moved to an unexpected address or an unexpected amount was transferred.

  ### Diagnosis

  **1. Find the transaction in the audit trail:**

  ```sql theme={null}
  SELECT * FROM audit_events
  WHERE event = 'signed'
  ORDER BY timestamp DESC
  LIMIT 10;
  ```

  **2. Reconstruct the full intent lifecycle:**

  ```sql theme={null}
  SELECT event, timestamp, data FROM audit_events
  WHERE intent_id = '<the-intent-id>'
  ORDER BY timestamp ASC;
  ```

  Look for:

  * `intent_received` -- Who submitted the intent? Check `data.source` for the adapter name.
  * `policy_evaluated` -- Was the policy decision `allow` or `require_approval`? If `allow`, the policy threshold was too permissive.
  * `approval_requested` / `approval_decided` -- Was user confirmation required? Was it actually confirmed by the operator?
  * `signed` -- The signing event with the transaction hash.
  * `broadcast` -- Was the transaction successfully broadcast?

  **3. Check if the approval was legitimate:**

  ```sql theme={null}
  SELECT data FROM audit_events
  WHERE intent_id = '<the-intent-id>'
    AND event = 'approval_decided';
  ```

  If `approved: true` but the operator did not authorize it, check:

  * Was `ISCL_APPROVAL_MODE=auto`? This bypasses human confirmation.
  * Was the approval submitted via the web API by an unauthorized party?
  * Was the approval TTL exploited (approved just before expiry)?

  ### Resolution

  <Warning>
    This is a critical security incident. Act immediately.
  </Warning>

  1. **Immediately** -- Rotate the wallet if compromise is suspected. Generate a new key and transfer remaining assets.
  2. **Review policy** -- Lower `requireApprovalAbove.valueWei` to require approval for smaller amounts.
  3. **Check approval mode** -- Ensure production is NOT running with `ISCL_APPROVAL_MODE=auto`.
  4. **Restrict access** -- If using web approval, ensure the `/approval-ui` endpoint is only accessible to authorized operators.
</Accordion>

<Accordion title="All Transactions Being Denied" icon="ban">
  **Severity:** Medium
  **Description:** Every transaction attempt returns HTTP 403 with policy denial.

  ### Diagnosis

  **1. Check the policy denial reasons:**

  ```sql theme={null}
  SELECT data FROM audit_events
  WHERE event = 'policy_evaluated'
    AND json_extract(data, '$.decision') = 'deny'
  ORDER BY timestamp DESC
  LIMIT 5;
  ```

  **2. Common denial reasons and fixes:**

  | Reason                         | Cause                                      | Fix                                                      |
  | ------------------------------ | ------------------------------------------ | -------------------------------------------------------- |
  | `"value exceeds max"`          | `maxValueWei` is too low (default `"0"`)   | Increase `maxValueWei` in policy config                  |
  | `"chain not allowed"`          | Chain ID not in `allowedChains`            | Add the chain ID to `allowedChains`                      |
  | `"token not in allowlist"`     | Token address not in `tokenAllowlist`      | Add the token or clear the allowlist (empty = allow all) |
  | `"recipient not in allowlist"` | Recipient not in `recipientAllowlist`      | Add the recipient or clear the allowlist                 |
  | `"contract not in allowlist"`  | Target contract not in `contractAllowlist` | Add the contract or clear the allowlist                  |
  | `"rate limit exceeded"`        | Wallet exceeded `maxTxPerHour`             | Wait for the window to expire, or increase the limit     |

  **3. Check if the default policy is in effect:**

  <Note>
    The default policy has `maxValueWei: "0"`, which denies ALL transactions. This is a safety default -- you must configure a policy with appropriate limits.
  </Note>

  ### Resolution

  Review and update the policy configuration. See [Configuration Reference](/reference/config-reference) for all PolicyConfig fields.
</Accordion>

<Accordion title="Transactions Failing on Broadcast" icon="tower-broadcast">
  **Severity:** Medium
  **Description:** ISCL signs the transaction but broadcast fails (`broadcastError` is non-null).

  ### Diagnosis

  **1. Check broadcast errors in the audit trail:**

  ```sql theme={null}
  SELECT data FROM audit_events
  WHERE event = 'broadcast'
    AND json_extract(data, '$.success') = 0
  ORDER BY timestamp DESC
  LIMIT 5;
  ```

  **2. Common broadcast errors:**

  | Error                                   | Cause                                                 | Fix                                                          |
  | --------------------------------------- | ----------------------------------------------------- | ------------------------------------------------------------ |
  | `"nonce too low"`                       | Another transaction with the same nonce was confirmed | Retry -- the next attempt will use the correct nonce         |
  | `"insufficient funds for gas"`          | Wallet doesn't have enough ETH for gas                | Fund the wallet with native ETH                              |
  | `"replacement transaction underpriced"` | A pending transaction exists with higher gas          | Wait for the pending transaction to confirm, or increase gas |
  | `"execution reverted"`                  | The transaction would revert on-chain                 | Check preflight simulation -- this should have been caught   |

  **3. Check RPC health:**

  ```bash theme={null}
  curl -s -X POST YOUR_RPC_URL \
    -H "Content-Type: application/json" \
    -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'
  ```

  If this fails, the RPC endpoint is down or unreachable.

  ### Resolution

  * For nonce errors: usually self-correcting on retry
  * For gas errors: send ETH to the wallet
  * For RPC errors: check provider status, switch to backup RPC
  * For persistent reverts: check the transaction parameters (insufficient token balance, expired deadline)
</Accordion>

<Accordion title="RPC Errors / 502 Responses" icon="plug-circle-exclamation">
  **Severity:** Medium
  **Description:** API returns HTTP 502 with `"no_rpc_client"` or RPC timeout errors.

  ### Diagnosis

  **1. Verify RPC configuration:**

  ```bash theme={null}
  # Check which RPC URLs are configured
  env | grep ISCL_RPC_URL
  env | grep BASE_RPC_URL
  ```

  **2. Test RPC connectivity:**

  ```bash theme={null}
  curl -s -X POST $ISCL_RPC_URL_8453 \
    -H "Content-Type: application/json" \
    -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}'
  ```

  **3. Check for provider rate limiting:**

  If you see intermittent 502s, your RPC provider may be rate limiting. Look for HTTP 429 responses in ISCL Core logs.

  ### Resolution

  | Cause                 | Fix                                                           |
  | --------------------- | ------------------------------------------------------------- |
  | No RPC URL configured | Set `ISCL_RPC_URL_{chainId}` environment variable             |
  | Wrong chain ID        | Verify the `chainId` in the TxIntent matches a configured RPC |
  | RPC provider down     | Switch to an alternative provider                             |
  | Rate limited          | Upgrade your RPC plan or reduce transaction frequency         |
</Accordion>

<Accordion title="Approval Requests Timing Out" icon="clock">
  **Severity:** Low
  **Description:** Users report "Transaction expired" when trying to approve transactions.

  ### Diagnosis

  **1. Check the approval TTL:**

  The default TTL is 300 seconds (5 minutes). If operators consistently need more time, the TTL may be too short.

  **2. Check for stale pending requests:**

  ```bash theme={null}
  curl -s http://localhost:3100/v1/approvals/pending | jq
  ```

  If this shows requests with low `ttlSeconds`, they are about to expire.

  **3. Check approval mode:**

  ```bash theme={null}
  echo $ISCL_APPROVAL_MODE
  ```

  If set to `cli` but no one is watching the terminal, approvals will time out.

  ### Resolution

  | Cause                        | Fix                                                         |
  | ---------------------------- | ----------------------------------------------------------- |
  | No one watching the terminal | Switch to `web` mode and use the browser dashboard          |
  | TTL too short                | This is not configurable in v0.1; the 300s default is fixed |
  | Web dashboard not open       | Open `http://localhost:3100/approval-ui`                    |
  | Telegram bot not connected   | Ensure the bot is running and can reach ISCL Core           |
</Accordion>

<Accordion title="Signing Fails &#x22;No key found&#x22;" icon="key">
  **Severity:** Medium
  **Description:** The approval succeeds but signing fails with a key-not-found error.

  ### Diagnosis

  **1. List available keys:**

  ```bash theme={null}
  npx clavion-cli key list --keystore-path ~/.iscl/keystore
  ```

  **2. Compare with the wallet address in the intent:**

  The `wallet.address` in the TxIntent must match an address in the keystore (case-insensitive).

  **3. Check keystore path:**

  Ensure the ISCL Core process can access the keystore directory. In Docker, the keystore is mounted as a volume.

  ### Resolution

  | Cause               | Fix                                                      |
  | ------------------- | -------------------------------------------------------- |
  | Key not imported    | Run `clavion-cli key import` or `key generate`           |
  | Wrong keystore path | Set `--keystore-path` or check the Docker volume mount   |
  | Address mismatch    | Verify the wallet address matches the imported key       |
  | Passphrase issue    | Ensure the keystore passphrase is available to ISCL Core |
</Accordion>

<Accordion title="Sandbox Skill Times Out" icon="hourglass-end">
  **Severity:** Low
  **Description:** A sandbox skill is killed (exit code 137) before completing.

  ### Diagnosis

  **1. Check sandbox events:**

  ```sql theme={null}
  SELECT * FROM audit_events
  WHERE event IN ('sandbox_started', 'sandbox_error', 'sandbox_completed')
  ORDER BY timestamp DESC
  LIMIT 10;
  ```

  **2. Look for timeout indicators:**

  ```sql theme={null}
  SELECT data FROM audit_events
  WHERE event = 'sandbox_error'
    AND json_extract(data, '$.error') = 'timeout'
  ORDER BY timestamp DESC;
  ```

  **3. Check manifest resource limits:**

  The skill's `sandbox.timeoutMs` in the manifest controls the deadline. The default cap is 60 seconds.

  ### Resolution

  | Cause                   | Fix                                                     |
  | ----------------------- | ------------------------------------------------------- |
  | Timeout too low         | Increase `timeoutMs` in the skill manifest (max: 60000) |
  | Skill has infinite loop | Debug the skill code outside the sandbox first          |
  | Memory limit hit        | Increase `memoryMb` in the manifest (max: 512)          |
  | Docker not available    | Ensure Docker daemon is running                         |
</Accordion>

<Accordion title="High Memory Usage" icon="memory">
  **Severity:** Low
  **Description:** ISCL Core process memory exceeds expected levels (>300MB).

  ### Diagnosis

  **1. Check Node.js heap usage:**

  ```bash theme={null}
  # If you have access to the process
  node -e "console.log(process.memoryUsage())"
  ```

  **2. Check for leaked pending approvals:**

  ```bash theme={null}
  curl -s http://localhost:3100/v1/approvals/pending | jq length
  ```

  A large number of stale pending approvals suggests the cleanup interval isn't running or requests are being submitted faster than they expire.

  **3. Check SQLite database size:**

  ```bash theme={null}
  ls -lh ~/.iscl/audit.sqlite*
  ```

  ### Resolution

  | Cause                | Fix                                                                |
  | -------------------- | ------------------------------------------------------------------ |
  | Many stale approvals | They auto-cleanup every 30s; wait or restart Core                  |
  | Large SQLite DB      | Run `VACUUM` during maintenance window                             |
  | Node.js heap growth  | Restart the process; check for memory leaks in custom integrations |
</Accordion>

<Accordion title="Preflight Simulation Fails" icon="flask-vial">
  **Severity:** Low
  **Description:** The preflight service reports simulation failure, blocking the transaction.

  ### Diagnosis

  **1. Check the preflight result in audit trail:**

  ```sql theme={null}
  SELECT data FROM audit_events
  WHERE event = 'preflight_simulated'
  ORDER BY timestamp DESC
  LIMIT 5;
  ```

  Look for `simulationSuccess: false` and the associated error message.

  **2. Common simulation failures:**

  | Simulation Error          | Cause                                                           |
  | ------------------------- | --------------------------------------------------------------- |
  | `"execution reverted"`    | Insufficient token balance, expired deadline, or bad parameters |
  | `"gas estimation failed"` | Transaction would revert -- check balances and allowances       |
  | `"RPC error"`             | RPC endpoint issue -- see "RPC Errors" section above            |

  ### Resolution

  * Verify the sender has sufficient balance for the transfer/swap
  * Check that ERC-20 approvals are in place for swap router interactions
  * Ensure the deadline hasn't passed
  * Verify the target contract exists on the specified chain
</Accordion>

***

## General investigation procedure

For any incident not covered above:

<Steps>
  <Step title="Get the intent ID">
    Find the intent ID from error messages, API responses, or logs.
  </Step>

  <Step title="Pull the full audit trail">
    ```sql theme={null}
    SELECT event, timestamp, data FROM audit_events
    WHERE intent_id = '<intent-id>'
    ORDER BY timestamp ASC;
    ```
  </Step>

  <Step title="Identify where the pipeline failed">
    Which event is the last one? What does its `data` show?
  </Step>

  <Step title="Check surrounding events for context">
    ```sql theme={null}
    SELECT event, intent_id, data FROM audit_events
    WHERE timestamp BETWEEN <failure_ts - 60000> AND <failure_ts + 60000>
    ORDER BY timestamp ASC;
    ```
  </Step>

  <Step title="Correlate with application logs">
    Cross-reference with pino structured logs for stack traces and detailed error context.
  </Step>
</Steps>

## Next steps

* [Audit Trail](/operations/audit-trail) -- Event types and SQL query patterns
* [Error Codes](/reference/error-codes) -- HTTP error shapes and recovery
* [Configuration Reference](/reference/config-reference) -- All configurable parameters
* [Observability](/operations/observability) -- Logging and monitoring setup
* [Performance Tuning](/operations/performance-tuning) -- Resource optimization