Agent checks

More and more of your users never read your docs themselves. They tell Claude Code or Codex "set up X with our app" and let the agent work it out from your docs, your CLI and your API. When a release renames a flag, moves a page or changes an error message, a person shrugs and searches; an agent gets lost, burns tokens, or quietly does the wrong thing, and you never hear about it.

An agent check catches that. On a schedule, Waitcher gives a real coding agent the task your users would give it, lets it work in a fresh sandbox with nothing but your public docs and tools, and then checks your product's own state to see whether the job got done. The verdict never comes from the agent's own report: an agent that says "done" but left nothing behind fails.

How a run works

1. Waitcher starts a fresh, single-use sandbox (a microVM on Cloudflare) with Claude Code 2.1.316 installed and an empty working directory.
2. Claude Code gets your task and works on it headless, with curl, Node and Python. It can reach your target's host and the hosts you declare, nothing else.
3. Its model requests carry a random placeholder, not your key. Waitcher's gateway outside the sandbox swaps in your key, only for the model you chose, and stops the agent at your time, turn and token limits.
4. The sandbox is destroyed. Your verifier, an ordinary api test, runs in a new one with its own secrets and checks what the agent left in your product.
5. The run passes or fails with the verifier. Its message starts with the agent's outcome, steps, model requests and tokens, and the evidence keeps the transcript.

Your model key stays out of the sandbox

The agent reads text you do not fully control, such as your docs, your pages and anything a third party can post there, and it has a shell. A key in its environment could be printed, encoded or sent away by a prompt injection. So it is never there:

• The key lives in a model connection: encrypted, write-only, scoped to one project environment, never shown again.
• Each run fetches it once, into the gateway that holds the agent's network traffic. The sandbox only ever sees a placeholder that is worthless outside that run.
• The gateway allows one API path and one model. Key management, credits and every other path are refused.
• Hosts you did not declare are refused and recorded, so the agent cannot send anything to them.
• Disable or change the connection and the next run gets no key.

The agent runs on your own OpenRouter key and your provider bills you for its tokens. Set a credit limit on that key in OpenRouter as a second safety net.

Set one up

You need a test account in your product that the agent may change, an OpenRouter API key, and a verifier test.

1. Deploy the project once, so its environment exists (any waitcher.json will do). The deploy apply output names the project and environment IDs.

2. Create a model connection in that environment. Write the configuration and keep the key in a private file:

{
  "name": "Agent checks",
  "provider": "openrouter",
  "projectId": "PROJECT_ID",
  "environmentId": "ENVIRONMENT_ID",
  "models": ["anthropic/claude-sonnet-5.5"],
  "enabled": true
}

waitcher connection create --file connection.json --credential-file ./private/openrouter-key

3. Store the test account's credentials as environment secrets: one the agent gets, one only the verifier gets.

waitcher secret set --env production --name CHECK_AGENT_TOKEN --value-file ./private/agent-token
waitcher secret set --env production --name CHECK_VERIFIER_TOKEN --value-file ./private/verifier-token

4. Add the check to waitcher.json. It is an api entry with an agent block:

{
  "key": "agent-creates-project",
  "type": "api",
  "entry": "checks/agent-creates-project.ts",
  "interval": 86400,
  "timeout": 30,
  "settings": { "envNames": ["CHECK_AGENT_TOKEN", "CHECK_VERIFIER_TOKEN"] },
  "agent": {
    "connectionId": "CONNECTION_ID",
    "configurationRevision": 1,
    "model": "anthropic/claude-sonnet-5.5",
    "task": "Install the Acme CLI as https://docs.acme.dev describes, sign in with the token in $CHECK_AGENT_TOKEN and create a project named agent-check. Nobody will answer questions; stop with a short summary.",
    "hosts": ["docs.acme.dev", "registry.npmjs.org"],
    "envNames": ["CHECK_AGENT_TOKEN"]
  }
}

5. Write the verifier. It checks the outcome through your API and resets the test account for the next run:

import { test, expect } from "e2e";

const api = `${process.env.TARGET_URL}api/projects`;
const headers = { authorization: `Bearer ${process.env.CHECK_VERIFIER_TOKEN}` };

test("the agent created the project", async () => {
  const projects = await (await fetch(api, { headers })).json();
  const created = projects.find(
    (p: { name: string }) => p.name === "agent-check",
  );
  expect(created).toBeTruthy();
  await fetch(`${api}/${created.id}`, { method: "DELETE", headers });
});

6. Plan, apply and resume as with any monitor (Repository workflow):

waitcher deploy plan --env production
waitcher deploy apply --env production
waitcher monitor resume --id MONITOR_ID

A run takes longer than a script: the agent's time plus two sandbox starts. waitcher run wait waits at most 180 seconds at a time, so call it again until the run completes.

A single check outside a repository works too: put the same fields, plus "schemaVersion": 2, the context with the project and environment IDs, and the built script, in a definition for waitcher monitor apply --file (Monitors).

Writing a good task

• Write what a user would type, not a script: "Set up uptime monitoring for our site with Acme", not "run acme init then acme deploy".
• Say where the docs are if a user would, and where the credentials are. Never put a secret in the task; name the CHECK_* variable instead.
• End with "Nobody will answer questions; stop with a short summary." The agent runs headless and would otherwise ask.
• Keep one outcome per check, one the verifier can read back. Two to five checks of your most common agent tasks cover a lot.
• Expect some variance. One failed run is a signal to read the transcript; the monitor's failure threshold (default two runs in a row) decides when it becomes an incident.

Reading the results

The run message reads like Agent success: 8 steps, 9 model requests, 139346 tokens. Verifier: 1 e2e test passed. Next to the verifier's usual report, the evidence holds three files:

The first failed command and the final message usually point straight at the docs page or CLI message to fix. A rise in steps or tokens with the same outcome is an early sign that something got harder to follow. Download a file with waitcher artifact get --id RUN_ID --path .e2e/agent-transcript.txt --file transcript.txt, or open it from the run in the web app (Results and evidence).

Limits

The gateway stops the agent at its token limit, which it can pass by one model request, since a request's size is known only once the provider answers. In our own runs one attempt of a typical docs task used 60,000 to 180,000 tokens, 0.04 to 0.08 USD with Claude Sonnet 5.5.

Agent checks are configured from waitcher.json or waitcher monitor apply; the web app shows their runs and evidence but has no form for the agent block, so edit them where you created them.