> ## Documentation Index
> Fetch the complete documentation index at: https://cntillion.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Agent testing API

> Manage reusable tests, run agent suites, and use verdicts and grouped results in CI

Use the testing API to maintain a workspace test library and run it against an
agent before publishing changes. Tests use the same configuration and results
as [Tests in the dashboard](https://cntillion.com/docs/agents/tests.md).

The base URL is `https://app.cntillion.com/api/v2`. Authenticate with a workspace
key from **Workspace settings → API keys**. A **Read only** key can read tests,
transcripts, and results. Creating or changing tests, starting runs, and
canceling them requires a **Full access** key. Keys can access only their own
workspace. The [Developer API](https://cntillion.com/docs/api-reference/developer-api.md) covers key expiry,
revocation, pagination, errors, and the 100-request-per-10-second rate limit.

## Create a reusable test

```bash theme={null}
curl 'https://app.cntillion.com/api/v2/tests' \
  -H "Authorization: Bearer $CENTILLION_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "configuration": {
      "name": "Explains the return window",
      "testType": "simulation",
      "simulatedUser": "A customer asks whether an order placed 12 days ago can be returned.",
      "successCriterion": "Explains the return policy accurately without inventing order details.",
      "maxTurns": 5,
      "expectedActions": [],
      "actionResponses": [],
      "mockStrategy": "all",
      "mockFallback": "error"
    }
  }'
```

The response is `201` with `data.id`, `data.configuration`, `data.revision`, and
creator metadata. Simulations can run for 1 to 50 turns. `testType` can also be
`next_reply` or `tool_invocation`; both take a `messages` context transcript
ending with a user message.

Read a saved test's `configuration` to copy or modify it. Configuration supports
multiple criteria, success/failure examples, JSON dynamic variables,
simulator/evaluator models, workflow starting nodes and transition assertions,
and mock strategies with conditional overrides. Each tool parameter can use
`anything`, `exact`, `regex`, or `llm` validation. Use action IDs returned by the
agent's test listing, including distinct IDs for each MCP subtool.

## Library and agent suites

| Method | Path | Behavior |
| - | - | - |
| GET | `/tests` | List workspace tests with `limit`, `cursor`, `search`, `folderId`, and `filter` |
| POST | `/tests` | Create a reusable test |
| GET | `/tests/{testId}` | Read configuration, revision, and creator |
| PUT or PATCH | `/tests/{testId}` | Update configuration and optional folder using the current revision |
| DELETE | `/tests/{testId}` | Remove the library entry; attached agent tests keep their configuration and history |
| GET / POST | `/test-folders` | List folders, or create one with `{ "name": "Checkout" }` |
| POST | `/tests/move` | Move `{ "testIds": ["test_id"], "folderId": "folder_id" }`; use `null` for the library root |
| GET | `/agents/{agentId}/tests` | Read the agent's suite, action catalog, and credits |
| POST | `/agents/{agentId}/tests` | Create an agent-local test with `{ "configuration": { ... } }` |
| GET | `/agents/{agentId}/tests/{testId}` | Read one attached or local test |
| PUT or PATCH | `/agents/{agentId}/tests/{testId}` | Update an agent test with configuration and revision |
| DELETE | `/agents/{agentId}/tests/{testId}` | Remove this agent's attachment/local test and its run history |
| POST | `/agents/{agentId}/tests/attach` | Attach `{ "testIds": ["workspace_test_id"] }` and return agent test IDs |

Library `filter` values are `all`, `created-by-me`, and `shared-with-me`, relative
to the API key's creator. Older tests without creator metadata appear under
shared tests. The workspace library allows 500 tests, and each agent can have
200 attached or local tests.

Updates include `configuration` and the latest `revision`. A stale revision
returns 409. Updating a shared test changes its definition for every attached
agent. Wait for active runs to finish before editing a test. A test or folder
ID outside the workspace returns 404 `TEST_NOT_FOUND` or `FOLDER_NOT_FOUND`.

## Start a run

```bash theme={null}
curl 'https://app.cntillion.com/api/v2/agents/YOUR_AGENT_ID/test-runs' \
  -H "Authorization: Bearer $CENTILLION_API_KEY" \
  -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: release-check-4821' \
  -d '{
    "testIds": ["WORKSPACE_TEST_ID"],
    "source": "workspace",
    "repeats": 3,
    "allowMessageCredits": false,
    "overrides": {
      "channel": "chat-bubble",
      "target": { "type": "draft" },
      "dynamicVariables": [{ "name": "customer_name", "value": "Sam" }]
    }
  }'
```

`source` defaults to `agent`, which takes IDs from the agent's suite. The
`workspace` source attaches missing library tests to the selected agent before
queuing them. `repeats` accepts every integer from 1 through 20, subject to a
maximum of 200 conversations per request.

A run requires an `Idempotency-Key` of 1 to 128 characters. Retry the same
logical request with the same key and body after a network failure. Replays
return the original response for 24 hours with `Idempotent-Replayed: true` and
create no additional conversations. Reusing the key with another body returns
`409 IDEMPOTENCY_KEY_REUSED`.

A successful request returns `202`:

```json theme={null}
{
  "data": {
    "requiresConfirmation": false,
    "batchIds": ["RUN_BATCH_ID"],
    "skipped": 0
  }
}
```

Tests already running are skipped. Check `skipped` when a release check requires
every selected test to run. If message credits would be needed, the API instead
returns `409` with `data.requiresConfirmation: true`, `data.estimate`, and
`data.credits`. Review the estimate, then explicitly set
`allowMessageCredits: true` and use a new idempotency key to proceed.

Execution overrides select a simulated channel, named environment, target,
agent configuration, starting workflow node, or dynamic variables. Targets are
`draft`, `published`, `branch`, or `version`; branch and version targets require
`id`. Branch targets use the branch's current published version. Named
environments belong to the test configuration and can provide variables and
secure HTTP/MCP connection resource mappings. Each run stores a private
configuration and tool snapshot; public results omit its stored credentials.

## Read results and cancel runs

| Method | Path | Behavior |
| - | - | - |
| GET | `/agents/{agentId}/test-runs` | Cursor-paginated runs, optionally filtered by agent `testId` |
| GET | `/agents/{agentId}/test-runs/{batchId}` | Configuration, transcripts, tool evidence, verdicts, and result groups |
| GET | `/agents/{agentId}/test-suites/{suiteId}` | Every test in one run, with verdicts and result groups. `suiteId` comes from the run response. |
| POST | `/agents/{agentId}/test-suites/{suiteId}/cancel` | Stop every pending conversation in the run. |
| POST | `/agents/{agentId}/test-suites/{suiteId}/resubmit` | Run the failed tests again, or the `testIds` you name, at the run's repeat count. Requires `Idempotency-Key`. |
| POST | `/agents/{agentId}/test-runs/{batchId}/cancel` | Stop pending conversations and return their count |

Resubmitting needs an `Idempotency-Key` and takes optional `testIds`,
`allowMessageCredits` and `overrides`. It starts a new run with its own
`suiteId`. Without `testIds` it reruns the tests that failed, and returns 409
`NOTHING_TO_RESUBMIT` when none did. A test from outside the run returns 404
`TEST_NOT_IN_SUITE`.

Poll each returned batch ID until all conversations leave `QUEUED` or
`IN_PROGRESS`. Terminal verdicts are `PASSED`, `FAILED`, and `NO_VERDICT`.
Treat both failures and no-verdict results as a failed release check.

`groupingStatus` is `pending`, `in_progress`, or `completed`. Once complete,
`resultGroups` contains shared failure reasons and their `runIds`. Groups retain
every conversation's original verdict. If semantic summarization cannot
complete, exact-reason groups remain available. Cancellation invalidates worker
leases, preserves completed verdicts, and retains usage already recorded.

## JavaScript client and CI

The Centillion Labs repository includes the dependency-free JavaScript client
`scripts/agent-testing-client.mjs` and CLI `scripts/agent-testing-cli.mjs` for
Node.js 22 or later. Store `CENTILLION_API_KEY` in your CI secret store.

```bash theme={null}
node scripts/agent-testing-cli.mjs run --agent YOUR_AGENT_ID \
  --repeats 3 --output test-results.json
```

The CLI runs all attached tests, waits for verdicts and result groups, and
writes JSON evidence. Select specific tests with repeated `--test TEST_ID`
flags. Use `--source workspace`, `--settings overrides.json`, or a stable
`--idempotency-key` when needed. `CENTILLION_API_URL` selects another API base
URL, including its path, for example `https://staging.example.com/api/v2`.

Exit codes are 0 when every selected conversation passes, 1 for failed,
no-verdict, or skipped tests, and 2 for request or timeout errors. The default
timeout is 1,800 seconds, configurable with `--timeout`; a timeout attempts to
cancel pending conversations. `--no-wait` returns queued batch IDs immediately.
Use `--help` for library CRUD, results, and cancellation commands. Saved evidence
contains transcripts and tool inputs. Restrict access to those CI artifacts.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.