Agent testing API

Use the testing API to maintain a workspace test library and run it against an agent before publishing changes. Tests use the same configuration and results as Tests in the dashboard. The base URL is https://app.cntillion.com/api/v2. Authenticate with a workspace key from Workspace settings → API keys. A Read only key can read tests, transcripts, and results. Creating or changing tests, starting runs, and canceling them requires a Full access key. Keys can access only their own workspace. The Developer API covers key expiry, revocation, pagination, errors, and the 100-request-per-10-second rate limit.

Create a reusable test

curl 'https://app.cntillion.com/api/v2/tests' \
  -H "Authorization: Bearer $CENTILLION_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "configuration": {
      "name": "Explains the return window",
      "testType": "simulation",
      "simulatedUser": "A customer asks whether an order placed 12 days ago can be returned.",
      "successCriterion": "Explains the return policy accurately without inventing order details.",
      "maxTurns": 5,
      "expectedActions": [],
      "actionResponses": [],
      "mockStrategy": "all",
      "mockFallback": "error"
    }
  }'

The response is 201 with data.id, data.configuration, data.revision, and creator metadata. Simulations can run for 1 to 50 turns. testType can also be next_reply or tool_invocation; both take a messages context transcript ending with a user message. Read a saved test’s configuration to copy or modify it. Configuration supports multiple criteria, success/failure examples, JSON dynamic variables, simulator/evaluator models, workflow starting nodes and transition assertions, and mock strategies with conditional overrides. Each tool parameter can use anything, exact, regex, or llm validation. Use action IDs returned by the agent’s test listing, including distinct IDs for each MCP subtool.

Library and agent suites

Method	Path	Behavior
GET	`/tests`	List workspace tests with `limit`, `cursor`, `search`, `folderId`, and `filter`
POST	`/tests`	Create a reusable test
GET	`/tests/{testId}`	Read configuration, revision, and creator
PUT or PATCH	`/tests/{testId}`	Update configuration and optional folder using the current revision
DELETE	`/tests/{testId}`	Remove the library entry; attached agent tests keep their configuration and history
GET / POST	`/test-folders`	List folders, or create one with `{ "name": "Checkout" }`
POST	`/tests/move`	Move `{ "testIds": ["test_id"], "folderId": "folder_id" }`; use `null` for the library root
GET	`/agents/{agentId}/tests`	Read the agent’s suite, action catalog, and credits
POST	`/agents/{agentId}/tests`	Create an agent-local test with `{ "configuration": { ... } }`
GET	`/agents/{agentId}/tests/{testId}`	Read one attached or local test
PUT or PATCH	`/agents/{agentId}/tests/{testId}`	Update an agent test with configuration and revision
DELETE	`/agents/{agentId}/tests/{testId}`	Remove this agent’s attachment/local test and its run history
POST	`/agents/{agentId}/tests/attach`	Attach `{ "testIds": ["workspace_test_id"] }` and return agent test IDs

Library filter values are all, created-by-me, and shared-with-me, relative to the API key’s creator. Older tests without creator metadata appear under shared tests. The workspace library allows 500 tests, and each agent can have 200 attached or local tests. Updates include configuration and the latest revision. A stale revision returns 409. Updating a shared test changes its definition for every attached agent. Wait for active runs to finish before editing a test. A test or folder ID outside the workspace returns 404 TEST_NOT_FOUND or FOLDER_NOT_FOUND.

Start a run

curl 'https://app.cntillion.com/api/v2/agents/YOUR_AGENT_ID/test-runs' \
  -H "Authorization: Bearer $CENTILLION_API_KEY" \
  -H 'Content-Type: application/json' \
  -H 'Idempotency-Key: release-check-4821' \
  -d '{
    "testIds": ["WORKSPACE_TEST_ID"],
    "source": "workspace",
    "repeats": 3,
    "allowMessageCredits": false,
    "overrides": {
      "channel": "chat-bubble",
      "target": { "type": "draft" },
      "dynamicVariables": [{ "name": "customer_name", "value": "Sam" }]
    }
  }'

source defaults to agent, which takes IDs from the agent’s suite. The workspace source attaches missing library tests to the selected agent before queuing them. repeats accepts every integer from 1 through 20, subject to a maximum of 200 conversations per request. A run requires an Idempotency-Key of 1 to 128 characters. Retry the same logical request with the same key and body after a network failure. Replays return the original response for 24 hours with Idempotent-Replayed: true and create no additional conversations. Reusing the key with another body returns 409 IDEMPOTENCY_KEY_REUSED. A successful request returns 202:

{
  "data": {
    "requiresConfirmation": false,
    "batchIds": ["RUN_BATCH_ID"],
    "skipped": 0
  }
}

Tests already running are skipped. Check skipped when a release check requires every selected test to run. If message credits would be needed, the API instead returns 409 with data.requiresConfirmation: true, data.estimate, and data.credits. Review the estimate, then explicitly set allowMessageCredits: true and use a new idempotency key to proceed. Execution overrides select a simulated channel, named environment, target, agent configuration, starting workflow node, or dynamic variables. Targets are draft, published, branch, or version; branch and version targets require id. Branch targets use the branch’s current published version. Named environments belong to the test configuration and can provide variables and secure HTTP/MCP connection resource mappings. Each run stores a private configuration and tool snapshot; public results omit its stored credentials.

Read results and cancel runs

Method	Path	Behavior
GET	`/agents/{agentId}/test-runs`	Cursor-paginated runs, optionally filtered by agent `testId`
GET	`/agents/{agentId}/test-runs/{batchId}`	Configuration, transcripts, tool evidence, verdicts, and result groups
GET	`/agents/{agentId}/test-suites/{suiteId}`	Every test in one run, with verdicts and result groups. `suiteId` comes from the run response.
POST	`/agents/{agentId}/test-suites/{suiteId}/cancel`	Stop every pending conversation in the run.
POST	`/agents/{agentId}/test-suites/{suiteId}/resubmit`	Run the failed tests again, or the `testIds` you name, at the run’s repeat count. Requires `Idempotency-Key`.
POST	`/agents/{agentId}/test-runs/{batchId}/cancel`	Stop pending conversations and return their count

Resubmitting needs an Idempotency-Key and takes optional testIds, allowMessageCredits and overrides. It starts a new run with its own suiteId. Without testIds it reruns the tests that failed, and returns 409 NOTHING_TO_RESUBMIT when none did. A test from outside the run returns 404 TEST_NOT_IN_SUITE. Poll each returned batch ID until all conversations leave QUEUED or IN_PROGRESS. Terminal verdicts are PASSED, FAILED, and NO_VERDICT. Treat both failures and no-verdict results as a failed release check. groupingStatus is pending, in_progress, or completed. Once complete, resultGroups contains shared failure reasons and their runIds. Groups retain every conversation’s original verdict. If semantic summarization cannot complete, exact-reason groups remain available. Cancellation invalidates worker leases, preserves completed verdicts, and retains usage already recorded.

JavaScript client and CI

The Centillion Labs repository includes the dependency-free JavaScript client scripts/agent-testing-client.mjs and CLI scripts/agent-testing-cli.mjs for Node.js 22 or later. Store CENTILLION_API_KEY in your CI secret store.

node scripts/agent-testing-cli.mjs run --agent YOUR_AGENT_ID \
  --repeats 3 --output test-results.json

The CLI runs all attached tests, waits for verdicts and result groups, and writes JSON evidence. Select specific tests with repeated --test TEST_ID flags. Use --source workspace, --settings overrides.json, or a stable --idempotency-key when needed. CENTILLION_API_URL selects another API base URL, including its path, for example https://staging.example.com/api/v2. Exit codes are 0 when every selected conversation passes, 1 for failed, no-verdict, or skipped tests, and 2 for request or timeout errors. The default timeout is 1,800 seconds, configurable with --timeout; a timeout attempts to cancel pending conversations. --no-wait returns queued batch IDs immediately. Use --help for library CRUD, results, and cancellation commands. Saved evidence contains transcripts and tool inputs. Restrict access to those CI artifacts.

Get started

Build your agent

Reach your customers

Website development

Customer data

Win customers back

Support

Workspace

Guides

API documentation

Helpdesk API

Developer guides

Create a reusable test

Library and agent suites

Start a run

Read results and cancel runs

JavaScript client and CI

​Create a reusable test

​Library and agent suites

​Start a run

​Read results and cancel runs

​JavaScript client and CI

Create a reusable test

Library and agent suites

Start a run

Read results and cancel runs

JavaScript client and CI