How do you test your agents?

We're building Ancha to help teams test AI agents before they reach customers. Tell us how you test today and what's hard about it.

Your agent

1What does your main agent do?Choose one

Customer support Sales or SDR outreach On-call or incident response Coding or pull requests Back-office work (finance, ops, HR) Legal or compliance Personal or executive assistant Something else

2Where is it today?Choose one

A prototype Piloting with a few customers In production with customers We're not building one yet

3How does it reach the systems it works in?Select all that apply

Our own tool functions over our own API clients Vendor SDKs or REST APIs, called directly (Stripe, Salesforce, Slack…) Vendor MCP servers Our own MCP server A browser or computer use Not sure

4How many external systems does one workflow touch?Choose one

One Two or three Four to six Seven or more

5Who else is involved besides the agent?Select all that apply

End customers or users Approvers or managers Teammates or colleagues Outside parties (prospects, vendors, partners) Other agents Nobody else

6Does a workflow wait on things that happen later?Select all that apply

Webhooks or events from other systems Human approvals Replies that take hours or days Schedules, deadlines or follow-ups CI, deploys or other long-running jobs No, it finishes in one session

How you test today

7How do you test it today?Select all that apply

Manual runs and reading transcripts Unit tests with mocked tools Recorded responses (cassettes or replays) Vendor sandboxes or test accounts A staging copy of our own systems LLM-simulated users An eval platform (Braintrust, LangSmith, Langfuse, Arize…) Production monitoring only Nothing systematic yet

8How do you decide whether a run passed?Select all that apply

We check the end state of the systems An LLM judge reads the transcript A person reviews it Assertions on tool calls or steps We mostly don't grade yet

9How often do you run your agent tests?Choose one

On every pull request Daily or nightly Before each release When something changes We don't run them regularly

10How do you test flows with several people or long waits?Choose one

We don't test them By hand, step by step With scripted fakes With LLM-simulated people With real colleagues in staging Our agent has no such flows

What's hard

11What's hardest about testing your agent?Pick up to three

Building realistic data and state Simulating the people on the other side External APIs: rate limits, errors, webhooks Anything that happens over time Reproducing a production failure Flaky, nondeterministic results Deciding pass or fail The cost of running evals Keeping tests current as the agent changes

12What kind of production failure has cost you the most?Choose one

It broke a business rule (a refund, a discount, a permission) It mishandled an external error or outage It acted on the wrong account or record Something was lost between people: it didn't ask, relay or follow up It missed a deadline or a follow-up It followed injected or malicious instructions It told someone something false No serious failure yet

13What do you spend on evals each month, tools and model calls together?Choose one

Nothing yet Under $500 $500 to $2,000 $2,000 to $10,000 Over $10,000 Not sure

What would help

14Which of these would you use?Select all that apply

Simulated customers, approvers and reviewers that react, and reveal facts only when asked A test world that passes messages and events between those people over time Stateful fakes of the vendor APIs we call Fault and attack injection: 429s, outages, prompt injection Pass or fail from the systems' end state, including whether the right people were told Turning a production failure into a repeatable test Reports across repeated runs (pass@k, pass^k)

15Would you talk with us for 20 minutes?Choose one

Yes Maybe later No

About you

Name Work email That looks like a personal address. A work email helps us follow up.

Website

We use your answers to decide what to build. We won't share your name or email, and we'll only contact you if you said yes or maybe to a call.

Thank you.

We read every answer.