How to compare AI testing tools: 10 questions to ask in a demo
Almost every testing tool now says it uses AI. These ten questions show the differences in a demo: where each step comes from, whether a green result can be trusted, what a run costs and where your data goes.
Almost every testing tool now says it uses AI. Some write tests from requirements, some heal broken locators, some let you describe a test in plain language and run it on a device. In a sales deck the labels look alike. The differences show up when you ask the right questions in a demo. Here are the ten we would ask, grouped by what they protect you from.
Is the test grounded in something real?
Where does each step come from? Ask the vendor to open a generated case and show the requirement, ticket, document or code each step is based on. A model that writes without sources will invent screens and rules that do not exist.
What happens when there is no evidence? A good agent says it could not find the rule. A weak one guesses, and the guess turns into a test that passes against nothing.
Who checks the expected result? Expected results should be verified against the live product, not copied from the prompt.
Can you trust a green result?
How are failures classified? A product bug, a broken test environment and a platform fault need different owners. If they all show up as "failed", your team spends its mornings sorting them out.
Can self-healing hide a bug? When an agent re-plans a step because the screen changed, it should record what it changed and why. Ask to see that record.
What evidence comes with a run? Screenshots, video, logs and the agent's reasoning for each step let you reproduce a failure without running it again.
What will it cost to run every day?
What does one step cost, and where is that shown? Model calls are a running cost. You should see tokens and money per step and per suite, and be able to set a monthly cap.
Do unchanged passing tests call the model again? A regression suite runs the same scenarios hundreds of times. Replaying a passing scenario as a script, with no model call, is the difference between a predictable bill and a growing one.
Where does your data go, and can you leave?
Which model runs the tests, and where? Screens, test data and sometimes source code reach the model. Ask whether you can use your own keys, a self-hosted model, a private cloud or an on-prem installation.
Can tests leave the platform? If a passing scenario can be exported as Playwright, Selenium or another framework you already run, you are not locked in.
How Cunda answers these questions
We built TestingOps and RunningOps around these questions. In TestingOps every step cites its evidence, and when the agents find none they say so. In RunningOps each run ends as passed, test failure, environment fault or platform fault. Every step shows its tokens and cost, scenarios that already passed replay with zero tokens, and a passing scenario can be exported as a Playwright script or to Selenium, Testinium (Gauge) and Maestro. Both run in the cloud, in a private cloud or on-prem, with the model you choose.
For the wider picture, from definitions to where to start, read our AI test automation guide.