GUIDE · AI TEST AUTOMATION
AI test automation: how agents write, run and maintain your tests
Scripted automation runs the steps someone wrote. AI test automation also decides what to test, writes the steps, keeps them working when the product changes and explains why a run failed. This guide covers what that means in practice, where it goes wrong and what to ask before you buy.
Last updated:
Definition
AI test automation is the use of large language models and AI agents to design, write, run and maintain software tests. Instead of a person scripting every step, agents read requirements and code, produce test cases with expected results, execute them on browsers, devices or APIs, repair broken steps and report failures with evidence.
Scripted, AI-assisted and agentic automation
Most tools sold as AI testing sit somewhere on this scale. The difference is how much of the work the person still does.
| Topic | Scripted automation | AI-assisted automation | Agentic test automation |
|---|---|---|---|
| Who decides what to test | Scripted automationA test engineer | AI-assisted automationA test engineer, with suggestions | Agentic test automationAgents, from requirements, code and tickets |
| How a test is written | Scripted automationCode (Selenium, Appium, Playwright) | AI-assisted automationRecorder or low-code, with AI locators | Agentic test automationPlain-language prompt or generated case |
| When the UI changes | Scripted automationThe script breaks until someone fixes it | AI-assisted automationLocators heal; flows still break | Agentic test automationThe agent re-plans the step and records why |
| When a run fails | Scripted automationSomeone reads logs | AI-assisted automationScreenshots and a short summary | Agentic test automationClassified result, video, reasoning and a filed defect |
| What it costs to run | Scripted automationInfrastructure only | AI-assisted automationInfrastructure and licence | Agentic test automationInfrastructure plus model tokens, which need limits |
What AI does at each stage of testing
AI test automation is not one feature. It changes each stage of the testing lifecycle in a different way, and each stage needs different evidence before you trust it.
- 01 · Design
TestingOps
Agents read requirements, repositories and tickets, design business scenarios and turn them into step-by-step cases. Each step cites the document or code it is based on.
TestingOps in detail - 02 · Devices
DeviceOps
Mobile tests need the phones your users have. Virtual Android devices booted from official OEM firmware, and digital twins of real phones, make that matrix reproducible.
DeviceOps in detail - 03 · Execution
RunningOps
Tests written as prompts run on web, Android, iOS and API targets. An agent drives the app, retries engine faults and records screenshots, video and its reasoning.
RunningOps in detail - 04 · Coverage
CoverOps
Runtime coverage shows which lines each automated or manual test executed, so after a change you re-run the tests that reach the changed code instead of the whole suite.
CoverOps in detail - 05 · Production feedback
ReplayOps
Session replay shows where real users get stuck. Those sessions point to the flows that need tests next.
ReplayOps in detail
Where AI test automation goes wrong, and what to ask
The same model that writes a useful test can also write a confident wrong one. These are the failure modes we see most often, with the question that exposes each one in a demo.
01Invented steps
Risk
The model writes a plausible flow for a screen or rule that does not exist, and the test passes against nothing.
Ask the vendor
Show me the source each step is based on. What happens when there is no evidence?
02Fake green
Risk
A self-healing agent works around a real bug and reports a pass.
Ask the vendor
How do you tell a product failure from an environment or platform fault? Can a healed step hide a defect?
03Uncontrolled model cost
Risk
Every run calls the model for every step, so a large regression suite becomes a large token bill.
Ask the vendor
What does one step cost? Do unchanged passing tests run without tokens? Is there a monthly cap?
04Data leaving your network
Risk
Screens, test data and source code are sent to a model provider you did not choose.
Ask the vendor
Can we bring our own model or run one on-prem? Can the whole platform run inside our network?
05Lock-in
Risk
Tests exist only as prompts inside one vendor's platform.
Ask the vendor
Can a passing scenario be exported as Playwright, Selenium or another framework we already run?
Where to start
You do not need the whole loop on day one. Start with the problem that costs your team the most and measure it before and after.
“We have no capacity to write tests and the backlog keeps growing.”
Let agents produce grounded cases for one product area and compare them with what your team would have written.
TestingOps“Script maintenance eats our week and mobile coverage is thin.”
Move one flaky regression flow to prompt-driven runs on real and virtual devices and track how often it breaks.
RunningOps“Regression takes hours and we do not know what it covers.”
Attach coverage agents to one service and see which tests actually reach the code that changed.
CoverOps“We cannot see why users get stuck in production.”
Record sessions on one journey and turn the worst-scoring ones into tickets and new test cases.
ReplayOps
Frequently asked questions
- What is AI test automation?
- It is the use of large language models and AI agents to design, write, run and maintain software tests. Agents read requirements and code, produce cases with expected results, execute them on browsers, devices or APIs, repair broken steps and report failures with screenshots, video and their reasoning.
- Does AI test automation replace QA engineers?
- No. It removes the repetitive part of the job: writing and repairing scripts, re-running regressions and filing defects. People still decide what risk matters, review what agents produce and own the release decision.
- Is it reliable enough for regression testing?
- It is when every result is classified honestly and every step is grounded in evidence. Ask whether the tool separates product failures from environment and platform faults, and whether a passing scenario can replay without calling the model again.
- How is it different from self-healing locators?
- Self-healing locators fix one broken selector. An agent re-plans a whole step from the goal of the test, so it survives redesigns, but it must record why it changed the step so a real bug is not hidden.
- Can we keep our Selenium, Appium or Playwright tests?
- Yes. Existing suites keep running in your pipeline, and a passing prompt-driven scenario can be exported as a Playwright script or to Selenium, Testinium (Gauge) and Maestro.
- Can AI test automation run on-premise?
- Yes. Cunda's products run in the cloud, in a private cloud or on-prem, and you choose the model: Anthropic Claude, OpenAI, Azure OpenAI, Google Gemini, open-weight models or a self-hosted endpoint.