AI testing is only as good as its knowledge base: RAG, test standards and summarized code

A model that writes tests knows only what you give it. Two teams can use the same model and get very different tests, because the difference is rarely the model. It is the knowledge behind it: requirements, business rules, documents, tickets and code. Retrieval-augmented generation, or RAG, is how that knowledge reaches the model at the moment it writes a test. Feed it badly and the agent writes confident, wrong tests.

Three things decide whether AI testing works in practice: a knowledge base that is fed correctly, tests that follow your test standards, and source code that is summarized before it reaches the model.

1. The knowledge base is your test basis

In testing terms, the knowledge base is the test basis: the material tests are derived from. For AI testing it should hold:

• Requirements and acceptance criteria, with their version.

• Business rules, limits and error messages, written down rather than remembered.

• API contracts and screen flows.

• Past defects and how they were fixed, because bugs cluster.

• Existing test cases, so the agent extends coverage instead of repeating it.

What you leave out matters as much. Outdated documents, abandoned drafts and three versions of the same rule give retrieval three answers, and the agent may pick the wrong one. Mark which source is authoritative, date every document and remove what is no longer true.

2. Feed RAG correctly

A knowledge base only helps if retrieval returns the right piece at the right time. A few rules make the difference:

• Split by meaning, not by size. One requirement, one endpoint or one screen per chunk keeps a rule and its context together. Fixed-size chunks cut rules in half.

• Add metadata. Product area, version, source type and owner let retrieval filter before it ranks.

• Cite the source in every step. If a step cannot point to the chunk it came from, it should not be in the test. When retrieval returns nothing, the agent should say so instead of guessing.

• Measure retrieval. Take a sample of requirements and check whether the right chunk comes back. If it does not, better prompts will not fix the tests.

• Keep it current. Re-index when a ticket changes or code is merged. A stale index tests last quarter's product.

3. Test standards still apply

A test written by an agent should meet the same standard as one written by a person. If your organization follows ISO/IEC/IEEE 29119 or ISTQB practice, put that standard into the knowledge base and into the agent's instructions, not into a reviewer's head.

• Every case has an identifier, preconditions, steps, an expected result and a link to the requirement it covers.

• Design techniques are chosen on purpose: boundary values for limits, equivalence classes for inputs, decision tables for rule combinations, state transitions for flows. A model left alone writes the happy path.

• Naming, priority and templates match what your test management tool expects.

• In regulated sectors, every change to a case and every run keeps an audit trail.

Then review a sample of generated cases against the same checklist you use for human ones.

4. You cannot give the model the whole codebase

Even a large context window cannot hold a real product's repositories, and a model given too much text loses the details in the middle. Source code has to be summarized before it is used for testing.

• Summarize in layers: repository, then service or module, then file. For each module record its purpose, public interfaces, data models, business rules, dependencies and error handling.

• Keep the facts tests need. A summary that drops "total greater than or equal to the limit goes to review" also drops the boundary test. Validation rules, limits, status codes and error messages must survive summarization.

• Update only what changed. Re-summarize the files a commit touched, not the whole repository.

• Combine summary and source. When writing a test, give the model the summaries of the relevant modules plus the actual code of the functions the change touched.

• Use runtime data to choose. Runtime coverage shows which code a flow really executes, which is a better guide to what belongs in the context than file names.

How we apply this at Cunda

TestingOps agents read your repositories, trackers and documents, and ground every test step on what they find there. Each step cites its evidence, and when the evidence is missing the agent says so instead of guessing. A knowledge graph links requirements to cases and cases to results, so you can see what is covered and what never ran. CoverOps records which lines each test executed on running services, so you can see which code a flow actually reaches. Both run in the cloud, in a private cloud or on-prem, so the knowledge base can stay inside your network.

A short checklist

• Is there one authoritative, dated source for each requirement and rule?

• Is the knowledge base split by meaning and tagged with metadata?

• Does every generated step cite its source?

• Is your test standard written into the agent's instructions?

• Is source code summarized in layers, with limits and rules kept intact?

• Is the index updated when tickets and code change?

For the wider picture, read our AI test automation guide.