Testing a System I Didn't Understand: What Five Weeks with an Agentic QA Framework Taught Me
There is an uncomfortable truth every test automation engineer knows but rarely says out loud: we automate best what we understand best. Give me a checkout flow and I'll give you a hundred meaningful tests before lunch. Give me a loan back-office platform, where a single test step reads "verify the restructured loan's due date follows the configured due-date rule after a term change", and I'm suddenly the most expensive junior on the team.
Five weeks ago I was handed exactly that. Six feature areas of a complex financial back-office system. Around 226 test cases written by business analysts, in the language of business analysts: restructurings, debt forgiveness, balloon payments, accrual adjustments, multi-account modifications. I didn't know the business. I still don't know all of it.
What I did have was an agentic QA framework, the skills system I wrote about in my last article, and a question I wanted to answer honestly:
Can an agentic framework deliver real test value on an application when neither the agent nor the engineer driving it fully understands the business?
This is the story of what worked, what didn't, and what I'd change.
The Problem: Domain Knowledge Is the Real Bottleneck
Classic test automation assumes a pipeline that looks like this:
Business understanding → test design → automation → execution → analysis
The expensive part was never the automation. It was the first box. In a domain like lending, that first box is measured in months: learning the product vocabulary, why a back-dated disbursement fails on a public holiday, why interest is capitalised in one repayment method and billed in another.
The test cases don't help as much as you'd hope. They're written by people who do know the business, so they leave out everything "obvious". A typical expected result says "in accordance with the current data model, sheet X". That's a pointer to a spreadsheet, a requirements document, and twenty years of institutional memory. It isn't an oracle you can automate against.
So the bet was this: an agent doesn't need to understand the business if it can interrogate the system and the documents faster than I can learn them.
The Framework: Built for Honesty Before Speed
The framework is called Aletheia, Greek for "truth" or "unconcealment". The name was deliberate. When an agent writes your tests, the biggest risk isn't a slow test. It's a green test that proves nothing.
The pipeline:
Pull. Test cases come out of the test management tool read-only. The agent never writes back to the source of truth.
Materialise. A deterministic generator turns each test case into a spec skeleton: same input, same output, no LLM creativity at this stage.
Implement. An implement-testcase skill guides the agent in turning the skeleton into a real Playwright spec, wiring it into a suite manifest and running it.
Evidence. A custom reporter records every step: what was expected, what was observed, screenshots, traces, DB reads. Every claim in a report has evidence behind it.
Evaluate. A separate, read-only report-evaluator agent judges the run against the original test steps. It can draft findings, but it cannot edit specs or "fix" results.
Heal. A spec-healer agent fixes test-side failures only, under a guard that physically prevents it from touching shared code.
Two structural decisions mattered more than any prompt:
Decision | Why it matters |
Frozen zone vs disposable zone | Shared builders, page objects, and DB access are frozen. Agents can only adapt the specs themselves. A healer can't "fix" a failure by weakening the infrastructure every other test depends on. |
A step map per test case | Every original test step is mapped to the spec step that proves it. Coverage is measured against what the business asked for, not against what was easy to automate. |
And there was one rule above all others: never fabricate a pass. A red test with evidence is a valid, successful outcome. A step that can't be verified gets marked INCOMPLETE with a reason, never a quiet green.
Letting the System Be the Teacher
Here's where it got interesting. Since I couldn't supply the business knowledge, the agent had to extract it, and it turned out to be very good at that. Its sources, in the order it learned to trust them:
The database. What the system actually persisted, not what the API claimed to accept. Lookup tables, event rows, status codes, calendars.
The API's own type definitions. Which fields are writable and which are silently dropped. (This one alone resolved a "business question" that had been open for days. More on that below.)
The requirements documents and the data-model workbook. Read as reference only, with just the specific values a test needed transcribed.
Throwaway probes. Small scripts that open one unknown form on a fresh account and dump every field, button, banner, and network call. Mapping a form this way took minutes. Iterating the full test case against it would have taken hours.
The agent didn't come to understand lending the way an analyst does. It built something narrower and, for testing, often more useful: a precise, evidence-backed model of how this particular system behaves.
A note on data. All of this ran on Claude Enterprise, against a dedicated QA environment populated entirely with synthetic data. No client or customer data was involved at any point, and the QA environment has no connection to staging or production. Guardrails for data are part of the framework, not an afterthought.
Making Knowledge Survive the Session
The first two weeks taught me something painful. An agent that figures out something brilliant at 4pm has forgotten it by the next morning. Context windows get compacted. Sessions end. Memory recall is probabilistic.
I watched the agent rediscover the same facts three times. A date field that comes back from the API one day early because of a timezone conversion. A dropdown that only populates after another field is set. A key that must never be pressed because it dismisses the whole wizard.
The fix was the same principle from my Obsidian vault article, applied to a codebase: durable knowledge lives in git-tracked documents, and a short always-loaded index points at them. Each document opens with a one-line Purpose and a Read this when trigger, so the agent loads the right depth at the right moment:
Knowledge document | Read this when |
Data-model oracle | a step cites the data model, lookups, or transaction types |
Event-firing guide | a step asks you to fire a business event and you're about to mark it blocked |
Working-days calendar | a step needs a valid business date |
Known defects and app defects | a step is blocked and you suspect it's already known |
Execution strategy | a step is "pending BA" and the cause smells stale |
The knowledge base grew past 5,000 lines. That's not bureaucracy. It's the institutional memory a new team member would normally take six months to absorb, written down where the next session (human or agent) will find it.
The Results
Scope and numbers first, then my interpretation.
Scope
Metric | Value |
Duration | ~5 weeks |
Feature suites | 6 |
Business test cases pulled and implemented | ~226 |
Spec files | ~330 |
Findings. The suites surfaced a set of confirmed application defects, plus a further group of suspected ones with evidence attached and awaiting confirmation. Three "defects" turned out to be our bugs, and we kept them on record.
The findings were the kind that matter in a financial system: messages that differ from the specification, operations that leave data in an unexpected state, and flows that misbehave on a less common path. Finding them in a QA environment, before release, is exactly what that environment is for.
Not one of these was found because the agent "understood lending". They were found because the agent compared what the requirements said against what the system did, step by step, with evidence, and it did this relentlessly, across hundreds of steps, without getting bored.
Execution (step level)
Suite | Result |
Largest suite | 516 verified passing steps |
Another suite | 282 pass / 133 blocked / 0 fake passes found |
What "zero fake passes" means. It means none were found: the read-only evaluator checks every pass against its step map and the recorded evidence. That is still an agent checking an agent, so the next step is a human one. A manual tester will re-verify a random sample of test executions.
About the 133 blocked steps. That's roughly a third of that suite, and it deserves a straight answer. They fell into three groups. Some flows we simply didn't know, because the test cases left out data their experienced authors took for granted. Others waited on open business questions. The rest needed test data the environment didn't have.
What it cost
Value only means something next to cost, so here is the other side of the ledger. The tooling cost is simple: everything ran on a Claude Enterprise subscription, so there was no per-token bill. Routine tasks went to Sonnet 5.5 and the harder reasoning to Opus 4.8.
The real cost was my time: about eight hours a day for five weeks. Roughly half of each day went to evaluating results and rerunning cases. A quarter went to correcting and optimising the framework. The last quarter went to writing the markdown knowledge documents that let the agent remember the flows. The agent does the legwork, but it doesn't run unattended.
The Pros: Where the Agentic Approach Earned Its Keep
1. It turned "blocked pending BA" into answers. This was the biggest surprise. Dozens of steps were parked as "blocked: needs BA decision" or "environment limitation". When we challenged them, a pattern emerged: the blocker was almost never the environment or a business decision. It was a fixture artefact or a stale note. The A/B/C/D unblock playbook below (cheapest fix first, escalating only when a single experiment proves it's needed) is how we made that challenge routine.
One example: an interest calculation didn't match the oracle and had been escalated as a business question about compounding. The agent checked the API's writable types and found that the field controlling interest handling was silently dropped by the API. The fixture had been wrong all along. With the field set through the UI instead, the calculation matched the oracle to the cent. No analyst time was needed.
2. It finds defects in the gaps between documents and behaviour. A domain expert reads past a slightly wrong message, because they know what it means. The agent compares the text to the specification literally. That's how we found several wording defects that had survived manual testing.
3. It scales the boring rigour. Checking 71 interest rows one by one instead of trusting a matching total. Validating every lookup value against a spreadsheet. Re-running a case three times on fresh data before calling it stable. Humans skip this under pressure. The agent doesn't feel pressure.
4. Knowledge compounds. By week five, a fresh agent session started out knowing things that had taken me weeks to learn.
The Cons: Where It Hurt
I promised myself this article would be honest, so here's the other column.
1. Agents want to be helpful, and "helpful" can look like a green build. Left unconstrained, an agent under pressure will widen a tolerance, reword a note, or reclassify a step to make a report look better. We found no fabricated results, but only because we made that structurally hard: read-only evaluators, step maps, evidence requirements, and explicit rules that a red run is a success. We even caught report notes drifting toward wording that would flip a step into a kinder bucket. The rule we had to write down: word the note to the real verdict, never to game the bucket.
2. Agents blame the application too quickly. Three of our "app defects" were our own bugs: a wrong locator, a stale test assumption carried over from another environment, and a construction gap combined with a misread requirement. Without a "ruled out" section in the defect ledger, those would have been filed against the development team. That costs credibility you don't get back.
3. A check that can't fail looks exactly like a check that passes. A pattern that never matches passes an "is absent" assertion forever. The report stays green, the evidence looks complete, and the step proves nothing. Agents produce these easily: syntactically perfect, logically empty. The only defence we found was to read the expected and actual values of every new case at least once with our own eyes.
4. The first implementation is always over-built. Before we standardised our approach, one test case had grown to 10 tests, 1,473 lines, and about 6 minutes of runtime, and it still verified only part of its steps. Agents happily write recovery code, retries, and workarounds for problems that a better test design would never create.
5. Financial functionality is hostile to autonomous testing. Fintech systems are built to be the opposite of disposable. Financial records can't be deleted, so every run leaves a permanent audit trail. Accounts get locked by abandoned forms, because concurrent changes to money aren't allowed. Strict session security signs the user out server-side, sometimes mid-test. Almost every rule depends on business dates, and the business calendar had only 64 working days configured for the test year, so a "random" date is usually a holiday. End-of-day batch processing, where interest and balances actually move, can't be triggered on demand. An agent can work around much of this, but each workaround is code you now maintain. The better answer sits at the environment level: restore a clean database backup at the start of every suite run, and give each test its own dedicated data instead of sharing it. Nothing needs deleting, nothing stays locked, and every run starts from a known state.
6. The agent cannot judge business intent. When the requirements were genuinely ambiguous, the agent could only do one useful thing: produce a precise, evidence-backed question. It could not answer it. Some steps remain honestly INCOMPLETE, waiting on a person who knows the business. That's correct behaviour, but it puts a ceiling on autonomy.
7. Guardrails must be policy, not preference. At one point a run touched a second test environment that we shouldn't have been using. Nothing broke, but the lesson was clear. "Prefer environment A" isn't a rule. "Environment A only, blanket policy" is. The same goes for irreversible writes: approval for one action doesn't carry over to the next.
How to Improve the Process: Two Playbooks That Changed Everything
By week three we'd distilled the lessons into two short playbooks. They did more for throughput than any model upgrade.
The A/B/C/D Unblock Playbook: cheapest-first, with a gate before each escalation
Phase | What you do | Cost |
A: Correct the documented intent | Re-read the requirement and the test step, and fix stale or wrong notes. "Blocked by end-of-day" was once just a misreading of a requirement section. | Minutes, zero risk |
B: One live diagnostic | Run the single cheapest experiment, on one fresh account, that proves or kills the hypothesis. Prefer non-mutating API calls. Compare row by row, not totals. | One account |
Gate | Clean fix → C. Partial fix → record the residual honestly. No change → it's a real finding. Stop. | — |
C: Productise | Turn the winning experiment into a re-runnable spec. | Hours |
D: Propagate | Apply the fix to sibling cases that share the mechanism. One root cause often unblocks several. | Hours |
Never build C before B confirms. Most blockers are cheaper to disprove than to build around.
The SEED Build Standard
Seed the starting state through the API, not the UI. The UI test starts at the first business step, not at "build the precondition".
Execute the test case as one sequential test, with soft assertions so every step gets its own verdict.
Every run on fresh data. No shared state, no stuck locks, no "consumed by the last run".
Deterministic waits. No sleeps. Wait for the real condition: the API going quiet, a field becoming editable.
The measured effect:
Test case | Before | After SEED |
Complex restructuring, part 1 | 10 tests, 1,473 lines, ~6 min, steps 11–25 not automated | 1 test, ~420 lines, ~2 min, 25/25 steps verified |
Complex restructuring, part 2 | 0 of 9 steps ("structural blocker") | 9/9 verified, ~2.3 min |
Multi-account modification, read-only case | 0 of 11 steps verified | 11/11, ~51 seconds |
Multi-account modification, state-changing case | 1 of 44 steps | 43/44 verified, and new findings surfaced |
That last row matters most. It went from "blocked" to finding new defects, because the test finally reached the parts of the system where the defects were.
What I'd do from day one next time
Write the knowledge index before the first test. Don't wait for the agent to forget something three times.
Keep a defect ledger with a "ruled out" section from the start. Your credibility depends on it.
Batch business questions. The agent is excellent at producing precise questions with evidence attached. Collect them and take them to the analysts weekly, instead of letting them block steps silently.
Ask for a test-data API early. Most of our hardest problems were really fixture problems.
Negotiate a dedicated environment with a restorable clean database, including for batch and end-of-day scenarios. Some behaviour simply can't be tested on a shared environment.
Check that a check can fail. Read the expected and actual values of every new case at least once with your own eyes.
So, Can You Test What You Don't Understand?
Partially. But that "partially" is far bigger than I expected.
An agentic framework can't replace domain expertise. What it can do is shrink the amount of domain expertise you need before you start producing value. It shifts the question from "what does this business rule mean?" to "does the system do what the document says, and can I prove it?" For a large share of real test cases, the second question is enough to find real defects.
The remaining share, the genuinely ambiguous and the genuinely business-judgement calls, still needs a human who knows the business. The framework's job there is to arrive at that human's desk with a precise question and the evidence attached, not to guess.
That's the shift I keep coming back to. In my Quality Guild article I asked whether "Test Management" is even the right title anymore. After these five weeks, my answer is that the tester's role moves toward designing the guardrails, curating the knowledge, and judging the evidence. The agent does the legwork. You remain accountable for the truth.
What's Next, and Where I'd Like Your Input
Three open problems on my radar:
Autonomy vs. authorisation for irreversible actions. Today every irreversible write needs explicit human approval. That's safe, but it's slow. Where do you draw the line between a self-provisioning suite and a human gate?
Business questions as a first-class artefact. I'm considering a structured "question ledger" that the agent fills and analysts answer, with each answer flowing straight back into the knowledge base. Has anyone built something like this?
Measuring evaluator honesty. How do you audit an agent that judges another agent's test runs? I'm exploring seeded "known-bad" runs to check that the evaluator catches them.
If you've tested a domain you didn't know, with or without AI, I'd like to hear how you approached it. Drop it in the comments or connect with me on LinkedIn.
And one question before you close this tab: when your automation goes green, how sure are you that it could have gone red?
Comments