Can AI Tell If Your Tests Are Actually Good? Using LLMs to Review Test Quality

A test suite can report 95% coverage and still be worthless. A test that calls a method and does assertTrue(true == true), or one that mocks every collaborator so thoroughly that the only thing under test is the mock itself, still counts as "covered" in a JaCoCo report. Coverage tells you which lines ran. It says nothing about whether that test would actually catch a regression.

In this article we look at using an LLM as a second reviewer for test quality — not to replace mutation testing or code review, but to sit alongside both in CI and flag the specific patterns that quietly erode a test suite’s value over time.

When auditing mature codebases, a recurring pattern surfaces: high line coverage paired with low confidence. Tests exist for nearly every class, yet many share the same skeleton — instantiate the object, call a method, assert the result isn’t null. No behavioural assertions, no edge cases, no verification of collaborator interactions.

Static analysis tools like SonarQube catch some of this (duplicate code, cyclomatic complexity), and mutation testing tools like PIT catch more of it by actually measuring whether tests fail when the code is deliberately broken. But mutation testing is slow on large suites, and its output — a list of surviving mutants — still needs a person to read the test and judge why it’s weak.

We wanted something capable of reading a test file the way a senior engineer would during a code review: does this assertion make sense, is this mock hiding real behaviour, does the test name describe what’s actually being verified? An LLM turned out to be a reasonable fit for that specific, judgment-heavy step.

1. The Tools

To improve test reliability, we employ a combination of tools, including Mutation Testing and LLM-based reviews. This complementary approach enables us to move beyond simple line coverage metrics toward more robust verification methods, ensuring tests are semantically meaningful and capable of catching real-world regressions.

Mutation Testing (PIT / pitest-kotlin)

• The problem: Line coverage doesn’t prove a test would catch a bug.
• The solution: Deliberately mutate source code and check that tests fail. Surviving mutants reveal weak tests.

An LLM as a Test Reviewer

• The problem: Even with surviving mutants, someone still has to read tests and explain what is weak.
• The solution: Pass the test file and class under test to an LLM with a structured rubric and consume JSON output in build tooling.

Neither tool replaces human review. Mutation testing tells you where to look; LLM review gives a first pass on why the test is weak, in review-like language.

Why use an LLM instead of custom static-analysis rules?

A natural question is why not simply implement these checks as custom SonarQube rules or static-analysis plugins.

The answer is that many of the weaknesses discussed here are semantic rather than syntactic. Determining whether an assertion is meaningful, whether mocks hide the behaviour under test, or whether a test name accurately reflects the scenario requires reasoning across the entire test rather than matching predefined patterns.

Static-analysis rules are excellent at detecting deterministic issues, while LLMs are better suited to contextual judgement. Rather than replacing existing tools, the two approaches complement each other: static analysis identifies objective violations, mutation testing measures whether tests actually detect faults, and an LLM provides reviewer-like feedback on aspects that are difficult to encode as fixed rules.

2. Why Coverage Numbers Hide Weak Tests

Code coverage only measures which lines ran during a test run — not whether the assertions are meaningful. A suite can hit 95% coverage while entirely missing edge cases, side effects, or incorrect return values. The patterns below are the most common ways weak tests inflate that number.

Tautological assertions (assertion cannot fail meaningfully):

@Testfun `should process refund`() {    val result = service.processRefund(refundRequest)    assertNotNull(result) // Tautological: always passes if non-null}

Over-mocking (test validates stubs, not logic):

@Testfun `should calculate discount`() {    val discountEngine = mock<DiscountEngine>()    whenever(discountEngine.calculate(any())).thenReturn(10.0)    val result = pricingService.applyDiscount(order, discountEngine)    assertEquals(10.0, result) // Only checks mock return value}

Assertion-free tests (pass if no exception is thrown):

@Test fun `test order 1`() { assertTrue(service.validate(order1)) }@Test fun `test order 2`() { assertTrue(service.validate(order2)) }@Test fun `test order 3`() { assertTrue(service.validate(order3)) }// ...similar variants, no edge-case coverage

Review Workflow

When a developer pushes and opens a PR, three quality checks run in sequence. SonarQube handles the deterministic side (code smells, complexity, and known patterns). PIT goes further by deliberately breaking the source code to check whether the tests actually notice. Finally, the LLM reads each changed test file alongside its source class and produces a JSON report:

If the score meets the threshold (≥ 70), the gate passes and the PR is ready for human review. If it falls below, the pipeline triggers an autonomous agent that picks up the JSON output, refactors the failing tests, and runs a compilation check to verify the code still builds. It then re-submits for LLM review and, if the new score passes, commits the refactored tests back to the PR branch.

This loop repeats up to a configured maximum, typically three iterations, before stopping and surfacing the remaining issues in the job summary for the developer to resolve manually. Either way, no changes are merged without a final human approval.

3. Building an AI Test Reviewer

There’s more than one way to wire this into the workflow. The approaches below go from least to most integration — and reflect how most teams are working today, where an AI coding assistant like Claude Code is already part of the development environment.

3.1. The Rubric: The Real Artefact

The review prompt is the central artefact. It lives in prompts/test-quality-review.txt, versioned alongside the code it judges, and the CI reads it at runtime:

You are reviewing a unit test file against the class or module it tests.First, read the CLASS UNDER TEST carefully and identify:- All public methods and their signatures- All conditional branches and error paths- All collaborators the class depends onThen score the test file from 0-100 on these criteria:- Assertions are behavioural, not tautological (checking a value that can never fail)- Collaborators are mocked only where necessary; the class's own logic is exercised- Test names describe the specific behaviour under test- Edge cases relevant to the method signatures are covered (nulls, empty collections)- All meaningful code paths identified in the class under test have a corresponding testThen identify coverage gaps:- Any public methods in the class under test with no corresponding test- Any conditional branches (if/when/try-catch) never exercised by the test suite- Any error paths or exception handling that is untestedRespond ONLY with JSON, no markdown fences, matching:{  "score": <0-100>,  "issues": [    { "line": <int>, "severity": "high|medium|low", "description": "<string>" }  ],  "coverage_gaps": [    { "element": "<method or branch>", "type": "missing_test|missing_branch|missing_error_path" }  ],  "summary": "<one sentence>"}

3.2. CI Step via Claude Code

The CI step reads the rubric from the file and passes it to claude --non-interactive together with each changed test file. Claude Code manages the review, refactor, and compilation check cycle natively—no curl calls, no manual JSON parsing:

name: Review and fix failing testsrun: |  RUBRIC=$(cat prompts/test-quality-review.txt)  for f in $(git diff --name-only origin/main -- '*Test.kt'); do    claude --non-interactive \      --max-turns 3 \      "$RUBRIC      Review $f against its class under test.      If score < 70, fix the tests in place.      Run ./gradlew compileTestKotlin after each change.      Stop if compilation fails."  doneenv:  ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

3.3. Agentic Loop — Auto-Refactoring on Failure

The --max-turns 3 flag is the direct equivalent of MAX_ITERATIONS=3 from a manual bash loop. Claude Code manages the cycle: review → refactor → compile check → re-review.

Two constraints remain non-negotiable: an iteration cap to avoid runaway costs, and a compilation check after each refactor to catch cases where the model breaks the code. Both are handled automatically.

The refactored file is committed back to the PR branch only if it passes both compilation and the quality gate. The commit message includes the final score so the reviewer can see at a glance what the agent achieved. No changes are merged automatically; a human must always review and approve the final result.

4. What It Caught in Practice

We ran the review against two tests for the same RefundService class: one deliberately weak, and one with three behavioural cases. This is Claude's actual output via the Anthropic API, unedited:

Weak Test Result:

{  "score": 8,  "issues": [    {       "line": 4,       "severity": "high",       "description": "The assertion is tautological: assertNotNull(result) cannot fail if processRefund always returns a non-null RefundResult."     },    {       "line": 3,       "severity": "high",       "description": "No verification that gateway.refund() or ledger.record() are called with the correct arguments, so the collaborator interactions are entirely untested."     }  ],  "coverage_gaps": [    { "element": "RefundService.processRefund", "type": "missing_test" },    { "element": "gateway.refund()", "type": "missing_branch" },    { "element": "ledger.record()", "type": "missing_error_path" }  ],  "summary": "The weak test provides little value: the assertion is irrelevant, and collaborator interactions and meaningful paths are not verified."}

Robust Test Result:

{  "score": 72,  "issues": [    {       "line": 10,       "severity": "medium",       "description": "Boundary: the case amount == 0 is not tested, which is distinct from negative values."     },    {       "line": 3,       "severity": "low",       "description": "Lack of verification that ledger is not called on rejection."     }  ],  "coverage_gaps": [    { "element": "RefundService.validateAmount", "type": "missing_test" },    { "element": "ledger.record()", "type": "missing_error_path" },    { "element": "gateway.refund()", "type": "missing_exception_path" }  ],  "summary": "The three tests cover the happy path and the main failure modes, but miss the zero-amount boundary, verification of side effects on rejection, and propagation of the transactionId."}

The score gap (8 vs. 72) matches what any human reviewer would say looking at both files. But the interesting part is in the detail of the second case. The “robust” test already covers the happy path and the two main failures with reasonable mocks — and the model still found six real improvement points that weren’t in our explicit rubric:

• The exact boundary at amount == 0 (distinct from a negative amount).
• Missing check that the ledger is not touched on rejection.
• Absence of a check on the returned transactionId.
• The most valuable: No test covers what happens if gateway.refund() throws an exception instead of returning a controlled failure.

That last point is exactly the kind of comment a senior reviewer would leave on a PR after thinking about it for a while, not something that falls out of mechanically applying the rubric.

5. Where This Breaks Down

Some honest limitations worth stating up front:

• It’s a heuristic, not ground truth: The model can misjudge intentionally simple tests (a smoke test that only checks “does this throw?”) as weak. Treat the score as a prompt for review, not a gate that overrides human judgment on its own.
• Cost and latency add up: Reviewing every test file on every commit is unnecessary; scoping the task to changed files in a PR keeps it fast and cheap.
• It doesn’t replace mutation testing: PIT tells you objectively whether a mutant survived. The LLM review explains likely reasons faster than reading raw mutant output, but the two are complementary, not interchangeable.

6. Conclusion

Coverage percentage answers “Did this line run?” Mutation testing answers “Would this test catch a bug here?” An LLM review sits between the two: it provides contextual feedback on why a test may be weak, highlighting issues such as tautological assertions, excessive mocking, poor behavioural verification, and missing edge cases using language familiar to code reviewers.

This approach is not intended to replace mutation testing or human review. Instead, it complements existing quality assurance techniques by reducing the effort required to identify low-value tests during continuous integration. Used as an advisory quality gate rather than an absolute authority, LLM-based review offers a practical way to improve test quality without significantly increasing build complexity or execution time.

At Parser, we are taking this further by integrating custom autonomous agents directly into our CI/CD pipelines. Rather than surfacing issues for a human to act on, these agents use specialied skills to propose concrete refactoring patterns inline — moving the workflow from audit toward automated remediation, with human review as the final gate. When the score falls below the threshold, the agent picks up the JSON output, refactors the failing tests, and resubmits for review — iterating until the quality gate turns green.

References

• Jia, Y., & Harman, M. (2011). An Analysis and Survey of the Development of Mutation Testing. IEEE Transactions on Software Engineering.
• PIT Mutation Testing Documentation
• JaCoCo Java Code Coverage Library
• SonarQube Documentation
• Anthropic API Documentation
• Martin Fowler. The Practical Test Pyramid.
• Google. Software Engineering at Google.
• OpenAI. SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?

‍