Mutation Testing

Testing Your Tests #

Code coverage metrics tell you what code your tests execute, but not whether your tests would actually catch bugs in that code. A test that executes a line but never checks the result achieves coverage without providing value.

Mutation testing flips the perspective: instead of measuring how much code your tests cover, it measures how well your tests detect faults. It does this by deliberately introducing bugs (mutations) into your source code and checking whether your test suite catches them.

If your tests pass when a bug is introduced, those tests are weak.

How Mutation Testing Works #

The process follows these steps:

1. Generate mutants. A mutation testing tool creates copies of your source code, each with one small change (mutation). Each modified copy is called a mutant.

2. Run tests against each mutant. The full test suite runs against every mutant.

3. Classify results:

   • Killed mutant — at least one test fails (good — your tests caught the fault)
   • Survived mutant — all tests pass (bad — your tests missed the fault)
   • Equivalent mutant — the mutation does not change program behavior (neutral — cannot be killed)

4. Calculate mutation score. Mutation score = killed / (total - equivalent) * 100%

Common Mutation Operators #

Mutation operators define the types of changes applied to the code:

Arithmetic Operator Replacement #

# Original
total = price * quantity
# Mutant
total = price + quantity

Relational Operator Replacement #

# Original
if age >= 18:
# Mutant
if age > 18:

Logical Operator Replacement #

# Original
if is_active and is_verified:
# Mutant
if is_active or is_verified:

Constant Replacement #

# Original
MAX_RETRIES = 3
# Mutant
MAX_RETRIES = 0

Statement Deletion #

# Original
def process(data):
    validate(data)      # This line removed in mutant
    transform(data)
    save(data)

Return Value Mutation #

# Original
return True
# Mutant
return False

Negation of Conditionals #

# Original
if not is_empty:
# Mutant
if is_empty:

The Coupling Effect and Competent Programmer Hypothesis #

Mutation testing rests on two theoretical foundations:

Competent programmer hypothesis: Programmers produce code that is close to correct. Real bugs are typically small errors — a wrong operator, an off-by-one boundary, a missing negation. Mutation operators simulate exactly these kinds of errors.

Coupling effect: Tests that detect simple faults (first-order mutants) will also detect more complex faults (higher-order mutants). This means testing with simple mutations is sufficient to assess test quality.

Mutation Score Interpretation #

A mutation score of 85% means your tests would catch 85% of the types of simple bugs that could be introduced. The surviving 15% point directly to testing gaps.

Mutation Testing Tools #

PIT (PITest) — Java #

PIT is the most popular mutation testing tool for Java. It integrates with Maven, Gradle, and most CI systems.

<!-- Maven plugin -->
<plugin>
    <groupId>org.pitest</groupId>
    <artifactId>pitest-maven</artifactId>
    <version>1.15.0</version>
    <configuration>
        <targetClasses>
            <param>com.example.service.*</param>
        </targetClasses>
    </configuration>
</plugin>

mvn org.pitest:pitest-maven:mutationCoverage

Stryker — JavaScript/TypeScript #

Stryker supports JavaScript, TypeScript, C#, and Scala.

npm install --save-dev @stryker-mutator/core
npx stryker init
npx stryker run

Other Tools #

• mutmut — Python mutation testing
• Infection — PHP mutation testing
• cosmic-ray — Another Python mutation tester
• cargo-mutants — Rust mutation testing

Performance Considerations #

Mutation testing is computationally expensive. If you have 1,000 lines of code and each line generates 3 mutants, that is 3,000 runs of your test suite. Strategies to manage this:

Incremental mutation testing. Only mutate changed files, not the entire codebase.

Test selection. Run only the tests relevant to the mutated code, not the full suite.

Parallel execution. Run mutant test suites in parallel across multiple cores or machines.

Sampling. Test a random subset of mutants instead of all of them.

Prioritize critical code. Run mutation testing on business-critical modules, not utility code.

Exercise: Analyzing Mutation Results #

Problem 1 #

Given this function and its tests:

def calculate_discount(price, customer_type):
    if customer_type == "premium":
        return price * 0.8    # 20% discount
    elif customer_type == "regular":
        return price * 0.9    # 10% discount
    else:
        return price          # No discount

# Tests
def test_premium_discount():
    assert calculate_discount(100, "premium") == 80

def test_regular_discount():
    assert calculate_discount(100, "regular") == 90

def test_no_discount():
    assert calculate_discount(100, "guest") == 100

A mutation tool generates these mutants. For each, predict whether it will be killed or survive:

1. Change price * 0.8 to price * 0.9
2. Change price * 0.9 to price * 0.8
3. Change customer_type == "premium" to customer_type != "premium"
4. Change return price to return 0
5. Change price * 0.8 to price + 0.8

Problem 2 #

Now consider a function with weaker tests:

def is_eligible(age, income, has_account):
    if age >= 18 and income > 30000:
        if has_account:
            return "APPROVED"
        else:
            return "PENDING"
    return "REJECTED"

# Tests
def test_approved():
    result = is_eligible(25, 50000, True)
    assert result == "APPROVED"

def test_rejected():
    result = is_eligible(16, 20000, False)
    assert result == "REJECTED"

Predict the outcome for each mutant:

1. Change age >= 18 to age > 18
2. Change income > 30000 to income >= 30000
3. Change has_account to not has_account
4. Change return "PENDING" to return "APPROVED"
5. Change and to or in the first condition

Equivalent Mutants: The Challenge #

An equivalent mutant produces the same output as the original for all possible inputs. Example:

# Original
i = 0
while i < 10:
    # ...
    i += 1

# Equivalent mutant
i = 0
while i != 10:
    # ...
    i += 1

Both loops execute exactly the same way. No test can kill this mutant because it behaves identically. Detecting equivalent mutants is undecidable in the general case (equivalent to the halting problem). Modern tools use heuristics to identify likely equivalent mutants and exclude them from the score.

Integrating Mutation Testing into CI/CD #

For practical adoption:

1. Start with critical modules. Do not run mutation testing on the entire codebase initially.
2. Set a threshold. Fail the build if mutation score drops below a target (e.g., 80%).
3. Run incrementally. Only mutate code changed in the current PR.
4. Use it for code review. Share mutation reports with reviewers to guide test improvement discussions.

# Example GitHub Actions step
- name: Run mutation testing
  run: npx stryker run --reporters html,dashboard
  if: github.event_name == 'pull_request'

Key Takeaways #

• Mutation testing evaluates test quality by introducing deliberate faults into source code
• Killed mutants = good tests; survived mutants = testing gaps
• Mutation score = killed / (total - equivalent) * 100%
• Common operators: arithmetic, relational, logical replacement; statement deletion; return value mutation
• Tools: PIT (Java), Stryker (JS/TS), mutmut (Python), Infection (PHP)
• Mutation testing is expensive — use incremental, parallel, and selective strategies
• Equivalent mutants cannot be killed and must be excluded from scoring
• Aim for 80%+ mutation score on critical business logic