Model-Based Testing

What Is Model-Based Testing? #

Model-based testing (MBT) is an approach where you create a formal model of the system’s expected behavior, and then use tools to automatically generate test cases from that model. Instead of manually writing hundreds of test cases, you build one model and let algorithms derive the tests.

The MBT workflow:

1. Analyze requirements — understand the system’s behavior
2. Build a model — represent behavior as a formal model (state machine, activity diagram, etc.)
3. Generate tests — use MBT tools to derive test cases from the model
4. Execute tests — run generated tests against the system under test
5. Evaluate results — the model serves as the oracle for expected behavior
6. Maintain the model — update the model as requirements change

Why Model-Based Testing? #

Manual test design problems:

• Test designers interpret requirements differently, leading to inconsistent coverage
• Important paths are missed because humans cannot enumerate all combinations
• Maintaining thousands of manual test cases is expensive
• Coverage criteria (path, transition) are hard to satisfy manually

MBT advantages:

• Systematic coverage — tools guarantee coverage criteria are met
• Automatic regeneration — change the model, regenerate all tests
• Defect detection in models — building the model reveals ambiguities in requirements
• Scalability — models describe behavior concisely; tests are generated, not written

Model Types for MBT #

Finite State Machines (FSM) #

The most common model type. States represent system conditions; transitions represent events or actions.

Example: ATM Session

States: Idle, CardInserted, PINEntry, Authenticated, Transaction, Dispensing, Error
Transitions:
  Idle → CardInserted [insertCard]
  CardInserted → PINEntry [cardValid]
  CardInserted → Error [cardInvalid]
  PINEntry → Authenticated [correctPIN]
  PINEntry → PINEntry [wrongPIN, attempts < 3]
  PINEntry → Error [wrongPIN, attempts >= 3]
  Authenticated → Transaction [selectWithdraw]
  Transaction → Dispensing [sufficientFunds]
  Transaction → Error [insufficientFunds]
  Dispensing → Idle [cashTaken]
  Error → Idle [ejectCard]

From this model, an MBT tool can generate tests covering:

• All states (state coverage)
• All transitions (transition coverage)
• All transition pairs (switch coverage)
• Specific paths (e.g., all error paths)

Extended Finite State Machines (EFSM) #

FSMs with variables, guards, and actions. More expressive but harder to analyze.

State: PINEntry
  Variables: attempts = 0
  Transition: wrongPIN [attempts < 3] / attempts++ → PINEntry
  Transition: wrongPIN [attempts >= 3] → Error
  Transition: correctPIN → Authenticated

UML Activity Diagrams #

Model workflows and business processes. Good for testing complex use case flows.

UML Sequence Diagrams #

Model interactions between components. Good for integration testing.

Markov Chains #

Probabilistic models where transitions have probabilities. Used for statistical testing and usage-profile-based testing.

Coverage Criteria for MBT #

Most MBT tools use transition coverage as the default, with options for stronger criteria.

MBT Tools #

Open Source #

• GraphWalker — Java-based, supports FSM and EFSM models, REST API for integration
• ModelJUnit — Extension of JUnit for FSM-based testing
• NModel — .NET model-based testing framework
• Spec Explorer — Microsoft tool for protocol testing

Commercial #

• Conformiq — Enterprise MBT with support for multiple model types
• Smartesting — Model-based test generation with UML
• MaTeLo — Markov chain-based MBT for statistical testing

GraphWalker Example #

public class ATMModel extends ExecutionContext {
    @Vertex()
    public void Idle() {
        // Verify ATM shows welcome screen
    }

    @Edge()
    public void insertCard() {
        // Simulate card insertion
    }

    @Vertex()
    public void PINEntry() {
        // Verify PIN prompt shown
    }

    @Edge()
    public void correctPIN() {
        // Enter correct PIN
    }
}

Run with: java -jar graphwalker.jar offline --model atm.graphml "random(edge_coverage(100))"

When MBT Works Best #

Good candidates for MBT:

• Protocol implementations (network protocols, APIs)
• Embedded systems with well-defined state behavior
• Business process workflows
• Systems with many valid paths and complex state logic
• Long-lived products that require continuous test maintenance

Poor candidates:

• Simple CRUD applications
• UI-heavy applications with complex visual behavior
• Systems where requirements change faster than models can be updated
• One-off projects where the modeling investment does not pay off

Exercise: Building an MBT Model #

Problem 1 #

Build a state machine model for an online shopping cart:

• Cart starts Empty
• Items can be added (Empty → HasItems, HasItems → HasItems)
• Items can be removed (HasItems → HasItems or HasItems → Empty)
• Cart can proceed to Checkout (HasItems → Checkout)
• Checkout can apply coupon (Checkout → Checkout)
• Checkout can submit order (Checkout → OrderPlaced)
• Order can fail (Checkout → PaymentFailed)
• User can retry payment (PaymentFailed → Checkout)
• User can return to cart (Checkout → HasItems)
• OrderPlaced is a final state

Then list all transitions and derive tests for transition coverage.

Problem 2 #

You are asked to evaluate whether MBT is appropriate for these projects. Justify your answer:

1. A banking API with 50 endpoints, complex state management (accounts, transactions, approvals)
2. A marketing landing page with a contact form
3. An IoT device firmware that manages sensor readings, alerts, and communication protocols

Challenges and Limitations #

Model maintenance: The model must stay synchronized with the system. Out-of-date models generate invalid tests.

Model complexity: Complex systems require complex models. If the model is as complex as the system itself, the benefit is lost.

Learning curve: Teams need training in modeling formalisms and MBT tools.

Oracle problem: The model defines expected behavior, but what if the model itself is wrong? The model must be validated against requirements independently.

Non-functional properties: MBT excels at functional testing but does not directly address performance, security, or usability.

Key Takeaways #

• MBT generates test cases automatically from behavioral models of the system
• Finite state machines are the most common model type for MBT
• MBT tools guarantee coverage criteria (state, transition, switch coverage)
• Building the model itself reveals requirement ambiguities — a valuable side effect
• Best candidates: protocol implementations, embedded systems, complex business workflows
• Poor candidates: simple CRUD apps, UI-heavy apps, short-lived projects
• The model serves as both test generator and test oracle
• Model maintenance is the biggest ongoing cost — keep models synchronized with the system