The DORA Metrics
The DORA (DevOps Research and Assessment) team, now part of Google Cloud, identified four key metrics that distinguish high-performing software delivery teams from low performers. These metrics are not just for DevOps — QA has direct influence on all four.
1. Deployment Frequency
What it measures: How often the team deploys to production.
| Performance Level | Frequency |
|---|---|
| Elite | On-demand (multiple times/day) |
| High | Once per week to once per month |
| Medium | Once per month to once every 6 months |
| Low | Less than once every 6 months |
QA influence: Fast, reliable automated tests enable frequent deployments. Slow or flaky tests force teams to batch changes and deploy less often.
2. Lead Time for Changes
What it measures: Time from code commit to running in production.
| Performance Level | Lead Time |
|---|---|
| Elite | Less than one hour |
| High | One day to one week |
| Medium | One month to six months |
| Low | More than six months |
QA influence: Pipeline speed depends on test execution time. A 2-hour test suite creates a 2-hour minimum lead time. Optimized test suites with sharding and test selection keep lead time short.
3. Change Failure Rate
What it measures: Percentage of deployments that cause a failure in production.
| Performance Level | Failure Rate |
|---|---|
| Elite | 0-15% |
| High | 16-30% |
| Medium | 16-30% |
| Low | 46-60% |
QA influence: This is QA’s most direct metric. Better testing = fewer production failures. QA reduces change failure rate through comprehensive test coverage, quality gates, and pre-deployment validation.
4. Mean Time to Recovery (MTTR)
What it measures: How quickly the team recovers from a production failure.
| Performance Level | MTTR |
|---|---|
| Elite | Less than one hour |
| High | Less than one day |
| Medium | One day to one week |
| Low | More than six months |
QA influence: Fast rollback procedures, production smoke tests, and good monitoring help detect and recover from failures quickly. QA designs and validates rollback procedures.
Quality Metrics That Connect to DORA
Test Suite Metrics
| Metric | Target | Why It Matters |
|---|---|---|
| Test pass rate | > 99% | Low pass rate blocks deployments |
| Test execution time | < 30 min | Long tests increase lead time |
| Flaky test rate | < 2% | Flaky tests erode trust in pipeline |
| Code coverage | > 80% | Gaps increase change failure rate |
| Test creation rate | > 0 per sprint | Growing coverage prevents regressions |
Pipeline Metrics
| Metric | Target | Why It Matters |
|---|---|---|
| Pipeline success rate | > 95% | Failed pipelines block deployments |
| Pipeline duration | < 30 min | Faster pipeline = faster lead time |
| Queue wait time | < 5 min | Runner contention slows everything |
| Rollback success rate | 100% | Failed rollback = extended MTTR |
Production Quality Metrics
| Metric | Target | Why It Matters |
|---|---|---|
| Escaped defects per release | < 1 | Direct measure of test effectiveness |
| Time to detect production issues | < 5 min | Faster detection = faster MTTR |
| Customer-reported bugs/month | Trending down | Lagging indicator of quality |
Building a Metrics Dashboard
The QA-DevOps Dashboard
Combine DORA metrics with quality metrics in a single dashboard:
Row 1: Delivery Performance
- Deployment frequency (weekly trend)
- Lead time (commit to production, weekly average)
- Pipeline success rate (daily)
Row 2: Quality Impact
- Change failure rate (monthly)
- Escaped defects per release
- Test pass rate trend
Row 3: Efficiency
- Test execution time trend
- Flaky test rate
- Time to detect production issues (MTTD)
Row 4: Recovery
- MTTR trend
- Rollback frequency
- Incident count (monthly)
Exercise: Analyze Team Metrics
Your team’s current metrics: Deploy 2x/week, lead time 3 days, change failure rate 25%, MTTR 4 hours. Test suite: 300 tests, 45-minute execution, 8% flaky rate, 72% coverage. Identify the biggest improvement opportunities and create an action plan.
Solution
Analysis
- Deployment Frequency: High level (2x/week). Good.
- Lead Time: 3 days is medium. Pipeline takes 45 min, so the bottleneck is elsewhere (code review, manual QA gates).
- Change Failure Rate: 25% is at the border of medium/high. Too many production issues.
- MTTR: 4 hours is high level. Recovery is good.
Root Cause: Change Failure Rate
25% failure rate with 72% coverage and 8% flaky rate indicates:
- Coverage gaps are letting bugs through
- Flaky tests are training the team to ignore failures
Action Plan
Priority 1: Reduce Flaky Rate (8% → < 2%)
- Quarantine the worst 10 flaky tests
- Fix or rewrite quarantined tests within 2 sprints
- Add flaky test detection to CI pipeline
- Timeline: 4 weeks
Priority 2: Increase Coverage (72% → 85%)
- Identify untested critical paths using coverage reports
- Add tests for top 10 escaped defects from last quarter
- Require minimum 80% coverage on new code
- Timeline: 8 weeks
Priority 3: Reduce Lead Time (3 days → 1 day)
- Implement test impact analysis (run only affected tests on PRs)
- Shard test suite across 4 runners (45 min → 12 min)
- Replace manual QA gate with automated quality checks
- Timeline: 6 weeks
Expected Results
- Change failure rate: 25% → 15% (through better coverage and less ignored failures)
- Lead time: 3 days → 1 day (through faster tests and automated gates)
- Deployment frequency: 2x/week → daily (enabled by faster, more reliable pipeline)
Key Takeaways
- DORA metrics measure delivery performance — QA directly influences all four
- Change failure rate is QA’s most direct metric — better testing = fewer production failures
- Test execution time affects lead time — fast tests enable fast deployments
- Flaky tests undermine the entire pipeline — treat them as high-priority bugs
- Measure, set targets, improve — what gets measured gets improved