Mean Time to Recovery (MTTR)
When I joined my first DevSecOps team five years ago, I learned a hard lesson about incident management during a catastrophic database failure. While our team scrambled for almost four hours to restore service, our CTO paced nervously behind us repeating one question: "What's our MTTR target here?"
I had no idea what he was talking about at the time. Now, after years of on-call rotations and countless incidents later, I not only understand MTTR intimately—I've built my career around optimizing it. Let me share what I've learned about this critical metric from the trenches.
What MTTR Really Means to Me as an SRE
Mean Time To Repair (MTTR) sounds like a dry statistical concept, but for me, it's the most tangible measure of an SRE team's effectiveness during crisis. It answers the question: "When things break—and they will—how quickly can we put them back together?"
In its simplest form, MTTR is the average time from when an incident begins until service is restored. But that definition hides the complex reality I've experienced while managing production incidents across multiple companies.
Breaking Down MTTR Through My Own Incident War Stories
Through hard-earned experience, I've come to visualize MTTR as four distinct phases:
1. Detection: Finding Out Something's Wrong
My worst MTTR story began with a subtle database degradation that our monitoring didn't catch. Users noticed performance issues a full 47 minutes before our alerts triggered. I learned that you can't fix what you don't know is broken.
What I did to improve detection:
Implemented multi-layered monitoring (infrastructure, application, and synthetic user journeys)
Created custom alerting based on user-facing SLIs rather than just system metrics
Developed "canary user" accounts that would immediately alert us if they experienced issues
After these changes, our average detection time dropped from 12 minutes to under 3 minutes.
2. Diagnosis: Understanding What Broke and Why
During a particularly frustrating outage, our team spent 90 minutes hypothesizing about a network issue that turned out to be a simple configuration change. The diagnosis phase is where I've seen the most time wasted.
What I did to improve diagnosis:
Created a "first five minutes" checklist that engineers follow for every incident
Built comprehensive dashboards that correlate events across our stack
Implemented detailed tracing across our microservices architecture
My favorite tool for diagnosis is a custom dashboard I call the "Incident Command Center" that gives a single-pane view of recent deployments, configuration changes, traffic patterns, and system metrics.
3. Repair: Actually Fixing the Problem
Early in my career, I made the classic rookie mistake of trying to fix a production issue with a complex solution when a simple rollback would have been faster. I've learned that repair time optimization is about having multiple resolution paths ready.
What I did to improve repair times:
Created automated rollback capabilities for all deployments
Implemented feature flags that could be toggled without deployment
Built "break glass" procedures for emergency access and actions
One of my proudest achievements was reducing our database failover time from 7 minutes to 45 seconds through automation and practice.
4. Recovery: Getting Back to Normal
After a major cloud region outage, our services were technically "up" but still experiencing elevated error rates and latency. I learned that repair isn't the same as full recovery.
What I did to improve recovery:
Implemented graduated traffic routing during recovery
Created automated cache warming procedures
Developed background task scheduling that prevents thundering herds during recovery
My MTTR Journey: From Hours to Minutes
When I joined my current company three years ago, our average MTTR for critical incidents was 4 hours and 26 minutes. Today, it's 37 minutes. Here's how we achieved that:
1. We Made MTTR Data Visible to Everyone
I created a prominent dashboard in our office showing:
MTTR for the last 5 incidents
MTTR trend over time
Breakdown of time spent in each phase (detection, diagnosis, repair, recovery)
Comparison to our target MTTR
This visibility created healthy competition among teams and made MTTR improvement a shared goal.
2. We Practiced... A Lot
My most effective MTTR reduction technique was implementing monthly "Game Days" where we deliberately break things in our staging environment. Each practice incident includes a full post-mortem where we ask:
How could we have detected this faster?
What slowed down our diagnosis?
Could we have repaired this more efficiently?
How could we have recovered more smoothly?
After each Game Day, we implement at least one improvement to our processes or tooling.
3. We Built Specialized Tooling
The custom tools that have had the biggest impact on our MTTR:
Incident Bot: Our Slack bot that automatically creates incident channels, adds the right people, starts a video call, and provides quick access to runbooks and dashboards.
Repair Automation Framework: A system that can execute pre-approved repair actions with proper safeguards and logging, allowing us to automate common fixes even for complex scenarios.
Recovery Tracker: A dashboard that tracks key metrics during recovery and predicts time to full normalization.
MTTR Measurement: What I've Learned About Getting It Right
Measuring MTTR accurately turned out to be harder than I expected. Here's what works for us:
// Simplified pseudocode for how we calculate MTTR
function calculateMTTR(incidents) {
let totalRepairTime = 0;
incidents.forEach(incident => {
const detectionTime = incident.detectionTime - incident.startTime;
const diagnosisTime = incident.rootCauseIdentifiedTime - incident.detectionTime;
const repairTime = incident.serviceRestoredTime - incident.rootCauseIdentifiedTime;
const recoveryTime = incident.fullRecoveryTime - incident.serviceRestoredTime;
const totalTime = detectionTime + diagnosisTime + repairTime + recoveryTime;
totalRepairTime += totalTime;
});
return totalRepairTime / incidents.length;
}What makes our measurement valuable:
We track the start of an incident from when the issue began, not when we detected it
We capture timestamps for each phase transition in real-time during the incident
We distinguish between "service restored" and "full recovery"
The Real Business Impact of MTTR Reduction
Improving our MTTR wasn't just about making our technical metrics look better. It delivered tangible business results:
Revenue Protection: We estimated that each minute of downtime cost $5,700 in lost transactions. Reducing our average MTTR by 3+ hours translates to approximately $1 million in protected revenue annually.
Customer Retention: Our NPS surveys showed that customers who experienced incidents with quick resolution (under 30 minutes) were 3x less likely to mention the incident as a reason for dissatisfaction than those who experienced longer outages.
Team Morale: Perhaps most importantly, our SRE team's satisfaction scores improved dramatically. Nobody likes being on call for extended, stressful incidents.
My Top MTTR Reduction Strategies for Different Incident Types
Through trial and error, I've learned that different types of incidents benefit from different MTTR strategies:
For Infrastructure Failures
Strategy: Maximize automation and redundancy.
Example: After a storage system failure took us 2 hours to recover from, we implemented automated failover with continuous testing. Our next similar incident took 3 minutes to resolve.
For Code Defects
Strategy: Invest in rapid rollback and deployment capabilities.
Example: We reduced deployment validation time from 20 minutes to 2 minutes while maintaining safety, allowing us to quickly roll back problematic code.
For Configuration Issues
Strategy: Implement progressive delivery with automatic rollback.
Example: We now deploy configuration changes to 5% of users, monitor for 10 minutes, then proceed or automatically roll back.
For External Dependency Failures
Strategy: Build graceful degradation paths.
Example: When our payment processor went down, we implemented an offline mode that queued transactions, reducing our effective MTTR from the provider's 3 hours to our 5 minutes of user impact.
Conclusion: MTTR as a Mindset, Not Just a Metric
After years of focusing on MTTR, I've come to see it as more than a number—it's a mindset that shapes how we build and operate systems. Every architectural decision we make now considers the question: "How will this affect our ability to detect, diagnose, repair, and recover from failures?"
The most valuable lesson I've learned is that stellar MTTR doesn't come from heroics during an incident—it comes from the careful work done before the incident ever occurs. It's about building systems that are designed to be quickly understood, easily diagnosed, safely repaired, and rapidly recovered.
For engineers just starting their SRE journey, my advice is simple: Track your MTTR components religiously, identify your biggest time sinks, and methodically eliminate them one by one. Your future self, frantically responding to a 2 AM incident, will thank you.
Last updated