Search for a command to run...
Part 1 of a 2 part series
Outdoorsman | Networking | SRE | Chaos & Platform Engineering | DevOps @toyotaracing | former @aws @splunk @verizon @gm | Thoughts are my own
Personal: Husband | GirldadX3 | BBQ | Outdoors | Nascar | Baseball | Triathlons
How to scale, modernize, and mature your observability practice
A Guide for Software Architects
Introduction In the rapidly evolving world of software development, Continuous Integration (CI) and Continuous Deployment (CD) have become cornerstone practices that ensure software quality and agility. CI/CD pipelines serve as the backbone of modern...
Introduction In software development, documentation is the spinach of coding—vital yet often sidelined. It's the unsung hero, the nutrient-rich foundation that sustains projects and teams. Imagine each software project accompanied by a precise recipe...
Howdy! Welcome to the most practical deep dive you'll read on observability this year. All of your wildest dreams are about to come true.
If you've been around the DevOps and SRE space, you've heard the term "observability" thrown around more than a football and some coors at a backyard BBQ. But here's the thing – conversations about O11y (that's observability for those keeping score) focus on the shiny tools and fancy dashboards. We don't talk about the human side of observability or the culture it takes to be successful. Thats where I come in:
I'm Kyle Shelton, and I've spent the better part of 15 years getting my hands dirty in the trenches of SRE, DevOps, and Network Engineering. These days, I'm a Senior Observability Architect at Grafana Labs, where I get to help organizations transform how they understand and operate their systems. But before we dive into the technical stuff, let me level with you – I didn't start out as an observability expert. I learned it the hard way, through late-night outages, angry customers, and more "what the hell is happening right now?" moments than I care to count.
In distributed systems, microservices, and cloud-native architectures, traditional monitoring doesn't cut it. It's like trying to understand a conversation by only hearing every fifth word – you catch the general topic, but you miss the nuance and context that matters. Observability gives us the ability to ask questions we didn't know we needed to ask, to understand not just what happened, but why it happened.
Think about it this way: when your system goes sideways at 2 AM (and it will), you don't just want to know that CPU is at 90%. You want to know which service is consuming that CPU, what user request triggered the spike, how that relates to the database performance you've been tracking, and why your auto-scaling didn't kick in like it should have. That's the difference between monitoring and observability – it's the difference between playing whack-a-mole with symptoms and actually solving problems.
Now, you might be wondering why you should listen to some guy from Texas ramble about observability. Fair question! Beyond my day job of helping organizations wrangle their chaos into something resembling order, I'm passionate about things that inform how I approach observability:
Chaos and Platform Engineering – There's something beautiful about intentionally breaking things to understand how they fail. It's taught me that the best observability strategies are built around failure modes, not success stories.
AI Agents – I'm fascinated by how we can leverage AI to make sense of the mountains of data our systems generate. The future of observability isn't about collecting data; it's about intelligent systems that can reason about that data.
Racing and Simulation – Whether it's on a track or in a simulator, racing has taught me that the difference between winning and losing often comes down to telemetry data and making split-second decisions based on incomplete information. Sound familiar?
BBQ and Fishing – Patience, timing, and understanding that good things take time. Also, both involve a lot of waiting around with occasional bursts of intense activity – much like incident response!
Audio Engineering and Music Production – There's a direct parallel between mixing a song and tuning observability. You need to understand how all the individual components work together to create something greater than the sum of its parts.
This blog is structured around five key areas that I've found make the biggest difference when organizations are trying to level up their observability game:
O11y Culture – Before you install a single agent or write your first query, you need to get your team and organization aligned on what observability means and why it matters. This isn't just about tooling; it's about changing how people think about systems and problems.
Speed – How do you move fast without breaking things? (Spoiler alert: you don't. You break things faster and recover faster.) We'll talk about how observability enables velocity while maintaining reliability.
Scale – What works for your startup doesn't work for your enterprise, and what works for your enterprise might kill your startup. We'll explore how to build observability strategies that scale with your organization and systems.
Migration and Modernization – You can't just rip and replace your monitoring stack overnight. We'll discuss practical strategies for evolving your observability practice without disrupting your business.
Ideal State and Maturity Model – Where are you trying to go, and how do you know when you've gotten there? We'll build a framework for measuring and improving your observability maturity.
Each section is going to be packed with real-world examples, war stories from the trenches, and practical advice you can start implementing tomorrow. This isn't academic theory – this is battle-tested strategy from someone who's been there, done that, and has the scars to prove it.
So grab your favorite beverage, settle in, and let's talk about how to win with observability. Trust me, by the end of this, you'll have a completely different perspective on what it means to truly understand your systems.
Let me tell you something that might sound crazy: the biggest observability problems I've seen in 15 years aren't technical. They're cultural. You can throw all the Prometheus, Grafana, and fancy APM tools you want at a system, but if your team doesn't fundamentally believe that observability matters, you're building on quicksand.
Brian Chesky, the co-founder of Airbnb, put it perfectly: "Culture is so incredibly important because it is the foundation for all future innovation. If you break the culture, you break the machine that creates your products."
That quote isn't about hospitality or travel – it's about any company trying to build reliable systems at scale. And observability? It's the nervous system of that machine.
Before we dive into tools and dashboards, let's talk money. Because at the end of the day, if you can't articulate why observability directly impacts the bottom line, you're going to struggle to get buy-in from leadership and resources from finance.
Here's the reality: every minute of downtime costs money. Every frustrated customer costs money. Every engineer spending hours debugging instead of building features costs money. But here's what most people miss – observability doesn't just prevent these costs, it creates revenue opportunities.
I worked with a fintech company that was hemorrhaging $50K per hour during payment processing outages. Before implementing proper observability, they averaged 6 hours to detection and resolution. That's $300K per incident. After building a culture around observability and implementing the right tooling, they got that down to 15 minutes. Do the math – they saved $287,500 per incident. The entire observability platform paid for itself after the first prevented outage.
But the real ROI came from what they could build next. With confidence in their system's reliability, they launched new payment methods 3x faster. They could experiment with pricing models because they understood exactly how changes affected system performance. Observability transformed from a cost center to a competitive advantage.
The key metrics that matter to executives:
Mean Time to Detection (MTTD) – How fast do you know something's wrong?
Mean Time to Resolution (MTTR) – How fast can you fix it?
Customer-impacting incidents – What actually affects revenue?
Engineering velocity – How much time do teams spend debugging vs building?
SLO: Service Level Objective (The Goal)
SLI: Service Level Indicator (Do you meet the goal)
SLA: Service Level Agreement (Agreement for what happens when you dont meet goal $$$$$)
Now that we've established the why, let's talk about the how. Building an observability strategy isn't about picking the shiniest tools – it's about understanding your organization's unique needs and maturity level.
Understanding Cross-Functional Needs
Every team needs different things from observability…. I REPEAT EVERY TEAM IS DIFFERENT:
Engineering wants detailed traces, metrics, and logs to debug issues quickly
Operations needs infrastructure monitoring and capacity planning data
Product wants user experience metrics and feature adoption data
Business requires uptime SLAs and revenue impact analysis
Security needs everything always all the time
The magic happens when these perspectives align. When product managers can see how a new feature affects backend performance in real-time, when business leaders can correlate system reliability with customer satisfaction scores, when engineers can proactively scale resources based on predicted load – that's when observability becomes transformational.
Assessing Organizational Maturity
Not every company is ready for the same observability approach. I use a simple maturity model:
Reactive (Fire Fighting) – You find out about problems when customers complain. Monitoring is basic resource utilization. Teams work in silos.
Proactive (Early Warning) – You have alerts for known failure modes. Basic dashboards exist. Some cross-team collaboration on incidents.
Predictive (System Intelligence) – You can forecast issues before they happen. Rich context from traces, metrics, and logs. Strong incident response culture.
Autonomous (Self-Healing) – Systems automatically detect and remediate issues. Observability drives product decisions. Full organizational alignment.
Be honest about where you are. Trying to jump from reactive to autonomous overnight is like trying to run a marathon when you can barely walk around the block.
Designing a Centralized Observability Organization
One of the biggest mistakes I see companies make is treating observability as everyone's job and no one's responsibility. You need champions. You need a center of excellence. You need people whose job it is to make everyone else successful with observability.
This doesn't mean building an ivory tower team that owns all the tools. It means creating a group that:
Defines standards and best practices/builds telemetry pipeline catalouges
Provides tooling and infrastructure
Trains and supports other teams, enablement is key to adoption
Measures and improves observability across the organization
Identifying Champions
Champions aren't necessarily senior engineers or managers. They're the people who already ask "why did this happen?" instead of just "how do we fix it?" They're curious, they care about reliability, and they have influence with their peers. Find them, empower them, and give them air cover to drive change.
Let me be blunt: tool selection is where most observability initiatives go to die. Teams fall in love with vendor demos, get caught up in feature comparisons, and lose sight of what they're actually trying to accomplish.
Inventory Before Investment
Before you buy anything new, understand what you already have. I've seen companies spend six figures on monitoring tools while ignoring the perfectly good telemetry already flowing through their existing systems. Map out:
What metrics, logs, and traces you're already collecting
Where the gaps are in coverage or quality
How well your current tools integrate
What your teams actually use vs what's available, spending a bunch of money on one user is not the best approach imo
Open Source vs Commercial: The Real Tradeoffs
The open source vs commercial debate isn't about cost – it's about capability and capacity. Open source tools like Prometheus, Grafana, and Jaeger are incredibly powerful, but they require expertise to operate at scale. Commercial solutions like Datadog, New Relic, or Grafana Cloud offer convenience but can get expensive fast.
The real question is: do you want to be in the observability infrastructure business, or do you want to focus on using observability to improve your products? There's no wrong answer, but be honest about your team's capabilities and priorities. It takes an advanced skillset to run open source software at an enterprise level. It also takes alot out of that advanced skillset when owning/maintaining an OSS stack. Let that sink in.
Integration and Alignment
Whatever tools you choose, they need to work together. Siloed monitoring tools create siloed teams, and siloed teams create fragile systems. Look for:
Shared data models and schemas
Common authentication and access controls
Consistent user experiences across tools
APIs that allow custom integrations
Tools don't create culture – people do. And creating a culture of observability means fundamentally changing how teams think about systems, problems, and responsibility.
Cultural Shifts That Matter
From Blame to Learning – When something breaks, the first question should be "what can we learn?" not "who screwed up?" Blameless post-mortems aren't just nice to have – they're essential for building psychological safety around observability data.
From Reactive to Proactive – Instead of waiting for alerts, teams should be continuously exploring their systems. Schedule "observability office hours" where teams dig into their dashboards just to see what's happening.
From Siloed to Shared – Observability data should be accessible to everyone who needs it. Product managers should understand system metrics. Engineers should see business KPIs. Break down the walls between technical and business data.
Alignment Across Teams
Product teams need to understand that feature flags and gradual rollouts aren't just development conveniences – they're observability strategies. Every new feature should come with hypotheses about its impact on system performance.
Engineering teams need to think beyond just "does it work?" to "how will we know if it stops working?" Observability should be part of the definition of done for every story.
Operations teams need to evolve from reactive firefighters to proactive system optimizers. The goal isn't just keeping the lights on – it's helping the business make better decisions.
Leadership teams need to understand that observability is a competitive advantage, not just a cost center. When you can deploy faster, debug quicker, and understand your users better than your competitors, you win.
Security teams need to protect the assets and IP amongst the company, they should balance risk/speed
Metadata Alignment
Consistent tagging and labeling across all your telemetry data is like having a common language. Every service, every deployment, every user action should have consistent metadata that allows you to correlate across metrics, logs, and traces. This isn't glamorous work, but it's the foundation that makes everything else possible.
SLO-Driven Roadmaps
Service Level Objectives aren't just SRE concepts – they're business tools. Define what "good enough" looks like for your users, measure against those objectives, and use error budgets to make deployment decisions. When your reliability metrics are aligned with business goals, observability becomes a strategic asset.
Machine Learning and Anomaly Detection
AI isn't going to replace observability engineers, but it's going to make them superhuman. Start simple with baseline alerting that learns normal patterns and alerts on deviations. The goal isn't to eliminate human judgment – it's to focus human attention on what matters most.
Observability as Code
Your dashboards, alerts, and SLOs should be version controlled, code reviewed, and deployed just like your applications. When observability configuration lives in code, it evolves with your systems instead of becoming stale technical debt. GITOPS is the way, I will die on that sword
Incident Response Culture
Every incident is a learning opportunity. Build runbooks that capture not just what to do, but how to investigate. Train people on your observability tools before they need them in an emergency. Practice incident response during calm periods so you're ready during storms.
The reality is that building an observability culture is hard work. It requires changing minds, not just installing tools. It requires patience, persistence, and a willingness to invest in capabilities that might not pay off immediately but will transform how your organization operates.
But here's what I know after 15 years in this space: companies that get observability culture right don't just have more reliable systems – they move faster, take smarter risks, and build better products. They turn uncertainty into confidence and chaos into competitive advantage.
And in a world where every company is becoming a software company, that might be the most important capability of all.
There's a misconception in engineering that speed and reliability are opposing forces – that you have to choose between moving fast and building stable systems. After 15 years of watching teams struggle with this false choice, I can tell you with certainty: that's complete nonsense. The fastest teams I've worked with are also the most reliable. And the secret ingredient? Observability.
Think about racing for a minute. A NASCAR driver doesn't slow down because they have more telemetry – they go faster because they can see what's happening. Every modern race car is loaded with sensors measuring tire pressure, engine temperature, fuel flow, G-forces, and dozens of other metrics. That data doesn't make drivers cautious; it makes them confident enough to push harder because they know exactly when they're approaching the limits.
Software development works the same way. When you can see what's happening in your systems in real-time, when you can understand the impact of changes immediately, when you can detect and resolve issues in minutes instead of hours – you don't slow down. You accelerate.
Let's be honest about where most organizations are today. I've walked into dozens of companies that are stuck in what I call "the fear cycle" – moving slowly because they're afraid of breaking things, which means they break things more often because they can't see what's happening.
Slow Release Schedules
I recently worked with a fintech company that was releasing code every two weeks. Not because they couldn't develop faster, but because they couldn't deploy safely. Every release required a three-hour maintenance window, manual testing in production, and a dedicated engineer babysitting the deployment.
Their competition was shipping multiple times per day. Guess who was winning in the market?
When you can't see the impact of your changes in real-time, every deployment becomes a gamble. Teams compensate by batching changes together, which makes deployments riskier, which makes teams more cautious, which slows down releases even more. It's a vicious cycle.
Long MTTX (Mean Time to Everything)
The pain isn't just in how long it takes to deploy – it's in how long everything takes:
Mean Time to Detection (MTTD): How long before you know something's wrong? In organizations without proper observability, this averages 4-6 hours. That's 4-6 hours of customers experiencing problems while you're blissfully unaware.
Mean Time to Resolution (MTTR): How long to fix issues once you know about them? Without observability, engineers spend 80% of their time figuring out what's wrong and 20% actually fixing it.
Mean Time to Context (MTTC): How long to understand what changed and why? When incidents happen, teams waste precious time playing detective instead of solving problems.
High TCO (Total Cost of Ownership)
Poor observability creates hidden costs everywhere:
Engineers spending nights and weekends firefighting instead of building features
Customer churn from reliability issues you can't detect or fix quickly
Over-provisioned infrastructure because you don't understand actual usage patterns
Technical debt accumulating because you can't see the impact of shortcuts
I've seen companies spend more on infrastructure over-provisioning than they would have spent on a world-class observability platform.
Unhappy Engineers
Here's the human cost nobody talks about: when engineers can't see what their code is doing in production, work becomes frustrating and stressful. They ship features into a black box and hope for the best. They get woken up at 2 AM to debug issues they can't understand. They spend days chasing symptoms instead of solving root causes.
Happy engineers write better code. Happy engineers stick around longer. Happy engineers innovate. Observability isn't just about system health – it's about engineer health.
Now let me paint a picture of what's possible when you get observability right. I've seen teams completely transform their velocity and happiness by investing in the right observability culture and tooling.
Faster Releases with More Features
The best teams I work with deploy code dozens of times per day. Not because they're reckless, but because they can see exactly what's happening and respond instantly if something goes wrong.
One e-commerce company I worked with went from monthly releases to 50+ deployments per day after implementing proper observability. Their time-to-market for new features dropped from months to days. Their competitive advantage shifted from "having the best features" to "learning and adapting fastest."
When you can deploy with confidence, you can experiment aggressively. When you can see the business impact of changes in real-time, you can iterate based on actual user behavior instead of assumptions.
Low MTTX Across the Board
With proper observability, those painful time-to-X metrics transform:
MTTD drops to seconds: Automated alerting based on real user impact, not just infrastructure metrics
MTTR drops to minutes: Rich context from traces, logs, and metrics means engineers know exactly what to fix
MTTC becomes instant: Deployment markers, change tracking, and correlation analysis show exactly what changed when
I've seen incident resolution times drop from hours to under 15 minutes. The same types of issues, the same engineers, but now they have the data they need to solve problems instead of guess at them.
Lower TCO Through Efficiency
Observability pays for itself through efficiency gains:
Right-sized infrastructure based on actual usage patterns, Just in time concept made famous by Toyota
Reduced firefighting means engineers build features instead of fixing things
Automated scaling and self-healing systems reduce manual intervention
Faster problem resolution reduces customer impact and churn
Happier Engineers
When engineers can see what their code is doing in production, work becomes satisfying again. They can validate that their features are working as intended. They can optimize performance based on real data. They can debug issues quickly instead of spending days playing detective.
More importantly, they can be proactive instead of reactive. Instead of getting woken up by alerts, they can see issues coming and prevent them. Instead of endless war rooms, they can solve problems with data and context.
Every championship team – whether in sports, business, or technology – has one thing in common: they make better decisions faster than their competition. Speed isn't just about going fast; it's about the velocity of learning and adaptation.
The Feedback Loop Advantage
In racing, the teams that win championships aren't necessarily the ones with the fastest cars on day one. They're the teams that can make the right adjustments fastest. They collect telemetry data during practice, analyze it between sessions, and make setup changes that give them an edge in qualifying and the race.
Software works the same way. The companies that win aren't the ones with perfect products on launch day – they're the ones that can learn from user behavior and adapt their products faster than competitors.
Netflix didn't beat Blockbuster because they had better movies. They beat them because they could see what users actually watched and recommend better content. Amazon didn't win because they had lower prices – they won because they could optimize the entire customer experience based on real behavioral data.
Competitive Velocity Through Observability
When your competition is still deploying monthly and debugging for hours, every improvement you make to observability creates competitive distance:
You can respond to market changes faster
You can experiment with new features without fear
You can optimize user experiences based on real data
You can scale efficiently as demand grows
The Compounding Effect
Here's where the racing analogy gets really interesting. In NASCAR, small advantages compound over time. A car that's 0.1 seconds per lap faster doesn't just win by 0.1 seconds – over 500 laps, that's 50 seconds. That's the difference between first place and last place.
Observability creates the same compounding effect. When you can deploy 10% faster, detect issues 10% quicker, and resolve problems 10% more efficiently, those improvements compound. Over months and years, they create massive competitive advantages. STONKS
Learning from the Track: Tire Tests and Victory
Speaking of racing, I had an experience last year that perfectly illustrates how observability drives performance. I had the opportunity to attend a tire test with TRD at Circuit of the Americas (COTA). For those who don't follow NASCAR, tire tests are where teams work directly with Goodyear to develop and validate new tire compounds for upcoming races.
What struck me wasn't just the complexity of the data collection – tire temperatures at dozens of points across each tire, suspension telemetry, aerodynamic pressure measurements, fuel consumption rates – but how that data directly informed strategy. The engineers weren't just collecting data for curiosity; every data point fed into decisions about tire pressure, suspension setup, and race strategy.
Fast forward a few months to the NASCAR Cup Series race at COTA, and Tyler Reddick – driving the same car I'd seen in testing – won the race. The connection wasn't coincidental. The data collected during that tire test, the understanding of how different compounds performed under various conditions, the insights into optimal setup configurations – all of that observability work translated directly into victory on race day.
I was at the race and here was the view of the burnout from the paddock:
That's the power of observability done right. It's not about having more data; it's about having the right data at the right time to make winning decisions. Whether you're optimizing tire pressure for turn 12 at COTA or optimizing API response times for your checkout flow, the principle is the same: see everything, understand everything, win everything.
Start with Deployment Visibility
If you can only instrument one thing, make it your deployments. Every deployment should create observable events that you can correlate with system and business metrics. When something goes wrong, the first question should be "what changed?" not "where should we start looking?"
Build Confidence Through Automation
Speed requires confidence, and confidence comes from automation. Automated testing gives you confidence in code quality. Automated deployment pipelines give you confidence in release processes. Automated monitoring and alerting give you confidence that you'll know immediately if something goes wrong.
Measure What Matters to Speed
Track metrics that directly correlate with velocity:
Deployment frequency and success rate
Time from commit to production
Mean time to detection and resolution
Feature flag adoption and experiment velocity
Engineer satisfaction with debugging and deployment processes
Speed isn't just about technical metrics – it's about team velocity, learning velocity, and business velocity. When you can see the connection between system performance and business outcomes, you can optimize for what actually matters.
The reality is that speed and reliability aren't opposing forces – they're complementary capabilities that both require observability to achieve. The fastest teams are also the most reliable because they can see what's happening and respond immediately when things go wrong.
But here's the secret: speed isn't just about moving fast. It's about moving fast in the right direction. And you can only do that when you can see where you're going and understand the impact of every decision you make.
That's the championship advantage that observability provides. Not just better systems, but better decisions. Not just faster deployments, but faster learning. Not just more reliable software, but more confident teams.
In the end, speed wins. And observability is what makes speed possible. Thats half of this series, next week I will go over scale, migrations/modernization, & the ideal mature state of observability. Thanks for making it this far, All of your wildest dreams will come true!
Kyle
