Search for a command to run...
Outdoorsman | Networking | SRE | Chaos & Platform Engineering | DevOps @toyotaracing | former @aws @splunk @verizon @gm | Thoughts are my own
Personal: Husband | GirldadX3 | BBQ | Outdoors | Nascar | Baseball | Triathlons
How to scale, modernize, and mature your observability practice
Part 1 of a 2 part series
A Guide for Software Architects
Introduction In the rapidly evolving world of software development, Continuous Integration (CI) and Continuous Deployment (CD) have become cornerstone practices that ensure software quality and agility. CI/CD pipelines serve as the backbone of modern...
Introduction In software development, documentation is the spinach of coding—vital yet often sidelined. It's the unsung hero, the nutrient-rich foundation that sustains projects and teams. Imagine each software project accompanied by a precise recipe...
Imagine yourself as a kid again, lining up with classmates as the shrill sound of the tornado drill fills the corridors. Or picture a more recent scenario - a fire drill at work, the building's pulse-quickening as everyone calmly but quickly heads for the exits. In both situations, the drill was all about being prepared for the unexpected. Scouts motto- "Be Prepared". also a great song from lion king
In the world of platform engineering, we have a similar approach to these drills - it's called Disaster Recovery (DR). It's not just an emergency protocol, but our metaphorical storm shelter. DR, in the context of IT and platform engineering, is a set of policies and procedures designed to prepare for and recover from potential threats that could buckle our business operations.
DR is not just about backing up your data - that's like knowing the evacuation route during a fire drill. Important, yes, but not the whole picture. Disaster Recovery is the full safety drill, the methodical plan designed to safeguard us from the catastrophic effect of a disaster. From a network outage to a natural disaster, it's our survival kit in the IT wilderness.
In the context of platform engineering, Cloud Disaster Recovery (CDR) is an essential concept to grasp. CDR involves storing and maintaining copies of electronic records in a cloud environment, thus facilitating efficient backup and recovery procedures.
When compared to traditional on-premise Disaster Recovery, Cloud-based DR exhibits significant advantages. On-premise DR solutions can be labor-intensive and expensive to maintain. They require substantial upfront investment in hardware, software, and infrastructure, not to mention the ongoing cost of operating and maintaining these systems.
On the other hand, Cloud DR offers scalability, cost-effectiveness, and automation. It allows businesses to adjust their DR resources based on actual needs, providing potential cost savings and flexibility. It also reduces the burden of manual tasks through automation, allowing IT teams to focus more on strategic tasks.
When I was working at Verizon, our on-premise Disaster Recovery systems required us to build our applications with a 40% overhead for compute and storage capacity. This was done to ensure that we could handle spikes in demand or recover from disasters effectively. However, this approach meant significant investment in infrastructure that was not always fully utilized, given that rare were the instances when we'd failover and maintain an 80% max.
In contrast, Cloud DR offers scalability, cost-effectiveness, and automation. It enables businesses to adjust their DR resources based on actual needs, thereby reducing wastage and providing potential cost savings and flexibility. Automation within Cloud DR also alleviates the burden of manual tasks, allowing Engineering teams to focus more on strategic tasks.
However, transitioning to Cloud DR isn't without its challenges. These include data security and compliance requirements, ensuring a reliable and robust internet connection, managing costs, and dealing with dependencies from providers.
In the following sections, we'll explore these concepts in more detail, providing a comprehensive understanding of how Cloud Disaster Recovery contributes to maintaining resilient and robust IT operations.
In the grand scheme of Disaster Recovery (DR), understanding key concepts is as important as understanding the strategy behind a complex chess game. These concepts dictate how we prepare for, respond to, and recover from disruptions. Let's tackle some of the most crucial ones:
RTO (Recovery Time Objective): Think of this as a countdown clock. It’s the targeted duration of time within which a business process must be restored after a disaster in order to avoid unacceptable losses.
RPO (Recovery Point Objective): This is your checkpoint in a video game. It defines the maximum tolerable period in which data might be lost due to a major incident.
SLA (Service Level Agreement): This is a commitment between a service provider and a client, outlining the level and quality of service to be provided. In our case, it defines the expected availability and performance of the DR solutions.
HA (High Availability): This is our goal post. It’s a characteristic of a system that aims to ensure an agreed level of operational performance for a higher than normal period.
MTTA (Mean Time to Acknowledge) and MTTR (Mean Time to Repair): These are our timers. MTTA is the average time it takes for a system to respond to a detected problem, while MTTR is the average time it takes to fix a failed component and return it to operational status.
Failover and Failback/Fallback: Failover is the process of switching to a redundant or standby system in the event of a failure. Failback is the subsequent process of returning to the original system once it is up and running again.
Redundancy and Replication: Redundancy is the duplication of critical components to increase reliability of the system, while replication is the frequent copying of data to a secondary site to enable quick recovery.
BCP (Business Continuity Planning): This is our broader strategy. It encompasses the process of creating systems of prevention and recovery to deal with potential threats to a company.
Hot Standby, Pilot Light, and Cold Standby: These are different DR strategies. Hot Standby involves having a duplicate system always running. Pilot Light keeps a minimal version of an environment always running that can be fired up like a pilot light on a heater, while Cold Standby only starts up the duplicate environment when a disaster is declared. I spoke about Chaos Engineering and these concepts in detail in a talk here: https://www.youtube.com/watch?v=9den8fe82ck
And one more for the road - Disaster Recovery as a Service (DRaaS): This is a cloud computing service model that allows an organization to back up its data and IT infrastructure in a third-party cloud computing environment and provide all the DR orchestration, all through a SaaS solution.
Understanding these terms is foundational to implementing a robust and resilient DR strategy.
Moving along our journey of understanding Cloud Disaster Recovery, let's park for a moment at a crucial pit-stop: planning and components of a Cloud Disaster Recovery Plan. This involves three key elements: Risk Assessment/Business Impact Analysis, DR Strategies, and DR Plan Testing.
Risk Assessment and Business Impact Analysis: Before you set out on any journey, you need to understand the potential roadblocks and challenges you might face. In our DR journey, this comes in the form of Risk Assessment and Business Impact Analysis. Risk Assessment is about identifying potential threats to your IT infrastructure, such as hardware failure, data breaches, or natural disasters. Business Impact Analysis, on the other hand, helps quantify the potential cost of these risks. It answers questions like, "What would be the financial impact of an hour of downtime?" or "What departments would be most affected by a server failure?"
DR Strategies: Once you've assessed the risks and understood their impact, the next step is to map out your journey, i.e., develop your DR strategies. There are several approaches you can take:
Backup & Restore: This is the most basic form of DR. It involves creating copies of your data at regular intervals and storing them off-site or in the cloud. In case of a disaster, you can restore your system from the latest backup.
Pilot Light: Imagine keeping a small replica of your IT environment always running. In the event of a disaster, this "pilot light" can be rapidly scaled up to replicate your production environment.
Warm Standby: A step up from Pilot Light, Warm Standby keeps a scaled-down version of a fully functional environment always running. In a disaster scenario, this environment can be quickly scaled up to handle the production load.
Multi-site: For businesses with a low tolerance for downtime, a multi-site approach might be the way to go. This strategy involves duplicating your IT infrastructure across multiple sites (which could be different geographical locations or different cloud regions). If one site goes down, the others can take over.
DR Plan Testing: A journey planned is only as good as its execution. Regular testing of your DR plan is crucial to ensure it works as expected when disaster strikes. It's the equivalent of a dress rehearsal before the main event. DR plan testing can uncover gaps or weaknesses in your strategy, giving you a chance to fix them before a real disaster occurs.
Remember, planning is an ongoing process and takes constant improvement or Kaizen as we call it at toyota. As your business changes and grows, so too will your risks and impacts. Regularly reviewing and updating your DR plan is key to ensuring you're always prepared for the worst.
Check out my FREE DR Guide and Notion Templates (Ones I use for consulting) for DR planning and Incident Command here: Guides and Notion Templates
Life's full of surprises and, unfortunately, not all of them are pleasant. Especially in the cloud, where anything can go wrong. Like a river guide preparing for white water, a good engineer must always expect the unexpected. Let's dive into various types of service events and levels of outages, as we try to navigate these unpredictable waters.
Types of Service Events / Levels of Outages:
Service events in the cloud can be categorized by their scope and severity, ranging from minor hiccups affecting a single instance, to major catastrophes taking down an entire region.
Instance or Service Level Outages: This is like having a flat tire on your road trip. It affects a single instance or a specific service within a cloud provider's offering. An example could be a failure of a single Amazon EC2 instance or a temporary glitch in Azure's Storage service.
Availability Zone Outages: Stepping up in severity, we have outages that affect an entire Availability Zone (AZ). Imagine if a power outage hit your whole neighborhood. A case in point is the AWS Sydney AZ outage in 2017, where a storm caused power loss to the entire zone.
Region-wide Outages: Now imagine if the power went out across your whole city. That's the equivalent of a region-wide outage. These are rare but significant events, like the GCP europe-west1 region outage in 2019, which affected all services across the region.
Provider-wide Outages: The most significant and rarest of outages, these affect multiple regions and sometimes even the entirety of a cloud provider's services. It's like a national power grid failing. Though rare, these can and have happened, such as the widespread Azure authentication outage in 2021, which affected users globally.
Cloud Provider Major Outages:
Even the best players in the field aren't immune to unexpected service events. For a better understanding, let's take a peek at the history books for AWS, Azure, and GCP. Each of these providers maintains an event history page, where you can learn about past incidents:
Azure: Azure Status History
Remember, no matter how well you plan, there's always an element of unpredictability in the cloud. The key is to learn from these events and adapt your strategies accordingly, ensuring your platform engineering efforts are resilient, robust, and ready to tackle whatever comes next.
Embarking further into our exploration of Cloud Disaster Recovery, we now tackle a critical component: creating a Disaster Recovery (DR) Plan and a Business Continuity Plan (BCP). These are your lifelines in the face of potential disaster, providing a blueprint and a navigation guide through the maze of disruptions.
Steps to Create a DR Plan:
Identify Critical Assets: Your journey begins by identifying the critical assets to your business. This could include data, applications, and infrastructure integral to your business operations.
Perform Risk Assessment and Business Impact Analysis: Equipped with a clear understanding of your vital assets, carry out a Risk Assessment and Business Impact Analysis. This helps to identify potential vulnerabilities, quantify their potential impact, and prioritize your recovery efforts.
Define Recovery Objectives: With your Business Impact Analysis in hand, you can define your Recovery Time Objective (RTO) and Recovery Point Objective (RPO), ensuring your recovery efforts align with your business needs.
Design and Implement Your DR Strategies: Pick the DR strategy that best aligns with your business needs, be it backup & restore, pilot light, warm standby, or multi-site, and then implement it.
Plan Testing, Review, and Updates: A plan is only as good as its execution. Regular testing of your DR Plan ensures its effectiveness while regular review and updates keep the plan relevant as your business evolves.
Steps to Create a Business Continuity Plan:
Business Impact Analysis: Expand your Risk Assessment from the DR plan to identify the broader implications of potential loss scenarios on your business processes.
Recovery Strategies: Develop recovery strategies to ensure the continuation of your business processes. This could involve relocation of operations, outsourcing to third parties, or any other viable means.
Plan Development: Craft your BCP document, which outlines the steps necessary for business process recovery.
Training, Testing, and Exercises: Your team should be well-versed in their roles during a disaster. Conduct training and tests of your BCP which could range from tabletop exercises, and drills, to full-scale exercises.
Plan Maintenance: Your BCP is a living, breathing document. As your business changes, your BCP should adapt. Regular updates and revisions keep your plan current and effective.
Importance of Documentation and Communication:
An excellent plan that no one knows about is as good as no plan at all. Document your DR Plan and BCP clearly and ensure they are readily accessible to all relevant personnel.
Similarly, effective communication is paramount during a disaster. Have a communication plan in place, specifying who will communicate, what information, to whom, and how during a disaster. Or better yet, if you have the budget, Hire a full on Incident Command Team.
As we near the end of our DR and BCP creation journey, remember that their creation is just the beginning. Keeping these plans effective requires regular reviews, updates, tests, and clear communication. But as any good platform engineer knows, the work doesn't stop here. Stay tuned as we move on to our next critical area - Incident Command and Management.
The importance of Incident Command (IC) in Cloud Disaster Recovery can be compared to the crucial role of a skilled captain navigating a ship during a storm. Inspired by the Incident Management Systems used by the military and firefighters, IC provides a structured approach to managing IT incidents that can turn the tide in favor of an organization during a disaster.
Building a Team Focused on Incidents and Incident Command:
Just like a well-oiled ship has a dedicated crew, a well-functioning Incident Command System requires a team of trained professionals. Drawing on my time as an SRE at Splunk, I can confidently attest to the importance of building a specialized team to manage incidents and incident command.
The team should include:
Incident Commander (IC): This is the person at the helm of the operation. They're responsible for making decisions, coordinating resources, and communicating with the rest of the team. The buck stops with them. They set time contracts, and push to the point of resolution.
Communications Officer: This team member manages all external and internal communications, ensuring that everyone is in the loop and updated about the status of the incident.
Note Taker: This role may seem minor, but it's actually crucial. The Note Taker is responsible for documenting everything that happens during an incident. This can be vital for post-incident analysis and improving future responses.
Technical Lead: This person brings technical expertise to the table and guides the team in resolving the technical aspects of an incident.
Executive Liaison: This individual is the bridge between the IC team and the organization's executive management. They keep the executives informed about the status of the incident and seek their support when necessary. They also keep the execs from throwing grenades into technical conversations. This is a very important role and requires good communication skills.
During my tenure at Splunk, our dedicated incident command team, comprising these roles, was instrumental in effectively managing disaster recovery for our Cloud SaaS product.
Incident Management System (IMS):
IMS is a standardized approach to managing incidents, regardless of their scale or complexity. It provides clear chains of command and communication, ensuring that all team members understand their responsibilities and can efficiently perform their duties under pressure.
Communication Styles and Executive Buy-in:
Incident Command isn't just about having the right team and following a proven methodology. It's also about effective communication and executive buy-in. Every incident should be treated as an opportunity to learn, improve, and get the executive team more involved in the incident management process. At Splunk, the executive team was always supportive and saw the value in our IC practices, which was key to the success of our incident response.
CANN Reports:
One effective tool in managing incidents is the CANN report. CANN stands for Condition, Action, Needs, and Next. It's a concise framework that keeps everyone updated about the status of an incident and the next steps. At Splunk, we found the CANN report immensely helpful in organizing our response and keeping everyone informed.
In our next segment, we'll conclude by revisiting the importance of Cloud Disaster Recovery and providing some key takeaways.
Now that we've navigated the high seas of Cloud Disaster Recovery, it's time to anchor down some key considerations and best practices:
Regular Testing and Auditing of the DR Plan:
Like any good adventurer, you need to know your gear inside and out. It's important to regularly test your DR plan and audit its effectiveness. This can expose vulnerabilities and areas that need improvement, ensuring your plan evolves and stays robust over time.
Considering Cost, Security, Compliance, and Business Needs:
When it comes to DR, it isn't a one-size-fits-all solution. Each organization has unique needs and considerations. Balancing cost, security, compliance, and business needs is crucial in building an effective DR plan. Remember, the goal isn't just to recover, but to ensure that recovery doesn't break the bank or compromise security.
Importance of Employee Training:
Even the best DR plan won't do much good if your crew isn't prepared to use it. Regular training for all relevant employees is key. This ensures that when disaster strikes, everyone knows their role and can execute the plan effectively.
From the schoolyard to the data center, the importance of a good evacuation (or in this case, recovery) plan has always been clear. In our world of platform engineering, the potential disasters might be virtual, but the consequences of not being prepared can be all too real.
Cloud Disaster Recovery is not just a lifeline—it's a beacon, guiding us towards a future where downtime becomes a ghost of the past. It's up to us, as engineers, SREs, and DevOps professionals, to continue learning, adapting, and innovating as technology evolves.
And remember—though Cloud Disaster Recovery might sound daunting, it's easier to grapple with than the task of explaining your prolonged downtime to your boss.
Before we end, here's a dad joke to lighten the mood: Why don't some engineers go on a diet? Because they can't resist a byte!
1. What is the difference between Disaster Recovery and a Backup?
While both disaster recovery and backup strategies aim to safeguard your data, they serve different functions. A backup is the process of making an extra copy (or copies) of data. You might think of it as a spare tire. Disaster recovery, however, is a strategy for responding to a catastrophic event. It's your car's entire emergency kit — it encompasses more than just data and may involve hardware, software, networking equipment, power, cooling, physical space, and people.
2. Is Disaster Recovery necessary for small businesses?
Regardless of the size of your business, data is probably one of your most valuable and critical assets. Therefore, ensuring that your business can continue to function during and after a disaster is vital. So, whether you're a one-person show or a multinational corporation, you need to have a disaster recovery plan in place.
3. How often should you test a Disaster Recovery Plan?
The frequency of DR testing varies depending on the needs and resources of your organization. However, best practices recommend conducting a full-scale DR test at least once a year. It's also beneficial to perform component testing, such as recovering individual applications, more frequently, perhaps every quarter.
4. Who is involved in a Disaster Recovery Plan?
While the IT department plays a major role, disaster recovery involves more than just the IT team. Executives should be invested in the process because it's a risk management issue that affects the entire business. It's also important to include representatives from various departments across your organization to ensure all aspects of your business are considered and included in the DR plan.
5. What's the role of cloud service providers in disaster recovery?
Cloud service providers play a pivotal role in disaster recovery. They offer services that can be leveraged to implement effective and efficient DR strategies. These may include data replication and backup, as well as resources for running applications in the cloud when on-premise infrastructure is unavailable. However, it's essential to remember that using cloud services doesn't absolve you of your responsibility for DR planning — you still need to set up and manage your recovery processes.
6. What are some common challenges in executing a DR plan?
Some common challenges in executing a DR plan include lack of understanding among staff, hardware compatibility issues during recovery, outdated DR plans, and lack of testing and updating of the DR plan. These challenges can be mitigated by training, regular testing, and updates to the DR plan.
7. Why do I need a Business Continuity Plan (BCP) in addition to a Disaster Recovery Plan?
A Business Continuity Plan and a Disaster Recovery Plan are two sides of the same coin. While a DR plan focuses on restoring IT infrastructure and systems to operation, a BCP ensures that the rest of your business operations can continue during a disaster. This includes everything from logistics and supply chain management to customer service and marketing operations.
