Last week we discussed the first four principles of SRE: Monitoring, toil, SLOs and risk. This week we are going to discuss the last 3: Automation, Release Engineering, and Simplicity. Automation moves the needle quite a bit when it comes to margins/productivity and should be at the core of every SRE. When hiring SRE’s I look for automation first in their resume. You have to be able to find ways to eliminate toil and reduce repetitive tasks. OPS WORK SUCKS, Nobody likes patching shit or updating IAM roles. Simply put (:D), these next three principles will make your life as SRE better. Oh and Automation should be the foundation of everything you build, I can’t say that enough.
SRE Principle 5: Automation
Automation creates and/or improves velocity by removing human input on tasks. Engineers or developers can focus on more high value/return areas of work for the business. Use cases include testing, deployment, incident response, and communication. Find the tasks that your team repeats and buy or build tools to automate those tasks. Continue to optimize and always look for areas of improvement. Having something automated doesn’t necessarily mean it’s the correct or most efficient way. Always have automation in mind even when in development. Make sure your applications can easily be integrated or implemented with your automated tools.
Early on in my career at Splunk we had automation in place via Ansible and it performed the remedial tasks required to build customer stacks. It worked for the most part but was a pain in the ass for our larger customers. The next iteration of automation was moving from Ansible to Puppet which made building stacks a simple PR and Jenkins run and BOOM. By making these changes we were able to build more stacks faster with less people which fueled the crazy growth going on at the time. There’s the saying lazy engineers are the best engineers and I have a problem with that statement. Just because someone doesn’t want to do something doesn’t make them lazy. I believe that efficiency naturally looks like laziness but do you think a bear is lazy because he hibernates? No fuck that cold and snow shit.
Automation is key to software delivery, point blank, end of story, thanks for listening to my TED talk.
SRE Principle 6: Release Engineering
Release engineering is building and deploying software consistently in a stable, repeatable way. Good release engineering has configuration management. You need to create a singular, agreed upon standard for how releases should be configured. Some releases may need changes, but they should be modified or appended to a baseline configuration. Like everything, having good process documentation is key to solid release engineering. This will reduce the toil of having to know what to do all the time and also contribute to reliability. Continually review the process to make sure they stay up to date. Automate the shit out of how you’re deploying your releases and make it quick. Make sure you also test releases via blue.green or canary deployment. TEST TEST TEST.
SRE Principle 7: Simplicity
KISS: Keep It Simple Stupid- Thats it…
Photo by Pablo Arroyo on Unsplash
Seriously, simplicity is crucial when dealing with large distributed systems. Reliability and simplicity go hand and hand. Simple systems are much easier to fix, observe, improve, and replace. Think about your workloads on both micro and macro levels. Look for unnecessary loads or steps that can be reused. Mature systems naturally will become more complex over time so always look to remove areas of unnecessary complexity. Build metrics to evaluate everything. Have targets to shoot for right? Always try to make the release as seamless as possible and never force your users to do things that they don’t want to or put them in a bad spot.
After reviewing the principles of SRE, maybe you’re ready to turn your idea into a software defined product.. Or maybe not and you want to know what the hell software defined means! Cool, next week I’’l dive into software defined X and how companies are transforming their businesses digitally. Cheers friends