90% of all DevOps initiatives without a culture component fail, and culture is not easily or quickly changed.
Last Tuesday, I participated in an online panel on the subject of Implementing a DevOps Culture, as part of Continuous Discussions (#c9d9), a series of community panels about Agile, Continuous Delivery and DevOps. Watch a recording of the panel:
Continuous Discussions is a community initiative by Electric Cloud, which powers Continuous Delivery at businesses like SpaceX, Cisco, GE and E*TRADE by automating their build, test and deployment processes.
Below are a few insights from my contribution to the panel:
What’s so great about implementing a DevOps culture?
Most of my DevOps culture-experience comes from GameDuell of course and I’m writing that openly here, because anybody can see that on my LinkedIn profile anyways. Back in the days we were doing pseudo-agile. We’ve had some teams working with proper scrum, a 2-week sprint cycle, which is pretty much the average, but for every week of development work, it took us another week to get stuff to production. That was a tremendous bite into our work time, the time we needed to actually produce software. So to me:
DevOps means a stop to throwing things over the fence, collaborating on deploying working software first try.
Here Mike Kavis had responded that one of the issues with agile movement was it being all about speed and neglecting proper architecture and DevOps originating from the willingness to keep the speed, but bring reliability back in.
We rushed to the next topic, but I can’t quite agree with that: In my opinion, proper agile with empowered Product Owners and capable developers builds quality in straightaway. Developers have the right and the responsibility to speak up if they think, know, assume some shortcuts had been taken (for the businesses sake) that make the software they built less reliable in production. After that it is an empowered POs responsibility to plan for repaying this technical debt.
Silo busting and system level thinking
Silo busting is applicable to any type of practice – it’s not only software development and ops, it’s also within software development, you often see conflicts between back-end and front-end, one practice not appreciating what the other does (“anyone can do front-end”), but you need real experts to do it properly, and you need these experts to work together.
For me silo busting is fundamentally important because it puts an end to throwing things over the fence – “Here’s where my responsibility ends and yours starts!”. That is never a distinct border, it’s always a gray zone, and people don’t go into that gray zone, they always leave responsibility behind and things crash in the end. Collaborating on getting something live, that’s what it’s about for me.
Freedom to experiment
Apply hot glue to the agile house of cards!
If you’re doing DevOps and agile, you need very good automation that gives you security when experimenting. Developers are always eager to try new technologies in a certain module, and if you have the monolith, you certainly can’t try new technology as easily. Ops tries to run things as smooth as possible so they don’t want to experiment, or they only experiment with the sole purpose of making things more reliable. These two things are in constant conflict.
That’s why you need this culture, not having the feeling of working against each other but working together – “We’re trying this new technology for this purpose.“, with ops supporting that, and vice verse, ops saying “We want to make this change to the system to run your software this much more smoothly“. That’s why automated tests are very important, and iterating in small chunks. A previous panelist mentioned that working on the monolith is like a house of cards, this way it’s adding one more card onto a house of cards which is already glued together.
Joys of rapid feedback
For me feedback is a fundamental thing for the culture. If something goes wrong in the live system somebody’s got to take the blame, and no human being steps forward and says, without knowing what happened, “That was me! It’s my fault!“. Rapid feedback takes out the blame game, you know exactly what caused the problem, it’s your name on the monitor, you broke the build, and you say yes, that’s on me, I’ll fix it immediately and in 10 minutes it’s fixed because I know the change that did it.
We still love you but we’re going to lunch without you because you’ve got to fix the build.
Also if it’s a small set of changes, you don’t need to go through a huge change set to find out who’s responsible. It can take weeks to find out who introduced what change, and the responsibility is gone, why are you bothering me with something I did two weeks ago. Rapid feedback takes out the finger-pointing, this goes hand in hand and is a prerequisite for a blameless culture. It’s not about blame anymore because it’s just a fact, it was me and I’ll fix it.
Bottom line I really enjoyed the discussion and after being skeptical beforehand I’m convinced in this being a great format and feel almost privileged for having had the chance to share experiences with this round of panelists!
Everything I described and praised I could live and contribute to and experience at GameDuell, where after a (sometimes painful) agile transition, followed by an even harder transition to CI everybody really lived that culture. A great work experience and a shining example for teamwork, agile, responsibility and all that 🙂