DevOps by the Data (Guests: Nicole Forsgren & Jez Humble)

[Music]
Ben Newton: Welcome to Masters of Data. The podcast where we talk about how data affects our businesses and our lives, and we talk to the people on the front-lines of the data revolution. And I'm your host Ben Newton. The DevOps movement is one of the most consequential, cultural and technological ideas to hit information technology and software engineering teams in the last couple of decades. At its core DevOps is a grassroots movement to reduce the friction between the people building applications, developers and the people keeping those applications humming, operations, in order to decrease the time to market for innovative products, all while maintaining quality.
DevOps brings some of the best elements from other movements before it like lean manufacturing and agile software development. This episode, we talk with two of the co-founders of the DevOps research and assessment LLC Dora, Nichole Forsgren, CEO and chief scientist and Jez Humble, CTO. Nicole and Jez are two of the key thought leaders in the DevOps movement, and have done more than most to evangelize and legitimize DevOps best practices.
Through their extensive research surveys and in-depth analysis at Dora, they along with her co-founder Jean Kim, have built a strong data backed foundation for DevOps. We will discuss their brand new book Accelerate, where they brought this research together and present some fascinating conclusions about DevOps, and the impact it can have on organizations when implemented well. So, without any further ado, let's get started.
Welcome everybody to Masters of Data, and I'm super excited about our guests today. They are co-founders of the DevOps research and assessment LLC. So, we have Nicole Forsgren, who's CEO and chief scientist over there, and we also have Jez Humble who is the CTO over there, and they're also co-author of a book that we're actually going to talk about today, called Accelerate. They did that with Gene Kim who's also over at DevOps research and assessment. I'm really excited to have you guys on today, thank you so much for coming on.
Jez Humble: Thanks so much for having us.
Nicole Forsgren: Sure thanks for having us.
Ben Newton: I know there's not everybody that's going to be listening to this is actually going to know what you guys have been doing with the DevOps research and assessment DRA, and what you guys have been doing with the company there. So, can you maybe Nicole, can you just maybe start off and kind of explain what you guys, what's the mission, what do you guys try to do, and tell us a little more about it.
Nicole Forsgren: Sure. We've been working together. So we, myself Jez and Gene, we've been working together as a group for the last four years on the state of DevOps reports. So if anyone here isn't familiar at that, the last four years we collaborated with a group at Puppet to do this giant research project, right. So, when I first got started I was actually a professor, and we were digging in to find academically rigorous ways to understand how to drive effectiveness and efficiency, and really excellence and high performance in software delivery, and then organizational performance, right.
Like, does technology actually make organizations better, and how do we make it better? And so, we've been doing not the last four years, and then a couple years ago we started Dora, so I just kind of shortened it to Dora, to really help continue that work, and then help organizations understand how to measure what they're doing, how to benchmark against the rest of the industry, and I think more importantly how to understand what pieces they should focus on to really help drive strategy and then accelerate their transformation.
Ben Newton: That's awesome Nicole. I haven't been in this space for a while myself. It's really nice to see what you guys have done to bring some rigor to space and data to space, which of course fits perfectly with this podcast, so I'm excited we're going to be talking about that today. And even before we get into a little bit more about what you guys have been doing recently, and with the book Accelerate, I'd really love to hear... I mean one of the things I like to do on this podcast is also, it's not just about the data, it's about the people.
And, it's about the people who create the data and work with the data. And I'd love to hear how you guys kind of came to this. You've both been involved in this space for a long time. Jez, you've written in a couple of books about this space, you both have been thought leaders for a while. I'd love to hear how you guys came in to, really how you got involved with DevOps and especially since you were there at the beginning. Go ahead yes, Jez want to start?
Jez Humble: Sure. So I kind of got involved in DevOps by doing it before it was a thing really. When I graduated from college in '99 I got a job at a startup in London. There was only two of us doing technical stuff, and so we had to do everything, whether it was racking servers in color [PHONETIC] facilities to doing systems configuration, network configuration, to building the software and operating it. We used to deploy from our machines into production by just FTP, no files.
We didn't have version control, I mean it was very scrappy. And then I joined Fulworks in 2004, which is a now pretty big IT consultancy firm. Got put on a really big project where there was many developers building stuff on Windows laptops, and the first time we tried to deploy to a production like environment, the whole thing just went horribly wrong. Took us two weeks to get it deployed, the software didn't work properly, and we had all these problems trying to even get the thing working in a production like environment, let alone actually getting it deployed to production. So, a lot of the problems that we had and ways that we've had to fix those problems went into the continuous delivery book, which me and Dave Farley based on our experiences and those of our colleagues, started in 2006 but only came out in 2010. And when it came out in 2010 it was the same year that the very first DevOps Days conference happened, and the DevOps movement just happened to come along at exactly the same time.
We were there because this is what we've been doing for the last several years. So, the reason that the develops moon became popular is just there was a confluence of a bunch of different parallel threads going on in the IT industry, in startups in Silicon Valley, in larger companies that were trying to solve this problem that really the DevOps movement is about. Which is, how do we build and evolve and operate, rapidly changing, secure, reliable, distributed systems at enormous scale, which is a question that we're still answering as a movement. But it turns out that a lot of people have that problem, which is why the DevOps movement has become so popular.
Ben Newton: Yes, I know that makes a lot of sense. I always think about, when I hear how you and other people started moving. I remember back in early 2000s I was doing some of the same things, and I remember I was working with a development group to try to get their code to production more quickly, and they actually, long story short, they actually caused 30 days of downtime for an application. And, ever since then I was so excited to see this... What you guys really brought into fruition, and that whole process of really thinking thoroughly about this, because it was it was amazing. I'm kind of you, I was I was doing DevOps before it was cool and it was it was hard. It's amazing to see how far we've come. And Nicole I know you came at it from a completely different direction, right. So how did you get involved in this?
Nicole Forsgren: I really did right so although I had a similar-ish start right. So I got into tech in late, late 90s as well. And then early 2000s I was in kind of a similar environment which Jez just described, right. So, I was racking servers, I was laying cable, I was configuring all my systems. I actually, my very first job was in mainframes. So we definitely didn't have version control. That was crazy. And then I went on to be a software engineer at IBM, and then I went out to do some consulting work where we were building out like workflow management systems, and of course like ad hoc and poorly defined, and it was fine, and then I went on to get a PhD.
Because, I wanted to help companies and teams, and organizations understand how to do this in kind of repeatable generalizable ways. Because, every time as a consultant you go in and you tell a team the answer, right. Well I've seen this work here, and I've seen this work here. Occasionally you would have someone say "Oh, well that won't work for my team." so, I wanted to find answers from a research standpoint. I ended up, I thought for sure I was going to be doing database research, because I loved databases and ways to kind of organize things. But, I ended up stumbling into I was still maintaining really large-scale systems, I was full-time at IBM at the time.
And I stumbled into a usability study for sysadmins. And, walked away from that saying like even IBM is working with people who are maintaining and managing these really large complex systems, they don't realize that that these people need slightly different ways of interacting with and using systems. And so I completely pivoted. Changed my dissertation research to try to understand how do we make and design systems, and how do we study the work of sysadmins. You know the back half of kind of DevOps, to help make their work more effective, more efficient, to deliver value in the best way we possibly can.
And then my postdoc expanded that to go farther up into the development side, so that we could look at development and then the handoff into operations. So, if we think about it that was sort of DevOps, right. So, how do we design all these systems to optimize effectiveness and efficiency, and delivering value to organizations and customers? And that was 07, 08 ish. It's really interesting because the timing was right on, it was right about the same time the DevOps movement was coming up in industry.
And so then when I hit academia I was a professor for a few years, kind of pushing this research, and going to industry conferences to try to make sure I understood what was happening. Then when we joined forces with Gene and Jez, and the team at Puppet in the end of 2013, really it was kind of this big movement of... I was kind of one of very, very few people who'd been studying this already for several years, and then they were the ones that were kind of in the heart of it, had been in the heart of it for several years, and it was this perfect storm which was super like nerd fun for me. It was great.
Ben Newton: It's like trying to get into the data and understand the people. How dare you. Who would think of that? [Laughter]
Nicole Forsgren: Right. Well and I mean at the same time, I'd also still been building systems and digging through log files and trying to make sense of all of this craziness, right. So yes, it was fun. I think that's one of the things I've always loved about the DevOps movement in general, is just how grassroots it was, and people just trying to understand what was going on right in front of them, and how to do things better. I think that's one of the reasons why it's been so effective. It seems we all kind of developed the skills at the same time. Because, I remember actually putting in expense reports on mainframe, so.
Nicole Forsgren: Oh yes. Every once in a while I get one of my own clients call me up, and they're like "Can you help with this thing?" and I'm like, "No, I should not." They're like, "You still have access." I'm like, "Revoke all of that access." [Laughter]
Ben Newton: Never ever again.
Nicole Forsgren: Take it all away.
Ben Newton: Oh, that's great. I love hearing the stories about how people... It definitely seems you guys came at a burgeoning moment where everything was kind of coming together, with the architectures, with cloud, with new technologies, with this kind of drive to build applications differently, the competitive pressures. We brought up the book Accelerate a little before I know I been a big fan of Gene Kim's as well for a while. The Phoenix project really actually helped me think through how to think through DevOps and now having read this book. I mean it's great by the way, I think you guys did an awesome job. Tell me a little bit more about how you guys got to writing this book, and why did you write it and, what was kind of driving that process?
Nicole Forsgren: Driving the process? [Laughter] I think there's a few things there. Do you want me to take this one Jez?
Jez Humble: Sure.
Nicole Forsgren: I think there are a few things there. So for me it was sort of a perfect storm of, we had four years of research under our belt, so I think we had a significant enough body of data and results to tell a good story. We had a lot of people coming up to us at conferences, or events, or even emailing us with questions saying, "What do you think about this, or what do you think about this?" and then me saying, "That's in 2013, or that's in 2014, or that's in 2016." And I was like "Ah. I have it in my head. I wish there was a way I could politely, or more politely say, "Download the last four reports. Read it all. Stitch it together." Or, someone would say, "I wish there was a way you could do this," or they would come hit me with so many science methodology questions, right.
And, just realizing it would be really great if I had a medium, right, a platform where we could really get into much more detail. I think the thing that really at least finally spurred me into action is I was going to the Agile Singapore Conference and DevOps Days Singapore, and I finally met Martin Fowler in person. So, we chatted before, we'd had a phone call. And he said, "You know, you really need to write this thing up." I said, "Yes, yes I know." Because he said... So when the first report came out that I was on, he asked just some questions about, you know, "You make all these assertions and I don't believe it. I'm pretty skeptical. You can't just come up with these numbers." And Jez said, "Oh, but wait. Nicole is on this project."
And so, we had this long lovely phone call where I outlined all of the really, really rigorous statistical methods we used. And so, Martin said, "I took really careful notes during that phone call, and anytime someone comes to me I pull out these notes from these questions, or sometimes I refer to them at a talk. You really should put those together in a book, and if you don't..." This was the kicker. "If you don't, I will." and I was like, "Whoa."
Ben Newton: [Laughter] Way to force your hand.
Nicole Forsgren: I know. I'm like "That's it. No one's writing my book." so, I came back from that trip and I was like, "Hey y'all. We're writing a book. Let's go." so, we kind of sketched it out and we got going. Is that a fair recollection Jez?
Jez Humble: Yes. I have nothing to add to that.
Nicole Forsgren: I think that's about right.
Jez Humble: Except that I think that was probably an empty threat on Martin Fowler's part, but a very effective one.
Nicole Forsgren: I was giving Martin a hard time, he goes, "No, I don't think I was going to write it. I think I was just going to like nudge you, or do a blog post. I wouldn't have written the book." so, I could be misremembering it, but my recollection is that I got off my keister I tell you what.
Ben Newton: Well that makes for a good origin story Nicole, remember that.
Nicole Forsgren: I started writing.
Ben Newton: And you're writing the history so that's always going to stand. That’s all that people remember, that's all that matters. How long has the book been out now? It's been a few months now, right.
Jez Humble: Two months.
Nicole Forsgren: Just under two months.
Ben Newton: Okay what's the reception been, i mean has this actually helped you? Because, I mean it seems part of what you guys are doing over at Dora is you have a mission around improving the way people are implementing DevOps. Did you did you find that having this in a book form and people can consumer that way. Has that helped you get your message across?
Jez Humble: Yes I mean the reception has been great. We're on, I think the fourth printing now.
Nicole Forsgren: Fifth.
Jez Humble: Fifth printing. So just did a printing of 15,000, which is for a book in to the second month is pretty extraordinary. So, it's been received very well. There's definitely a success from the perspective of IT books any IT book that publishes more than 10,000 copies is considered a success. We've exceeded that in our first two months so we're obviously delighted about that, and very grateful to everyone who's bought it and spread the word. But yes, there's obviously a big market for this, and I think it's because people want to know that what we're doing, and the things that we talk about actually really work, and they want data and evidence. That's really been lacking in our industry.
Now, for a long time you talked about how Agile is an empirically driven movement, but it's very hard to gather data. You can't do randomized control trials in the context of software development. Firstly, because there's so many variables and they're incredibly difficult to control in the context of teams building software. A lot of the studies that have been done have been done using small teams of students where you can't really generalize the results. But having some studies done using open source projects, but again very hard control these variables, and then if you did want to do a randomized control trial in a commercial context, you can't really do that, because you can't say... It's very hard to induce companies to create two teams and try and control the variables, in the noise that one of them might fail and cost a bunch of money.
So we have a fundamentally different approach to studying these systems. It was a chance. I remember Nicole telling me that she found it very hard to actually find someone to supervise her thesis because it was so innovative what she was doing. But, I think the results that we've been able to produce really speak for themselves. I think that they've been very widely and happily received, because we were able to actually provide real data and research on these things that we've been talking about for many years now.
Ben Newton: That makes a lot of sense. I can I can definitely sense it myself. A lot of people that I've had those conversations about DevOps, a lot of it is about, you basically giving them a platform of results and data that they can use to drive that adoption in their own teams. I mean that's great, because otherwise it's a lot of people talking about opinions, and that only works in Silicon Valley. That won't work where we are at. You guys have shown that that's not the case, and that's great.
Nicole Forsgren: Or they'll say something like, Well, that only works in startups, that doesn't work in highly regulated fields." Which we've also shown isn't true.
Ben Newton: Well and that was actually one thing that as was I was reading through the book, that I just found really interesting as well. Because you guys have basically shown right, it's not only not only Stardust but not only one type of architecture either like microservices is this, or they have to be cloud. You've actually seen high-performing groups kind of across the board, right?
Jez Humble: Yes. You can do this stuff in mainframes and be successful. So, yes. Absolutely. People tend to look at the wrong things in terms of the elements for success, which is natural but architecture is really interesting, because people think, "Well, we have to re-architect, and then they buy all the latest technology, or they re platform to Docker and Kubernetes and whatever. And not to bash those technologies, I think those technologies that are really, really amazing, but it's the outcomes that those technologies enable that's critical, which is teams being able to perform deployments on demand, not having to go through these very complex orchestrated deployments.
Seems being able to run tests locally, and get comprehensive test results without having to wait for a complex integrated environment that takes days or weeks to provision, those outcomes are what's important. And, you can achieve those outcomes with mainframes, and you can do all the latest stuff with Docker and Kubernetes and not achieve those outcomes. So, that's something we've been able to show very clearly.
Ben Newton: Right. That makes a lot of sense. And one thing that I found really interesting about this, because having kind of watched from the sidelines as this was developing. It seemed a lot of times, that people that haven't [PHONETIC] interact with it, the tendency of us technologists is to go towards the technology, right. It was easy to describe early on in DevOps, is like, "Oh, Puppet versus Chef. How are you automating?" and all that was really important, but at least from my perspective it came down to the culture. What kind of culture did you have in place, and you guys spend a lot of time on that in the in the book. I think if I remember right there was one place where you said that, "Culture could be measured."
Which I just found fascinating, because that's one of the things I would have thought would have been a struggle for you guys in this process. Is like "How do you measure the impact of culture?" So, maybe you guys can talk a bit about that. You had a whole section in the book that was kind of about culture and about leadership. Maybe talk a bit more about how you guys approached that, and what you saw the impact of culture was.
Nicole Forsgren: I think one thing that's important to start with, is anytime you want to measure something, you first have to start by defining it, right. So we need to understand what it is we're talking about when we say culture. Because so often people say, "I'm with you. Culture is important. We need to make sure we have a good culture when we talk about DevOps." Well, so often people are talking about very different things. Are you talking about national identity and culture, because that is a thing that has actually been shown to have big impacts in certain types of literature, like in financial research?
Well, that's not normally what we're talking about when we talk about DevOps transformations. When we talk about that kind of a culture, we're usually talking about things like, we hear phrases like, "breaking down silos," and "fast feedback," and "accepting failure," and "doing post-mortems," right. And so, what we did is we tried to keep that in mind as we thought about the type of culture that we were going to be researching. And what we did is, we came across a definition of a culture that was defined by a sociologist named Dr. Ron Westrom. He had been researching human factors in system safety, particularly in high-risk complex environments like healthcare and aviation. This particular definition and kind of typology was found to be predictive of performance outcomes in these environments.
We're like, "Well, this is great right. This is a really fantastic place to start." And this definition of culture from him included things like level of cooperation among three different kind of groupings like low, medium. Oh, I guess you could say it's low, medium and high, but he calls them pathological or power-oriented, bureaucratic and rule-oriented, or generative performance oriented, or maybe even mission oriented. He talks about if the messenger is shot, you blame the person who brings you bad news, the messenger is neglected or the messenger is trained, right, to bring... You want that person to bring you bad news, you can uncover bad things that are happening. Responsibilities are shirked, you have narrowly defined responsibilities, or risks are shared. You talk about bridging is discouraged, bridging is tolerated, or bridging is encouraged.
That sounds a lot breaking down silos, right. Failure leads to scapegoating, failure leads to justice, or failure leads to inquiry. That sounds a lot implementing post mortems. Novelty is crushed, novelty leads to problems, or novelty is implemented. Sounds a lot like encouraging experimentation, right. So this was resonating a lot with the types of things when we were hearing people say culture was really important. So, what we did was we used psychometric methods and we turned this into laker [PHONETIC] type questions. We kind of anchored on the high end. We turned into questions like information is actively sought, right.
So you can say on my team, "Okay is information sought? I can strongly disagree with that. I can kind of neither agree nor disagree, or I can strongly agree." we say responsibilities are shared, because they cross-functional collaborations encouraged and rewarded, failure causes inquiry. Then we collected a bunch of data, and I went through several different types of statistical tests to see if all of these questions together actually measure one core idea of culture, if they don't measure anything else. And, if everyone is reading them in relatively the same way. And, what we found is that it does indeed capture a really good, we call it a latent construct, a similar underlying idea of a thing.
Because, it's all the same thing, I'm going to call it culture. Now there are several teams around the industry that are doing this like six, seven item measure kind of on a quarterly cadence, so they can see what kind of their team culture measure is.
Ben Newton: No, I found that absolutely fascinating. Because, it's like reading through this pathological bureaucratic and generative system. I was lie, Oh, yes I remember that from one place I worked. It was exactly like that."
Nicole Forsgren: Yes.
Jez Humble: We find that it really resonates.
Nicole Forsgren: Yes. Well the thing I think I like the most about it is, Dr. Westrom's research gave us a really nice definition of culture that we found was applicable to technology. Another thing our research did, was it gave us a nice way to measure that. We kind of extended Dr. Westrom's research that way. Another thing that our research did was it really gave us a nice way to see how culture impacts software delivery performance and organizational performance. And, it showed us how we can impact and change and alter culture. And Jez, I know this is one of your favorite things to talk about so maybe I'll let you talk about this, how we can act our way to a better culture.
Jez Humble: Right. So, I've been interested in the lean movement for a long time. Obviously the lean software development movement was really pioneered by Mary and Tom Poppendieck in a series of books on lean software development, which I read when they came out in the early thousands, when I was working at Thought Works. So, this is something I've been interested in for a while. Then as a result of that I got interested in the history of lean manufacturing which actually happened very near where I currently live in California.
So the NUMMI factory was a joint venture between General Motors and Toyota. That was a factory that's about 30 minutes away from my house which is now the Tesla factory. So the NUMMI story is really interesting. I'm not going to cover it now, but if you're interested, and I really recommend listening to podcasts on this American Life, there's a transcript available as well which talks about the story of NUMMI, and how the people who build that joint partnership took a workforce which was producing really bad cars for GM. Worker management relationships had broken down. They took that workforce and basically completely turned it around, to the external label producing the best cars in GM North America's whole manufacturing capability, and as good as the cars being produced in Japan.
The guy who designed the training program for that team was a guy called John Shook who was the very first American employee of Toyota Japan, and he's hugely important in the history of the lean movement, which came from that joint partnership. That's where the words lean manufacturing, or the term lean manufacturing comes from, is the people who worked in that NUMMI factory. He then went on to write about their experiences there, and one of the things he says is, the way to change culture is to change behavior.
To change the way that people work and what they do. And so our hypothesis was, well maybe that's true in technology as well. Maybe implementing technical practices like tinies [PHONETIC] integration, test automation. Maybe implementing lean practices like working in small batches, using visual displays, the lean product management stuff that's come from lean startups. Things like building MVPs, working building prototypes, maybe the implementing these practices will change culture, and that's indeed what we found.
We found that implementing the practices changes the culture. Changing the culture impacts not just software delivery performance, but also organizational performance both in terms of commercial measures such as profitability, market share, and productivity, also non-commercial measures, such as the quality the products produced, the ability to achieve mission goals and so forth. So, we know that changing the way you work changes culture, which changes performance. Which is, I think a pretty big result.
Ben Newton: No, absolutely. I mean that was that was definitely the part of the book, when I was really fascinated where you guys pull it together. And one thing it kind of reminds me of some stuff that I'd read up too is that I've always been interested in Skunk Works, and how Lockheed Martin developed these kind of amazing aircraft in the in the 50s and the 60s. I remember reading that and understanding how they did it and like how they did team structure, and how they work together, and putting engineers and mechanics together on the floor, and then reading about how Elon Musk was doing it at Tesla, and then reading how you guys are doing here. It really is remarkable to me that it's not...
There's a tendency when we're in these industries to think, "Oh well, this is the way we do it. This way we think about it. We don't need to think about how people are doing it over there." But, I love how you guys have pulled in these threads and strands from different parts of the industry and be like, "Well, we can learn something from that over there, we can learn something from that over there," and I think that's a really great contribution to how we're doing technology. Because, it seems like a lot of times in the technology industry we tend to ignore what goes on in other places.
Jez Humble: I'm so glad you said that because it's something that really winds me up. I'm not going to go on a big rant now, but I absolutely could.
Nicole Forsgren: [Laughter] Now hold, on hold on. Research Nicole is going to have to step in, because not everything applies the same when technology is introduced. For example, we know that communication patterns can change. So, early on in my academic career I studied nonverbal communication. As an aside, this is interesting when people lie face to face, their communication patterns change in certain predictable ways. But when people lie using technology, their communications patterns change in completely different ways.
Jez Humble: Can you give an example of that?
Nicole Forsgren: Okay. I'm going to have to try to remember right now. When I lie to you... I might get this slightly wrong because I wasn't prepared for this. When I lie to you in person I do things like hedge, and I want to try to remember if my word count goes up or my word count goes down, but it's one of them. Now if I lie to you over chat, or over email the opposite happens. So, whichever happens in person is one thing, but the opposite happens when I do it over a chat. Like a text base either asynchronous or synchronous text-based communication. So, there is a very, very real thing that happens where once you introduce a technology, we really do have to test that hypothesis, right. Even if you think and we assume, there's a presupposition that the technology context looks similar, we really do have to test it out.
Because, sometimes technology throws a wrench in the game. It really does change the way people act and behave and interact with things. Sometimes it doesn't, sometimes it looks the same. So here's another thing, right. And Jez's going to know this. Jez is going to be smiling and nodding his head. Manufacturing is a case where we know what happens, like queue times and WIP limits and all the saying, right. And I was an accounting professor for years, so I could talk about the way people behave in the case of manufacturing. However, when we're talking about technology, we're dealing with inventory that it's completely invisible.
Like in some ways there's the case for saying, "Well, of course," right. This is just inventory, the same principle should apply. However, people may act and behave completely differently because there is no way to see inventory. There is no way to see the assembly line behave in the same way. So, we actually really needed to test a lot of these things out. And one example is WIP limits. So, we looked at lean principles, and we drew from WIP limits. And WIP limits do help predict and support software delivery performance, but they don't have the strong effect that we would normally assume until we pair and combine it with other lean accounting or lean principles, such as visual display boards, and combination of monitoring to drive business decisions. And then we see the amplified effect in technology.
Jez Humble: This is a theme of our work in general, is that you can't ignore things in other industries. There's a thing in tech that I really don't like which is like, "We are evolving a completely new way of doing things, and nothing else supplies from other industries." Which I find insufferably arrogant because what we're doing in technology is in many ways repeating the mistakes of generations that came before. My favorite example of this, I've got a book on my bookshelf called The Human Side of Enterprise by a guy called McGregor, which came out in the 60s, which talks about starts of management. He talks about Theory X management where you, and again I I'm going to have this problem where I can't remember which is which, but there's Theory X and Theory Y which are terrible names.
Because, obviously I can't remember which is which, but one of them is you know carrot-and-stick ways of managing people. The other one is you assume that people want to do good work, and help encourage them do that work. And the thing about it is, you get what you expect. If you treat people as fungible resources who are interchangeable, and who responds a carrot and stick incentives, they all behave that way. And Theory Y, if you treat people as real human beings, that they will respond that way. So, however you behave, your predispositions will be confirmed. So, this is written about in the 60s, and we're still learning that in management more than 50 years later. So, I find that really annoying.
But, absolutely. Want to affirm what Nicole said. You can't just take these things that have been shown in other industries, and then just apply them willy-nilly. You've got to retest and revalidate it, because they're complex social systems, and technology has an impact on complex social systems. I think what the Poppendiecks did with the lean software development was amazing. They took something from a completely different paradigm and said, "What happens when we apply this to the context of software development?"
And it was hugely influential, but I think for us to be able to come along and actually take a scientific approach to studying what really works and what doesn't, I think that that's been hugely... Just going through this project has taught me an enormous amount and hopefully that will be true of people reading the research as well.
Ben Newton: Yes. Absolutely. I do again love the way that you guys kind of pulled that in. I couldn't agree with you more Jez on the fact that we tend to ignore what's come before us. There's also a tendency of those who came before us like, "Well, can nobody can do any better." it's always that kind of balance, right. And it does seem to come down to... A lot of this is that there's some core cultural changes need to be accepted about grassroots small teams that have devolved decision-making, that is opposed to this kind of command and control structure like you might have had earlier in the twentieth century. So, it's endlessly fascinating. Nicole I do have to ask you one thing. So is research Nicole like an alternate personality that comes out every once in a while, or?
Nicole Forsgren: [Laughter] Yes. I mean there's a joke that. When Nicole says, "Research says..." Like it's like, "Take a shot."
Jez Humble: Yes. I have to say that all those things about the application of new research methods to old ideas, Nicole has been hugely influential in understanding how that works for me. That thing that she talked about how you have to test those hypotheses, that's not in my nature. I just like, "Well, this obviously works, so let's run with it." The fact that she takes this very rigorous analytical method to testing those hypotheses, and looking at these nuances which have huge impacts on actually what happens on the grounds, that's completely changed my way of thinking about how to apply these theories to what's going on in the real world in practice. I just wanted to give her a shout-out for that.
Nicole Forsgren: Thank you. And it's been lovely working with the team. Jez was much more familiar with the lean movement than I was, even with my accounting background, and definitely some of your tech background. It's been a fantastic experience working with the team and I also have to say thanks for your patience. I'm pretty sure I'm really lucky I didn't get kicked off the project that very first year when I kept saying, "No you can't do that. No you can't do that. No, that's not how this works, no, that's not actually a hypothesis." [Laughter] It was great though.
Jez Humble: It was enormously frustrating, but if you hadn't done that we wouldn't have proper science. This is hilarious, because I'm doing ending science projects with my kids right now, who are six and eight. And I'm saying, "Okay, before you design the hypothesis you have to create a theory. Lets..." and my kids are rolling their eyes and my wife's rolling her eyes. [Laughter] And I'm like, "Listen, I'm just channeling Nicole here. Got to do it, right." So, yes.
Nicole Forsgren: Sorry y'all.
Jez Humble: I'm passing it forward, what can I say.
Nicole Forsgren: Love it.
Ben Newton: I like that. I'm going to... I'm never going to... I have a seven year old girl and a three year old boy, so we actually went to the Maker's Faire yesterday, and just seeing them discover these things, I think it's amazing to see how their mind just kind of naturally understand this whole experimental thing, and the kind of scientific approach if you just find a way to explain it to them, so that's pretty cool. Now one thing is we're kind of... So this has been great talking about what... Kind of the wrap-up maybe the culture discussion about there's another thing that I found really interesting. Having been in, worked for enterprise companies for a long time.
I don't know how many leadership classes I've either willingly or unwillingly been forced to go through. I was really interested to get to the point in the book where you guys talked about, and used a term transformational leadership. I was not actually really expecting that. I was expecting more to talk about culture like we just went through, and I found that really interesting. So, I'd love to kind of hear a little bit about how you guys, how you kind of came to that idea and where have you seen that come out in the research. I found that personally to be fascinating.
Nicole Forsgren: I think this was sort of the culmination of a couple years. So, we've been looking at the Westrom culture typology for a few years. We've been batting around what leadership might look like. I've been digging into the literature. Lots of people at conferences were saying like, "How do we make a difference? Is it org design, is it something else?" Traditionally DevOps was a grassroots movement, but now that it's getting all of this traction, organizations are trying to make a difference, so what does leadership look like? So, I started digging into the literature. See what might be a really good application and set of hypotheses to test. And some people were saying, "Maybe it's transactional leadership, maybe it's servant leadership."
But it turns out servant leadership is nice, but that doesn't tend to have predictive results. It doesn't tend to have outcomes that really drive something for an organization. But, transformational leadership does, particularly in contexts that look like something like what we would be looking for in an obviously transformational sense, right. But, technology transformation, you're trying to move from one state to another. And so we kind of pulled up, or I pulled up a model of transformational leadership from, I want to say a paper from Rafferty and Griffin, that had five different dimensions. And so it has these five characteristics of a transformational leader that at least looked like it was really promising.
These dimensions were vision - all right, you have a clear understanding of where the organization's going and where it should be in five years. Inspirational communication: you can communicate in a way that inspires and motivates even in an uncertain or changing environment. I'm like, "Okay, well that's tech." intellectual stimulation: you can challenge followers to think about problems in new ways, which is exactly what we want our teams to do, right. We want our teams to be able to do it, not just the leaders. Supportive: you can demonstrate care and consideration of your team's personal needs and feelings.
And then personal recognition: you do want leaders that can praise and acknowledge the achievements, goals and improvements, and work quality. And, can personally compliment others when they do outstanding work. So, we tested this in the 2017 study. I was really kind of excited about the results, because what we found was that leadership does play a role. It does have an impact. But, what it does is it has an impact in supporting all of the rest of the work that happens.
It kind of has this bolstering effect and this amplification effect, right. Helps support the technical work, it helps support the process work. Which then in turn, has follow-on effects to everything else that happens. Which then in turn, helps with software delivery performance, and in turn helps with organizational performance. However, the leaders alone can't do it themselves. So, it's definitely worth investing in in creating and building strong leaders.
It was interesting that you said you've been to a whole bunch of leadership classes, which is great, but I've also seen companies where they're in this kind of hyper growth stage. They have to build up their tech team so quickly, but they're just promoting team leads and technical leaders as fast as they can, but they never learn how to be leaders. Or you're still in this grassroots movement stage, and you're not totally sure what to do. Like what does it mean to be a leader? Maybe you don't even have people reporting to you, you don't have to be a manager to be a leader. What types of things should you be focusing on?
Jez Humble: And again, this is something that I think we really see a lack of in the tech industry. Firstly, this idea that "Well you don't really need management, you don't really need leadership. You just need to hire the best people." I think that's completely false. One of the things that our study on culture and on leadership, and echoed by for example, Google's Project Aristotle research where they looked at how to build great teams at Google, is that what's important is the relationships between people on the teams, not just the individuals themselves.
The cult of tech which is that you hire the A players and then it takes care of itself, that's not borne out by the evidence. And then this other thing that Nicole talks about, where we're just going to find the great technical people and promote them. You know it just becomes the Peter Principle really, really quickly, and the same things that make you a great engineer, are not the same things that make you a great manager in any way at all. And I think that's borne out by what we see. So, I just wanted to call that out in particular, just because it does adhere to things that people in the tech industry believe are true that really aren't.
Nicole Forsgren: Oh, and that's such a great point Jez, right. Now technical skills are important, but they are skills, and so are leadership skills. This is something that you should absolutely try to learn and grow, and is something that can be taught. So, this is really an investment that organizations should be making in their people, and in their leaders if they really want to be excelling, and growing, and amplifying the rest of the efforts throughout their organization.
Jez Humble: And by the same measure technical skills are things that you can learn. The idea but you're just going to hire these people who were born to be brilliant is completely false. Technical skills can be learned just as well as any other skills, and I think that the third problem we see in our industry in this area, is companies you know wanting to hire people with the right skills, rather than investing in helping people develop the necessary skills. Which is tremendously short-sighted in an industry where the technologies and the skills change on such a frequent basis.
Nicole Forsgren: Well it's short sighted and it's expensive, and it's crazy. I mean we quote Adrienne Cockroft in our book. When he was at Netflix still, right, everyone would look at him and they would say, "Where do you hire these amazing engineers?" And he would look back at them and he would say, "Well, I hired them from you." So, I mean, sure, you can spend a ton of money, you can go out and you can poach incredible engineers for an ungodly amount of cash, or you could grow your own people. And then you know what? They know your company, they know your infrastructure, they know the context and they're loyal.
Ben Newton: And as the transformational leadership research shows, just hiring the best people and setting them free isn't enough. This was actually what McKinsey tried to do for a number of years in the 90s. There was a very big and very famous organization that followed McKinsey's research very rigorously and just hired the best people and let them go on with it, that company was called Enron. [Laughter]
Nicole Forsgren: And they did. They made a ton of money. Didn't go so well.
Ben Newton: I think that's... You had your hot button issue. I think that's definitely one of my hot-button issues. Just as the idea of the management, and hiring and promotional practices within technology. Because, it tends to be attached more to performance in your current job, than this idea. There's characteristics that really can drive performance from a leadership perspective, and there's characteristics that can drive performance from a team perspective, an individual perspective. And it's good to see that the technology hopefully is industry is moving in the right direction, and start thinking about these type of things.
Because, Nicole, that quote that you mentioned about Netflix. I was just listening to an interview about Netflix just a couple days ago, and talking about the culture that they developed there, and this is so important. Because, if you can't... You can make all these structural changes, you can make all these technology changes, and if you don't think about the culture you can end up deadening the impact of all this places you invested, right. And one thing you guys said too, that actually was something a little unexpected for me, was the idea of the correlation between this DevOps culture and job satisfaction.
So I've heard that in other places particularly having been on the product side, about Agile and job satisfaction. But, I liked how you guys pull that in as well. That you did an NPS survey as part of this, right. About how satisfied people were with their current environment. Is that right?
Nicole Forsgren: Yes. We included a few kind of satisfaction and happiness and identity outcome measures, and NPS was one of them.
Ben Newton: Yes. Because it seemed to be that that strong correlation in some ways or like at least in in the ways that I've encountered. You know, the arguments for DevOps it's usually been about, "Well, if you implement develops you'll be more effective and you'll get code out quicker," and I mean that's great, but to then go in and say, "Hey, you want to retain those people that you hired, and you want to have a high job satisfaction and you actually want to attract those better, you need to implement this." Seems like you can make an argument that this is one of the most effective ways to job satisfaction for these companies that are trying to go this direction. Sound right?
Nicole Forsgren: Absolutely. And so many companies right now are trying to hire. And we found that high-performing teams and organizations, their employees were 2.2 times more likely to recommend their organization as a great place to work, which is huge if you're trying to hire. And then we reference other research that found that having a high Employee Net Promoter Score that ENPS drove revenues. I think that was researched by Bain & Company. So it ends up having kind of this double on effect of it also driving revenues. So it's a smart investment.
What we're finding to kind of sum this up again, investing in smart technology transformation, whether you want to call it DevOps or tech transformation, or digital transformation. Which involves more than just tech right. So it's tech, and it's process, and it's culture, gives you strong ability to transform your technology so you can delight your customers, and drive revenue. It also makes the work better for the people. It decreases burnout, it decreases deployment pain, it makes your culture better, it makes your Employee Net Promoter Score better which makes it easier to hire. It has this great turnaround effect to kind of... It's almost like creating this halo effect that makes a lot of things better. It does require some investment though. It is an investment in technology, but it really, at the end of the day, it's an investment in your organization.
Jez Humble: And I think what you see, I'm sure a lot of people listening to this will have heard this before, where the teams know that they need to invest in improvement work. Whether that's building test automation, or refactoring, or improving the design of the software, or doing things which are not just you know delivering features. And, they're told, "Listen, we don't have time for that. We've got to deliver the features." and I think what the research shows is that that's a really false economy even in the short term. There's definitely an argument for getting things to market quickly in the knowledge that you're going to incur some, what's called often called technical debt in the literature.
And Martin Fowler has a good blog post about this. He calls it the technical debt quadrant diagram. But, when that keeps happening and you never have time to do anything, a.) It slows things down. Because it becomes harder to change the software, which is the reason why you can't get things done, by the way. Which is why you're going so slowly. So, saying, "Well, let's continue down this path of not doing improvement work." You're going to end up with your feet stuck in concrete. But, also just it makes people miserable when people don't get to... Actually, imagine you're living somewhere and it's covered in garbage, and you're like, "Well, maybe we should go and clean up the garbage," and people are told, "Well, no. You can't do that. We're not going to give you the resources to do that.
We're just going to pile up more garbage." It's going to make you miserable, and so investing in improvement work and not only will make it go faster and improve things, it will also make people happier, and then they're more likely to enjoy their jobs. And also, it's a virtuous cycle. So, it's not just the immediate benefit that there's knock on benefits. I mean, we've never seen people... Yes, I used to work on projects where I would get rolled off the project, onto another project before the project actually went live. And that's fundamentally a miserable experience.
When you're practicing things like continuous delivery, and you're getting feedback from customers, and you see your features go live, and then you get the feedback and then you improve them, that's really motivating. No one who will practice continuous delivery, continuous deployment ever said, "Let's go back to releasing every six months." That's just not something that happens. So, these things are all self-reinforcing, and it's very dumb to continue down the path of like, “Let's just build the features." and think that you won't have anything other than the worst possible outcome.
Ben Newton: I like that. I like the way you describe it and it's a great note to wrap this up on. It's DevOps making the world a better and happier place one life at a time. [Laughs] I think you guys have written a great book here. I love how you guys approached this. Like everybody that's listening, definitely go check out their book Accelerate that they, Nicole and Jez co-authored with Gene Kim. Great read. And it kind of opened my eyes up to a lot of different ways to improve performance on the team, and I learned a lot. Statistics was not my best class in college so I appreciate your skill to kind of pull this data together. And Nicole Forsgren and Jez Humble, thank you again for coming on here. Really appreciated having you on the show and thank you for this great discussion.
Jez Humble: Thanks so much for having us.
Nicole Forsgren: Thanks.