James Governor - The Culture Change Observability Caused in Modern Tech (Observability Series - Part 3)

James Governor: We're well beyond the, "Oh, you have an app dev team and they build something and then send it to a testing team and then they send it to a team to operate and manage the application." Those days... No wait, I was going to say those days are gone. Plenty of organizations are still doing plenty of that, but they probably should be gone, right?

Ben Newton: Welcome to the Masters of Data podcast, the podcast that brings the human to data. And I'm your host, Ben Newton. Welcome everybody to another episode of the Masters of Data podcast. We're now a couple episodes into our observability series, and I'm very honored to have our guest today. He's James Governor. He is the co founder of RedMonk. Welcome to the show, James. Good to have you here.

James Governor: Hey, it's a pleasure to be here, Ben.

Ben Newton: Well, absolutely. I think your name has definitely been associated with this topic recently, so I'm really excited to dig in and hear more about any of your thoughts. But particularly, like we always do, we love to start off and find more about you as a person and where you come from, so tell us a little bit about your background. I mean, how did you end up where you're at, in particular, starting RedMonk and what you guys focus on. And I'd love to have you go through that a little bit.

James Governor: To be honest, the story now at RedMonk is so long that I almost don't think there was anything before RedMonk. We've been at this for a while. My background before starting the firm with Steven O'Grady, I had basically been a tech journalist, actually. I was always interested in going deeper. I always wanted to go deeper. I always wanted to get further into the tech, to better understand what was going on. And, as a journalist, you're not necessarily, you don't always have the time to do that anymore. You're under so many time pressures. And I was interested in doing something that was more research based and this industry analyst space looked interesting to me, and so basically pursued that. And yeah, pretty quickly, Steven had joined me at another firm and we were working together and thinking what worked well and maybe what didn't work so well. And it was pretty clear to us that there was room for a new kind of research firm with a slightly different focus. We were growing up in that era of open source, just a different way of working and building communities and communicating with people and learning from them. And so we packaged all that up together in this company, RedMonk. And the thing for us is that there are a lot of research companies that look at the world through the lens of purchasing. And I think that's fine, but for us, we're a little bit more forward looking and we really focus on developers. We want to understand developers, what makes them tick, developer communities. And really that's how we build our models and our understanding of what the world is, and indeed what it should be. I mean, I think we've always had some pretty strong opinions on some things and we brought that to the table, so I think that's RedMonk. I mean, we're the developer focused industry analyst company.

Ben Newton: Yeah. I know, I know. Absolutely. I first came in contact with... just sort of agree with the way you guys think about the world with the book, The New Kingmakers, and I think I've reread that book multiple times. But that concept of developers, the changing power of developers within corporations. And it seemed like with what you guys are doing, it was really tied to that the purchasing power and who makes the decision, where the budgets are going, has really been changing in the cloud era. Is that how you'd phrase it?

James Governor: Yeah. I mean, that's exactly right. We used to be in an industry in which you required permission to do anything. So, we're moving to increasing self surface and I know that people, if they've grown up with AWS, I guess, AWS natives don't remember what it used to be like... But yeah, having to ask for a different team in the organization, you know what I mean? It used to be that we didn't have open source software, so you needed budget to buy database licenses. It was all very top down, command and control, permission based purchasing activities. But with the Cloud, with open source, with social coding, you look at the incredible impact GitHub has made. That need to ask for permission has gone away. And that's had a dramatic effect on IT organizations. It's just been very interesting to us, Fortune 500 companies, financial services companies, government organizations talking about the need to better serve software developers. And, I think that was not the case when we started out. We were the crazy people. I think one of the phrases we tend to use is that the CIO is the last to know, and that's actually as it should be, high performing teams, ideally choosing their own tools, within constraints, getting stuff done. And that's what we've seen from successful organizations. So yeah, huge changes in just how technology is chosen and used in anger, I guess.

Ben Newton: So James, kind of taken a different tack on it. I mean, we're talking a little bit about the culture, the different buying patterns, adoption patterns, which is super interesting. But, particularly, when we're talking about this term of observability, which we're getting to, it also seems to be about the way that people really develop or software developers build and support applications. So maybe talk a little bit about how that changed. Not just how they adopt technology, but how they're building and thinking about these high- scale applications. Does that make sense?

James Governor: Oh yeah, it definitely does, and I think that's foundational. If we ask ourselves what we mean about, by observability and what are we talking about. I think there is a definition which says, Oh, it's three pillars. We're going to look at your logs, we're going to look your metrics, your traces. We're going to mash them all up in one database and then we're going to have observability nirvana. I think that's possibly a little bit simplistic, but, from a marketing perspective, you would certainly expect that vendors of logging APM or tracing tools might like that story. But the truth is, I think that observability, one of the reasons it's interesting, one of the reasons we're talking about it is that we are thinking about different ways of working, and different ways that teams are constituted. And, certainly, we've moved... We're well beyond the, Oh, you have an app dev team and they build something and then send it to a testing team and then they send it to a team to operate and manage the application. I mean, those days... No wait, I was going to say, those days are gone. Plenty of organizations are still doing plenty of that.

Ben Newton: They should be gone, maybe.

But they probably should be gone, right? But yes, modern software development, I think there is much more of a sense of manageability by the team. Now, clearly any of this should have been done historically. But when I'm thinking about observability: I'd like to think of observability as ... a key part of observability is a love letter to the future, a love letter to your future self, and a love letter to your friends and those that may need to support the application after you. So, really thinking about the tags, really thinking about the telemetry, and really thinking about... If you think about a log and an error message. Some error messages, you read them, and you're not going to have a clue what that means, but a well- written error message is just such gold. And so I think that observability, one of the things that I see is teams thinking about, look, we know that systems fall over, that's going to happen. Let us be able to deal with as many of the known- unknowns and so on, so that when we do have an unknown- unknown, we're able to troubleshoot that. So I really do think that observability, it is about troubleshooting. It's not about an operator sitting back and just looking at a dashboard. And so from that perspective, coming at this from the perspective of, what can we do during the process of developing an application so that in production we feel more comfortable? That's super important. And I think even that, just even, what is the question about production? I mean, we live in a world of dark launches. I talk a lot about progressive delivery, which is my term for thinking about that set of disciplines, AB testing, Blue- green deployments, Canary deployments, and so on. Once you are successfully moving forward with a good testing and CICD strategy, what are some of the other things that you might do? Let's think about rolling out the application to a particular subset of users before it's rolled out more broadly. Maybe we'll roll it out to our internal users first. Maybe we'll roll it out when we're developing in the morning in San Francisco. Maybe we should put it into production in Tokyo when no one's actually using the system so that we can see it in production, see how it works, learn from that, and then release the service or application more broadly, so that Canarian notion. I think observability really ties into that. I think, for me, it is about a different way of thinking about applications. One is, I say that love letter to the future, but the other thing, because of the world we're living in where we are, in effect, testing in production, that future may be in five minutes, or 10 minutes, or tomorrow. Right? So I think there were some different dimensions of time, frankly, in observability that are also worth really giving consideration. Historically, logging for example, was a post facto activity, so it was about working out what happened after the fact, and some of it might be about compliance. Oh, we had to save the logs so that we could look at this stuff after the fact. But really, I think, everything needs to be much more real time because that's, frankly, the world that we're living in. If we're a modern software organization, we're doing multiple production deploys a day and, it's a different way of thinking. And that's why I think that articulation, I mean, I guess it's overused. I guess, Everything in modern tech is actually a culture change, right? And observability is yet another culture change.

Ben Newton: Yeah. No, absolutely. And, to some degree, that's what makes it more difficult, right, because if it was just inserting a technology here or something like that, it'd be a lot easier to implement. But because you're talking about people having to learn new skills, and learn new ways to talk to each other, and work with each other, it almost feels generational in some ways. In the same way that dev ops early on was, where we're seeing a cultural shift from that command and control type of perspective to these team- based very multifunctional, cross functional teams. You're not having these strict roles that never change, people doing a lot of different things. Does that ring true to you?

James Governor: Yeah, I think so. I think that's fair. I mean, I've got enough gray in my beard now that I'm a little bit cautious about terms like generational. But from an IT management perspective, yes, there is a new generation of technology building and management. Although, there might be multiple generations of folks involved in driving that change. So yeah, I do think that's reasonable. And that's one of the challenges for enterprise organizations, for organizations. I mean, one of the challenges for, frankly, a lot of SAS companies to, or anyone building software. How do you get everybody doing this? Is it just one team that's living in the future and everyone else is not? I think that's one of... We have all the jokes about the hardest problem in computer science. To my mind, one of the hardest problems in computer science is just getting everyone else to do the new stuff and learn those practices and change. So yeah, there's a generational shift going on for sure.

Ben Newton: I remember sitting down with one of the developers when I was a product manager, and I remember two things that always stuck with me, and I've used this story before, is one, the concept of him not being on call was just very strange to him. It was very natural to him because that's the environment he'd grown up in. He went straight from school into a development team, they was on call. And then the second thing was, is what he was doing would have been classified as database administration back when I first started out, but he would not have done it. He did not think of himself as a one trick pony, like DBA. He was like, well, this is what I'm doing today. I'm going to do something different tomorrow. And he did. That's always struck with me, the fluidity of the way he thought about his job, which is much more the way a software engineer would think about it. But then also the fact that supporting his own code, not doing that, just seemed strange to him.

James Governor: Yeah. Yeah. And again, that gets to that notion of generational shift that you're talking about, and it is. We just live in an era where no developers are taking more responsibility for the management of the applications and services that they build. And as you say, that on call notion, it is about responsibility. That goes back to the love letter because, if you are on call, you definitely want to make your life as easy as possible at that point. But from an observability perspective, that's definitely related.

Ben Newton: So maybe we take a bit of a turn here. The way I've always thought about it, a lot of this is being driven by the changes in the way we develop software, like Cloud, making it easier to adopt new technologies, try new technologies, the rise of the open source movement. We've talked through a lot of that. So I think, James in particular, you and RedMonk have a pretty good vantage point to really understand what's going on. I mean, what do you see changing right now? Where do you see things going? What are the trends and rivers of change that you're watching?

James Governor: Well, yeah. I mean, I think that some of it is what we're talking about now, observability. If it's a different way of working, and thinking, and composing an organization, then that is stuff that RedMonk is interested in. I think from our perspective, we've been on this journey to... Everything as code, and that just continues to provide more and more value as a framework for getting things done. Definitely through configurationist code and automating everything now, is something that is super interesting as a framing, which is to GitOps. Which I think is really interesting at the moment, where you start thinking about all of the, frankly, the automation you do should be done through Git. So Git becomes the single source of the truth. IT environments are complicated enough already without having developers, or operators, or whoever the role is, running around and SSH- ing into servers and spinning up instances of VMs or containers or whatever, without you knowing what's happening. This GitOps notion where you do a pull request of your information, that gets banished to you paratively, instead of that desired state kind of way. I mean, I've been around long enough that I've heard, desired state is the term a lot, but I think it does make a fair bit of sense in this sort of container world, at least. So I think that's something that I'm... I'm interested in the intersection of progressive delivery, GitOps, and observability, but then, out of left field, you also have to think about all of that stuff in the context of server- less and what's going on there. I'm not a big believer in no ops. Because I think everyone has to take responsibility for building the system, for managing the systems that they build. But clearly for a lot of developers, they're finding server- less, and certainly the Lambda modelers as an extremely... For them, high productive way of establishing minimum viable products, rolling out new services. And so I think, that those, well frankly, I guess those sorts of, I guess, mega trends are plenty to be getting on with. I mean, I came up with that, or I came across, sorry, an interesting term recently. I was really thinking about what is, what is... everyone's using Cloud. We've had this idea of multi cloud for a while. I think some of it was as a promise of, Oh, it's about building an app and having portability, but I'm much more interested in the, choose the best service for the job view of the world. So like, I think that organizations could be an Amazon shop wall- to- wall and still be heavy, heavy users of, of GitHub and GitHub Actions. Is that multi cloud? I think it kind of is, people are obviously going to be using Stripe, and Altzero, and Algolia to get specific jobs done. And so I'm... The term there is, that they're using is only smoke stack. So it's like service fall, which is the server- less word for those sorts of services Mashable, Open, and now I'm going to hopefully forget what the K is. Oh, Komposable. So it's of course coming from the Coobernetti's world, anything C actually starts with a K. So that's this idea that services that run, applications that are composed of services that are running in different places. This composite world, that raises all sorts of interesting issues and challenges for observability.

Ben Newton: Yeah, no, it's interesting. You actually bring to mind one thing that I'd be interested to hear your perspective on because, even calling it like generational and stuff like that makes things seem like they're like these sudden shifts in time where it's like, it was one thing and it's another, and that's not the way reality works. It's always kind of like these very blended, gradual changes in a lot of ways. I mean, sometimes there's big point changes, but a lot of times it's just very gradual. And one thing that has struck me, is that when I started out, there was the people that developed the code, they tested it on their laptops. Maybe they test it in a little environment and then they handed off the ops and ops had to make the best they could of it. And you saw now, where there was more of like a supporting your code through production and stuff like that, but what's interesting me in particular, when you talk about server- less, it seems like in some sense as the application, end to end application development process, meaning like from writing code, like birth to death, your code goes out there, you're supporting it. There's these new stratas that are emerging. Because in some sense, when I hear people talk about site reliability engineering, or platform engineering, even the way that QE, a quality assurance like QA has shifted into QE, and like a lot of these changes, it seems like... you're still have to solve the same problems. Like you just said, it's like, there's no such thing as no ops, you still have to support your applications. You still have to do all these things. But instead of it being these kind of, I almost think in, like visually, like vertical silos, they're becoming almost like horizontal layers. It's like, okay, well, we are the team that is going to think about tooling, or we're the team that's going to think about the platform we're running on. We... Is it, are you... Is that actually resonate with what you're seeing? Or what do you think?

James Governor: Yeah, yeah. You know, I think that in all of that, the entire organization can make this change. I think that at any given point there are pendulums involved, and stuff swings one way, and then swings the other, in all of that the entire organizations can rethink what they're doing and find... And be able to work in new ways. Now we've had this period of intense innovation in some technology stacks. And certainly, if we look at what happened with Coobernetti's, there's a lot of complexity there, enough complexity, that I jokingly talk about the new role in the IT shop as the chief Coobernetti's officer. That's a joke, but of course, how do we make people productive? We have... What we've seen in some of the organizations, frankly, that have been adopting. You sign a deal, with Red Hat or whatever, but then you don't actually know how to use Openshift, and you end up with empty clusters. So is everybody really going to run around doing HLM shots, using YAML to describe everything, the full stack world, I think, makes it very, very hard to cross the chasm. And so definitely there's some... There absolutely are platform teams and, we're going to see within the enterprise, I'm sure we'll see Tansu from VMware, we'll see Openshift, then they'll be using, then there'll be coordinating something around, whether it's EKS, or AKS, or whatever. There will be, I think, a return to some specialization. So it's one thing to say, Oh, fluidity and everything else. But platforms, there does tend to be a bit of specialization there, in order to get everybody able to do this thing. Specialization, certification, people that... I mean, on call is one thing, but a lot of people, they like to just do that job and, we need to find ways to make their skills relevant. So yeah, I do think that will happen. I think the one crucial part there, and what we're seeing in teams is that, having a platform team doesn't mean that platform team is the only people that do ops. Platform team will curate services. Platform team will enrich services. Platform team will think about making developers more productive, but they are not taking response. You know, the platform team and the app dev team, both do dev ops, both think about the platform, is about managing the platform. The app people have to go about managing that application. So there will be some specialization, but I think as those layers you mentioned settled down a bit. I do think there's room for some horizontal specialisms. It's just too hard to expect everyone to know everything. If you're Uber, or Netflix, or whoever and you're scribe, you're paying whatever, like north of two 50 for an engineer. I'm sorry, but that's not going to fly in Peoria. That's not going to fly in Berlin. That's not going to fly in Tokyo. So yeah. I mean, I do think that we need a bit of specialization just so that everyone else can start taking advantage of some of the new technologies and approaches. I don't, I just don't think that everybody can be... And it's interesting, that word full stack. Some people think that's an insult. Some people think that's an aspiration. I just think that frankly, it's just too hard to expect everybody to know everything.

Ben Newton: Right? No, no, exactly. And I mean maybe in some sense, it's like, I like the way you put it, as everybody has a shared responsibility for operations, whatever stack it is, that you've decided to work on you. It is a shared responsibility, but also the commonality, maybe is that it's not just about people twiddling widgets and supporting stuff they don't understand. It's like there's a sense of, I may specialize in this platform bed, or I may, but you know, I'm an actual software engineer. Like I'm going to write code, I'm going to support this. There's a, I don't know it feels like you're solving the same problems, but you're solving them in a different way. That requires more, that kind of everything is code approach, it's-

James Governor: Yeah. I think that is fair. So, and again, there is definitely some gen shift there. I talk about some people just want to do their jobs for a living, but people are going to have to learn some new skills if they want to be relevant, as these stacks are increasingly used by the companies they work for. So that, one hopes, that people are willing to learn and more pertinently, I guess, one hopes that the organizations they work for are ready to invest in them learning new skills, being increasingly relevant, because it's a little bit fair, unfair, I think, to expect everybody to be learning on their own time, on their own dime. So I think the organizational training is super important.

Ben Newton: Yeah. Yeah. When it comes right back to where we started, at the end of the day it's culture and people like, are you willing to support your people that have a learning culture? Are you willing, you have a collaborative culture? Are you... It's not just a generational thing. It's about our, you know-

James Governor: Right. But you'll have organizations that say they have a learning culture and then you ask them what's the training budget for each.

Ben Newton: What training budget?

James Governor: Each employee. And they'll be like, Oh yeah, about that.

Ben Newton: Well, what we mean by that is you're allowed to learn on your own time. That's what we mean.

James Governor: Yeah. There was a brilliant framework. I think it's ABN AMRO. They do something, they have a 70, 20, 10. 70% of your time should be spent on writing business logic. But 20% of your time should be on, they call it, IT for IT, which is basically hacking automation to improve some IT processes that touch the work you do. And then 10% of your time should be spent on training. And there is budget that every developer or operator has in order to undergo that training. I thought that was quite an interesting framework and way of looking at it.

Ben Newton: Yeah. I like that. I like that a lot. It's all about just being more purposeful about the investment. Absolutely.

James Governor: Yeah. I mean, it's just difficult. Yeah, exactly. Being purposeful and investing in people. I mean, any organization that doesn't support their employees and wanting to learn more about technology, that is extremely shortsighted. And I would say the strongest piece of advice I would give on this podcast is don't do that.

Ben Newton: James, as always, it's always a pleasure to talk to you. I think you have a great perspective. And I really, I think we covered a lot of landscape here, so I appreciate your time and thank you for coming on the podcast.

James Governor: Awesome. Ben, that was super fun. Thanks for inviting me.

Ben Newton: Absolutely. And thanks everybody, as always, for listening and find us and rate us on iTunes or your favorite podcast app, so other people can find us. And as always, thanks for listening and see you next time.

Speaker 3: Masters of data is brought to you by Sumo logic. Sumo logic is a cloud native machine data analytics platform delivering real time, continuous intelligence as a service to build, run, and secure modern applications. Sumo logic empowers the people who power modern business for more information, go to Sumologic. com, for more on masters of data, go to mastersofdata. com and subscribe, and spread the word by rating us on iTunes or your favorite podcast app.