Today's Methods of Data Security and Erasure (Guest: Nathan Jones)

Nathan Jones: Are we talking about my personal hard drive, or maybe yours? No, we're talking about drives that merit state- sponsored attacks. This is critical infrastructure. This is a big Fortune 100 data center where they've got information that's worth millions and billions of dollars.

Ben Newton: Welcome to the Masters of Data Podcast, the podcast that brings the human to data. And I'm your host, Ben Newton. Welcome everybody to another episode of the Masters of Data Podcast. And we are going to be exploring a different part of the data experience here, which I think is going to be a lot of fun, kind of the end of the data where data goes to die. So we're talking with Nathan Jones today. He's a data security expert. And we were talking about it before, he has a very particular set of skills about data. So, welcome to the podcast, Nathan, it's good to have you here.

Nathan Jones: Hey, Ben. Good to be with you.

Ben Newton: Absolutely. And so, before we start digging into that, how did you get into that? What's your story? How did you end up where you're at?

Nathan Jones: Well, I grew up in the northwest. My father was a Microsoft employee. It was his dream that I'd follow in his footsteps and be an engineer. He wanted me to code, all the toys he bought me growing up, all tried to point me in that direction. And I completely failed him because I found out that I wasn't really that good at coding. So I ended up taking a turn to the dark side of the sales and marketing aspect of things.

Ben Newton: So what made you decide that that wasn't for you? You just didn't enjoy programming? Or there-

Nathan Jones: No, there was a moment when I was in a lab late on a Saturday when everyone else was out having some fun, and I was trying to discover where I left out a semicolon in my code, and I just got so angry. I've never been so angry in my life where I just... And I was looking around and there was nobody else there. And I was like," I can't do this. I can't do something where I'm just completely average at it." And so that was my moment of clarity was, I knew that I could probably still hack a career, but I knew that I wasn't going to be that good at it and I probably needed to find something else.

Ben Newton: Oh, yeah. No, I can understand that. I definitely had some moments like that in college. And you remind me, I remember my first programming project I ever did in school, it was a... you were playing... I don't remember what the game was. It was some sort of two- person game, and you all had to program a program to compete against each other. And my program was beating this guy, and then it hit a bug, and then it went down flaming. And that was not a good feeling. Somehow I still ended up staying in that direction, but... So, you realized that programming itself is not for you. So what did you decide to do? What direction did you take it from there?

Nathan Jones: I was still fascinated with the field and I love new technology. And the way I paid my way through school was I sold pest control door- to- door, and I was good at getting people excited about things, and so I wanted to stay in the industry. And I was at a point where I could hang with the technical crowd and I really enjoyed the first part of coding, which was," All right, let's figure out how this is going to go. Let's figure out the algorithms we're going to use, what's going to hand off what data to what." And that part, I loved. And I actually didn't even mind coding it up, but once it was just kicking out a bunch of compile errors, that's where it just clearly broke down for me. So I knew enough about it and I really enjoyed it, so I just found more of a corner that I was more comfortable with.

Ben Newton: Right. So, how'd you end up in data security specifically? Did you get into the data center game or something? Or-

Nathan Jones: No, there were all kinds of ads for," Hey, have an internship here, if you're wanting to dip your toe in the water and see what programming is actually like." And so I took a job at a software company that had just split off. One of the biggest forensic companies out there is AccessData. They have a tool called the Ultimate Forensic Toolkit, widely used by law enforcement. And they had just splintered and had kind of an offshoot called WhiteCanyon that left them that was specialized, not on the forensic side, but on the data erasure side, the exact opposite of forensic, which is making sure that no one could ever recover data from a device. And they decided that those two things were incompatible, and so they split the company in two. And once that company had split off, they were hiring on additional engineers. And I took a job where... getting paid just a little bit, but it was more to see if I would enjoy it. And once I decided I wasn't going to be a programmer, I was on my way out the door. And one of the CEOs stopped me and said," Hey, we all like you, and we know that you're good on the sales side. Maybe you want to work for us in a different capacity." And so it was just one of those moments where my life took a turn and it worked out really well. I've been doing this now for 14, 15 years now. And it was all because of, when I was on my way out the door, I had made enough friends that they would try to find a spot for me in something that was kind of different.

Ben Newton: Yeah. No, that's really interesting. So I guess that's the way a lot of these things happen is you don't really see it coming, and then one of those forks in the road where you have to make a big choice. So you picked data erasure as the place for you. One of the things, too, is earlier in my career I did work around data center stuff. I actually did government contracts. And so when we were first talking and I was looking at what you specialize in, that special set of skills... I think a lot of people would not, off the top of their head, think about that being so important, but anyone who's had to deal with government work or particularly before it was just a Cloud and stuff magically happened... I mean, this is complicated. We were having to put nails through drives and burning stuff and just all these complicated things that we had to go to. So, talk a little bit about it. I mean, what makes this complicated? Why is it such a big issue?

Nathan Jones: Well, it's basically you've got all this data, it's encrypted or not, but you have all this data on drives. What are you going to do with it when it reaches its end of life? And are you going to allow that to be repurposed or reused? Or are you just going to try and grind it up, degauss it, pulverize it, incinerate it. Companies do all kinds of things. And where we come in as our specialty is that we're doing a non- destructive means of forensically removing that data from the drive. So we're going through and we're overriding all of the various locations on the drive, we're doing specialized verification to make sure that no one could ever recover that data from there, we're doing it to NSA specifications. And then that's really the crusade that I'm on, is that these drives with this information can be securely overwritten. They don't need to be physically destroyed. They don't need to have a nail put through the middle of it or have them degaussed or incinerated, but that a software tool can securely remove the data from there. And in fact, in a lot of cases, a software tool is a superior solution to that. And we've gotten a lot of big Fortune 100 companies on board with this, but there's still a lot of holdouts that just, they like having their drives destroyed. They enjoy it or it's just something that's more satisfying when you're physically destroying the drives, as opposed to wiping the data from them.

Ben Newton: Well, I will say, I did put a nail through a drive once and it was kind of fun. So I can tie it onto that view. But yeah, there's a physical connection to things which is... it's a natural human connection. You want to see something happen and software sometimes doesn't feel real.

Nathan Jones: For sure. And the NSA will put out different specifications every year, and they'll say," You need to grind this type of drive up to this size of dust." And it gets progressively smaller every year, meaning that," Hey, the reason why we're changing this is because we think that if someone was convinced that there was data on these little pieces of shred, that if they had the means to do it, they could still recover some important information." So they're constantly saying," You've got to do more on the physical destruction side." Now, are we talking about my personal hard drive or maybe yours? No, we're talking about drives that merit state- sponsored attacks. This is critical infrastructure. This is a big Fortune 100 data center where they've got information that's worth millions and billions of dollars. So, when you're dealing with that kind of information where there is... you fail once and that's it. There is zero tolerance for any type of failure in this. And so, if you let something get out... There's no room for mistakes in this.

Ben Newton: Yeah, no, no. It makes a lot of sense, because it's like a... not quite drive, but I always remember the stories about, in Iran, how they shredded the documents and then people spent I think months or years piecing the documents back together. If someone's motivated enough and it's important enough, the amount of expense they'll go to reverse engineer this stuff is big. So what would seem over the top in a normal situation is different when you're dealing with this level of data, I would think.

Nathan Jones: And people will still continue to go over the top where they'll use our tool, which is certified by the NSA to completely remove any type of forensic recovery possibility, and even after that's done and it's been certified, they'll still go through the physical destruction side. And they'll take it even beyond that, where they'll say," We can't have a single point of failure as far as personnel goes, so we need to have multiple people in different rooms. One of them doing the wiping, one of them doing the verification, then one of them doing the physical destruction. And these three people are never allowed to communicate with each other." It gets pretty serious in some cases.

Ben Newton: Yeah. Well, all I'm imagining now is what they need is that... What's the big, bad guy in the Avengers that has all the rings and all the stones-

Nathan Jones: Thanos.

Ben Newton: ...and he can... Thanos, thank you. If you have Thanos, you can take care of all of it.

Nathan Jones: But that'd only get rid of half the data, right? So there'd still be stuff.

Ben Newton: Yeah, that's true. Absolutely right. Okay, you got me. So, it's one of those things after worked in environments like that, I've still got drives... We just moved recently and I was digging through my boxes, putting stuff up, and I've still got drives that are probably like 15 years old, because I'm afraid to do anything with them, because it got beat into me that you had to think about stuff like that. So it's a big deal. So talk to me a little about the software side of this, because like we were talking about, you've got systems that are end of life, for whatever reason they're being put aside, they have to think about the data. And I know that it's not just government stuff, you've got these even big internet companies that are just going through thousands or tens of thousands of drives that they have to deal with. And it makes a lot of sense, I'm assuming, like we were saying, to put hammers through them and stuff like that, but it's actually expensive, it's time- consuming. So what does it actually mean from a software perspective? Because I can understand, anyone can understand, you put a nail through a drive, you degauss or whatever, something physical that you can imagine happening to it. But what does it mean from a software perspective to erase data like that? What do you have to think about?

Nathan Jones: Yeah. So the core principle is that we're just going to go through and in every addressable location, whether it be a platter base drive or a solid state drive, we're going to go to every addressable location and we're going to overwrite useless information across there. And different countries and different industries will have different standards as to how they want it to be done, if they want it to be done just once, properly, or if they want to be done multiple times. You have the old DoD 5220.22-M standard that came out around 2001 that said you needed to override it three or seven times, depending. And then the NIST 800- 88r1 standard came out about four or five years ago that said," No, that's not necessary." However, the DoD standard wasn't taking into account solid state drives, which have what's called wear leveling areas. So if you've got a one terabyte SSD, the actual size of that's like 1. 18 or 1.2. That additional space is needed to prolong the life of those SSDs. And if you're not hitting that space, well, then someone could actually pull the chips off of that SSD and read the raw data. So we weren't taking that into account around 2001. So the standards are evolving, but the principle is still the same. We're just making sure that we're overriding every location on that device, no matter how the drive controller is trying to protect it, even in higher- end labs, even if you're going to try and throw them under an atomic microscope, make any type of recovery impossible.

Ben Newton: And I guess talk a little bit about that. Because I remember the first time that I saw this. You think about it, even beyond the obvious, it's like," I delete data through my computer and I say delete this or something like that." But even if you get to the deeper level, I remember dealing with drives where you needed to override it all with ones and zeroes or random stuff or things like that. But why do you even have to think about multiple times like that? Why does that matter?

Nathan Jones: It's a fun question. It's kind of an older question, but it's a fun one. So around 2000, the way that the hardness technology was coming along was, the way that we were writing to a location, we would write a one and then we would write a zero over the top of the one. And if you put that platter under an atomic microscope, you could see," Hey, there used to be a zero there, and now there's a one." And you can actually see up to four or five or six levels deep of what used to be there. And they'll call that bit shadows. Now, this is an incredibly expensive and... I mean, it's far worse than the 79 Iranies putting the embassy's shredded documents back together. This would be incredibly painful, but the US military saw," Hey, wait a minute. If somebody really was adamant about trying to recover from it, we need to make sure that our drives in the military are being overwritten multiple times to eliminate the possibility of somebody looking into the past, so to speak, and seeing those bit shadows and saying,'Hey, there was a one there. Even though they overwritten it six times, there was a one there, and that one we can use. And now let's go to the next.'" It would be unbelievably painful to try and do a recovery like this, but it was theoretically possible. There was actually a scientist named Peter Gutmann that said to be 100% certain with drive technology as it was, you had to override it 35 times. And if you do that, you're basically just burning out your drive anyway. So it was more of an academic exercise than anything else, but that's why you'll hear," Hey, you need to overwrite your drives multiple times." It's a vestige of that thinking that was around 2000.

Ben Newton: Okay. Got it. Got it. So, you said it was kind of where it was back then. What do people have to think about today? What's really where it's at right now, if you're going through this?

Nathan Jones: Today it's about doing one proper overwrite and not missing anything. There's different ways that drive controllers will try and hide data from you or not allow you to access the entire drive. We talked about one, the wear leveling areas, but there's also post- protected areas, device configuration overlays, hidden partitions, you have remapped sectors on drives. These are all ways that the controller is saying," Hey, you're not going to use this space." But if you were really trying to do some high level recovery, you could go back and you could get that data, if you were to put it under a microscope. Or some of those things can be even recovered with software tools, so that makes it incredibly easy. So, the state of the art now is making sure that the tool that you're using is piercing every type of hurdle that the controller's throwing up to prevent you from accessing the entirety of the drive. Does that make sense?

Ben Newton: Yeah, no, I think. So, if I'm understanding you right, it's basically the last few decades of building hard drives and storage devices in general, the manufacturers have created all these optimizations and the complexity that makes what you think you're seeing is not what's actually really happening on the device itself. And you have to get around those optimizations to actually get at the reality beneath the surface. Is that right?

Nathan Jones: Exactly. And then when you're dealing with servers and you've got drives and RAID arrays, now you're dealing with another controller that's trying to control how you're accessing that data. And so, we've got to be able to pierce that RAID to be able to issue firmware level commands to each of the individual physical hard drives to be able to do this properly. Otherwise, it's possible that the RAID controller is hiding something. And so, we've got to go basically down to bare metal and issue firmware level commands to these drives to be able to do a proper job. And then, if we can't do that, then we'll flag it and we'll say," Hey, we weren't able to get past this RAID controller. We were just wiping a logical drive and we'll flag it." And then you'll likely still want to physically destroy that drive. But if we certify that a drive has met the standard, it's better than anything else out there. And it means that you don't need to worry about it. You can donate it to a school library, charity, sell it to your employees. It can leave your organization. And that's what we're seeing with our large banking customers and tech customers is if it's meeting the high standard of the NIST 800-88r1 purge, that's the gold standard, then they'll let these devices actually leave. And now they don't need to be physically destroyed, they can be reused.

Ben Newton: Yeah. Because we were talking about the extreme things in the intelligence community and military, which is going to be completely different, because they have to destroy it anyway. But really the thing is if you're an organization that... this would actually end up being cheaper for you, because number one, you wouldn't have to go through the... Because I'm assuming physically restoring these things, you correct me if I'm wrong, but I'm assuming that's got to be expensive. And then you can recoup some of the costs later, too.

Nathan Jones: It is. Yeah. You're constantly having to sharpen blades and it's breaking down all the time. Yeah. It's very expensive. I mean, if you're contracting out to have this done, you're seeing prices between say $5 and $12 a drive. But then if you're also talking about," Hey, these are enterprise class drives and they still sell for 350 bucks a pop on the open market." So now you're not talking about $5 to $12, you're talking about $360 a drive. And then there's also just the environmental factor where you're putting a bunch of scrap in a landfill that doesn't need to be there, because it could still be used.

Ben Newton: Yeah. No, that actually makes a lot of sense, because that seems to be in some degree how this stuff has changed. Because I would assume a lot of your corporate clients, they're thinking about this in a much broader way than we might've thought about it 20-plus years ago. They're thinking about all those factors, and they have to.

Nathan Jones: I'm always a student of things that are a little strange. I'll tell you, it's one of the strange things we've seen, is actually the US has been moving more quickly towards software destruction versus physical destruction. And you would think that Europe, where they pride themselves more on being environmentally friendly, they're very much still physically destroying everything. And it's our tech industry that's really taking the lead on this and embracing the software destruction more quickly.

Ben Newton: Now, is that partly because of the money-saving aspect of it? Because everything you described, it sounds like this could really save your bottom line, not having to... particularly when you're... Sorry, go ahead.

Nathan Jones: Going back to the beginning of our conversation, it is holding onto, "Oh, I feel safer when I see the totes of scrap metal coming out. And that makes me feel better than seeing intact-looking drives leaving." It's very much a psychological thing, because the types of tests that we go through and the types of tests that our customers put us through, where they'll send them off to world- class labs, they'll spend thousands of dollars for recovery on all these drives. And they'll say," If you can recover anything from these drives, then we're not going to use this process." 100%. And we've been doing this for so many years and we realized we don't have any room for error on this. And so it's something that we take very seriously. But the European piece is kind of a puzzle to me. When I go over there, sometimes I'll speak at shows, it seems like they're not going to lead the way on this particular thing. Although it's kind of sad because in a lot of ways they're ahead of us on the environmental side of things.

Ben Newton: Yeah. And one thing I would think too is that, as I've seen... because I remember when I was actually on a government contract in like the early 2000s, we got our first, I don't know, it was like 10, 20 terabyte drive array. And we're like," Oh my God, this is amazing." And now you're talking about petabytes and petabytes and petabytes. And particularly for some of these data- heavy industries, it seems like there's more data infrastructure than there is compute. So, I would assume that the problem we're talking about here, it's like one of those exponential problems that you really can't... I mean, if you're one of these large internet data companies or a banking company where you're just dealing with just massive amounts of data, the physical method of dealing with this just wouldn't scale, I would think.

Nathan Jones: Absolutely. And then one of the nice things about having a software solution is," Okay, we can just plug directly into the back of these cabinets and we can be wiping thousands of drives simultaneously. And then, if anything doesn't meet the standard, then it gets pulled. And then it goes into a bin at a later point. And now we're physically destroying 5% of the drives that we're decommissioning as opposed to 100%. And these 95% are meeting the higher purge standard. And now they're able to be resold. And we're rolling that back into new equipment that we're purchasing." It's a virtuous thing all around. It's kind of strange that we have such an uphill battle. It's the one part of my job that feels even slightly noble. I don't get to do a lot of things where I'm like," Oh, I'm doing some good," but this is the one part that is. The crusade that I'm on is that physical destruction is unnecessary until a certain point. Now, obviously at some point those drives are not worth anything and now they just need to be processed and you need to be... Now, there's a lot that goes into, "Hey, let's separate out the metals and the rare earths and all that properly." And there's some important points around that, but that's much further down the road. That's not after it's been in a data center for two years and the drive's still worth 400 bucks. It's that, after it's had a second and third life, and now it's turning up with a bunch of remap sectors, now is a time that physical destruction is the way to go.

Ben Newton: Yeah. No, that makes a lot of sense. So you basically take these investments and it's kind of like the secondhand store for drives. Like somebody else can wear that. Somebody else can use that drive.

Nathan Jones: Yeah. And these big companies, they're absorbing the majority of the costs for getting this very expensive infrastructure, but then there's a lot of mid- sized players that would love to be able to get equipment that's performing well that's just a couple of years old, and they don't need to have the top of the line. So it's starving them for the kind of equipment that would be perfect for their use case.

Ben Newton: Yeah. Yeah. No, that makes a lot of sense. So where do you see this industry going? You clearly have been doing this a while now and you have an interesting vantage point in all this. What do you see are the big trends that you're watching, that you find interesting?

Nathan Jones: Yeah. So there was different components in systems that have persistent memory. So for example, we've just rolled out the [inaudible] TPM chips that are holding incredibly important details about the encryption. So we're flashing that as part of our process. Some of the exciting things we're working on is we're working on predicting the life expectancy of hard drives. So we'll take all the data that we know about your drive and we'll say," Hey, this drive is 90% likely to last another three years." So we'll help you to decide whether or not that drive actually has a second life beyond that. We're working on doing faster wipes where... We work extensively with the military, work with the US House, Senate, White House, Department of State, and they would love to have a wipe that's almost instantaneous. Not possible on platter base drives, but what we can do is we can target the parts of that drive that are most essential. Instead of just going in sequence around the drives, we'll hit the most critical parts of the drive first, and then we'll work our way out to the less important. So in a matter of 15 to 20 seconds, we've hit the majority of what you would want to erase. And then that way, if that process were to get interrupted, we've still done as much damage to that data as possible. So those are some of the exciting things we're working on, on our side.

Ben Newton: No, that sounds super interesting. Yeah, I could see how that can matter a lot in some of these field situations with the government. Yeah. Well, this is super interesting, Nathan. Like I said, having started out my career working in data centers and seeing the complexity with this, I think what you guys are doing is super interesting. So I appreciate you taking time to come on the podcast and talking with us.

Nathan Jones: Ben, been a pleasure talking with you.

Ben Newton: Absolutely. And as always, thanks everybody for listening. And take some time to go on iTunes and rate and review it so other people can find us. And catch you on the next episode. Thanks, everybody.

Speaker 3: Masters of Data is brought to you by Sumo Logic. Sumo Logic is a Cloud- native, machine data analytics platform delivering real time, continuous intelligence as a service to build, run, and secure modern applications. Sumo Logic empowers the people who power modern business. For more information, go to sumologic. com. For more on Masters of Data, go to mastersofdata. com and subscribe. And spread the word by rating us on iTunes or your favorite podcast app.