The Backup Wrap-Up

Subbiah Sundaram, VP of Products at HYCU, joins Mr. Backup and Prasanna Malaiyandi on the podcast to explain the evolution of the HYCU product line. They have expanded well beyond their original product that was aimed at Nutanix AHV. In addition to VMware, they've added support for Google Cloud Platform, Azure, Kubernetes, and Microsoft 365. They are also clearly focused on delivering their offerings as a service. We then have a very interesting discussion about the use of the multi-region object storage that Google Cloud offers.

Mr. Backup and Prasanna grill George Crump, CMO of StorOne, a software-defined storage company, about their product. We start with an overview, alon with George's claims about not needing backup... .then we challenge him on that topic. George used to be Curtis' boss, so this one's a fun one.

We are super excited to have the Emmy Winning sound mixer Larry Blake on the podcast to talk about his upcoming book "Solving The Digital Dilemma." It is a response to "The Digital Dilemma," which is a paper released in 2007 by the Academy's Sci-Tech Council that discusses long term archiving of a movie. (It describes a very NON-digital solution.) He has edited the sound on over 60 motion pictures, over 30 of which with Oscar-winning director Steven Soderbergh. This is both a very entertaining and very informative podcast about the about the art and science of archiving motion pictures.

Wolfgang Goerlich, Advisory CISO, explains the current state of information security, and why he thinks many environments are focusing on the wrong things. We speak about ransomware, extortionware, and phishing, even giving examples where we know we have personally been phished! He explains how this illustrates his point that we need more emphasis in different areas of information security.

Mr. Backup and Prasanna discuss the Kaseya attack that happened over the July 4th weekend of 2021. First, we talk about how bad actors use long weekends for attacks, then discuss various things you can/should do to ensure that your environment would not be vulnerable to such an attack. We talk about the kinds of questions that even an unskilled person can ask to help mitigate this risk, including (especially) the all-important questions about backups. We also talk about the need for establishing a "bat-phone" type connection between your environment and any providers you may be using. We have a short talk about the impact that all these attacks are having on ransomware costs, and how you can use your DR site to test patches.

Joe Jurneke has been designing the innards of magnetic devices since 1973, and now he's here to answer our questions. He started with disk, and moved over to tape over thirty years ago, and is now retired – but consults with the tape industry from time to time. We talk in detail about how tape drives work they way they do, their reliability and more. We take down a couple of myths from the guy that was there, even correcting a misunderstanding Mr. Backup has had for many years. If the phrase "magneto crystalline anisotropy" has been used on any other podcast, we'd be surprised. This is a good one you won't want to miss.

We cover my latest book, Modern Data Protection, by O'Reilly & Associates on this podcast. I give an overview of the book, which covers the "Why, What, Where, and How" of backup, archive, and disaster recovery. After giving an overview of the book, I talk about why I wrote, and what it's like to write a book for O'Reilly. Prasanna acted as a tech editor on the book, so he offered his unique perspective as well.

Prasanna and Curtis discuss whether or not can (or will) OVH properly redesign their backup infrastructure to prevent another incident like what happened in March, where many customers lost their sites forever. As we discussed in our previous podcast, OVH had a backup service already that people paid for, and it was not up to the task. OVH"s CEO made an 8-minute video where he discussed some of the things they were going to do to make things better, and we discuss what he said. We talk about their idea of a centralized region just for backups, and whether or not that's a good idea. We also talk about how big of a job they have in front of them. We applaud what we see, but have many concerns that the brief video do not address. We also talk about how this plan is supposed to take five years, and what do OVH customers do in the meantime?

We talk to Mike Johnson of ComplyTrust, who says they "remember those you are supposed to forget." We talk about the data management challenges created by data subject access requests (DSARs), right-of-erasure (ROE, AKA right to be forgotten, or RTBF) requests, and the fact that we have many parts of the datacenter that are much better at remembering than forgetting. Backups are a particular challenge, but Mike brings up other challenges, such as mergers and acquisitions, and salespeople importing old data. ComplyTrust SaaS offering has a unique solution to this problem by remembering (on your behalf) those you are supposed to forget, and continually checking to see that they stay forgotten.

This week we discuss further lessons from the OVH fire, which starts with an admission by the CEO that some customers who paid for the backup service lost their backups in the fire. It then morphs into a discussion about designing resilient systems, starting with the concept of designing for failure. You have to protect against both physical and logical damage to your apps and data. We talk about using both cloud-native apps that have resiliency built in, vs having to add resiliency to your own app. Most importantly, know how your app/data is protected, and don't tolerate wishy-washy terminology in your service agreements. Above all, test, test, test!

This week we discuss a topic brought up by the OVH fire. It appears some people actually had a contractual backup service that wasn't doing it's job. How do you verify that a service you're paying for is real, and is doing what it claims to be doing? Especially how do you make sure they are storing data offsite? We've got some ideas.

Datacenter manager Dan Frith (@penguinpunk) joins us on the podcast for our first discussion of the #OVHFire. A massive fire destroyed a datacenter of a large cloud provider in Europe, and millions of websites disappeared. We talk about the lessons we can learn from this event.Dan talks about how outsourcing the servers doesn't outsource the responsibility for data protection. I make the point that this fire shows what happens when you completely rely on a single entity for both production and data protection. We end up talking about the 3-2-1 rule and how it applies in this scenario. I also give a discount code during the podcast for my new O'Reilly book Modern Data Protection, which is now available for purchase. If you use the URL below and the code I give on the podcast, you can get 35% of the retail price. https://shop.aer.io/oreilly/p/Modern_Data_Protection_Ensuring_Recoverability_of_/9781492094050-9149

It only took us 100 episodes, but we finally got Dave Russell, VP of Enterprise Strategy at Veeam, as our guest on the podcast. Dave and Curtis go way back, and this was a great discussion. We cover the proper use of tape, and what it was like for Dave when he went to Veeam. Another big discussion point was Dave clearing up misconceptions (some of which may have come from this podcast) about what Insight Partners acquiring Veeam really meant. We then get into a great discussion about how Veeam works, ending that discussion with Dave explaining what Veeam is doing to address concerns about Windows and ransomware.

Our anonymous guest this week is from a Fortune 100 company who is considering swapping out their backup product. Our guest has been at the company for over 20 years, and remembers swapping out NetWorker for NetBackup many years ago. Now he is considering swapping out NetBackup to address his challenges with that product. We discuss a number of topics, including the age-old argument of who should be in charge of database backups, as well as the challenges of moving to a modern backup product when you are still using operating systems not usually supporting by such products. Our guest's final thoughts center on the importance of a good relationship with the vendor in question.

A Veeam user warns of what he felt is a confusing option in Veeam Backup for Microsoft 365. He says he likes the product, but that the first retention setting mentioned in the documentation (item-level retention) might not do what you think it does. He thinks everyone should use snapshot-level retention, which behaves more traditionally. We also discuss IBM Spectrum Protect (AKA TSM) a little bit, as they also use that product. Our guest is speaking on conditions of anonymity, so we gave him a fake name (Puddleglum) and altered his voice in the recording. (Want to talk about your environment, but don't want to use your name? We'd love to have you on and we'll keep you anonymous too!)

Adi Ruppin, founder of Ananda Networks, joins us on the podcast to discuss how they secure – and increase the performance of – network traffic without deploying a VPN or SD-WAN. He talks about how the technologies we use for networking are actually very old designs that come with a lot of downsides. Ananda Networks aims to address those downsides while giving you everything a VPN and SD-WAN do – and more. Faster and more secure internet connections without the technologies we usually use for such things.

It's hard to believe, but this is our 100th episode! Prasanna and Curtis discuss the favorite topics we've had over 100 episodes, as well as the many things we've learned along the way. We talk about containers and K8s, tape, COVID-19, election security, and how recoveries are impacted by other factors. We also talked extensively about ransomware, and talked to someone who had actually been through a recovery from an attack. We talked about DDI, cloud backups, the importance of segregating your Windows backup server, and many things about the 120+ database products that have to be backed up. Thank you so much to our listeners! We look forward to 100 more episodes!

Russ Cantwell (@rcantw3ll), CTO of SHI Corp, joins us to talk about Kubernetes, the Container Storage Interface (CSI), and backups. Before we get to that, however, we talk about how he, his pregnant wife, and two-year old child all got COVID-19. We discuss how that went and continues to go, and then we talk about Kubernetes. (If you're not interested in our banter, and just want to hear about K8s, just fast-forward to 11:50.)

Another industry veteran, Jose Calhariz. joins us this week to explain how he uses the community versions of two open-source backup tools to meet his university's backup needs, while saving money. We have a very interesting discussion that includes coverage of dump, tar, and Amanda. Jose also tells a great story of a huge recovery he had to do, that took several days.

Darren McBride, CEO of Highly Reliable Systems (http://high-rely.com/ ) joins us on this podcast to talk about their product, which he says is designed for SMBs to get reliable offsite backup without using tape or an Internet connection. They have purpose-built appliances that support mirroring, but with removable disks. This product has been on the market for almost twenty years, but is having a surge in demand due to the desire by many companies to have an air-gapped backup for ransomware protection purposes.

In May of 2020, Tony Mendoza of Spectra Logic found out his company had been attacked by ransomware. Hear his harrowing tale of how long it took just to get the data center ready for a restore, and then the various tools they used to bring things back online. He did not want to pay that ransom! Spectra Logic is actually a tape vendor, so Tony has a unique viewpoint. We thank him so much for being so candid about his experience. You will learn a lot.

Rob Morrison joins us from Bacula Systems, the commercial arm of the open-source backup product, Bacula. It's tagline is that it roams the datacenter at night and sucks the vital essence from your computers. Bacula Systems has come a long way since I first saw them years ago. Check out what they're been up to.

This week’s episode is dedicated to my friend Jim Bougor, who passed on this week. Jim, this week’s guest (Darryl Baker), and I all worked at Collective Technologies back in the day. Darryl comes with over 30 years of backup and IT experience, and he and I walk down memory lane about the way things used to be with backups. Apropos for women’s history month, we also talk about Grace Hopper and Ada Lovelace a little bit. The Grace Hopper speech Darryl references can be seen here: https://www.youtube.com/watch?v=9eyFDBPk4Yw&ab_channel=funbury We talk about all sorts of tape drives from the old days including: 9-track, VHS, QIC-180, 8mm, AIT, 4mm, TK-70s, DLTs, and LTOs. We discuss the concept of coercivity and how that relates to magnetic media. We also discuss the difference between helical scan and linear tape drives, and Darryl’s theory as to why helical scan disappeared.

Chainkit Founder & CEO Val Bercovici returns to the podcast to build on what we learned last week. This week we talk about how distributed ledger technology (such as the one in BlockChain, but there are others), can be used to increase security. We talk about the SolarWinds hack and how that could have been prevented using such technology.

We invite blockchain expert Val Bercovici, Founder & CEO of ChainKit, on the podcast to explain the basics of distributed ledgers, as well as the biggest distributed ledger – BlockChain. He explains what a distributed ledger is and why you might want one. We then talk about why someone would contribute to such a ledger, meaning why you would volunteer your resources to be part one – a process known as "mining." Then, as a precursor to our next episode, we talk a little bit about the security possibilities of a distributed ledger.

Prasanna and Curtis discuss the importance of occasionally refreshing your hardware (or virtual hardware) and how important backup in in that scenario. There are many modern tools that can help you migrate from one thing to another (e.g. iPhone migration), but you'd better also have a decent backup. We also discuss the pros and cons of TimeMachine. It's nice, but not perfect. (Still better than anything in WIndows, though.)

Julie Ulrich, Systems Engineer at Farm Bureau Insurance of Michigan, joins us on this episode to talk about her experiences with NetBackup and Rubrik in her world. She’s been working in backups for over 25 years, so has seen a number of iterations of both products. We talk about many of the challenges she had with NetBackup that led her to considering Rubrik, as well as the pros and cons of using Rubrik. We also talk a little about her concerns about Microsoft 365.

These are the kinds of stories that keep you up at night. UK police deleted hundreds of thousands of records. Sysadmin accidentally deleting thousands of users. A new backup "feature" that made the backups worthless. The infamous story of Toy Story getting deleted with no backups! All this and more!

In another fascinating look into a very different world (backup-wise), we are delighted to have Timo Piiparinen from Multicom in Finland. He’s been in the IT industry for over 40 years at the same company! TImo gave us a fascinating look into a very different backup world. He makes a case for tape in the backup system (something I gave up on a while ago) because he’s using a backup software vendor that didn’t give up on the medium and actually innovated for it. Using a unique multiplexing setup that used flash disk as a big cache for the tape, very large block sizes, and what he called read-optimized writes, they run these tape drives at over 650 MB/s during backups and can guarantee restore speeds! He and I spar a bit over the value of disk during DR, and his position was that this was only necessary for the most critical systems. He’s a fan of tape, which is hard to find these days. TImo will be back.

We welcome Andrew Wertkin, Chief Strategy Officer of BlueCat, a DNS, DHCP, & IPAM (DDI) security company. Like backup, properly securing these parts of your infrastructure are both extremely important – and everyone thinks it's boring. I knew nothing about DDI before this recording and I learned a ton about the ways that bad actors use these technologies to either attack or exploit your company. Using technologies like Bluecat can actually stop ransomware in its tracks! Andrew explains exactly how ransomware attackers use DNS for Command and Control, and how products like Bluecat can be used to stop them in ther tracks. This is a great episode with a lot of really good information.

Vanessa Toves joins us again to explain exactly why you need to backup Microsoft 365 and similar services. We talk about how Microsoft is only responsible for that platform; you are responsible for the data. Particular attention is given to the idea that somehow Retention Policies are a substitute for backup. She explains exactly why that is not the case. In fact, her experience with such policies has her struggling to wrap her head around why someone would want to do that.

Vanessa Toves, a Microsoft 365 expert, joins us on Restore it All to explain the unique architecture of Microsoft 365 (formerly known as Office365). We talk about Teams and Groups, and how many people misuse both. We cover Sharepoint's role in this as well, and how Exchange Online figures into the picture. This was originally going to just be an overview, but our architecture questions just kept coming, and Vanessa kept answering them. This episode lays a good foundation of what we will cover in the next episode, which will be dedicate to backing up Microsoft 365.

Prasanna and Curtis wax philisophical at the end of 2020, and try to predict what they think will happen next year. We look forward to the end of 2020 (don't we all?) and look with a positive outlook to 2021. We talk about the vaccine, and what it's going to be like getting back to work, and whether or not there will be a new normal there that's not like what it was. We talk about SaaS products and their prevalence, as well as the emergence of a number of backup products based on Kubernetes. 2020 has been quite the year; let's hope 2021 is better.Here is the FTC page I referenced in one of the editor's notes: https://www.ftc.gov/tips-advice/competition-guidance/guide-antitrust-laws/single-firm-conduct/predatory-or-below-cost

Fathom Analytics (https://www.usefathom.com) is a privacy-first analytics engine, and they were attacked in November 2020 via a very big DDOS attack that threatened their core business. Jack Ellis, co-founder of Fathom Analytics, is a developer and first used the developer mindset to address this threat. He quickly realized he was out of his depth and decided to call for help. The AWS Shield Advanced team came to the rescue and helped them shut down the attack, and helped to stop additional attacks that were still on the horizon. Jack wrote about his experiences in the blog post "Someone attacked our company," and its URL is below. Now he shares his experience with us. What's it like to be attacke by an unknown assailant bent on doing your company harm – and to not be sure how to stop it? Join us on this episode to find out!Someone Attacked Our Company: https://usefathom.com/blog/ddos-attack

We get a closer look at the eye of the storm. A few episodes ago, we had an anonymous guest we called Harry Potter, who told us what it was like to manage a DR remotely. This week we hear from the man on the ground, recovering two datacenters after a hurricane wiped out the island where they resided. We learn all sorts of lessons, especially about how so much of a disaster recovery has very little to do with what backup people think of when they think about a "recovery." In fact, we spend almost this entire episode NOT talking about backup! We talk about network connections, places to sleep, how to eat, and how to recover your datacenter when the neighborhood power won't even stay on. This guest is also remaining anonymous, and since he is Harry Potter's friend, we called him Ron Weasley. (My apologies to the Potterheads out there.)

Cornelia Davis, 30-year industry veteran and CTO of Weaveworks, explains what cloud-native design is and why it matters. She explains that you do not have to go fully cloud-native on day one if you are moving an app to the cloud, but everything you do "refactor" to a cloud-native architecture will be more resilient, scalable, and cost you less to operate. We talk about Kubernetes and how that figures into everything, but how it is not the only way to be cloud-native.Listeners to the podcast can get a 40% discount on Cornelia's book by using our code "podrestore20" https://www.manning.com/books/cloud-native-patterns?query=Cloud%20Native%20Patterns

Lindsey Schulz MD/MPH joins once more on the podcast to discuss the great news we have had in the last few weeks around the Coronavirus (COVID-19). There are THREE vaccines that have been announced that all exceed the goals the medical community set, and will all be applying for emergency use authorization from the FDA. She explains the pros and cons of each of the three vaccinees we know about at this point, as well as giving a little info about another vaccine that is expected to announce soon. This is great news!

Lindsey Schulz MD/MPH visits us again to give us an update on the current state of COVID-19. There is good and bad news, here, as we are in the midst of a third wave of cases and deaths. But the good news is that health care professionals have learned a lot about the virus in the last several months that have helped to reduce the death rate somewhat, and improve quality of life for those suffering from the illness. Next week's episode will focus on the vaccines that we now know about, but this week we will just talk about the current state of the disease itself.

In a departure from our normal coverage, I decided to bring on Mark Thompson, who was the Site Manager for the election polling site I volunteered at last week. He gives us insight into how elections are managed in the US, and what we do to prevent fraud and ensure the overall integrity of the system. We talk about the technology used on the front end of the polling process, which in San Diego county including Electronic Poll Books (EPBs) and Ballot Marking Devices (BMDs). We then also talk briefly about the checks and balances in the actual counting process. Although this is a departure from our normal fare, I think a lot of people will benefit from the info we discuss.Also, I also posted a blog post about this topic here: http://www.backupcentral.com/why-its-really-hard-to-rig-a-national-election/

Prasanna and Curtis talk about two recent ransomware attacks on hospitals and what we can learn from them. They also discuss things you can do to protect yourself from such attacks, and how to prepare to respond if you get one. We especially talk about the 3-2-1 rule and the remote desktop protocol (RDP) and how these figure into protecting yourself from such things.

This isn't a rebuttal to last week's podcast, but it might seem that way. Last week we talked about the advantages of tape for very long term retention (e.g. 10 years), one of which is a significant cost advantage. This week we will discuss how backup, recovery, and disaster recovery are very different use cases, and why disk and cloud is a much more appropriate place for that use case. Joining us to discuss this topic is Druva's CTO, Stephen Manley, who has spent many years at companies that use disk for this purpose.

Matt Starr, CTO of Spectra, comes on the podcast to discuss the advantages of tape for long term storage. We talk about how tape is actually better at holding data long term than disk is – 10,000 times better if you compare it to SATA disk. We also talk about the advancements in tape in the last 10-20 years that have made libraries like Spectra's even more reliable than they used to be. Finally, we talk about the Spectra T-Finity library that can now hold an Exabyte of data in a single unit! What started this whole idea of bringing Matt on was Spectra's eBook that said that the T-Finity tape library was significantly cheaper than Glacier Deep Archive if you store your data for a long period of time (e.g. 10 years or more). Here's the eBook that got the conversation started: https://bit.ly/37BtTkK

Daniel Rosehill, a self-described "backup anorak," joins us to discuss how difficult it can be to backup consumer SaaS services, such as Evernote. Daniel used to use EverNote on Linux, and sent a message to their support system on how to backup its data. They had no answer unless you were using it on Windows. We talk about Google Drive, Dropbox, and other consumer-grade cloud services, and how the challenges of backing them up should be a concern for any users of these platforms – not just backup anoraks. (An anorak is a slightly prejorative term that refers to someone who is interested in a not-so-leading idea – like backups.)

The Palo Alto Networks’ Unit 42 threat hunting team found that a big customer of theirs had misconfigured two critical Amazon Web Services (AWS) services. If these misconfigurations were exploited by hackers, it could have created a data breach that could have cost the customers tens of millions of dollars. Prasanna Malaiyandi and W. Curtis Preston (Mr. Backup) discuss this misconfiguration, and what you can learn from it to protect backups you store in the cloud.

A mystery guest from a multinational corporation that experienced a disaster during a recent hurricane describes their mostly-disk NetBackup environment and how they used it to recover from a hurricane that ravaged an island. (Company and location names are changed or not mentioned so "Harry Potter," our mystery guest, can speak freely.)

Shalabh Goyal(@goyalshalabh) joins us on this podcast to discuss sharded databases – and MongoDB and Cassandra in particular – and how one might go about backing them up. Suffice it to say that it is complicated. We learned a lot from recording this podcast and you will learn a lot listening to it!

Tony McGarry, Senior Principal Engineer at Druva, joins W. Curtis Preston and Prasanna Malaiyandi to talk about backing up large, multi-node, sharded NoSQL databases like DynamoDB, Cassandra, and MongoDB.

Adam Fisher (@BonzoVT), Cloud & DevOps Engineer at RoundTower Technologies, visits the podcast to talk about VMware Cloud on AWS (AKA VMC), and what it's like to actually use and administer it in a production environment. We also talk about how people backup VMC.

Carol Nichols and Jake Goulding, the authors of the Rust in Motion video series, join us to talk about Rust and why it makes such a "safe" programming language. We discuss what it means to be a safe programming language and how Rust accomplishes that.We also discuss the upcoming free Live@Manning Rust Conference.Finding Rustaceans weird but intriguing? Secretly wanting to become one? Tune-in, Sep 15, to the live@Manning #Rust conference to find your #Rustlang pincers! http://mng.bz/qNoASomewhere in the podcast we also give out a 40% discount code for anything at Manning.com.

Gina Rosenthal (@gminks) joins us on this week's podcast to talk about the edge, Kubernetes, and how to protect it all. Good conversation that also includes som Texas Brisket talk!We also mention the Women in Tech conference that will go live on October 13. Here's what they have to say about themselves:When the girls get coding!. Join us on your screens, Oct 13, for the live@Manning “Women in Tech” conference to celebrate the rising movement of women in technology. http://mng.bz/7GZmWe still have a long way to go to achieve diversity, inclusion and equality in technology. Our contribution is the live@Manning “Women in Tech” online conference, Oct 13, starring the women rocking the tech boat! http://mng.bz/7GZmCloud navigators and serverless gurus; algorithm sorceresses and community advocates; we proudly bring you the women creating the tech world we live in. Oct 13, live@manning “Women in Tech” Twitch conference! #womenintech #womenwhocode