The Backup Wrap-Up

This week, Prasanna and Mr. Backup (W. Curtis Preston) review a series of posts made by Snorkel42, who previously appeared on this podcast in the episode called "Security expert rips Okta for their response to hack." Things were recorded out of order, so this is the episode where we discovered him on Reddit, and tried our best to distill several thousand words into about 30 mins of advice on how to protect against ransomware. We talk about how to prevent getting it in the first place, how to limit its damage if you do get it, and how to respond and restore your data once that happens. There is a ton of really good advice here, so check it out! Here are the three posts:https://www.reddit.com/r/sysadmin/comments/tdvbp4/security_cadence_okay_fine_lets_talk_ransomware/https://www.reddit.com/r/SecurityCadence/comments/tedapy/security_cadence_ransomware_part_2_actions_on/https://www.reddit.com/r/SecurityCadence/comments/tfm927/security_cadence_ransomware_part_3_the_worst_case/ 

Vast is a massively-scalable storage system designed around multiple pieces of technology that weren't available just a few years ago (e.g. NVMe, Storage class memory, QLC) that offers both file and object functionality, immutable snapshots, and integration with the cloud to address the "smoking hole" problem. Their typical sale (of which they've made many) is north of $1 million, and they have many exabytes of disk in the wild. It's a scale-out storage system without all the typical East-West traffic such systems have. We do our best to poke holes in their offering, but Howard Marks goes toe-to-toe quite well. This one went a little long (one hour) but we truly were fascinated with the Vast story Howard was telling.

This episode is a unique look inside the Conti ransomware group, courtesy of a four-part series from Krebs on Security. We review the interesting takeaways from Brian Kreb's series of over 12,000-words from quite a bit of research. The series was inspired by a hack of Conti that resulted in a traunch of internal documents being made public. This gives a unique view into how the organization thinks, how it is laid out just like any other business, the weapons it uses to spread ransomware, and its attempts to branch out to other areas of cybercrime. If you enjoy the episode, be sure to check out the articles that inspired it:https://krebsonsecurity.com/2022/03/conti-ransomware-group-diaries-part-i-evasionhttps://krebsonsecurity.com/2022/03/conti-ransomware-group-diaries-part-ii-the-officehttps://krebsonsecurity.com/2022/03/conti-ransomware-group-diaries-part-iii-weaponryhttps://krebsonsecurity.com/2022/03/conti-ransomware-group-diaries-part-iv-cryptocrime

This week we are joined by John "Ricky" Martin, Director of Strategy at NetApp (and former owner of a tape recovery business), to talk about his paper that declares that backup is fundamentally evil and done in an unintelligent way. Mr. Backup wasn't sure how this one was going to go, and there were at least one or two arguments along the way. No blows were thrown, though. We definitely talk about what a tape recovery business is, and what it was like to do that. We also talk about tape backup, full backups, multiplexing, tape handling, and other elements of how backup is still done today by many people. It's a fun episode where you should learn a lot.

We have none other than Snorkel42 from Reddit on the podcast today. He has 20 years experience in InfoSec, and is a prolific writer on Reddit under the handle Snorkel42. (Check out his posts here: https://www.reddit.com/user/snorkel42/). (We will not be using his given name during the recording.).He thinks Okta managed to turn a mole hill into a mountain by incorrectly handling the hack that happened in January – that we just learned about last week. That's right, we just found out about a hack that actually happened in January! We dive deep into what happened, what it means, and how the worst problem of all is how Okta responded to it. Our expert says he no longer trusts Okta, and gives advice to customers on what to do next. This is a very timely episode that you will really enjoy – unless you're an Okta customer or employee.

Gary Williams tells a great story about earlier in his career that taught him the value of testing backups and updating documentation. He explains how he thought his backups were fine, until a "new guy" came onto the scene and dared to ask the question, "When was the last time you tested your backups?" As Gary explains, sometimes new people have the best perspective. They let him do the first test, and .... it failed spectacularly! It all came down to the documentation they were so proud of. Hear Gary's story and learn from his mistake – one that defined his career. (Mr. Backup also tells the story that defined his career as well!)

When you back up your SaaS apps (because you know you are supposed to), should you back them up to a SaaS service or on on-premises backup system? After defining what SaaS is and isn't, Prasanna and Curtis discuss this important question. First they look at how sizing a SaaS system for backup is different than when you do it in a datacenter, and how that creates challenges for backup design. Does it make sense to use on-premises backup to backup a cloud resource like SaaS?

Learn from others' mistakes by reviewing last year's worst ransomware attacks with Mr. Backup and Prasanna Malaiyandi. Listen to them review the 10 worst attacks from 2021, then discuss lessons learned: Colonial Pipeline, BrennTag, Acer, JBS, Quanta, NBA,AXA, CNA, CD Projekt, and Kaseya. Then they discuss the trends they see, and the lessons we can all learn from these horrible attacks.

W. Curtis Preston (Mr. Backup) and Prasanna Malaiyandi weigh in on a reddit thread that started with a simple question from a user. He has MSPs trying to sell him a solution to back up Microsoft 365, and he's wondering if that's even necessary. As usual on reddit, there are many opinions. Mr. Backup debates the various points being made by the anit-backup crowd, including an article arguing the same point. We start with an overview of why it's important, then we take on the various arguments used to support not backing it up. We talk about how retention policies are not backup, and why. Even retention lock doesn't help, and neither do lagged copies of Exchange Online. We also talk about how the e-discovery tool is NOT a restore tool and will not give you what you're looking for. Finally, we talk about the idea that backup tools can't do DR for Exchange online, and how that is used to bash them as well. Great discussion on this one.

On this week's episode of Backup Central's Restore it All, Mr. Backup himself becomes the guest, while Prasanna Malaiyandi takes over as host. W. Curtis Preston explains the backup configuration of the website behind the Restore it All podcast, and how bit rot caused him to have to restore part of it. We talk about bit rot, the 3-2-1 rule, off-site backups, backups stored in S3 and Google Drive, and what it's like to restore part of a MySQL database. Luckily, the folks at LiquidWeb were very helpful. Watch Curtis explain how practices what he preaches over at BackupCentral.com.If you want to watch the video version of this episode, it's here: https://www.youtube.com/watch?v=I3285etiYBs

This is a response to Tom Hollingsworth's (@networkingnerd) video "Disaster Recovery is a Security Function," found here: https://gestaltit.com/tomversations/tom/disaster-recovery-is-a-security-function-tomversations-episode-25/. I respectfully disagree w/Tom's assertions in his video, and decided to use this as the first episode I'm going to publish a video version of. You can listen to the podcast on all the usual podcast channels, or watch the video version on youtube here: https://youtu.be/ym_ibNWVjgATom said that backup and security are very closely related, and suggested that if we reported to the same team, we could perhaps accomplish more together. While I understand the point he is making, I disagree with it, and Prasanna and I discuss it on this episode. We believe Tom's opinion comes from an outdated concept of how security works in backup systems; we haven't worked like that in quite some time. I explain how modern backup systems work from a security perspective, then talk about the idea of backup folks reporting to security folks. I think it's a bad idea for several reasons.

This week's episode is about an incident that happened at the University of Kyoto, Japan, where they lost 77 TB of research data forever. What can we learn from what happened to them? First we discuss the concept of "we can't afford backup," that seems to be prevalent in a lot of universities and research institutions. We then ask and answer the question of whether or not it is every OK to not backup data, along with whose responsibility is it? We pause the recording for what appears at first to be a spam call, but you'll have to listen to hear that. We talk about what happened there, including a letter from HPE apologizing for what happened. Kudos to HPE for that. We also discuss a story from my very first week on the job in 1993; it's not pretty.

Vinicius "Vinny" Grippa, the co-author of O'Reilly's Learning MySQL (now in its second edition) talks MySQL and MongoDB, as well as that all-important topic of how to back them up! We first learn a little bit about Percona, where Vinny works, as they consult in the database space. We then discussed a hot topic from Curtis, which is this idea of companies that say they don't want an IT department. We then discuss the book, Learning MySQL 2nd edition, and Vinny's top 3 performance suggestions for MySQL, including a discussion about the differences between MyISAM and InnoDB tables. We then discuss the typical ways people back up MySQL and MongoDB, followed by a discussion of two free tools that Percona makes available: Xtrabackup and PerconaBackup for MongoDB. It is a fascinating discussion you won't want to miss.

This week we celebrate Data Privacy Day, which is an international event that occurs every year on 28 January. According to its website, "The purpose of Data Privacy Day is to raise awareness and promote privacy and data protection best practices. It is currently observed in the United States, Canada, Nigeria, Israel and 47 European countries." Prasanna and Curtis discuss the latest in privacy practices and regulations, drawing on Prasanna's new experience protecting the privacy of Zoom's customers. We talk about the difference between PII and Personal Data, different regulations around the world, and some new tech features you can use to protect your privacy. We also discuss a few gotchas out there, such as Verizon's new system that they require you to opt out of! Happy Data Privacy Day, everyone!

Things got a little tense on this week's podcast when James Strong (@strongjz, Co-Author of O'Reilly's Networking & Kubernetes) hinted at DR being a thing of the past with K8s. Mr. Backups was having none of that. No blows were thrown, mostly because it was all online, but it was a really good conversation that K8s and DR enthusiasts alike will find interesting. We also cover the new book, Networking and Kubernetes, by James Strong and Vallery Lancey, including why the decided to write it, and what it covers. We talk about monitoring K8s networking, and James uses at least 25 acronyms that may be new to some listeners. Don't worry: we make him explain all of them.

Bob Plankers, resiliency specialist from VMware, joins us on this week's podcast, as we examine last year's major Facebook outage that took out Facebook, WhatsApp, and Instagram all at once. We discuss what we believe happened, just how bad it got, and our thoughts as to what we can learn from this huge outage. This isn't schadenfreude, and we acknowledge that we are Monday morning quarterbacking. It's an attempt to LEARN from the misfortunes of others – not to take joy in them. Solid discussion with @plankers.

On this first business day of 2022, let’s take a look back at the year that was. It was, of course, another year of COVID. In fact, Curtis contracted COVID right at the end, despite being boosted. This is also the year of the OVH fire that we talked about for three episodes: https://www.backupcentral.com/datacenter-manager-dan-frith-discusses-the-ovh-fire-restore-it-all-podcast-105/https://www.backupcentral.com/how-do-you-prove-your-backup-service-is-real-restore-it-all-podcast-106/https://www.backupcentral.com/ovhs-backup-service-didnt-work-restore-it-all-podcast-107/We think our most interesting episode of the year goes to Paul VanDyke from Kodiak Island, who deleted his whole environment and then tested his backups: https://www.backupcentral.com/it-admin-deletes-entire-datacenter-then-tests-his-backups-restore-it-all-podcast-135/We even had a barbecue episode. No backups, just beef and BBQ. https://www.backupcentral.com/no-backups-just-beef-bbq-restore-it-all-bonus-episode/Prasanna said he learned a lot about tape this year from these three episodes:https://www.backupcentral.com/tape-drive-designer-schools-mr-backup-on-tape-restore-it-all-podcast-111/https://www.backupcentral.com/deep-dive-into-why-tape-still-has-a-future-in-storage-restore-it-all-podcast-129/https://www.backupcentral.com/fujifilm-tape-evangelist-explains-past-present-future-of-tape-lto-restore-it-all-podcast-132/We think our best “get” was Peter Krogh, who talked about how he coined the term “3-2-1 Rule” while writing the first edition of The DAM Book: Digital Asset Management for Photographers. https://www.backupcentral.com/peter-krogh-who-coined-the-3-2-1-rule-on-our-podcast-restore-it-all-podcast-131/This led to a brief discussion about solar flares, where I mention a talk I watched by Intel. This is a link to that talk:https://techfieldday.com/event/eicd16/We also had two Druva competitors on this year, Veeam and HYCU.https://www.backupcentral.com/dave-russell-answers-our-questions-about-veeam-restore-it-all-podcast-104/https://www.backupcentral.com/veeam-reps-explain-defense-against-conti-ransomware-restore-it-all-podcast-127/https://www.backupcentral.com/hycu-vp-explains-their-service-to-mr-backup-2/The big winner of the year was ransomware. We talked about it a lot:https://www.backupcentral.com/ransomware-victim-tells-his-story-restore-it-all-podcast-96/https://www.backupcentral.com/is-entity-level-encryption-the-answer-to-exfiltration-ransomware-restore-it-all-podast-119/https://www.backupcentral.com/protecting-your-network-from-ransomware-restore-it-all-podcast-122/https://www.backupcentral.com/restoring-quickly-from-a-ransomare-attack-with-a-long-dwell-time-restore-it-all-podcast-123/https://www.backupcentral.com/veeam-reps-explain-defense-against-conti-ransomware-restore-it-all-podcast-127/Curtis this everyone should be looking into an intelligent DDI (DNS, DHCP, IP management) system that will spot (and stop) ransomware when it tries to reach out to its command and control servers. https://www.backupcentral.com/stop-ransomware-in-its-tracks-with-dns-dhcp-ipam-restore-it-all-podcast-87/You also need to monitor your bandwidth to look for exfiltration:https://www.backupcentral.com/securing-speeding-up-network-traffic/https://www.backupcentral.com/stop-ransomware-attacks-in-seconds-restore-it-all-podcast-126/Finally, we talked a little about the book, and the upcoming 2022. https://www.backupcentral.com/why-you-need-a-copy-of-modern-data-protection-restore-it-all-podcast-110/Happy New Year, everyone! Here’s to a better 2022!

This week’s guest tells the most incredible story we’ve ever had on the podcast. We’ve had ransomware restores, disaster recoveries after a hurricane, but we’ve never had someone who deleted their entire computing environment and then restored it using their backups. (Backups that had never been tested to this degree, BTW.) Paul VanDyke is the IT Supervisor at the Kodiak Island Borough in Alaska, which is the second largest island in the US and has to satisfy its backup and DR needs while staying on the island. Cloud resources are not a possibility due to bandwidth concerns, so he’s doing things “old school.” We first talk about the kinds of things they are protecting from, including tsunamis, fires, and strong winds. They are primarily based on tape, and for DR they store copies of all backups in a nearby safe. We discussed ways they could improve their resilience, such as shipping some tapes to a location on the mainland. But the highlight of this episode is the story of when Paul intentionally destroyed his entire environment and then tested his backup system! He learned many valuable lessons, starting with “don’t ever do that again!” Luckily, his test was successful, albeit not without some challenges. He wiped the storage arrays on five servers: two domain controllers, an email server, a file server, and an application server and then restored them. (He had his reasons for doing it this way, which he goes into in the podcast.) One big thing he learned was how restores are often slower than backups. So he prioritized critical apps (e.g. email, fileserver, logins) and got them up by Monday morning. Then it took him a few more days to get the application server up and running due to a more complicated restore. We have a really good discussion on how Paul could have done things better, including a really good idea that Prasanna came up with it. Curtis also tells a similar story about the first time he “tested” backups when he actually needed them, versus doing it in advance. We cover a number of topics and questions on this podcast:What was an Exabyte Mammoth (M2) tape drive?What is a helical scan tape drive?What is multiplexing?Why can restores be slower than backups?What happens when you rebuild a RAID array?Should you have a post-mortem after a large incident? How important is recovery testing? How important is it to set expectations in IT, especially when it comes to recovery times?

The founder of rsync.net, John Kozubik, joins us on the podcast this week. It's a unique offering: a ZFS filesystem running in a private cloud – accessible only via SSH – that is designed just for sending your backup data to. They support anything that can run over SSH. Use rsync, scp, etc. to copy your data unencrypted, or something like restic, duplicity, or borg, if you want your backups to be encrypted. (All backups are encrypted in flight, of course, because they are all over SSH.). The servers are completely locked down except for the SSH port, so they're about as secure as they can be for what they are. You can configure ssh to behave the way you want it (e.g. passphrase, MFA, etc.), and the ZFS filesystem automatically creates daily snapshots of the backups you send there. (More complicated schedules can also be created.)You pay by the gigabyte ($.025/GB/mth) for the size of the ZFS filesystem and its associated snapshots, but they urge you to NOT over-provision. Provisioning is easy and non-disruptive, so only add storage when you need it. For an extra fee ($.017/GB/mth), they can also replicate your backups to another region.It's a no-nonsense offering that seems to be unique out there – especially when you add the ZFS features. Check out the website and rsync.net, and you'll see they aren't spending any money on being flashy. They just want to build a rock-solid ZFS syncing destination that is separate from any cloud provides.

This week, we talk to Nick Craigwood, the creator and principal developer of rclone, a very popular open-source tool for copying data to and from cloud providers. Rclone is downloaded roughly 250,000 times each month, and has over 30,000 stars on GitHub. There are six core developers, and a great community of users and other developers at rclone.org. We talk a little bit about Nick’s development philosophy, which is that he doesn’t mind adding features - as long as they don’t break backwards compatibility. Then we talk about how rclone works, and what it’s like to sync a filesystem to an object store – including support for multi-part uploads and downloads. We also talk about rclone’s encryption support, while Nick was “relaxing” on holiday. We then talked about how rclone can be used to minimize the risk of backing up to any one cloud provider, preventing things like what happened during the OVH fire earlier in 2021. We also discuss some strategies, such as backing up directly to two different clouds, versus backing up to one, then syncing to another – and how CloudFlare’s R2 might figure into things. Finally, we talk about Nick’s plans for rclone’s future, such as making their web UI better to increase usability for many more people – while not sacrificing the command line. Join us for a fascinating episode, the first one where we’re talking to the creator of the tool in question.Don’t forget the drawing for a free e-book version of Modern Data Protection. All you have to do to be eligible is sign up for my newsletter at https://www.backupcentral.com/subscribe-to-our-newsletter/

Fujifilm's tape evangelist, Rich Gadomski, joins us for an interesting discussion on tape and LTO. We talk about the different subtrates that have been used over the years, and how that changed things. We then talk about LTO-9 and what that brings to market. We also talk about how tape has seen a bit of a resurgence in interest in the backup market due to the advent of ransomware. Always fun to talk to someone that can talk at this depth on such things.

The term "3-2-1 rule" comes up on almost every episode, and we have the guy that coined it with us on the podcast! How exciting! Peter Krogh coined the term fifteen years ago. He is now Chief Product Officer at Tandem Vault, but this week he is talking to us. He first talks about how he coined the term “3-2-1 Rule” while writing the first edition of The DAM Book: Digital Asset Management for Photographers, now in it’s 3rd edition. He didn’t invent the idea of three copies and offsite backup, but he did distill it down to what we now refer to as the 3-2-1 rule. (Three copies on two media types, one of which is offsite.) We’ve played with it a bit over the years, but that is the core idea.) He explains how digital photographers were some of the first to need significant amounts of storage -- and to have the need to protect that storage so they don’t lose everything. Hard drives were too small to hold your whole collection, so what do you do? Like a lot of folks in this space, his love for good backups goes back to a moment when he thought he lost it all. Curtis then tells his very similar story of how his company almost lost the company’s purchasing database, which also launched his career in backups. Peter then explains the incredible importance of metadata, and the huge importance it plays in the overall value of an image.Then we get into the nitty-gritty of what the person who coined the term “3-2-1 rule” was thinking for each of the numbers. And interestingly enough, Mr. Backup had a slightly different understanding of the 2! Peter feels that the “2” refers to different media types. (This led to a very interesting discussion about how you do what he’s asking for in today’s cloud world.) One idea he talked about is that if you have two hard drives on the same network, they’re still subject to many of the same risks, which isn’t really keeping in line with the original idea of the 2. We then talk about those that believe that RAID is backup, and follow that with a discussion about how SaaS services aren’t backing up your data – unless they specifically say they do so in your contract. Then we get into a discussion of Peter’s company, Tandem Vault, and how they have designed the next generation of Digital Asset Management and delivered it as a SaaS offering.

Ever had questions about SQL Server, Azure, SQL Server ON Azure, how to backup SQL Server, or how to backup Azure? This is the episode for you. Denny Cherry, a SQL Server and Azure specialist and author of seven books, talks to us about both of these technologies. Before talking about anything important, we tackle the mystery of how you pronounce Azure. Surprise! I was pronouncing it wrong, according to Denny, who talks to Microsoft people all the time. We first talk about performance tuning, and Denny explains some things that most DBAs can do to improve performance, starting with indexes. (He also explains what an index is for those that don’t know.) We then talk about how bad query code needs to be in order to justify looking into that, and he gives us a few examples. We also (of course) talk about backing up SQL Server, starting with the political discussion of WHO should own the backup process: a backup admin or a DBA? Denny and Curtis clearly do not agree on this one, but the discussion is a good one. Grab your popcorn! One of Denny’s best quotes is that he feels one of the primary jobs of the DBA is to be able to restore the database if something happens and if you can’t do that, nothing else matters. So beautiful. Then the topic of dedupe comes up and things get heated again; our guest hates dedupe and Curtis loves it. That was another good discussion. Short version: make sure you have more than one copy of a deduped data store.We continue the discussion of different ways to backup SQL Server, and Denny definitely prefers the native backup capabilities of SQL Server, and he explains why. Curtis then makes a suggestion on a way for DBAs and backup admins to both get what they want, but it doesn’t sound like Denny is taking the bait.After a brief discussion on SQL Server vs Oracle, we move into the various ways one can use SQL Server in Azure. Denny’s gives advice as to what makes sense for most customers – and his opinion on the question of whether or not you save money in the cloud. Short answer: not usually, but you get a lot more power, flexibility and ease of use. Regarding Azure vs AWS, it appears that Azure is very equivalent to AWS in overall functionality at this point, and there appears to be a number of cost and functionality advantages to running SQL Server in the cloud. One of the biggest advantages is that you can use an on-prem license of SQL Server in the PaaS version of it in the cloud. That’s pretty cool. We also talk about how roughly half of the VMs in Azure run Linux, and why that might be the case. All-in-all it’s a really interesting podcast, even though we almost came to blows once or twice. (OK, not really.) But really good discussions about SQL Server, Azure, and backups of both.

Mark Lantz, Manager CloudFPGA and Tape Technologies for IBM, joins us on this week’s podcast to talk about how he feels that tape still has a future in data storage. We talk about past and future advancements in the substrates tape uses, as well as how tape has not approached the superparamagnetic limit, the way we have with disk. (This is the limit at which you cannot increase the storage capacity of a particular magnetic medium without creating more problems.) We have reached this limit on disk, where the magnetic grains have gotten so small, they can’t get any smaller without assistance. One such method of assistance is heat-assisted magnetic recording (HAMR), which we discuss – and how HAMR comes with its own problems. By contrast, tape hasn’t come even close to the superparamagnetic limit. In fact, tape can scale the aerial density 100X before it starts getting close. We also discuss coercivity and bit error rate (BER), which are extremely important concepts to understand. Another topic we talk about is how tape is getting better at scaling capacity faster than speed, because most people do not need faster tapes. (We talk about how and why we can’t stream the ones they have.) We finish out the podcast with an explanation of why helican scan drives (e.g. 8mm, 4mm, & AIT) all disappeared overnight. We cover a lot of territory in this episode, so buckle up!

Every wondered what you're supposed to do with all your old backups, now that you've moved on to another backup product? Simon Brown from StoneRam believes he has the answer to this problem that has plagued backup customers (and vendors) for ages. He's able to transfer backup data out of common backup formats and into your new product, or restore backups from your old product without having to maintain that infrastructure. It's a fascinating approach to this age-old problem. Check them out at https://www.stoneram.com.

After the recent stories about Veeam customers being directly targeted by the Conti ransomware group, we invited Rick Vanover and Dave Russell from Veeam to discuss the topic on the podcast. The stories in the press seemed to focus on the attack, as well as how ruthless the Conti ransomware gang tends to be. We thought we'd give Veeam a chance to explain exactly what Veeam customers can do to protect their backups from being exfiltrated and deleted. It seems that Rick, Dave, and company are doing everything they can to explain to all Veeam customers that this is something they should pay attention to. The following are two resources they said should prove useful: Ransomware in 2022: 7 Capabilities You Need for Rapid and Reliable Recoveryhttps://bit.ly/3m32gI85 Ransomware Protection Best Practiceshttps://bit.ly/3nh7aAx

This week we are joined by Greg Edwards, CEO and founder of CryptoStopper, to discuss once again the important topic of ransomware. We talk about the challenges typically experienced by ransomware victims, especially exfiltration and potential exposure of sensitive data. The only way to stop that particular attack is stop the data from being exfiltrated in the first place. CryptoStopper has a way to detect that a ransomware attack has begun, but stopping it before it does any actual damage.

This one will get you talking! Jody Bruchon, author of “ZFS won’t save you: fancy filesystem fanatics need to get a clue about bit rot (and RAID-5),”, joins us on the podcast. The blog post went viral, resulting in three times as many words in the comments as the original article had. We start with an explanation of bit rot, why it happens, and why ZFS won’t be able to fix all bit rot. (For more information on bit rot, check out Episode 111 of this podcast here: https://www.backupcentral.com/tape-drive-designer-schools-mr-backup-on-tape-restore-it-all-podcast-111/). Jody then explains how ZFS needs disk redundancy in order for its self-healing features to work, and how if you don’t have that, you’re going to need backup to repair a ZFS volume damaged by bit rot. (We also talk about how it’s possible for a bit to be flipped without being noticed – even with ZFS.) Jody’s main concern is that people talk about how ZFS can be used to repair data corruption – without explaining how you need RAID-Z (or something) to use those features. He also explains why he prefers RAID-5 or RAID-10 to RAID-6. We then discuss “shucking,” the practice of buying external drives and ripping the drive out of them – to save money. Curtis (Mr. Backup) then gets into an argument with Jody about the merits of Blu-Ray vs disk vs tape as a backup medium. Jody has some good points, but Curtis was unconvinced. If you want to read Jody Bruchon’s original article, you can do so here: https://www.jodybruchon.com/2017/03/07/zfs-wont-save-you-fancy-filesystem-fanatics-need-to-get-a-clue-about-bit-rot-and-raid-5/

Inspired by the article "How good is your backup, really?" by Sandra Vogel, we discuss how to evaluate and potentially redesign your backup system. We talk about different kinds of backup systems, and how that impacts how you evaluate them. We also talk about how important it is these days to ensure that your backups are impervious to ransomware. We also talk about the importance of including recovery testing in your evaluation, and what kinds of restores to test.Here's a link to the original article: https://www.itpro.com/server-storage/backup/357713/how-good-is-your-backup-really

We talk to Celeste Kinswood from Druva about cyber resilience, and specifically ransomware attacks that have long dwell times – which is most of them. The median dwell time (the time between infection and you finding out you have ransomware) is 23 days, and the average is around 90. It's encrypting files during that entire time, and responding to that is beyond difficult. Celeste talks about a novel approach that Druva is taking to solve this growing challenge.

We review and discuss Mark Dargin's Network World article "Credible threat: how to protect networks from ransomware." His article lays out several steps, each of which we discuss and expand upon: train your people, update your servers and apps, antivirus tools on endpointsm backup your data (of course), test your backups, and conduct vulnerability assessments (Pen tests).Read the original article here: https://www.networkworld.com/article/3218708/how-to-protect-your-network-from-ransomware-attacks.html

We all know how important it is to verify your backups. But many companies simply lack the technical ability or time to do such a thing. Chris Marshall's company, VerifiedBackups.com to the rescue. They have a service that is aimed at companies with under 100 GB of SQL data. and will do an end-to-end verification of your SQL backups, guaranteeing that they are recoverable and safely stored offsite. I'm a fan of "as a service" anything, but this really takes the cake. Such an important thing to do, and he makes it happen.

Brian Greenberg and Cameron Laghaeian argue the point that individually encrypting each entity in a database storing personal information is the only true way to stop ransomware attacks that have exfiltrated data and threaten to release it. While Mr. Backup likes the idea, there was a pretty heated discussion on this episode, because he believes that doing this will roll back all advancements in backup in the last twenty years. Thanks to an olive branch from Cameron, though, they may have come to a hybrid solution that makes both sides happy. This is a great episode with heated discussion and good news for companies trying to protect themselves from ransomware attacks that include exfiltration of data.

In this privacy-themed podcast, we start with Apple's bombshell of a new feature, where they are scanning your pictures to see if you have any child porn on your Apple devices. While stopping child porn is a laudable goal, we discuss the privacy ramifications of this feature. We also discuss how Apple's iCloud backups aren't encrypted! The next topic is Luxembourg's huge $900M GDPR fine levied against Amazon for failure to get permission to track users. Then finally we discuss Prasanna's experience with using CCPA to find out how a company got his cell phone number. Spoiler alert: it's a sad ending. We then round out the podcast by discussing whether or not we need a more Federal response to privacy in the US.

Dave Lukens from Cockroach Labs joins us to discuss CochroachDB, a highly resilient SQL ACID-compliant database. We discuss the CAP theorum, which says you can choose any two from Consistency, Availability, and Partition Tolerance – but you can't have all three. We talk about which two they chose, and why they believe they're the most resilient SQL database. We also, of course, discuss how you back this monster up! (We also discuss why it's named CockroachDB, and it's exactly what we thought!)

In this bonus episode of Restore it All, Mr. Backup talks BBQ! A few months ago, Curtis did a 15-stop BBQ tour of Texas, and made a series of youtube videos (https://www.youtube.com/wcpreston) about it. This is a bonus episode where Prasanna interviews him about his experience. If you have no interest in BBQ or beef, you can skip this one. Our regular episode will go live next Monday.

Subbiah Sundaram, VP of Products at HYCU, joins Mr. Backup and Prasanna Malaiyandi on the podcast to explain the evolution of the HYCU product line. They have expanded well beyond their original product that was aimed at Nutanix AHV. In addition to VMware, they've added support for Google Cloud Platform, Azure, Kubernetes, and Microsoft 365. They are also clearly focused on delivering their offerings as a service. We then have a very interesting discussion about the use of the multi-region object storage that Google Cloud offers.

Mr. Backup and Prasanna grill George Crump, CMO of StorOne, a software-defined storage company, about their product. We start with an overview, alon with George's claims about not needing backup... .then we challenge him on that topic. George used to be Curtis' boss, so this one's a fun one.

We are super excited to have the Emmy Winning sound mixer Larry Blake on the podcast to talk about his upcoming book "Solving The Digital Dilemma." It is a response to "The Digital Dilemma," which is a paper released in 2007 by the Academy's Sci-Tech Council that discusses long term archiving of a movie. (It describes a very NON-digital solution.) He has edited the sound on over 60 motion pictures, over 30 of which with Oscar-winning director Steven Soderbergh. This is both a very entertaining and very informative podcast about the about the art and science of archiving motion pictures.

Wolfgang Goerlich, Advisory CISO, explains the current state of information security, and why he thinks many environments are focusing on the wrong things. We speak about ransomware, extortionware, and phishing, even giving examples where we know we have personally been phished! He explains how this illustrates his point that we need more emphasis in different areas of information security.

Mr. Backup and Prasanna discuss the Kaseya attack that happened over the July 4th weekend of 2021. First, we talk about how bad actors use long weekends for attacks, then discuss various things you can/should do to ensure that your environment would not be vulnerable to such an attack. We talk about the kinds of questions that even an unskilled person can ask to help mitigate this risk, including (especially) the all-important questions about backups. We also talk about the need for establishing a "bat-phone" type connection between your environment and any providers you may be using. We have a short talk about the impact that all these attacks are having on ransomware costs, and how you can use your DR site to test patches.

Joe Jurneke has been designing the innards of magnetic devices since 1973, and now he's here to answer our questions. He started with disk, and moved over to tape over thirty years ago, and is now retired – but consults with the tape industry from time to time. We talk in detail about how tape drives work they way they do, their reliability and more. We take down a couple of myths from the guy that was there, even correcting a misunderstanding Mr. Backup has had for many years. If the phrase "magneto crystalline anisotropy" has been used on any other podcast, we'd be surprised. This is a good one you won't want to miss.

We cover my latest book, Modern Data Protection, by O'Reilly & Associates on this podcast. I give an overview of the book, which covers the "Why, What, Where, and How" of backup, archive, and disaster recovery. After giving an overview of the book, I talk about why I wrote, and what it's like to write a book for O'Reilly. Prasanna acted as a tech editor on the book, so he offered his unique perspective as well.

Prasanna and Curtis discuss whether or not can (or will) OVH properly redesign their backup infrastructure to prevent another incident like what happened in March, where many customers lost their sites forever. As we discussed in our previous podcast, OVH had a backup service already that people paid for, and it was not up to the task. OVH"s CEO made an 8-minute video where he discussed some of the things they were going to do to make things better, and we discuss what he said. We talk about their idea of a centralized region just for backups, and whether or not that's a good idea. We also talk about how big of a job they have in front of them. We applaud what we see, but have many concerns that the brief video do not address. We also talk about how this plan is supposed to take five years, and what do OVH customers do in the meantime?

We talk to Mike Johnson of ComplyTrust, who says they "remember those you are supposed to forget." We talk about the data management challenges created by data subject access requests (DSARs), right-of-erasure (ROE, AKA right to be forgotten, or RTBF) requests, and the fact that we have many parts of the datacenter that are much better at remembering than forgetting. Backups are a particular challenge, but Mike brings up other challenges, such as mergers and acquisitions, and salespeople importing old data. ComplyTrust SaaS offering has a unique solution to this problem by remembering (on your behalf) those you are supposed to forget, and continually checking to see that they stay forgotten.

This week we discuss further lessons from the OVH fire, which starts with an admission by the CEO that some customers who paid for the backup service lost their backups in the fire. It then morphs into a discussion about designing resilient systems, starting with the concept of designing for failure. You have to protect against both physical and logical damage to your apps and data. We talk about using both cloud-native apps that have resiliency built in, vs having to add resiliency to your own app. Most importantly, know how your app/data is protected, and don't tolerate wishy-washy terminology in your service agreements. Above all, test, test, test!

This week we discuss a topic brought up by the OVH fire. It appears some people actually had a contractual backup service that wasn't doing it's job. How do you verify that a service you're paying for is real, and is doing what it claims to be doing? Especially how do you make sure they are storing data offsite? We've got some ideas.

Datacenter manager Dan Frith (@penguinpunk) joins us on the podcast for our first discussion of the #OVHFire. A massive fire destroyed a datacenter of a large cloud provider in Europe, and millions of websites disappeared. We talk about the lessons we can learn from this event.Dan talks about how outsourcing the servers doesn't outsource the responsibility for data protection. I make the point that this fire shows what happens when you completely rely on a single entity for both production and data protection. We end up talking about the 3-2-1 rule and how it applies in this scenario. I also give a discount code during the podcast for my new O'Reilly book Modern Data Protection, which is now available for purchase. If you use the URL below and the code I give on the podcast, you can get 35% of the retail price. https://shop.aer.io/oreilly/p/Modern_Data_Protection_Ensuring_Recoverability_of_/9781492094050-9149

It only took us 100 episodes, but we finally got Dave Russell, VP of Enterprise Strategy at Veeam, as our guest on the podcast. Dave and Curtis go way back, and this was a great discussion. We cover the proper use of tape, and what it was like for Dave when he went to Veeam. Another big discussion point was Dave clearing up misconceptions (some of which may have come from this podcast) about what Insight Partners acquiring Veeam really meant. We then get into a great discussion about how Veeam works, ending that discussion with Dave explaining what Veeam is doing to address concerns about Windows and ransomware.

Our anonymous guest this week is from a Fortune 100 company who is considering swapping out their backup product. Our guest has been at the company for over 20 years, and remembers swapping out NetWorker for NetBackup many years ago. Now he is considering swapping out NetBackup to address his challenges with that product. We discuss a number of topics, including the age-old argument of who should be in charge of database backups, as well as the challenges of moving to a modern backup product when you are still using operating systems not usually supporting by such products. Our guest's final thoughts center on the importance of a good relationship with the vendor in question.