Software Engineering Daily

React started as just a view layer–it was the V in MVC. React has moved down the stack, with Flux, Redux, GraphQL, and Relay providing opinions for how React applications should structure their data flow. Jared Forsyth works at Khan Academy, which uses React on the front end. At Khan Academy, Jared has experimented with many different ways of handling data flow for a React application, and in today’s episode we not only discuss the conventional tools for React applications, but also ClojureScript, Reframe, and Om/next, which are solutions for React data handling, that are outside of the world of raw JavaScript.

Applications on your phone can talk to each other, but the way that apps interact is still far from the way that web pages interact. Mike Wakerly is the CTO of Button, a service that connects mobile apps to each other. By embedding a component from Button into your app, you can easily request the functionality of an app while you are in a different app. In this episode, Mike and I discuss the future of mobile, and the problems of app stores today. We also explore how mobile application context switching has improved, and how that creates business opportunities that are much more interesting than advertising.

The Changelog is one of the most popular Software Engineering Podcasts in existence. Open source software moves fast, and The Changelog helps developers keep up with that rapid pace by publishing consistent podcasts and newsletters. When I started Software Engineering Daily, Changelog was one of the shows that I looked at for inspiration on how to succeed as a software podcaster. Adam Stacoviak and Jerod Santo are the hosts of The Changelog, and they join me today to talk about how open source has changed since they started the podcast in 2009, and how it will change in the near future.

Virtual reality is a new platform for software engineers to work with. Best practices for VR development have not become widespread throughout the developer community. If you are developing VR software, you need to know how to avoid making the user sick, and this requires an understanding of the hardware limitations. Andreia Gaita is a software engineer at github. On today’s episode of Software Engineering Daily, Andreia takes us through the developer experience of VR, and gives some analysis and predictions about consumer VR. For anyone who is looking for an introduction to virtual reality development–this episode is for you.

Patients with Type-1 diabetes need to frequently pump insulin into their bodies. In order to know when to pump insulin, these patients have a continuous glucose monitor alarm which detects incorrect levels of blood glucose. When the alarm goes off, the diabetes patient administers insulin manually through the pump. In an ideal world, the alarm would communicate with the insulin pump, creating a closed loop. But the world of outdated medical devices is not ideal. Dana Lewis is a developer of an open-source, closed-loop artificial pancreas called the Do-It-Yourself Pancreas system. Dana joins the show to discuss how she hacked together her artificial pancreas using a Raspberry pi plugged into these medical devices. This is a fascinating story of reverse engineering, internet of things, and the hacker mentality. Sponsors

At Netflix, building a user interface requires data about movies, actors, users, accounts, and much more. If each piece of data is coming from a different source, a programmer has to call each of the services that has access to that data. Falcor is a piece of technology designed at Netflix to solve this problem. Falcor creates a unified JSON model of all of the data you could want–and Netflix engineer Brian Holt joins us today to discuss how Falcor works. We also discuss React, ES6, and JavaScript at Netflix, and Brian explains why Falcor is extremely similar to GraphQL, and why that is OK.

Kubernetes is an open-source distributed operating system that allows normal engineers to manage their resources at the scale that Google does. With Docker providing the APIs to make containers easy to work with, Kubernetes allows processes to be easily distributed and replicated across a data center. Kubernetes was created at Google, and today’s guest Kelsey Hightower is a staff developer advocate working at Google. He joins me in a conversation about how Kubernetes works, and why it is important to an average developer. We touch on how software engineering is going to look in the next 5-10 years, and how a lot of the work around distributed systems will become much easier over time.

Exception monitoring services and log management services are two sides of a gradient. Exception monitoring services capture and aggregate the problems that occur on your application. Log management services aggregate all of your logs, so that you can decide for yourself what constitutes a problem. Brian Rue from Rollbar joins the show today to talk about Rollbar’s exception monitoring architecture, and the competitive landscape of these technology products. Every software engineer wants track the problems with an application, but some developers need more information than others–and that ends up changing how these error aggregation services are architected. This is an interesting conversation on the business of SaaS products for developers, and the architecture of a distributed system designed to monitor and aggregate errors.

As software engineering gets more popular, the resources that we use to read and understand software are growing and improving. Scale Your Code is an organization that seeks to improve accessibility to the world’s programming knowledge, and today’s guest Christophe Limpalair joins me to discuss Scale Your Code. Christophe is a podcaster and blogger who runs Scale Your Code.

Docker is an open platform for distributed applications. Docker allows for standardized container runtimes and will be an important piece of infrastructure for many years to come. But if you are new to Docker, the technology can be confusing and intimidating. What is a container? Why do I want to put my application in a container? Today’s guest Prakhar Srivastav wrote a long tutorial explaining the fundamentals of Docker. Prakhar explains why Docker is useful, how Docker works, and how you can get started.

Apache Kafka has become the most popular open-source solution for persistent replicated messaging in the Hadoop ecosystem. But some software engineers who are working with “big data” don’t want to deal with the configuration and set up of Kafka. One way to side step this problem is to go with a managed solution, like Microsoft Azure Event Hubs. Dan Rosanova is today’s guest. He joins us to discuss persistent replicated messaging, and the features that Azure Event Hubs provide. This is an interesting discussion about distributed pub sub messaging, and we also get into a great conversation about platform as a service versus open source software, and the future of cloud software.

Kubernetes has been a focus of several previous shows on Software Engineering Daily. The architecture and management of our distributed systems have changed, and widespread adoption of containers and container management systems is just beginning. Joseph Jacks from Kismatic joins the show today to give us a perspective on the past, present, and future of container management. He also talks about his company Kismatic, which provides enterprise support for Docker and Kubernetes.

“Graphing when your Facebook Friends are awake” was trending at the top of Hacker News when Alex Hogue first posted it. Alex wrote a blog post about this piece of software he built that interacts with public Facebook APIs to detect when any of his friends are awake or asleep. As I read that post, I wanted to talk to Alex, because it was such a creative and technical piece of software–but it is surprising that someone would take the time to randomly inspect the elements of the Facebook page, reverse engineer the way Facebook sends data about statuses, mine data from that reverse engineering, and create a bunch of graphs about that data. Today, Alex joins me to explain how and why he built a tool to graph when people are awake or asleep using Facebook data.

Applications crash, and engineers need to be able to quickly find the root cause of a crash. Apps have become distributed, and debugging workflows have changed. Developers need better tools to identify and troubleshoot problems with their apps. Ben Curtis joins us today to discuss Honeybadger, an exception and monitoring tool for web applications. We discuss how software tools can be used to detect problems faster and improve the debugging workflow.

Kubernetes is an open source system for automating deployment, operations, and scaling of containerized applications. Google developed Kubernetes after fifteen years of running containers in production. Brendan Burns is a founder of the Kubernetes project, and he joins us to talk about the lessons learned as Google has built containerized applications to distribute across its massive infrastructure. We talk about Docker, Borg, Kubernetes, and other distributed systems technologies. Applications crash, and engineers need to be able to quickly find the root cause of a crash. Apps have become distributed, and debugging workflows have changed. Developers need better tools to identify and troubleshoot problems with their apps.

Elixir is a programming language built on top of the Erlang virtual machine. Elixir allows metaprogramming, polymorphism, and a web framework called Phoenix that has drawn positive comparisons to Ruby on Rails. Jose Valim is today’s guest. He built Elixir to augment a language that he loved–Erlang. On Software Engineering Daily, we interviewed Joe Armstrong, the creator of Erlang, which was a very popular show–I encourage any listeners who are fans of Erlang to check it out. Erlang was built with concurrency in mind, and it has been rising in popularity as more of our applications are written to be distributed. In today’s episode we will discuss what Jose is building on top of Erlang.

Engineers who want to add search to their application usually deploy Elasticsearch, or write their own search engine that uses TF-IDF. These solutions work well for large documents, but are less effective for large volumes of small records–which is how many modern web and mobile applications are structured. In today’s show, Julien Lemoine discusses how his company Algolia thinks about search. Algolia is a search as a service company that gives developers an easier way to search on their websites and applications.

CDN stands for content delivery network. A content delivery network is a system of distributed servers that delivers web pages and other web content. Without CDNs, the internet would be much slower, because CDNs function as a caching layer for most web resources. Carl Gustas is an engineer at CacheFly, a popular content delivery network. He joins us today to discuss how CDNs work, and the different methods an engineer can use to take advantage of caching in a CDN.

Google has published papers on distributed systems such as BigTable, Chubby, and the Google File System. During this episode, we focus on a product that takes inspiration from Google’s Spanner project, a database that is built on a distributed monolithic sorted map. CockroachDB is a scalable, survivable, consistent SQL database. Today’s guest Ben Darnell is the CTO of Cockroach Labs, and he joins us today to discuss SQL and NoSQL distributed databases. Ben explores the tradeoffs between these database types, and explains how CockroachDB provides many of the best features of both a SQL and a NoSQL distributed database.

Wiki technology was invented decades ago to improve how software developers communicate. Today, Wikipedia has taken the ideas of the wiki to a new level, creating a free knowledge graph for the world to learn from. Ward Cunningham developed WikiWikiWeb in 1994. He joins us today to discuss the first wiki, and how wikis have changed the way information propagates. Ward is not just an expert on how to write software–he is an expert on the software development experience, and we discuss how to build a community where software developers can overcome the fear of building something that has not been built before.

Software applications are constantly generating logs. These logs are necessary to understand how an application is functioning, and logs are key to debugging. As applications have gotten more complex, logging infrastructure has become complex as well. Storing and managing all of our log data is such a big task that several companies have been started to tackle this problem. Today’s guest is Christian Beedgen, CTO at Sumo Logic. Sumo Logic is a cloud-based log management company. We discuss the elastic log-processing platform Sumo Logic has built to help software engineers with log management. It’s a great conversation about distributed systems, machine learning, and debugging applications.

Frontend web development was simpler in the past–CSS, HTML, and JavaScript were all you needed to know. Today, we have mobile web, React, Angular, PHP, JQuery, and so much more. Marc Grabanski focuses on what he believes is timeless–pure JavaScript. Marc is the founder of Frontend Masters, a training site for developers who want to build quality web interfaces. He joins us today to discuss the shifting nature of front end development.

Nadia Eghbal joins us as the host of today’s show to discuss the past, present, and future of open-source software. This monologue was adapted from Nadia’s article, We’re in a brave, new post open source world. In this episode, Nadia describes the origins of open-source in the Free software movement, its rise to popularity, and today’s golden era where mainstream and popular technologies like React, Spark and Docker are all open source projects. Yet there are several serious issues that threaten the foundation upon which the current open-source renaissance is happening. Nadia discusses what these are, and how to resolve these issues so that we can continue to have a vibrant ecosystem for developers. Thank you to Nadia Eghbal for coming on the show and for standing up for open source software.

When you spend money online, you expect a receipt to come in your email. When you register for a new web site, you need to verify your sign up in your email. These types of emails are called “transactional email” and sending these types of email at scale is a complex engineering task. J.R. Jasperson is the chief architect at SendGrid, a transactional email platform. On this episode, we discuss how email works–from the basics to the massive scale that SendGrid operates on. We also talk about email spam and fraud in detail.

Git is the most popular version control system. If you have been programming for less than a decade, it’s likely that you haven’t used any other method of version control. The git workflow of a software team defines how that team collaborates, builds, and ships software. Tim Pettersen is a developer advocate at Atlassian, where he has been building software around git for many years. He joins us on today’s episode to talk about strategies for git–including branching, merging, continuous integration, and software as a service.

Application delivery has become more complex as software architectures have moved into the cloud. Data center infrastructure has turned into code to be manipulated, and software engineering teams are adjusting their strategies. HashiCorp is a company that builds open-source software for application development and deployment. Mitchell Hashimoto is the founder of HashiCorp, and he joins us to discuss a modern approach to application delivery, and the tools HashiCorp is developing.

Code analysis tools can help a developer understand code. One tool for code analysis is bitHound, which provides code and dependency analysis for NodeJS applications. On today’s episode, we discuss how to use a code analysis tool–and we also talk about how to build one, by breaking down the distributed architecture of bitHound’s backend.

Uber is a transportation company with a high volume of temporal spacial data, constantly being collected from the devices of its users. At any given time, the engineers and data scientists at Uber need to be able to query the system, and understand what is going on with drivers and riders. The unique real-time engineering requirements of Uber lead to an interesting architecture. Danny Yuan joins us today to discuss Uber’s data engineering stack and how the company makes use of its streaming data.

The last editorial we did was 10 Philosophies for Engineers. Listeners enjoyed that episode, so we decided to do another. 10 Philosophies was a collection of beliefs I have about the software engineering industry, and how we can find fulfillment in our work as engineers. The philosophies that I discussed were rooted in my experiences working as a computer science student and as an engineer at several different companies. Before writing any lines of code, I played poker competitively for five years, from the age of 15-20. This was a formative experience. Playing poker changed my perspective on money, risk, and statistics. It has been 7 years since I stopped playing poker full time, but the lessons of the game are still with me. In this editorial, I will discuss some beliefs I have that relate to both poker and software engineering. As with the 10 Philosophies episode, these beliefs my own opinions. If you disagree with them, I would love to know why. At Software Engineering Daily we actually want to know any thoughts you have on our content. We make content for our listeners and our readers. Please tell us how we can improve by emailing us, or by filling out the listener survey. As a poker player becomes a software engineer, certain trends about human-computer interaction become apparent: This post explores each of these trends, explaining why these trends are important to poker players, software engineers, and everyone else. Automated Games In 2008, poker was the perfect sport for human-computer symbiosis. What Tyler Cowen said about freestyle chess also applied to poker: In poker, a human with a statistical “heads-up” display can make decisions that are more mathematically justified than a human without such a tool. Heads-up displays create the poker version of “human-plus-machine teams”. One thesis of Average is Over is that a human will only be employable in the future by finding a career where human reasoning provides defensible value to the problem solving process of a computer. If the human’s responsibilities are not defensible, the human will beobviated. In a subsequent blog post, Cowen addresses the “flip” that can occur when a computational problem no longer requires human assistance: Poker players have been increasingly defeated by machines for the past 10 years. It is no surprise that Google’s AlphaGo has defeated human champion Lee Sedol. If Google decided to beat humans at poker, it would be a trivial exercise for the researchers. Poker seems different than Go or Chess, since there is nondeterminism. You start with two cards, but you don’t know how the board will develop. It would seem that fate is in control, unlike Go and Chess, which have no random elements. With just 4 suits and 13 ranks, a poker game has a trivial branching factor. The nondeterminism is so minimal for a computer to plan around that it is effectively deterministic. Imagine if AlphaGo had to learn to play a version of Go with the following rule: at the beginning of each turn, flip a coin. If you lose the flip, you don’t get to move. Adjusting to this rule would be trivial. That is the magnitude of nondeterminism within poker. Poker, Chess, and Go have small decision spaces. The rules never change, the game pieces never change, there is minimal nondeterminism. A computer can assess a hand of poker as it would a hidden Markov model, but it will take work on par with the AlphaGo team for a computer to be trained to build a model accurately. The job of a professional poker player has been a bad long-term choice for a human to pursue for more than a decade, because it is vulnerable to automation. Games like Go, Poker, and Chess can be automated with machine learning techniques we understand today. The rules, game piece schema, and objectives are easy to define, so these games are ripe for supervised learningand reinforcement learning. Yann LeCun protested against the hype around the AlphaGo victory: Poker is vulnerable t...

Applications can and will crash — it is increasingly important for developers to have visibility into the reasons how and why the crash occurred. James Smith is the guest on the show today, and joins Jeff to discuss why modern applications crash, and how developer products are improving to tighten the gap between QA/support and dev. James is the founder of Bugsnag, an automated crash monitoring system.

Ten years ago, building a highly scalable image delivery service would require millions of dollars in upfront costs, and hours of work configuring hardware server infrastructure. Today, it is possible to bootstrap this type of service, with minimal investment. Today’s episode is about building a content delivery network for images and video. Today’s guest is Itai Lahan, CEO of Cloudinary. We discuss Cloudinary’s early product infrastructure, and how they have evolved as a company since then. We also talk in detail about the venture capital landscape of Silicon Valley today, and how to strategize about raising money. Full disclosure, Cloudinary is a sponsor of Software Engineering Daily.

Memory is king. The cost of memory and disk capacity are both decreasing every year–but only the throughput of memory is increasing exponentially. This trend is driving opportunity in the space of big data processing. Alluxio is an open source, memory-centric, distributed, and reliable storage system enabling data sharing across clusters at memory speed. Alluxio was formerly known as Tachyon. Haoyuan Li is the creator of Alluxio. Haoyuan was a member of the Berkeley AMPLab, which is the same research facility from which Apache Mesos and Apache Spark were born. In this episode, we discuss Alluxio, Spark, Hadoop, and the evolution of the data center software architecture.

Cloud service providers like Amazon, Google, and Microsoft provide both infrastructure as a service and platform as a service. Infrastructure as a service gives developers access to virtual machines, servers, and network infrastructure. Platform as a service is the software that runs on top of that infrastructure as a service–this includes things like Amazon DynamoDB, Microsoft Azure Machine Learning, and Google App Engine. OpenStack is an open-source cloud operating system. Today’s guest is John Purrier, a founder of OpenStack and the CTO of Automic Software.

In today’s episode, Ben Halpern interviews Juan Buritica, VP of Engineering at Ride. They discuss Ride’s migration from a monolith to microservices, the challenges of running a distributed team and preventing developer burnout.

GreyOrange Robotics builds robots for warehouse automation of logistics and ecommerce companies for quicker deliveries. Today’s episode features Akash Gupta, the CTO of GreyOrange.

DevOps has been the subject of many episodes of Software Engineering Daily. And yet–the question we continue to ask ourselves is “what is DevOps?” In order to understand the present, we must look into the past. On today’s show, we break down the history of dev ops with Demir Selmanovic, the lead technical editor at TopTal. From waterfall software development, to the agile manifesto. From lean manufacturing to the productive pinnacle of DevOps nirvana. Demir also wrote a blog post called What the Hell is DevOps, which he posted on the TopTal blog. Full disclosure: TopTal is a sponsor of our show, but this episode doesn’t discuss TopTal very much so hopefully that doesn’t impact the trust of our listeners.

Today’s guest is Calvin French-Owen, the CTO of Segment, a tool that companies use to aggregate their analytics into once place. As Segment has scaled, the company has had to restructure its etire technical architecture. Microservices, containers, Amazon Web Services, and dev ops are a few of the topics that Calvin and I explore in our conversation, so this is a great episode for anyone who is trying to understand the relationships between those different subjects. Segment’s product unifies analytics from different services and puts them into one centralized place. Full disclosure: Segment is a sponsor of Software Engineering Daily. For most of this episode, we don’t even talk about the product, we talk about the back end engineering behind the product.

CodeNewbie is a community of programmers and people learning to code. There are so many people learning about software today, and CodeNewbie gives them a place to hang out, socialize, and become comfortable with the world of software. CodeNewbie has an excellent podcast, and if you like Software Engineering Daily you should check it out. Saron Yitbarek is the guest today, she is the creator of CodeNewbie. In this episode of Software Engineering Daily, Saron sits down to talk about learning to code after she had already had several careers, spanning biology, public radio, and marketing.

Mic.com is a media company focused on news for millennials. Anthony Sessa is the VP of product at Mic.com, and he joins us to talk about the engineering of a modern news organization tailored to young people. We discuss data engineering, frontend technologies, and how to migrate away from WordPress. We also explore how to build a successful media company today, and how millennials want to consume news and software.

Software developers have been socializing on chat rooms for decades. In the nineties, we began using IRC and AOL instant messenger. In the early 2000s, we turned to Google Hangouts and Yammer. Today, we are using modern apps like Slack and Hipchat. On today’s show, we take a deep dive into Gitter, the chat client specifically designed for developers. Our guests are Mike Bartlett and Andrew Newdigate, the creators of Gitter. Gitter is a highly scalable, real time social application for developers to talk about writing their software. This is a great episode that spans topics like front end development, back end distributed systems, how to compete with Slack, and how to scale a chat room to tens of thousands of active users.

Military veterans have the right set of skills to become programmers. Technical expertise, emotional resilience, psychological persistence, and teamwork–these are the qualities of the US army and they are the qualities of the best programmers. More and more veterans who leave the army are becoming coders, and the mission of Operation Code is to help veterans make that transition. Dave Molina is the founder of Operation Code. His organization helps army veterans become coders. Dave came on the show to discuss coding boot camps, military, software education, and what it’s like to become a civilian technologist after spending years in the army.

Building a software business today requires lots of decision making. Building software isn’t just about choosing a programming language, or a framework, or a database. Developers have to choose the right cloud service provider, the best issue tracking service, the best hosted code repository, the clearest data visualization tools. We need a platform to compare the options between different products, all in one place. In today’s episode, we talk to Ayan Barua from Siftery, a platform for software products and the companies who use them. Siftery’s mission is to help technologists find the best products for their job, whether they are a sysadmin or an advertising account manager. My discussion with Ayan covers the question of build versus buy, how to build a recommender system, and how software will be built in the future; this is also a great episode for anyone who is thinking of how to build a two-sided marketplace business.

Software engineering is a deterministic field. We write lines of code, and feed data into that code, expecting to get a certain answer. Computing is deterministic because humans developed it–we understand computers from top to bottom. The same cannot be said about biology. Matt Might is an associate professor with a PhD in computer science. When his son was diagnosed with an extremely rare illness, he was confronted with the uncertainties of human biology. In this episode, we discuss Matt’s quest to solve the puzzle of his son’s disease–computer science meets genetics, on this episode of Software Engineering Daily.

In many ways the engineering job market is broken. Engineers cannot find the right jobs where they are valued properly and enjoy their work. Employers have difficulty filling their positions and finding the right people that fit their cultures. Traditional matchmakers like Monster and generic recruiting agencies don’t cut it — you know this by the generic LinkedIn messages you get looking for a Java expert… when you have JavaScript as a skill on your profile. Hired.com is trying to fix this broken process by using a high-touch approach to recruiting. Hiten Parmar, an engineer and product strategist at Hired joins us today to discuss the issues at hand, as well as how Hired is approaching the solution. We chat about the challenges facing engineers, which skills are in demand, and the inner workings of Hired. Full disclosure: Hired is a sponsor of the show, but we would still have done this interview because we genuinely want to discuss the job market, and figure out how companies are addressing this issue.

Stack Overflow is used by developers to find out how to build software. Stack Overflow is both a tool and a community, and today’s guest Jeff Atwood has made a career out of building tools and communities. As the co-founder of Stack Exchange and Discourse.org, Jeff has been solving the problem of civilized online communication for seven years. In today’s episode of Software Engineering Daily, Jeff Atwood talks about building online communities from the perspective of an engineer as well as a sociologist.

On Software Engineering Daily, we often discuss big data in terms of data engineering and data science. Data engineering is the infrastructure and pipelines that handle massive amounts of data and puts that data in a data lake. With a data infrastructure in place, a data scientist can study the data and take action on it. A data scientist can also create visualizations–which is the subject of today’s episode. Aurelia Moser has a specialty in mapping and data visualization. We discussed how to create effective visualizations of our data, and the tools that can be used to collect and present our data. Whether you work at the New York Times or a small tech company, data visualizations are important, because visualizations can be used to communicate important trends across an organization. Aurelia will also be speaking at the upcoming Strata + Hadoop World Conference in San Jose. We’re partnering with O’Reilly to support this conference – if you want to go to Strata, you can save 20% off a ticket with our code PCSED.

Big data is yesterday–fast data is now. FiloDB is a reactive columnar OLAP database that is built on Cassandra and Spark. Today’s guest is Evan Chan, creator of FiloDB. In our discussion today, we talk about the use cases of an OLAP data store. Evan explains how to tackle the problem of video analytics–if you have ever found yourself asking how a company like YouTube or Netflix or Ooyala performs analytics on millions of users watching millions of videos, this episode is for you. By combining the database features of Cassandra with the data processing power of Spark, Evan created FiloDB to help solve this type of analytics problem. Evan will also be presenting at Strata + Hadoop World in San Jose. We’re partnering with O’Reilly to support this conference – if you want to go to Strata, you can save 20% off a ticket with our code PCSED.

Apache Cassandra can serve as both the real-time data store for online transactional applications, as well as the read-intensive database for data warehousing operations. In order to combine these two use cases into a single database, Apache Cassandra required lots of innovation. In today’s episode of Software Engineering Daily, we discuss the internals of Cassandra. Tim Berglund takes us through how Cassandra performs reads and writes, how Cassandra offers tunable consistency, and what inspirations Cassandra took from the Amazon Dynamo paper as well as the Google BigTable paper. Tim will also be presenting at Strata + Hadoop World in San Jose. We’re partnering with O’Reilly to support this conference – if you want to go to Strata, you can save 20% off a ticket with our code PCSED.

Hadoop was created in 2003. In the early years, Hadoop provided large scale data processing with MapReduce, and distributed fault-tolerant storage with the Hadoop Distributed File System. Over the last decade, Hadoop has evolved rapidly, with the support of a big open-source community. Today’s guest is Mike Cafarella, co-creator of Hadoop. Mike takes us on a journey from past to present. Hadoop was based on the Google File System and MapReduce papers, and so Mike and I talk about what it was like to work on a distributed file system in 2004, and the challenges of implementing real software systems based on white papers. We also discuss YARN, and the wave of innovation that YARN enabled within the Hadoop ecosystem. Mike will also be presenting at Strata + Hadoop World in San Jose. We’re partnering with O’Reilly to support this conference – if you want to go to Strata, you can save 20% off a ticket with our code PCSED.

When a company gets big enough, there is so much data to be processed that an entire data engineering team becomes responsible for managing this data and making it available to other teams. Airbnb is one such company. Max Beauchemin works on the data engineering team at Airbnb, where he creates infrastructure and tooling for managing data. In this episode of Software Engineering Daily, we talk about AIrflow, a workflow scheduler that assists in job processing. If you don’t know what a workflow is, or a job, we will explain that in this episode. Max and I also talk about Panoramix, a data slicing and visualization tool that helps data scientists and business analysts understand large volumes of data. Max will also be presenting at Strata + Hadoop World in San Jose. We’re partnering with O’Reilly to support this conference – if you want to go to Strata, you can save 20% off a ticket with our code PCSED.