Software Engineering Daily

Ten years ago, databases were much simpler. Most companies would only have one or two types of databases in production. Today, the age of one-size-fits-all is over. Companies have multiple databases to deal with different types of use cases, and databases have become distributed to multiple nodes in order to be scalable. Ozgun Erdogan of Citus Data joins the show to give us a modern look at databases. Ozgun suggests that PostgreSQL alone can perform most of the work that we are trying to get from our variety of databases. We discuss how Citus Data scales Postgres, and Ozgun contrasts an all-Postgres architecture with other types of databases such as the “NewSQL” class of databases, like MemSQL and VoltDB.

Kubernetes is the container management platform that came out of Google’s experiences managing data centers. Kubernetes abstracts away many of the frustrations of distributed systems management. OpenShift is a platform built on top of Kubernetes to provide an additional layer of usability. Clayton Coleman is the lead engineer of OpenShift, and in our conversation today, we start with the basics of Kubernetes, then talk about OpenShift–Clayton explains why we need another abstraction on top of Kubernetes. Near the end of the conversation, we discuss the current state of cloud products–which can be confusing. Mesosphere, Swarm, Kubernetes, OpenStack, ECS–why do we need all of these different products? Clayton gives his perspective, and explains why it is not going to get any less confusing any time soon.

Coding boot camps are a subject of controversy. Critics of boot camps defend the conventional university system, and argue that boot camp graduates do not have enough experience to write quality software. But the reality is that some boot camp graduates have found success from this new educational path. After graduating high school, Kenny Tran attended one coding boot camp, then spent some time living at home absorbed in his personal projects. Eventually, he went to second coding boot camp–his form of graduate school. During his second boot camp, Kenny worked on PurifyCSS, a module that can reduce the size of front-end projects by 60%. Today, Kenny works at the groundbreaking company Mesosphere–completing a career arc that proves coding boot camps are enough of an education to rival traditional university learning.

Infrastructure-as-code is a trend that has been popularized over the past decade, as cloud computing and distributed systems have become a part of every technology company. Tools like Salt, Puppet, Chef, and Ansible allow us to manage servers and processes from the command line. David Boucha works at SaltStack, the company that makes Salt. Salt is a platform that provides configuration management and remote execution. In our conversation today, we discuss how the distributed systems architecture of Salt works, and how it can be used in practice. We also discuss infrastructure-as-code in the context of modern infrastructure tools like containers and Kubernetes.

Solar energy is a growing market. Improvements in hardware have led to some people predicting that solar energy will be powering the world within the next few decades. Undoubtedly, a large percentage of our current energy infrastructure will be replaced by solar in the near future. Replacing our old, inefficient power grid requires massive investment. On Software Engineering Daily, the resources we usually talk about are memory, bandwidth, storage, and other computer science concepts. David Reiss joins us today to talk about energy, which is a resource so fundamental that we usually don’t even consider it. David is the CTO of Wunder Capital, a fintech company that facilitates investments in solar power. Much of this conversation centers on the economics of the solar energy market, but we also discuss software topics–machine learning and why Wunder Capital’s software architecture is a monolith by design today. Sponsors

Minecraft is a sandbox video game in which players build constructions out of 3-D cubes in a procedurally generated world. Minecraft is the best-selling PC game of all time. But Minecraft is not just a game. It is a platform for creativity, used by players within the game as well as programmers outside of it. Gabriel Simmer is a 16-year-old programmer who build NodeMC, a tool that wraps around the Minecraft server process. NodeMC can be used to build dashboards for Minecraft, and spin up additional Minecraft servers. In our conversation, Gabriel explains why people are so excited about Minecraft, how people are hacking Minecraft, and what the future of Minecraft is now that Microsoft has acquired it.

Rust is a systems programming language being developed at Mozilla. Rust has features of a high-level functional language like Scala and a low-level, performance-driven language like C++. Steve Klabnik is a developer program member with Mozilla. In this episode, he discusses how Rust looks at memory management, type safety, mutability, and concurrency. We also dive into a discussion of the low level virtual machine, also known as the LLVM, which the language Swift is built on.

The process of building a software project requires us to make so many architectural decisions. Which programming languages should be used? Which cloud service provider? Which database? A newer type of building block is the analytics platform. Companies need to track events, aggregate metrics, and change the user’s experience based on aggregated data. Dan Kador is a co-founder of Keen IO, and he joins us to discuss the rise of analytics platforms. Keen’s architecture is based on Storm, Cassandra, and Kafka, and Keen uses these different building blocks to create a scalable, reliable analytics backend. On today’s episode, we explore the usage of analytics, the architecture of Keen’s backend system, and the business model of an analytics as a service company.

Erlang is a programming language with primitives that help software engineers build distributed systems. When a process is malfunctioning in Erlang, the philosophy of the language is to let the process crash–and in a distributed system where unexpected faults happen on a regular basis, this philosophy of “let it crash” simplifies how we reason about an Erlang system. Other distributed systems advantages of Erlang include the garbage collection strategy. Each process in Erlang has its own garbage collector which means makes it easier to construct systems without a stop-the-world garbage collection. Francesco Cesarini is the founder of Erlang Solutions, and he joins the show today to discuss the book that he wrote with Steve Vinoski, called Designing for Scalability with Erlang/OTP.

Google has built a microservices architecture on top of the internal container management system called Borg. These services communicate over an internal protocol known as Stubby. Borg and Stubby are tightly coupled to Google’s infrastructure–it would not make sense for Google to open source them–but Google has worked with the open source community to develop open source projects with the core functionality of Borg and Stubby. These projects are known as Kubernetes and gRPC. Sandeep Dinesh works on the Google Cloud Platform team as a developer advocate. Our conversation explores how a client application request from an app like Gmail would communicate with Google’s servers, where the request is handled by a network of microservices. We also talk about where Google Cloud Platform is evolving, and how it offers a competitive, differentiated model from Amazon Web Services.

At Netflix, 500 billion events and 1.3 petabytes of data are ingested by the system per day. This includes video viewing activities, error logs, and performance events. On today’s episode, we dive deep into the data pipeline of Netflix, and how it evolved from their 1.0 version to the modern 2.0 version. Before listening to this episode, check out the blog post that inspired it.

Dropbox has been storing files on Amazon Web Services for 8 years, and Dropbox’s core business is storing files. For the past three years, Dropbox has been working on a project to migrate its file storage from Amazon Web Services to its own custom built infrastructure. Magic Pocket is the name of Dropbox’s new infrastructure layer, and it gives Dropbox more control and improved economics. James Cowling leads the storage team at Dropbox. On today’s episode, James takes us into the architecture of Dropbox and explains how the team moved all of the user file storage from Amazon S3 to Dropbox’s Magic Pocket infrastructure. Dropbox’s architecture is built with a focus on simplicity–and there are numerous challenges to maintaining that simplicity in the face of an extremely complex problem like this.

Almost a year ago, Software Engineering Daily aired a week of shows about decentralized technologies like Bitcoin, Ethereum, and IPFS. Bitcoin has established itself as a stable network, but it can only be used for financial transactions. Ethereum is a global computer built on a blockchain, but it does not have the adoption of Bitcoin. IPFS is a distributed data store with an incentivization layer where you can pay strangers to store your content. Today’s guest is Karl Floersh, an engineer working on decentralized technology. Karl has written several posts about how to build decentralized applications, as well as how the future might look once these decentralized technologies have gotten traction.

Distributed systems products are often marketed with terms like “real-time data” and “hassle-free scaling”, but what do those terms actually mean? Is data in a distributed system ever reliably “real time”? Do we ever have strong enough plans about our scalability strategy to say that scaling will be “hassle free”? Camille Fournier joins us today to discuss distributed systems in practice. Like everything in else in computer science, distributed systems are all about tradeoffs–and picking the right sets of tradeoffs in our distributed system will affect the entire organization that is building that system. We also discuss the Cloud Native Computing Foundation, which is similar to the Apache Foundation, but specifically for cloud technologies. The CNCF is likely to have strong impact on the way we build software for a long time to come.

Distributed databases are difficult to operate, and Crate.io wants to change that. Crate is a fast, scalable, easy-to-use SQL database that is built to run in containerized environments. An average software company runs several databases–MySQL for relational store, MongoDB for a document database, HDFS for blob storage and data warehouse, elastic search for search. On today’s show, Jodok Batlogg from Crate discuss the distributed architecture of Crate, and breaks down the use cases, from Microservices to data warehousing.

Microsoft has built a suite of technologies on top of its Azure infrastructure as a service. Today, we discuss Azure Stream Analytics, a real-time event processing engine developed at Microsoft. Azure Streaming allows for constant querying of incoming data streams, and my guest Santosh Balasubramanian discusses Azure and the movement from batch processing to stream processing.

Apache Spark is a framework for fast, distributed, in-memory analysis. Apache Cassandra is a distributed database management system that provides high availability and fast throughput. Today, we are collecting fast, big data streams from user behavior, smart phones and sensors, and the disk checkpointing of and query language of Hadoop MapReduce is no longer adequate. Tim Berglund from Datastax came on Software Engineering Daily to explain how Apache Cassandra in a popular episode a few weeks ago. On this episode, Tim returns to discuss how Spark and Cassandra can be used together to provide a stack with the analytics and storage we need for today’s distributed computing environment.

React started as just a view layer–it was the V in MVC. React has moved down the stack, with Flux, Redux, GraphQL, and Relay providing opinions for how React applications should structure their data flow. Jared Forsyth works at Khan Academy, which uses React on the front end. At Khan Academy, Jared has experimented with many different ways of handling data flow for a React application, and in today’s episode we not only discuss the conventional tools for React applications, but also ClojureScript, Reframe, and Om/next, which are solutions for React data handling, that are outside of the world of raw JavaScript.

Applications on your phone can talk to each other, but the way that apps interact is still far from the way that web pages interact. Mike Wakerly is the CTO of Button, a service that connects mobile apps to each other. By embedding a component from Button into your app, you can easily request the functionality of an app while you are in a different app. In this episode, Mike and I discuss the future of mobile, and the problems of app stores today. We also explore how mobile application context switching has improved, and how that creates business opportunities that are much more interesting than advertising.

The Changelog is one of the most popular Software Engineering Podcasts in existence. Open source software moves fast, and The Changelog helps developers keep up with that rapid pace by publishing consistent podcasts and newsletters. When I started Software Engineering Daily, Changelog was one of the shows that I looked at for inspiration on how to succeed as a software podcaster. Adam Stacoviak and Jerod Santo are the hosts of The Changelog, and they join me today to talk about how open source has changed since they started the podcast in 2009, and how it will change in the near future.

Virtual reality is a new platform for software engineers to work with. Best practices for VR development have not become widespread throughout the developer community. If you are developing VR software, you need to know how to avoid making the user sick, and this requires an understanding of the hardware limitations. Andreia Gaita is a software engineer at github. On today’s episode of Software Engineering Daily, Andreia takes us through the developer experience of VR, and gives some analysis and predictions about consumer VR. For anyone who is looking for an introduction to virtual reality development–this episode is for you.

Patients with Type-1 diabetes need to frequently pump insulin into their bodies. In order to know when to pump insulin, these patients have a continuous glucose monitor alarm which detects incorrect levels of blood glucose. When the alarm goes off, the diabetes patient administers insulin manually through the pump. In an ideal world, the alarm would communicate with the insulin pump, creating a closed loop. But the world of outdated medical devices is not ideal. Dana Lewis is a developer of an open-source, closed-loop artificial pancreas called the Do-It-Yourself Pancreas system. Dana joins the show to discuss how she hacked together her artificial pancreas using a Raspberry pi plugged into these medical devices. This is a fascinating story of reverse engineering, internet of things, and the hacker mentality. Sponsors

At Netflix, building a user interface requires data about movies, actors, users, accounts, and much more. If each piece of data is coming from a different source, a programmer has to call each of the services that has access to that data. Falcor is a piece of technology designed at Netflix to solve this problem. Falcor creates a unified JSON model of all of the data you could want–and Netflix engineer Brian Holt joins us today to discuss how Falcor works. We also discuss React, ES6, and JavaScript at Netflix, and Brian explains why Falcor is extremely similar to GraphQL, and why that is OK.

Kubernetes is an open-source distributed operating system that allows normal engineers to manage their resources at the scale that Google does. With Docker providing the APIs to make containers easy to work with, Kubernetes allows processes to be easily distributed and replicated across a data center. Kubernetes was created at Google, and today’s guest Kelsey Hightower is a staff developer advocate working at Google. He joins me in a conversation about how Kubernetes works, and why it is important to an average developer. We touch on how software engineering is going to look in the next 5-10 years, and how a lot of the work around distributed systems will become much easier over time.

Exception monitoring services and log management services are two sides of a gradient. Exception monitoring services capture and aggregate the problems that occur on your application. Log management services aggregate all of your logs, so that you can decide for yourself what constitutes a problem. Brian Rue from Rollbar joins the show today to talk about Rollbar’s exception monitoring architecture, and the competitive landscape of these technology products. Every software engineer wants track the problems with an application, but some developers need more information than others–and that ends up changing how these error aggregation services are architected. This is an interesting conversation on the business of SaaS products for developers, and the architecture of a distributed system designed to monitor and aggregate errors.

As software engineering gets more popular, the resources that we use to read and understand software are growing and improving. Scale Your Code is an organization that seeks to improve accessibility to the world’s programming knowledge, and today’s guest Christophe Limpalair joins me to discuss Scale Your Code. Christophe is a podcaster and blogger who runs Scale Your Code.

Docker is an open platform for distributed applications. Docker allows for standardized container runtimes and will be an important piece of infrastructure for many years to come. But if you are new to Docker, the technology can be confusing and intimidating. What is a container? Why do I want to put my application in a container? Today’s guest Prakhar Srivastav wrote a long tutorial explaining the fundamentals of Docker. Prakhar explains why Docker is useful, how Docker works, and how you can get started.

Apache Kafka has become the most popular open-source solution for persistent replicated messaging in the Hadoop ecosystem. But some software engineers who are working with “big data” don’t want to deal with the configuration and set up of Kafka. One way to side step this problem is to go with a managed solution, like Microsoft Azure Event Hubs. Dan Rosanova is today’s guest. He joins us to discuss persistent replicated messaging, and the features that Azure Event Hubs provide. This is an interesting discussion about distributed pub sub messaging, and we also get into a great conversation about platform as a service versus open source software, and the future of cloud software.

Kubernetes has been a focus of several previous shows on Software Engineering Daily. The architecture and management of our distributed systems have changed, and widespread adoption of containers and container management systems is just beginning. Joseph Jacks from Kismatic joins the show today to give us a perspective on the past, present, and future of container management. He also talks about his company Kismatic, which provides enterprise support for Docker and Kubernetes.

“Graphing when your Facebook Friends are awake” was trending at the top of Hacker News when Alex Hogue first posted it. Alex wrote a blog post about this piece of software he built that interacts with public Facebook APIs to detect when any of his friends are awake or asleep. As I read that post, I wanted to talk to Alex, because it was such a creative and technical piece of software–but it is surprising that someone would take the time to randomly inspect the elements of the Facebook page, reverse engineer the way Facebook sends data about statuses, mine data from that reverse engineering, and create a bunch of graphs about that data. Today, Alex joins me to explain how and why he built a tool to graph when people are awake or asleep using Facebook data.

Applications crash, and engineers need to be able to quickly find the root cause of a crash. Apps have become distributed, and debugging workflows have changed. Developers need better tools to identify and troubleshoot problems with their apps. Ben Curtis joins us today to discuss Honeybadger, an exception and monitoring tool for web applications. We discuss how software tools can be used to detect problems faster and improve the debugging workflow.

Kubernetes is an open source system for automating deployment, operations, and scaling of containerized applications. Google developed Kubernetes after fifteen years of running containers in production. Brendan Burns is a founder of the Kubernetes project, and he joins us to talk about the lessons learned as Google has built containerized applications to distribute across its massive infrastructure. We talk about Docker, Borg, Kubernetes, and other distributed systems technologies. Applications crash, and engineers need to be able to quickly find the root cause of a crash. Apps have become distributed, and debugging workflows have changed. Developers need better tools to identify and troubleshoot problems with their apps.

Elixir is a programming language built on top of the Erlang virtual machine. Elixir allows metaprogramming, polymorphism, and a web framework called Phoenix that has drawn positive comparisons to Ruby on Rails. Jose Valim is today’s guest. He built Elixir to augment a language that he loved–Erlang. On Software Engineering Daily, we interviewed Joe Armstrong, the creator of Erlang, which was a very popular show–I encourage any listeners who are fans of Erlang to check it out. Erlang was built with concurrency in mind, and it has been rising in popularity as more of our applications are written to be distributed. In today’s episode we will discuss what Jose is building on top of Erlang.

Engineers who want to add search to their application usually deploy Elasticsearch, or write their own search engine that uses TF-IDF. These solutions work well for large documents, but are less effective for large volumes of small records–which is how many modern web and mobile applications are structured. In today’s show, Julien Lemoine discusses how his company Algolia thinks about search. Algolia is a search as a service company that gives developers an easier way to search on their websites and applications.

CDN stands for content delivery network. A content delivery network is a system of distributed servers that delivers web pages and other web content. Without CDNs, the internet would be much slower, because CDNs function as a caching layer for most web resources. Carl Gustas is an engineer at CacheFly, a popular content delivery network. He joins us today to discuss how CDNs work, and the different methods an engineer can use to take advantage of caching in a CDN.

Google has published papers on distributed systems such as BigTable, Chubby, and the Google File System. During this episode, we focus on a product that takes inspiration from Google’s Spanner project, a database that is built on a distributed monolithic sorted map. CockroachDB is a scalable, survivable, consistent SQL database. Today’s guest Ben Darnell is the CTO of Cockroach Labs, and he joins us today to discuss SQL and NoSQL distributed databases. Ben explores the tradeoffs between these database types, and explains how CockroachDB provides many of the best features of both a SQL and a NoSQL distributed database.

Wiki technology was invented decades ago to improve how software developers communicate. Today, Wikipedia has taken the ideas of the wiki to a new level, creating a free knowledge graph for the world to learn from. Ward Cunningham developed WikiWikiWeb in 1994. He joins us today to discuss the first wiki, and how wikis have changed the way information propagates. Ward is not just an expert on how to write software–he is an expert on the software development experience, and we discuss how to build a community where software developers can overcome the fear of building something that has not been built before.

Software applications are constantly generating logs. These logs are necessary to understand how an application is functioning, and logs are key to debugging. As applications have gotten more complex, logging infrastructure has become complex as well. Storing and managing all of our log data is such a big task that several companies have been started to tackle this problem. Today’s guest is Christian Beedgen, CTO at Sumo Logic. Sumo Logic is a cloud-based log management company. We discuss the elastic log-processing platform Sumo Logic has built to help software engineers with log management. It’s a great conversation about distributed systems, machine learning, and debugging applications.

Frontend web development was simpler in the past–CSS, HTML, and JavaScript were all you needed to know. Today, we have mobile web, React, Angular, PHP, JQuery, and so much more. Marc Grabanski focuses on what he believes is timeless–pure JavaScript. Marc is the founder of Frontend Masters, a training site for developers who want to build quality web interfaces. He joins us today to discuss the shifting nature of front end development.

Nadia Eghbal joins us as the host of today’s show to discuss the past, present, and future of open-source software. This monologue was adapted from Nadia’s article, We’re in a brave, new post open source world. In this episode, Nadia describes the origins of open-source in the Free software movement, its rise to popularity, and today’s golden era where mainstream and popular technologies like React, Spark and Docker are all open source projects. Yet there are several serious issues that threaten the foundation upon which the current open-source renaissance is happening. Nadia discusses what these are, and how to resolve these issues so that we can continue to have a vibrant ecosystem for developers. Thank you to Nadia Eghbal for coming on the show and for standing up for open source software.

When you spend money online, you expect a receipt to come in your email. When you register for a new web site, you need to verify your sign up in your email. These types of emails are called “transactional email” and sending these types of email at scale is a complex engineering task. J.R. Jasperson is the chief architect at SendGrid, a transactional email platform. On this episode, we discuss how email works–from the basics to the massive scale that SendGrid operates on. We also talk about email spam and fraud in detail.

Git is the most popular version control system. If you have been programming for less than a decade, it’s likely that you haven’t used any other method of version control. The git workflow of a software team defines how that team collaborates, builds, and ships software. Tim Pettersen is a developer advocate at Atlassian, where he has been building software around git for many years. He joins us on today’s episode to talk about strategies for git–including branching, merging, continuous integration, and software as a service.

Application delivery has become more complex as software architectures have moved into the cloud. Data center infrastructure has turned into code to be manipulated, and software engineering teams are adjusting their strategies. HashiCorp is a company that builds open-source software for application development and deployment. Mitchell Hashimoto is the founder of HashiCorp, and he joins us to discuss a modern approach to application delivery, and the tools HashiCorp is developing.

Code analysis tools can help a developer understand code. One tool for code analysis is bitHound, which provides code and dependency analysis for NodeJS applications. On today’s episode, we discuss how to use a code analysis tool–and we also talk about how to build one, by breaking down the distributed architecture of bitHound’s backend.

Uber is a transportation company with a high volume of temporal spacial data, constantly being collected from the devices of its users. At any given time, the engineers and data scientists at Uber need to be able to query the system, and understand what is going on with drivers and riders. The unique real-time engineering requirements of Uber lead to an interesting architecture. Danny Yuan joins us today to discuss Uber’s data engineering stack and how the company makes use of its streaming data.

The last editorial we did was 10 Philosophies for Engineers. Listeners enjoyed that episode, so we decided to do another. 10 Philosophies was a collection of beliefs I have about the software engineering industry, and how we can find fulfillment in our work as engineers. The philosophies that I discussed were rooted in my experiences working as a computer science student and as an engineer at several different companies. Before writing any lines of code, I played poker competitively for five years, from the age of 15-20. This was a formative experience. Playing poker changed my perspective on money, risk, and statistics. It has been 7 years since I stopped playing poker full time, but the lessons of the game are still with me. In this editorial, I will discuss some beliefs I have that relate to both poker and software engineering. As with the 10 Philosophies episode, these beliefs my own opinions. If you disagree with them, I would love to know why. At Software Engineering Daily we actually want to know any thoughts you have on our content. We make content for our listeners and our readers. Please tell us how we can improve by emailing us, or by filling out the listener survey. As a poker player becomes a software engineer, certain trends about human-computer interaction become apparent: This post explores each of these trends, explaining why these trends are important to poker players, software engineers, and everyone else. Automated Games In 2008, poker was the perfect sport for human-computer symbiosis. What Tyler Cowen said about freestyle chess also applied to poker: In poker, a human with a statistical “heads-up” display can make decisions that are more mathematically justified than a human without such a tool. Heads-up displays create the poker version of “human-plus-machine teams”. One thesis of Average is Over is that a human will only be employable in the future by finding a career where human reasoning provides defensible value to the problem solving process of a computer. If the human’s responsibilities are not defensible, the human will beobviated. In a subsequent blog post, Cowen addresses the “flip” that can occur when a computational problem no longer requires human assistance: Poker players have been increasingly defeated by machines for the past 10 years. It is no surprise that Google’s AlphaGo has defeated human champion Lee Sedol. If Google decided to beat humans at poker, it would be a trivial exercise for the researchers. Poker seems different than Go or Chess, since there is nondeterminism. You start with two cards, but you don’t know how the board will develop. It would seem that fate is in control, unlike Go and Chess, which have no random elements. With just 4 suits and 13 ranks, a poker game has a trivial branching factor. The nondeterminism is so minimal for a computer to plan around that it is effectively deterministic. Imagine if AlphaGo had to learn to play a version of Go with the following rule: at the beginning of each turn, flip a coin. If you lose the flip, you don’t get to move. Adjusting to this rule would be trivial. That is the magnitude of nondeterminism within poker. Poker, Chess, and Go have small decision spaces. The rules never change, the game pieces never change, there is minimal nondeterminism. A computer can assess a hand of poker as it would a hidden Markov model, but it will take work on par with the AlphaGo team for a computer to be trained to build a model accurately. The job of a professional poker player has been a bad long-term choice for a human to pursue for more than a decade, because it is vulnerable to automation. Games like Go, Poker, and Chess can be automated with machine learning techniques we understand today. The rules, game piece schema, and objectives are easy to define, so these games are ripe for supervised learningand reinforcement learning. Yann LeCun protested against the hype around the AlphaGo victory: Poker is vulnerable t...

Applications can and will crash — it is increasingly important for developers to have visibility into the reasons how and why the crash occurred. James Smith is the guest on the show today, and joins Jeff to discuss why modern applications crash, and how developer products are improving to tighten the gap between QA/support and dev. James is the founder of Bugsnag, an automated crash monitoring system.

Ten years ago, building a highly scalable image delivery service would require millions of dollars in upfront costs, and hours of work configuring hardware server infrastructure. Today, it is possible to bootstrap this type of service, with minimal investment. Today’s episode is about building a content delivery network for images and video. Today’s guest is Itai Lahan, CEO of Cloudinary. We discuss Cloudinary’s early product infrastructure, and how they have evolved as a company since then. We also talk in detail about the venture capital landscape of Silicon Valley today, and how to strategize about raising money. Full disclosure, Cloudinary is a sponsor of Software Engineering Daily.

Memory is king. The cost of memory and disk capacity are both decreasing every year–but only the throughput of memory is increasing exponentially. This trend is driving opportunity in the space of big data processing. Alluxio is an open source, memory-centric, distributed, and reliable storage system enabling data sharing across clusters at memory speed. Alluxio was formerly known as Tachyon. Haoyuan Li is the creator of Alluxio. Haoyuan was a member of the Berkeley AMPLab, which is the same research facility from which Apache Mesos and Apache Spark were born. In this episode, we discuss Alluxio, Spark, Hadoop, and the evolution of the data center software architecture.

Cloud service providers like Amazon, Google, and Microsoft provide both infrastructure as a service and platform as a service. Infrastructure as a service gives developers access to virtual machines, servers, and network infrastructure. Platform as a service is the software that runs on top of that infrastructure as a service–this includes things like Amazon DynamoDB, Microsoft Azure Machine Learning, and Google App Engine. OpenStack is an open-source cloud operating system. Today’s guest is John Purrier, a founder of OpenStack and the CTO of Automic Software.