Software Engineering Daily

GitLab is an open source platform for software development. GitLab started with the ability to manage git repositories and now has functionality for collaboration, issue tracking, continuous integration, logging, and tracing. GitLab’s core business is selling to enterprises who want a self-hosted git installation, such as banks or other companies who prefer not to use a git service in the cloud. The vision for GitLab is to provide a platform for managing the full software development lifecycle, from code hosting to deployment–as well as tools for observability and project management. Sid Sijbrandij is the CEO of GitLab and he joins the show to talk about the product, the business, and the company’s vision for the future. GitLab’s strategy is to offer a set of tools that work for developers out of the box, cutting down on time spent integrating each individual vendor.

An operating system kernel manages the system resources that are needed to run applications. The Linux kernel runs most of the smart devices that we interact with, and is the largest open source project in history. Shuah Khan has worked on operating systems for two decades, including 13 years at HP and 5 years at Samsung. She has worked on proprietary operating systems and a variety of Linux operating system environments, including mobile devices. Shuah joins the show to discuss her work within Linux and her experience contributing to open source. Shuah has made significant contributions to kselftest, a set of tests for Linux. Testing the Linux kernel is complicated. Because there is so much depth to the codebase, and such a variety of ways that Linux can be used, there is also a variety of ways that the operating system gets tested. There is smoke testing, performance testing, and regression testing. There are trees of tests, and as a developer you may only want to run a subset of the tests in that tree. The conversation with Shuah ranged from the low level practices of testing the kernel to a high level discussion of how the Linux kernel can reveal dynamics of human nature.

Malware is malicious software that makes money for the creator of that software. Malware can appear onto a user’s computer if that user visits a malicious website or installs malicious software by accident. There are many types of malware. Spyware sits on your machine and logs your data in order to sell it. Ransomware can lock your computer and demand that you pay money to unlock it. Adware serves you popup ads that you don’t want to see. Cryptojacking is a newer form of malware. Cryptojacking software uses your computer to mine Bitcoin and other cryptocurrencies. Cryptojacking can occur when you visit a website that is running JavaScript that is executing along with the rest of the webpage. When you visit a website with a cryptojacker, your computer will become slower, because your CPU is being taken over to mine cryptocurrency. Cryptojacking can occur anywhere that code runs–and there is a lot of code running on cloud providers. Cloud providers themselves are very secure. But a cloud provider cannot force its customers to be secure. Users who host an insecure application on a cloud provider may get infected with a cryptojacker. If I host a large, complex website on a cloud provider, and I’m serving millions of users, I’m already paying a lot in cloud costs. But when my application gets infected with a cryptojacker, my costs could shoot up. And if I don’t know why my costs are increasing, I might leave the cloud provider. Estaban Vargas is the co-founder of SafeTalpa, a company that provides defense against cryptojackers. Estaban joins the show to explain how cryptojackers work and why cloud providers have trouble defending against them.

Advertising fraud occurs when a brand pays for an advertisement online and that advertisement is shown to an automated bot account that has been created to view ads. Advertising fraud is rampant on the Internet. It’s not possible to know how much money is lost to ad fraud, but the costs are in the billions of dollars. Praneet Sharma and Shailin Dhar are the founders of Method Media Intelligence, a company that builds solutions around improving advertising quality. In previous shows, Praneet and Shailin have described the online advertising ecosystem in detail. They have told stories of bot farms, replay attacks, and adtech companies. In today’s episode, Praneet and Shailin return to the show to discuss how advertising fraud is getting worse–not better. Praneet and Shailin worked with BuzzFeed reporter Craig Silverman, who was a previous guest on the show to talk about his remarkable findings about mobile advertising fraud, which accounts for hundreds of millions of dollars in theft every year.

The demand for electricity is based on the consumption of the electrical grid at a given time. The supply of electricity is based on how much energy is being produced or stored on the grid at a given time. Because these sources of supply and demand fluctuate rapidly but predictably, energy markets present profit opportunities for traders. Minh Dang and Corey Noone are engineers with Advanced Microgrid Solutions, a company that builds software to help traders capture better opportunities in the energy markets. Minh and Corey join the show to talk about how their company builds and deploys machine learning models for market prediction. We discussed data infrastructure, machine learning model deployments, and the dynamics of the energy markets.

Cloud computing started to become popular in 2006 with the release of Amazon EC2, a system for deploying applications to virtual machines sitting on remote data center infrastructure . With cloud computing, application developers no longer needed to purchase expensive server hardware. Creating an application for the Internet became easier, cheaper, and simpler. As the cloud has become popular, new ways of deploying applications have emerged. A developer with a web app has so many different options. You can host your app on an Amazon EC2 server, which will require you to manage cloud infrastructure in case your server crashes. You can deploy your app to Heroku, which gives your cloud deployment better uptime guarantees for a higher price than Amazon EC2. Or you can use Linode, or Microsoft Azure, or Google Cloud. There is such a large market for cloud computing that the world of cloud providers serves more niches every year. In past episodes we have explored a variety of different cloud providers, and the markets they target. Pivotal Cloud Foundry is for managing complex distributed systems applications, typically with large teams. Firebase is a cloud provider that simplifies the developer experience for applications with small teams. Spotinst is a cloud provider that emphasizes low cost. Zeit is a cloud provider that is built to manage applications through serverless “functions-as-a-service” like AWS Lambda. In today’s episode, Mathias Biilman Christensen, CEO of Netlify, joins the show. Netlify is a cloud provider that was built for modern web projects. Netlify represents the convergence of several trends in software development converging: static site deployment, serverless functions, a desire to have a “no-ops” deployment with minimal management, and the rise of newer tools like GraphQL and Gatsby. Mathias explores these trends in detail, and explores the technical challenges of building Netlify. He was a great guest, capable of talking about difficult backend problems that require writing C++, as well as the frontend world of JavaScript frameworks. One announcement before we begin: we are having a $5000 hackathon. The $5000 hackathon is for a new product we’ve been working on: FindCollabs. FindCollabs is a platform for finding collaborators and building projects. Whether you are an engineer, a musician, a designer, a videographer, or an artist, FindCollabs lets you find people and collaborate. To try out FindCollabs, just go to FindCollabs.com, you can make a project or you can join someone else’s project. And it’s very easy to make these projects–you don’t need to have anything built yet–you need to have a vision for what you want to build. And to find out about the hackathon, go to findcollabs.com/hackathon. We are giving away $5000 in cash to the coolest projects that get built before Sunday April 14th. So I recommend getting started early, finding some people to collaborate with, and building some cool stuff!

Monitoring tools are used by every area of an organization. Business development teams use monitoring to understand the metrics for product performance. Finance teams need to understand how the costs of cloud computing resources are changing. Site reliability engineers use monitoring dashboards that applications are up and running without problematic latency. Product managers evaluate the results of AB tests based off of the monitoring data of how users are reacting to new features. A monitoring system needs to be able to handle large volumes of data that are being generated at a high velocity. The data needs to be queryable in an aggregated format, which might require an ETL system for getting data into columnar format. Alexander Pucher is an engineer at LinkedIn, where he works on a monitoring platform called ThirdEye. ThirdEye is built on top of Apache Pinot, a distributed columnar storage engine that ingests data and serves analytical queries at low latency. Pinot uses RocksDB, and is comparable to Apache Druid. Alexander joins the show to discuss ThirdEye, and explain why Pinot is a useful building block for monitoring infrastructure.

WebAssembly is a runtime that lets languages beyond JavaScript to execute in frontend web applications. WebAssembly is novel because most modern frontend applications are written entirely in JavaScript. WebAssembly lets us use languages like Rust and C++ after they have been compiled down to a web assembly binary module. Language interoperability is only one part of why WebAssembly is exciting. The execution environment for WebAssembly modules has benefits for security and software distribution and consumption as well. In previous shows, we’ve given an overview of WebAssembly and explored its future applications as well as its relationship to the Rust programming language. In today’s episode, we explore the packaging and execution path of a WebAssembly module, and some other applications of the technology. Syrus Akbary is the CEO and founder of Wasmer, a company focused on creating universal binaries powered by WebAssembly. Wasmer provides a way to execute WebAssembly files universally. He joins the show to talk about the state of WebAssembly, and what his company is building.

Cryptocurrencies enable a large number of applications. Trustless reputation systems, decentralized identity tools, micropayments, non-fungible Internet items, borderless currencies, just to name a few. But cryptocurrencies have not yet impacted daily life, for most of us. Why is that? One reason is that it is still very hard for developers to build within the cryptocurrency ecosystem. The programming languages, such as Solidity, are not widely used by software engineers. Building and deploying smart contracts is not as easy as deploying a simple Ruby on Rails webapp. The open source tooling is immature, as are the paid developer tools. Sean Li is the CEO of Fortmatic, a company that is building tools to improve the Ethereum developer experience. Fortmatic simplifies wallet creation, user identity management, security, and money transfer for Ethereum developers. Before starting Fortmatic, Sean was the founder of Kitematic, a company that made the developer experience of Docker easier. Kitematic was acquired by Docker. Sean is one of the few people with significant experience in both the enterprise container ecosystem and the cryptocurrency ecosystem. Sean joins the show to discuss his time in the Docker ecosystem, his new company Fortmatic, and his perspective on how to build tools for developers. Someday there will be hundreds of thousands of developers building applications around cryptocurrencies, just like people use cloud computing today. The road to getting there is unclear, and Sean provides useful insights and predictions for the future.

Computational integrity is a property that is required for financial transactions on the Internet. Computational integrity means that the output of a certain computation is correct. If I deposit money into my bank, my bank sends me a number that represents the new balance in my account. I assume that the number they have sent me is correct. The bank could be lying to me–maybe this bank is not actually trustworthy. But I use a bank with a good reputation. If the bank stole money from its users, it would quickly go out of business. Therefore, I feel safe by trusting a bank with my money, because the bank needs to maintain its reputation. The problem with reputation-based systems is that they are opaque. It’s not easy for us to audit the bank and prove the bank actually has the money that it claims to have. Most of the time, the reputation-based systems work fine. But occasionally, we have catastrophic events–think of the 2008 financial crisis, or the Bernie Madoff financial scandal. These circumstances would have been avoided if the financial institutions could have been continuously audited for their solvency. With blockchains and cryptocurrencies, we now have tools that allow us to maintain computational integrity without the opaque systems of reputation. We no longer have to trust a central authority–we can verify computational integrity with math. Eli Ben-Sasson is a co-founder and chief scientist at StarkWare Industries, a company that is bringing zero-trust technology to market. Implementations of zero-trust technology include zk-STARKs, zk-SNARKs, and bulletproofs. StarkWare is focused on the application of zk-STARKs, which can be used to improve scalability and privacy. Eli joins the show to discuss the topic of computational integrity, and how STARKs can be used to provide scalable, secure infrastructure to blockchain applications.

Collaboration on the Internet creates innovation. New inventions, new art, and new products–built by people working together on the Internet. FindCollabs is a product we have been working on to enable people to find and collaborate with each other. If you want to try it out, you can go to FindCollabs.com. FindCollabs is for finding people to create your projects with, and getting those projects built. Whether you are a programmer, a writer, a musician, a game designer, an actor, a videographer, or a project manager–you can find people to collaborate with. I love to work on so many different types of projects, and I love to collaborate with other people. For me, the best way that I learn new skills is by building things. I like to write music, create software, make podcasts, and build businesses. I like to create all the time. For big projects, it’s easier to build your project with a team. Finding team members can be very hard. I value other people who are creative and reliable. FindCollabs lets you find and invite people to your projects, so that you can put together a team to build your project. FindCollabs also has a reputation system. When you work on a project, your collaborators rate you. As you make contributions to projects, you show the FindCollabs community that you are reliable and productive–and other people will want to work with you because of that. The only way to build huge, ambitious projects as a team is for people to trust each other. If you are unreliable, people will not want to work with you. When you join a project on FindCollabs, you are committing to doing work that will add value to that project. If you like to build projects and be creative, you might like FindCollabs. To get started, you can go to FindCollabs.com, log in, and post a project. Or join someone else’s project. If you are ever confused about anything, you can always send me an email. We are sponsoring a series of hackathons on FindCollabs. These hackathons are for anyone with a creative project–whether you want to make a music video, a virtual reality game, an acoustic guitar song, a cryptocurrency whitepaper, a mobile app, a commercial–anything creative. Our first hackathon starts today, March 3rd 2019, and ends at 11:59 PM PST on Saturday March 16th. On March 17 2019 we will announce the winners of the first hackathon, and send them emails. We will also announce the details of the second hackathon. Prizes In the first hackathon, the prizes are not very big. But they will get bigger over time. If you like the idea of FindCollabs, it might benefit you to get involved now, so that you can build your reputation and find better people to work with in the future. 1st place: $500 divided evenly among the winning team; SE Daily hoodies for each member of the team 2nd place: SE Daily hoodies for each member of the team Most valuable feedback on the product: SE Daily Towel Most helpful community member award: SE Daily Old School Bucket Hat The FindCollabs hackathons will be judged by a panel of investors, entrepreneurs, podcasters, artists, and technologists. We will announce the judges of the first hackathon in the next few days. These judges will be voting based on which projects they like the most. Every project on the FindCollabs site before 11:59 PM PST on Saturday March 16th will be entered to win the contest. To find our detailed terms and conditions, go to findcollabs.com/terms. Thanks for taking the time to read through this post. If you get a chance, check out FindCollabs and feel free to send me feedback. I’d love to know what you think, and any suggestions you have.

The Internet has transformed humanity. The Internet is the result of a long series of innovations from military, academia, business, and the open source community. In his book, How The Internet Happened: From Netscape to the iPhone, Brian McCullough tells the story of the last 25 years of Internet development through the lens of companies like ebay, Amazon, Google, and Apple. Whereas other books have focused on the trajectory of these individual companies, Brian explains how innovations in one company often lead to success in another. Without the lessons from Napster, we might not have Spotify. Without the trust model pioneered by ebay, we would not have marketplaces like Airbnb. Brian is also the host of The Internet History Podcast and the Techmeme Ride Home podcast. In The Internet History Podcast, Brian interviews entrepreneurs and engineers who were firsthand witnesses to the developments that led to our modern Internet, including early employees at Amazon, Tesla, and TheGlobe.com. In his other podcast, the Techmeme Ride Home, Brian gives a daily overview of the day’s Internet news. Through his podcasts about the Internet’s past and present, Brian has also accumulated an intuition about the future. He joins the show to discuss his book, the art of podcasting, and the historical lessons of technology.

Advertising fraud steals billions of dollars every year. BuzzFeed reporter Craig Silverman reports on advertising fraud and its impact on the Internet. In one investigation, Craig uncovered a mobile advertising fraud scheme in which four people stole millions of dollars (perhaps as much as $75 million or even $750 million) by serving advertisements to automated users on mobile apps. The scheme worked as follows: This scheme was easy to pull off. It did not require much sophistication in terms of engineering or business skills. If a group of four people can generate tens of millions of dollars, how much ill-gotten capital is being generated by large corporations that are deeply involved in the advertising market? Craig’s article went viral, and he has followed it up with other pieces about ad networks, fraud investigations by Google, and the potential for mobile apps to be used for large scale surveillance of Americans by the Chinese. Craig is the most dedicated reporter covering advertising fraud today. His work is invaluable because he is asking difficult questions about the economics of our Internet. As we discuss in the episode, there is currently no effective automated means of detecting a bot from a human on the internet. We have also discussed this in detail on previous episodes about ad fraud, the advertising industry, advertising analytics, and the techniques of ad fraud. Ad fraud is not the fault of any one party. It is an emergent result of the way that our Internet is set up. It is as hard to imagine a world without advertising fraud as it is to imagine a world without email spam.

Podcast listeners usually find out about a new podcast in one of two ways: either a friend recommends that podcast or the Apple podcast charts rank that new podcast highly. The Apple podcast charts are created using an algorithm that is not public. Many people believe that the chart ranking of a podcast is based on the number of podcast subscribers, the number of podcast downloads, and the reviews that are written about the podcasts on iTunes. Jack Rhysider is the host of Darknet Diaries, a podcast about the dark and strange elements of the Internet. Darknet Diaries is told in a high quality, narrative audio format. Jack is a security engineer with a deep understanding of technology, and has been blogging for a long time. As Jack has built a following with his podcast, he has spent more time looking at the iTunes podcast charts. He has seen the rank of Darknet Diaries increase–but he has seen the rank of other podcasts increase much faster. Some of these podcasts have low quality content. The audio quality is poor, the host is unprepared–these are the kinds of podcasts you would listen to once, and never subscribe to. And yet, numerous podcasts with low quality were somehow able to game the rankings and make it to the top of the charts. In episode 27 of Darknet Diaries, Jack investigated the phenomenon of fraudulent podcast chart manipulation. It was one of my favorite podcast episodes ever (and this is coming from someone who has listened to a lot of podcasts). The investigation went to several unexpected places, but Jack did solve the riddle of how low quality podcasts climb the iTunes charts. Jack joins the show to talk about fraudulent–and the broader implications of the fake Internet. Today’s episode is a simple example of how easily Internet platforms can be gamed–for a deeper dive into the fake Internet, listen to our past episodes on advertising fraud, or tomorrow’s episode with ad fraud investigative journalist Craig Silverman, which I am very excited about.

Many devices in our world are not “smart.” Air conditioners, electric guitars, power outlets, and factory conveyor belts, just to name a few. There are exciting software applications that we could build around these devices, but we need to be able to interface with them programmatically. We need to be able to know the state of these devices. We need to be able to save that state, and then we can use that state data to perform actions that change the state of those devices. To make these devices smart, we can use a microcontroller, a small device with a constrained amount of CPU, memory, and I/O. Device data can be sent to the cloud or processed locally, and that data can be used perform predictive maintenance, or create machine learning models, or create simple dashboards so human operators can understand the state of their hardware. Dirk Didascalou is the VP of Internet of Things with Amazon Web Services. Dirk joins today’s show to discuss the strategy and philosophy of the AWS Internet of Things set of tools. We talk about a wide-ranging set of topics–including IoT security, edge deployments, and machine learning.

Edge computing refers to computation involving drones, connected cars, smart factories, or IoT sensors. Any software deployment that is not a large centralized server installation could qualify as an edge device–even a smartphone. Today, much of our heavy computation takes place in the cloud–a set of remote data centers some distance away from our client devices. For many use cases, this works fine. But there are a growing number of use cases with lower latency and higher bandwidth requirements at the edge. A simple example is video. Let’s say you want to record a video stream, and detect people in that video stream in real time. Based on who those people in the video stream are, you want to do different things–maybe you want to send them a text message, or report to the police that a dangerous person has entered the premises. This video stream could be captured by a drone, or by a smart car, or by a video camera mounted somewhere. Where is the video stream getting stored? Where is the machine learning model running? How do you deploy new machine learning models to the operating system with the machine learning model? This is a simple example, and there are many open questions as to how to best solve such a problem. With the increased resource constraints at the edge, there is a need for new hardware and software to power these edge applications. This led to the creation of LF Edge, a new open source group under the Linux Foundation. The goal of LF Edge is to build an open source framework for the edge. Arpit Joshipura is the general manager of networking, orchestration, edge computing, and IoT with the Linux Foundation. He joins the show to describe the state of edge computation, and the mission of LF Edge. This episode was exciting for several reasons. After seeing the rise of Kubernetes for container orchestration, we know that a popular open source technology that solves a widespread problem can have dramatic influence on the software world. And when multiple large companies get involved in that open source project, it can gain traction quite quickly. Edge computing has a large set of unanswered questions, but telecom providers like AT&T and large infrastructure companies like Dell EMC are getting heavily involved with the Linux Foundation Edge group. This represents a significant expansion of the open source model, and a suggestion of further investment into open source projects in the near future.

React Native allows developers to build native applications for iOS and Android using components written in the React JavaScript framework. These ReactJS components render to native application code by going over a JavaScript bridge, a message bus that communicates between JavaScript code and native iOS or Android runtimes. For most mobile application use cases, React Native works well. But in some cases, the platform suffers from performance issues due to the functionality of the JavaScript bridge. For example, mobile games with high demands on graphics, networking, and fast real-time updates to the UI can stutter when using React Native. To address the performance issues of React Native, the core team working on React Native at Facebook is rearchitecting the React Native runtime within a project called Fabric. Fabric consists of changes to the threading model, the data handling system, and the JavaScript bridge. Chris Severns and Lee Johnson work at G2i, a group of React and React Native specialists. Chris and Lee join the show to discuss the rearchitecture, including the engineering history of React, the technical debt within the React project, and the vision that the React team has for the future. We also discuss Google’s Flutter project, a cross-platform native framework with a different architectural model than React Native.

In the early days of YouTube, there were scalability problems with the MySQL database that hosted the data model for all of YouTube’s videos. The state of the art solution to scaling MySQL at the time was known as “application-level sharding.” To scale a database using application-level sharding, you break up the database into shards–disjoint regions of data. When you want to query the database, you need know which shard to query. In your application code, you have to issue the query to a specific shard. The solution of application-level sharding does scale your database. But the downside is that every application that interfaces with the database now has to include code that is aware of the sharding schema. If you are an application engineer, you don’t want to have to worry about the way that the database is sharded, because it adds significant complexity to your code. The engineers at YouTube decided to fix this problem with a project called Vitess. Vitess abstracts away the details of sharding by orchestrating reads and writes across the distributed database. In a previous episode, we covered the architecture, read and write path, and the story of Vitess in detail. In today’s episode, Jiten Vaidya and Dan Kozlowski of PlanetScale Data join the show to give their perspective on MySQL scalability, and their work taking Vitess to market as a solution to scaling relational databases.

Zoox is a full-stack self-driving car company. Zoox engineers work on everything a self-driving car company needs, from the physical car itself to the algorithms running on the car to the ride hailing system which the company plans to use to drive around riders. Since starting in 2014, Zoox has grown to over 500 employees. Ethan Dreyfuss is a software infrastructure engineer at Zoox. He joins the show to discuss scaling an engineering team for self-driving. Machine learning was a big part of our conversation, because there are so many different approaches that an engineering team can take when it comes to machine learning for cars. Can you take computer vision algorithms from academic papers and apply them to cars? Can you use the computer vision APIs from the cloud providers for anything useful? What about physical world mapping companies like Mapillary? How do you do data labeling, and data management? And how do you manage the interactions across the stack, from mechanical engineering to user interface design? We touched on some of these areas, but barely scratched the surface of the self-driving car domain.

DoorDash is a food delivery company where users find restaurants to order from. When a user opens the DoorDash app, the user can search for types of food or specific restaurants from the search bar or they can scroll through the feed section and look at recommendations that the app gives them within their local geographic area. Recommendations is a classic computer science problem. Much like sorting, or mapping, or scheduling, we will probably never “solve” recommendations. We will adapt our recommendation systems based off of discoveries in computer science and software engineering. One pattern that has been utilized recently by software engineers in many different areas is the “word2vec”-style strategy of embedding entities in a vector space and then finding relationships between them. If you have never heard of the word2vec algorithm, you can listen to the episode we did with computer scientist and venture capitalist Adrian Colyer or listen to this episode in which we will describe the algorithm with a few brief examples. Store2vec is a strategy used by DoorDash to model restaurants in vector space and find relationships between them in order to generate recommendations. Mitchell Koch is a senior data scientist with DoorDash, and he joins the show to discuss the application of store2vec, and the more general strategy of word2vec-like systems. This episode is also a great companion to our episode about data infrastructure at DoorDash.

The nature of software projects is changing. Projects are using a wider variety of cloud providers and SaaS tools. Projects are being broken up into more git repositories, and the code in those repositories are being deployed into small microservices. With the increased number of tools, repositories, and deployment targets, it can become difficult to manage software policy. “Policy” defines how different parts of an application can behave. Which parts of your application can access an Amazon S3 bucket? Which parts of your application can communicate with the authentication microservice? Which developers are allowed to push a new build to production? Shimon Tolts is the CTO and co-founder of Datree, a platform for policy enforcement and code compliance. He joins the show to talk about continuous delivery, configuration management, and policy enforcement. He also explains the motivation for his company Datree, which performs analysis across a user’s GitHub repo to map the committers, code components, and repositories.

Ethereum allows developers to run decentralized applications. But the tooling for building and managing those decentralized applications is immature. Experienced software engineers have difficulty getting started with writing Ethereum applications because the stack of tools is so unfamiliar and different than traditional software tools. Whether or not Ethereum itself succeeds, developers in the future will probably be building some decentralized apps. We will be treating money as a first-class citizen and architecting software that transfers financial value as easily as we transmit JavaScript today. “Web3” will be a world in which many more software applications will be possible. As we move towards Web3, many new tools will be built. Web2 was the result of Ruby on Rails, Amazon Web Services, the iPhone, and other software tools that made it easier to deploy web servers and consume Internet services. In the world of Web2, we saw the birth of Airbnb, Uber, and Netflix. In the world of Web3, we will see new types of gig economy apps, sharing economy platforms, and social networks. These new applications will arrive gradually as the tooling improves, and makes it easier for developers to hack together businesses and side projects built on cryptocurrencies. Brian Soule is the founder or Ethsimple, a company that makes tools for Ethereum developers. Brian joins the show to talk about the state of cryptocurrencies, the tooling that developers have access to, and his company Ethsimple. We cover high-level ideas, such as Bitcoin maximalism and also talk about some more technical areas of the Ethereum ecosystem, such as the Ethereum Name Service.

A Kubernetes cluster presents multiple potential attack surfaces: the cluster itself, a node running on the cluster, a pod running in the node, a container running in a pod. If you are managing your own Kubernetes cluster, you need to be aware of the security settings on your etcd, your API server, and your container build pipeline. Many of the security risks of a Kubernetes cluster can be avoided by using the default settings of Kubernetes, or by using a managed Kubernetes service from a cloud provider or an infrastructure company. But it is useful to know about the fundamentals of operating a secure cluster, so that you can hopefully avoid falling victim to the most common vulnerabilities. Liz Rice wrote the book Kubernetes Security with co-author Michael Hausenblas. Liz works at Aqua Security, a company that develops security tools for containerized applications. In today’s show, Liz gives an overview of the security risks of a Kubernetes cluster, and provides some best practices including secret management, penetration testing, and container lifecycle management.

Cloud computing has been popular for less than twenty years. Large software companies have existed for much longer. If your company was started before the cloud became popular, you probably have a large, data center on your companies premises. The shorthand term for this software environment is “on-prem”. Deploying software to your own on-prem servers can be significantly different than deploying to remote servers in the cloud. In the cloud, servers and resources are more standardized. It is often easier to find documentation and best practices for how to use cloud services. Many of the software vendors who got started in the last decade created their software in the cloud. For example, Readme.io makes it easy for companies to create hosted documentation. Their early customers were startups and other cloud-native companies. All of those companies were happy to consume the software in the cloud. As time went on, Readme found that other customers wanted to use the Readme product as a self-hosted, on-prem service. Readme needed to figure out how to deploy their software easily to the “on-prem” environment. It turns out that this is a common problem. Software vendors who want to sell to on-prem enterprises must have a defined strategy for making those deployments to on-prem infrastructure–and those deployments are not always easy to configure. Replicated is a company that allows cloud-based software to easily deploy to on-prem infrastructure. Grant Miller is the founder of Replicated and he joins the show to discuss on-prem, cloud, and the changing adoption patterns of enterprise software companies.

Uber manages the car rides for millions of people. The Uber system must remain operational 24/7, and the app involves financial transactions and the safety of passengers. Uber infrastructure runs across thousands of server instances and produce terabytes of monitoring data. The monitoring data is used to understand the health of the software systems as well as relevant business metrics, such as driver efficiency, daily revenues, and user satisfaction. Uber adopted the Prometheus monitoring system to manage their monitoring data. Prometheus regularly scrapes metrics across infrastructure to gather time series data about the state of everything across Uber. As the usage of Prometheus has grown within the company, Uber has had to figure out how to scale their monitoring platform. M3 is a monitoring system built at Uber to scale Prometheus and provide a platform that can effectively scale the data storage as well as the query serving. Rob Skillington is a staff software engineer at Uber, and he joins the show to talk about monitoring at Uber–from the requirements of the system to the implementation of M3. At Uber, M3 powers dashboards, ad-hoc queries, and alerting. M3 was open sourced to give other users access to a scalable Prometheus solution. In a previous episode with Brian Boreham, we discussed one strategy for scaling Prometheus. Today’s episode covers another scalability solution, with M3.

Data infrastructure is advancing beyond the days of Hadoop MapReduce, single-node databases, and nightly reporting. Companies are adopting modern data warehouses, streaming data systems, and cloud-specific data tools like BigQuery. Every company with a large amount of data wants to aggregate that data into a data lake and make the data available to developers. All of this data can be used to power machine learning models which can potentially improve every area within a company where they have historical data. “Data pipeline” is a term used to describe the process of preparing data, building machine learning models, deploying those models, and tracking the results of those models. Pachyderm is a company and open source project that is focused on deployment, management, and scalability of data pipelines. Pachyderm allows developers to version data, track the state of data sets, backtest machine learning models, and collaborate on data. It also tackles the very hard problem of machine learning auditability. Joe Doliner is the CEO of Pachyderm and joins the show to discuss his experience building Pachyderm over the last five years. Data infrastructure has changed a lot in five years, and the world has moved in a direction that has benefitted Pachyderm, with more infrastructure moving to containers and more data teams advancing beyond a world of just Hadoop MapReduce. In today’s show, Joe talks about modern infrastructure, data provenance, and the long-term vision of Pachyderm.

Infrastructure software is having a renaissance. Cloud providers offer a wide range of deployment tools, including virtual machines, managed Kubernetes clusters, standalone container instances, and serverless functions. Kubernetes has standardized the container orchestration layer and created a thriving community. The Kubernetes community gives the cloud providers a neutral ground to collaborate on projects that benefit everyone. The two forces of cloud providers and Kubernetes have led to massive improvements in software quality and development practices over the last few years. But one downside of the current ecosystem is that many more developers learn how to operate a Kubernetes cluster than perhaps is necessary. “Serverless” tools are at a higher level than Kubernetes, and can improve developer productivity–but a risk of using a serverless tool is the potential for lock-in, and a lack of portability. Knative is an open-source serverless platform from Google built on top of Kubernetes. Ville Aikas is a senior staff engineer at Google who has worked at the company for eleven years. With his experience, Ville brings a rare perspective to the subjects of Kubernetes, serverless, and the infrastructure lessons of Google. Ville joins the show to discuss Knative, the motivation for building it, and the future of “serverless” infrastructure.

Virtualization software allows companies to get better utilization from their physical servers. A single physical host can manage multiple virtual machines using a hypervisor. VMware brought virtualization software to market, creating popular tools for allowing enterprises to deploy virtual machines throughout their organization. Containers provide another improvement to server utilization. A virtual machine can be broken up into containers, allowing multiple services to run within a single VM. Containers proliferated after the popularization of Docker, and the Kubernetes open source container orchestration system grew to be the most common way of managing the large numbers of containers that were running throughout an organization. As Kubernetes has risen to prominence, software infrastructure companies have developed Kubernetes services to allow enterprises to use Kubernetes more easily. VMware’s PKS is one example of a managed Kubernetes service. Brad Meiseles is a senior director of engineering at VMware with more than nine years of experience with the company. He joins the show to discuss virtualization, Kubernetes, containers, and the strategy of a large infrastructure provider like VMware.

Real estate is an asset that is not straightforward to invest in. Real estate can generate excellent returns for investors, but can require much more time and expertise than stocks. Cadre is a company that allows users to invest in real estate more easily and intelligently. Cadre provides users with lots of data about potential investments and enables investments in those opportunities within the platform. Leonid Movsesyan is the head of engineering at Cadre and joins the show to talk about the problems being solved by the company in areas of product development, infrastructure engineering, hiring, and data science. To build a platform for evaluating real estate investments, Cadre ingests and merges lots of data sets–some public and some private. This gives investors a detailed picture of the value of investments. Fintech Daily is a new podcast from Software Engineering Daily covering payments, cryptocurrencies, trading, and the intersection of finance and technology. We are looking for volunteer hosts for Fintech Daily, and if you are interested in working with us to conduct interviews, send an email to [email protected]. You can find the podcast on iTunes, Google, and everywhere else, and if you are interested in hosting, don’t hesitate to reach out.

RocksDB is a storage engine based on the log structured merge tree data structure. RocksDB was developed at Facebook to provide a tool for embedded databases. The code for RocksDB is a fork of LevelDB, an embedded database built by Google for the Chrome browser. Every database has a storage engine. The storage engine is the low level data structure that manages the data in the database. RocksDB is widely used in database applications where a log structured merge tree is preferable to a b-tree. These tend to be write-heavy workloads. In past shows, we have explored applications of RocksDB in our coverage of databases like TiDB, data intensive applications like Smyte, and data platforms like Rockset. In today’s episode, Dhruba Borthakur and Igor Canadi join for a deep dive into how RocksDB works. Dhruba was the original creator of RocksDB, and Igor is a former Facebook engineer who worked on RocksDB in its early days. Both Dhruba and Igor work at Rockset. We talk about the log structured merge tree, discuss why an LSM has higher write throughput than storage engines based on a b-tree, and evaluate some of the use cases for RocksDB.

Hashicorp was founded seven years ago with the goal of building infrastructure tools for automating cloud workflows such as provisioning, secret management, and service discovery. Hashicorp’s thesis was that operating cloud infrastructure was too hard: there was a need for new tools to serve application developers. Hashicorp founders Mitchell Hashimoto and Armon Dadgar began releasing open source tools to fulfill their vision of better automation. Terraform, Vagrant, Consul, and other tools created by Hashicorp gained popularity, and Hashicorp began iterating on their business model. Today, Hashicorp makes money by offering enterprise features and support to enterprises such as Pinterest, Adobe, and Cruise Automation. Over the last seven years, enterprise software infrastructure has changed rapidly. First, enterprises moved from script-based infrastructure automation to container orchestration frameworks. Then, the container orchestration world consolidated around Kubernetes. Today, large enterprises are rapidly adopting Kubernetes with a mix of public cloud and on-prem vendors. At the same time, these enterprises are also becoming more willing to consume proprietary tools from the public cloud providers. Hashicorp has benefitted from all of this change. Their different tools fit into a variety of workflows, and are not closely coupled with any particular cloud provider or platform solution. Armon and Mitchell join today’s show to discuss the business model and the product philosophy of Hashicorp. We also touch on service mesh, zero trust networking, and their lessons from the container orchestration wars.

Tyler Cowen’s book Stubborn Attachments outlines a framework that individuals can use to make decisions grounded in economic philosophy. In his previous books, Tyler examined recent economic history. Stubborn Attachments gives his perspective for navigating the future. Tyler is a professor of economics at George Mason University. He is also the host of Conversations with Tyler, a podcast that includes guests such as Ethereum creator Vitalik Buterin, Stripe co-founder Patrick Collison, and Coinbase CTO Balaji Srinivasan. Tyler blogs frequently at Marginal Revolution. Tyler’s previous appearance on Software Engineering Daily centered around his earlier books, including The Complacent Class. In this episode, Tyler describes the philosophy outlined in Stubborn Attachments, then we discuss how his philosophy relates to software engineering, podcasting, and economics. To find all 900 of our old episodes, including past episodes with writers, entrepreneurs, and venture capitalists, check out the Software Engineering Daily app in the iOS and Android app stores. Whether or not you are a software engineer, we have lots of content about technology, business, and culture. In our app, you can also become a paid subscriber and get ad-free episodes–and you can have conversations with other members of the Software Engineering Daily community.

Artificial intelligence is reshaping every aspect of our lives, from transportation to agriculture to dating. Someday, we may even create a superintelligence–a computer system that is demonstrably smarter than humans. But there is widespread disagreement on how soon we could build a superintelligence. There is not even a broad consensus on how we can define the term “intelligence”. Information technology is improving so rapidly we are losing the ability to forecast the near future. Even the most well-informed politicians and business people are constantly surprised by technological changes, and the downstream impact on society. Today, the most accurate guidance on the pace of technology comes from the scientists and the engineers who are building the tools of our future. Martin Ford is a computer engineer and the author of Architects of Intelligence, a new book of interviews with the top researchers in artificial intelligence. His interviewees include Jeff Dean, Andrew Ng, Demis Hassabis, Ian Goodfellow, and Ray Kurzweil. Architects of Intelligence is a privileged look at how AI is developing. Martin Ford surveys these different AI experts with similar questions. How will China’s adoption of AI differ from that of the US? What is the difference between the human brain and that of a computer? What are the low-hanging fruit applications of AI that we have yet to build? Martin joins the show to talk about his new book. In our conversation, Martin synthesizes ideas from these different researchers, and describes the key areas of disagreement from across the field. To find all 900 of our old episodes, including past episodes with authors and artificial intelligence researchers, check out the Software Engineering Daily app in the iOS and Android app stores. Whether or not you are a software engineer, we have lots of content about technology, business, and culture. In our app, you can also become a paid subscriber and get ad-free episodes–and you can have conversations with other members of the Software Engineering Daily community.

Mars is a cold, inhospitable planet far from earth. It presents one of the most complex challenges faced by engineers: how can we create a new world? To create a new world, first we have to get there. We can build new rockets with improved propulsion systems. We can build ships that allow us to survive the long, grueling trip from Earth to Mars. We can build robots that will help us construct our new home. And this is just the beginning. Mars could be warmed, and could develop a hydrologic cycle like the system of clouds and oceans on earth. Mars could be a place for new ideas and new cultures, unfettered by the conventions of Earth. Mike Solana is the host of Anatomy of Next, a podcast about technologies and philosophies of the future. He’s also a vice president at Founder’s Fund. In a previous episode, Mike joined the show to talk about artificial intelligence, genetics, and robotics. Today, we discuss Mars. The latest season of Anatomy of Next explores the science that is bringing us closer to exploring other planets. On his podcast, Mike speaks with engineers, researchers, and entrepreneurs about the state of the art of space technology–as well as the challenges that remain unsolved. Mike returns to the show to discuss this dream of a new world. Why should we go to Mars? And why should the software engineers listening to this podcast even care? To find all 900 of our old episodes, including past episodes with venture capitalists, futurists, and philosophers, check out the Software Engineering Daily app in the iOS and Android app stores. Whether or not you are a software engineer, we have lots of content about technology, business, and culture. In our app, you can also become a paid subscriber and get ad-free episodes–and you can have conversations with other members of the Software Engineering Daily community.

Social media has transformed our lives. It has also transformed how wars are fought. P.W. Singer’s new book “Likewar: The Weaponization of Social Media” describes the far-reaching impact of social media on the tactics and strategies used by military, business, and everyday citizens. We have all read about stories such as Russian bots and Cambridge Analytica, but Likewar covers many more cases that are surprising and mildly frightening. From the Gaza Strip to the streets of Chicago to Taylor Swift’s Instagram feed, Likewar describes just how pervasive the effect of social media has been on warfare. Likewar also provides historical context. For software engineers, the repurposing of social media as a weapon is disconcerting. Many of us are working on products with a social networking component. Does this make us complicit in building weapons? We can find some reassurance in the fact that this has happened before: from the newspaper to the television, every new invention has been used repurposed for war. In a war, a new piece of technology always presents a new vector to gain an advantage in a conflict. Because the stakes are so high in a war, there is a large incentive to find creative ways to use technology to undermine your adversaries and to help your allies. P.W. Singer has written about robotics, cybersecurity, and modern warfare for a decade. In a previous episode, we discussed subjects like Stuxnet, drones, and social media manipulation. In today’s show, P.W. returns to talk about his book Likewar: The Weaponization of Social Media.

Infrastructure software can be a great business. An infrastructure software company sells core technology to a large enterprise such as a bank or insurance company. This software has near zero marginal cost and generates a large annuity for the infrastructure software company. Once a bank has purchased your infrastructure software, the bank is likely to renew every year and never remove the software. Selling infrastructure software is like selling concrete or steel, except the software is cheaper to produce, easier to distribute, and generates an annuity rather than being a one-time sale. The fundamental economics of enterprise infrastructure software are extremely appealing, and every year more businesses enter the space–but few businesses ever leave. If you are starting an infrastructure software company, you can expect a complex battle for market share. There is no easy trick to get it into the hands of your target customer. Martin Casado studied computer science at Stanford before founding Nicira, a company that pioneered software-defined networking and virtualization technology. In 2012, Nicira sold to VMware for $1.26 billion. Martin now works as a general partner at Andreessen Horowitz. Martin writes about the modern strategies of building a successful infrastructure software company. He describes two methods of selling into an enterprise: bottoms-up and top-down. In a bottoms-up model, engineers within an enterprise start using your product to solve a well-defined problem, such as API management. As more and more employees within the organization start to use your product, you can begin to engage the enterprise about becoming a paying customer for your product. Since the enterprise is already using your product, the sales conversation is much easier. In the top-down model, you engage the CIO, CEO, or CTO directly and try to convince them that your product is worth paying for. When the senior leadership of a bank buys into your product idea, you can count on that senior leadership to convince their developers to use your product within the bank. It is a rare occurrence that your infrastructure software company will be able to fit cleanly into either of these models–bottoms-up or top-down. More often, there will be some bottoms-up usage, and some top-down buy-in for your product. But you will have to evangelize the product on all fronts. You will have to convince both the engineers and the senior leadership. Your product probably won’t speak for itself. You will have to develop expertise in sales, marketing, and consultancy. And in many cases, you might end up in an unending chasm. The unending chasm describes a mode in which an infrastructure company must function as both a product company and a consultancy. Your consultancy is necessary to integrate your product into the enterprise, and ensure that your software actually gets used. But it reduces the appealing economics of a pure software company. The unending chasm does not prevent you from being successful. Companies who have had very successful IPOs remain in the unending chasm. But it’s useful to know whether you are heading for an unending chasm–or if you are already in one. Martin Casado joins the show today for a discussion of product development, software engineering, and go-to-market strategy. To find all 900 of our episodes, including past episodes with a16z partners, check out the Software Engineering Daily app in the iOS and Android app stores. Whether or not you are a software engineer, we have lots of content about technology, business, and culture. In our app, you can also become a paid subscriber and get ad-free episodes–and you can have conversations with other members of the Software Engineering Daily community. Image Source

When TensorFlow came out of Google, the machine learning community converged around it. TensorFlow is a framework for building machine learning models, but the lifecycle of a machine learning model has a scope that is bigger than just creating a model. Machine learning developers also need to have a testing and deployment process for continuous delivery of models. The continuous delivery process for machine learning models is like the continuous delivery process for microservices, but can be more complicated. A developer testing a model on their local machine is working with a smaller data set than what they will have access to when it is deployed. A machine learning engineer needs to be conscious of versioning and auditability. Kubeflow is a machine learning toolkit for Kubernetes based on Google’s internal machine learning pipelines. Google open sourced Kubernetes and TensorFlow, and the projects have users AWS and Microsoft. David Aronchick is the head of open source machine learning strategy at Microsoft, and he joins the show to talk about the problems that Kubeflow solves for developers, and the evolving strategies for cloud providers. David was previously on the show when he worked at Google, and in this episode he provides some useful discussion about how open source software presents a great opportunity for the cloud providers to collaborate with each other in a positive sum relationship.

When a user interacts with an application to order a ride with a ridesharing app, the data for that user interaction is written to a “transactional” database. A transactional database is a database where specific rows need to be written to and read from quickly and consistently. Speed and consistency are important for applications like a user ordering a car, and riding around in that car, because the user’s client is frequently communicating with the database to update their session. Other applications of a transactional database would include a database that backs a messaging system, a banking application, or document editing software. The data from a transactional database is often reused in “analytic” databases. An analytic database can be used for performing large scale analysis, aggregations, averages, and other data science queries. The requirements for an analytic database are different from a transactional database because the data is not being used for an active user session. To fill the data in an analytic database, the transactional data gets copied from the transactional database in a process called ETL. The separation of the transaction data store from the analytic data store causes problems for data engineering. To address these problems, some newer databases combine transactional and analytic functionality in the same database. These databases are often called “NewSQL”. TiDB is an open source database built on RocksDB and Kubernetes. TiDB is widely used in China by high volume applications such as bike sharing and massively multiplayer online games. Kevin Xu works at PingCAP, a company built around TiDB. He joins the show to talk about modern databases, distributed systems, and the architecture for TiDB.

React, Vue, and Angular are the most popular frontend JavaScript frameworks. Each of these frameworks lets frontend developers build components. A component is a high level visual abstraction that is used to compose a user interface. Frontend development has moved towards component-driven-development. At a typical technology company, a designer will put together a design file of different user interface elements, and the frontend engineer will take those UI elements and program code that can render those designs as components. As organizations have started to reuse their components and share them across the organization, the efficiency of design and frontend engineering is improving. User interface is gaining more of an emphasis with organizations and new tools are allowing frontend engineers and designers to work together more productively. One of these tools is Storybook, a system for sharing components and the code that renders those components. Zoltan Olah joins the show to talk about Storybook, and his company Chroma. Chroma is building tools to allow design-driven teams to work more effectively. We talked about how the relationship of designers and frontend engineers has some resemblance to the relationship between “dev” and “ops” before the DevOps movement. There are some frictions in the process of moving between design and engineering implementation, and in talking to Zoltan, I got an understanding for how much the UI layer could improve through better tooling.

Netflix has thousands of service instances communicating with each other. When a Netflix client on a smartphone makes a request for a movie, that request hits Netflix’s backend, where the request is fulfilled by a chain of requests through different services. Services and clients communicate using several different interaction patterns. A service might send a single request and expect a single response. Or it might fire and forget, not expecting a response. A service also might send a single request and expect a stream of messages to be sent back over the network. In a highly interactive application like Netflix, there is a frequent use of “streams” of data. RSocket is a protocol that makes reactive streams easier to work with. Ryland Degnan is the CTO of Netifi, and he joins the show to discuss reactive streams and service-to-service networking. Ryland worked at Netflix on the Edge Platform team for four years, and he shares his experience working at Netflix, the challenges of networking at scale, and the company he is building around RSocket.

Prometheus is an open source monitoring system and time series database. Prometheus includes a multi-dimensional data model, a query language called PromQL, and a pull model for gathering metrics from your different services. Prometheus is widely used by large distributed systems deployments such as Kubernetes and Cloud Foundry. Prometheus gathers metrics from your services by periodically scraping those services. Those metrics get gathered, compressed, and stored onto disk for querying. But Prometheus is designed to store all of its records on one host in one set of files–which limits the scalability and availability of those metrics. Cortex is an open source project built to scale Prometheus. Cortex effectively shards Prometheus by parallelizing the “ingestion” and storage of Prometheus metrics. Cortex can take metrics from multiple Prometheus instances and store them across a distributed NoSQL database like DynamoDB, BigTable, or Cassandra. Bryan Boreham is an engineer at Weaveworks, where he works on deployment, observability, and monitoring tools for containers and microservices. He wrote much of the code for Cortex, and we met up at KubeCon North America to talk about the motivation for creating Cortex, the broader landscape of Kubernetes monitoring, and other approaches to scaling Prometheus.

When a developer provisions a cloud server, that server is called an “instance”. These instances can be used for running whatever workload a developer has, whether it is a web application, a database, or a set of containers. The cloud is cheap to get started on. New applications with few users can often be hosted on infrastructure that is less than $10 per month. But as an application grows in popularity, there is more demand for CPUs and storage. A company will start to buy more and more servers to scale up to the requirements of their growing user base. The costs of running infrastructure in the cloud will increase, and the company will start to look for ways to save money. One common method of saving money is to buy “spot instances”. A spot instance is an instance that is cheaper than “reserved instances” or “on-demand” instances. The reason that there are different instance types is because a giant cloud provider has a highly variable amount of work that is being demanded from that cloud provider. If you are in charge of AWS, you have to make sure that at any given time, you can give server resources to anyone that asks for it. Your data centers need to have physical machines that are ready to go at any time. This means that much of the time, you have server resources that are going unused. If you are a cloud provider, how can you get people to use your compute resources? You can make them cheaper. So a user can come along and buy your compute at the discounted “spot” price. But this presents a problem for the cloud provider. If you start to give away your compute at cheaper prices, and then the overall demand for your cloud resources go up once again, you are going to miss out on profits. As the cloud provider, you need to kick people off of your spot instances, so that you can take those same instances and sell them to people at the higher market prices. And this presents a problem for the user. If you buy a cheap spot instance, that instance is only available until the cloud provider decides to kick you off. You have a tradeoff between cost and availability of your instances. Because of this, spot instances are typically used only for workloads that are not mission critical–workloads that can afford to fail. Spotinst is a company that allows developers to deploy their workloads reliably onto spot instances. Spotinst works by detecting when a spot instance is going to be reclaimed by a cloud provider and re-scheduling the workload from that cloud provider onto a new spot instance. Amiram Shachar is the CEO of Spotinst. He joins the show to talk about the different types of instances across cloud providers, the engineering behind Spotinst, and how the usage of containers and the rise of Kubernetes is changing the business landscape of the cloud.

If a business has been operating successfully for a few years, that business has accumulated a high volume of data. That data exists in spreadsheets, CSV files, log files, and balance sheets. Data might be spread across local files on a user’s laptop, databases in the cloud, or storage systems in an on-premise data center. Older businesses have more data, in more places, in more formats. Legacy systems and old batch processing jobs that have been running for years are taking data from one place and porting it to another. Every mature company needs to access and analyze the data in all of these different places–whether they are a publication with millions of readers like The Economist or a fast growing infrastructure provider like Twilio. “Business intelligence” is a term often used to describe tools for analyzing data in the form of charts, graphs, and reports. Business intelligence applications are crucial to the success of a business because they are used by everyone in an organization–whether you are a business analyst forecasting sales for the next quarter, an engineer who is determining how many servers to provision, or a CEO trying to decide what the best area of your business to focus on is. There have been several generations of business intelligence tools. Each generation of business intelligence is built for the trends and infrastructure of that generation. Looker is a more recent business intelligence tool that was built in light of several trends in software: the growth in volume of data; the growth in the number of systems that users need; the changing types of users that need to access data; and the need to share business intelligence across social workplace tools like Slack, Asana, and email. Daniel Mintz joins the show to describe his experience using business intelligence tooling and his work at Looker, as well as the landscape of business intelligence, ETL, and data engineering.

Robots are making their way into every area of our lives. Security robots roll around industrial parks at night, monitoring the area for intruders. Amazon robots tirelessly move packages around in warehouses, reducing the time and cost of logistics. Self-driving cars have become a ubiquitous presence in cities like San Francisco. For a hacker in a dorm room, or a researcher in a small lab, how do you get started with robotics? There are drones and other small options like AWS DeepRacer–but what is the equivalent of the Raspberry Pi for large, human-sized robots? Zach Allen is the founder of Slate Robotics, a company that makes large, human-sized robots that are at a low enough cost to be accessible to tinkerers, researchers, and prototype builders. Zach joins the show to talk about the state of robotics and why he started a robot company. What Zach is doing is quite hard–he is a solo founder who has bootstrapped a robotics company from scratch. He is set up in a strip mall in Missouri, where he has set up a row of 3-D printers to create the parts for his robots. He programs and assembles these robots himself. Whether you are interested in robots are thinking about starting a hardware company, this episode could be useful to you.

Netflix has petabytes of data and thousands of workloads running across that data every day. These workloads generate movie recommendations for users, create dashboards for data analysts to study, and reshape data in ETL jobs, to make it more accessible across the organization. Over the last ten years, data engineering has become a key component of what makes Netflix successful. There are many different engineering roles who interact with the data infrastructure–including data analyst, machine learning scientist, analytics engineer, and software engineer. Data engineering at Netflix has come a long way from the days of Hadoop MapReduce jobs running nightly, and generating reports of the most popular movies. As data engineering and data science has grown, the tooling has expanded. The people in different data roles at Netflix might use Apache Spark, Presto, Python, Scala, SQL, and many other applications to study data–but in recent years, there is one tool that has stood out for its ability to be distinctly useful: Jupyter Notebooks. A Jupyter Notebook lets users create and share documents that contain live code, visualizations, documentation, and many other types of components. In some ways, it is like a shareable IDE, that allows other people to see how you are working with your code and why you are making certain decisions. It is also a tool for building interactive, user-friendly applications–you can embed videos and images in a Jupyter notebook. A Jupyter Notebook stores both the code and the results together in one place. By combining code with results in one document, you can have context around why a certain result came out the way it did. Matthew Seal is a senior software engineer at Netflix, where he builds infrastructure and internal tools around Jupyter Notebooks. He joins the show to explain what problems Jupyter Notebooks solve for Netflix, and why they have quickly grown in popularity within the company. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Chinese Internet companies operate at a massive scale. WeChat has over a billion users and is widely used as the primary means of payment by urban Chinese consumers. Alibaba ships 12 million packages per day, which is four times the amount of Amazon. JD.com, a Chinese ecommerce company, has perhaps the largest production Kubernetes installation in the world. China’s rapid adoption of Internet services, combined with a large population and a growing middle class has led to the creation of Internet giants on par with the social networks, ecommerce sites, and ridesharing startups of the United States. Last November, I attended the first KubeCon China and saw firsthand how the Chinese Internet companies are using open source software to scale their infrastructure. Despite the differences between the US and China, the culture of technologists at KubeCon felt familiar. In some ways, it was just like any other Kubernetes conference that I have attended: large numbers of engineers trying to find the cutting edge of technology, and learning how to solve the problems they are facing back at the office. There were presentations on scaling databases and service meshes and machine learning on Kubernetes. Outside of these presentation halls, there were tables where you could pick up a translation device so that Chinese-only and English-only presentations could be understood by the other nationality. Dan Kohn joins the show to talk about Chinese Internet companies and how they are adopting Kubernetes. Dan is the executive director of the Cloud Native Computing Foundation, an organization within the Linux Foundation that organizes KubeCon. Before joining the CNCF, Dan worked as an entrepreneur, engineer, and executive at several technology companies.

Amazon Web Services changed how software engineers work. Before AWS, it was common for startups to purchase their own physical servers. AWS made server resources as accessible as an API request, and has gone on to create higher-level abstractions for building applications. For the first few years of AWS, the abstractions were familiar. S3 provided distributed, reliable object storage. Elastic MapReduce provided a managed Hadoop system. Kinesis provided a scalable queue. Amazon provided developers with managed alternatives to complicated open source software. More recently, AWS has started to release products that are unlike anything else. A perfect example is AWS Lambda, the first function-as-a-service platform. Other newer AWS products include Ground Station, a service for processing satellite data and AWS DeepRacer, a miniature race car for developers to build and test machine learning algorithms on. As AWS has grown into new categories, the blog announcements of new services and features have started coming so frequently that it is hard to keep track of it all. Corey Quinn is the author of “Last Week in AWS”, a popular newsletter about what is changing across Amazon Web Services. Corey joins the show to give his perspective on the growing, shifting behemoth that is Amazon Web Services–as well as the other major cloud providers that have risen to prominence. He’s also the host of the Screaming in the Cloud podcast, which you should check out if you like this episode.

Serverless computing is a technique for deploying applications without an addressable server. A serverless application is running on servers, but the developer does not have access to the server in the traditional sense. The developer is not dealing with IP addresses and configuring instances of their different services to be able to scale. Just as higher level languages like C abstracted away the necessity of a developer to work with assembly code, serverless computing gives a developer more leverage by letting them focus on business logic while a serverless platform takes care of deployment, uptime, autoscaling, and other aspects of cloud computing that are fundamental to every application. “Serverless” can several different things: backend-as-a-service products like Firebase, functions-as-a-service like AWS Lambda, and high-level APIs such as Twilio. Zeit is a deployment platform built for serverless development. In Zeit, users model a GitHub repository in terms of the functions within their application. Zeit deploys the code from those functions onto functions-as-a-service and allows you to run your code across all the major cloud providers. Guillermo Rauch is the founder of Zeit, and he joins the show to discuss his vision for the company and the platform as it looks today. Guillermo was previously on the show to discuss Socket.io, which he created.

Functions-as-a-service allow developers to run their code in a “serverless” environment. A developer can provide a function to a cloud provider and the code for that function will be scheduled onto a container and executed whenever an event triggers that function. An “event” can mean many different things. It is a signal that something has changed within your application. When you save a file to an Amazon S3 bucket, that creates an event. When a user signs up for your app, that can create an event. Functions-as-a-service are allowing people to build applications completely out of managed cloud infrastructure. Apps can be fully “serverless”, with managed databases, queueing systems, and APIs tied together by event-triggered functions. Today, there is not a consistent format for events across different applications and cloud providers. This makes it more difficult to stitch together events across these different environments. Ideally, events would be lightweight, easy to deserialize, and easy to interoperate with. The Cloud Events specification is a project within the Cloud Native Computing Foundation with the goal of creating a standard format for events. Doug Davis is the CTO for developer advocacy of containers at Microsoft. He joins the show to discuss how events and event-based programming works, and the need for a common format across cloud events.

Most applications today are either deployed to on-premise environments or deployed to a single cloud provider. Developers who are deploying on-prem struggle to set up complicated open source tools like Kafka and Hadoop. Developers who are deploying to a cloud provider tend to stay within that specific cloud provider, because moving between different clouds and integrating services across clouds adds complexity. Ben Hindman started the Apache Mesos project when he was working in the Berkeley AMPLab. Mesos is a scheduler for resources in a distributed system, allowing compute and storage to be scheduled onto jobs that can use those resources. In his time at the AMPLab, Ben collaborated with Matei Zaharia, creator of Apache Spark. Ben founded Mesosphere based off of his work on Apache Mesos, and since 2013 he has been building a company to bring it to market. In the meantime, several market forces have influenced the enterprise market. Enterprise businesses built on virtual machines and on-prem hardware are trying to migrate to containers, Kubernetes, and Spark. Cloud providers like Google and Microsoft have risen to prominence in addition to Amazon’s continued growth, and enterprises are increasingly willing to adopt multiple clouds. I spoke with Ben Hindman at Kubecon North America. Today, the company that he co-founded works to provide tools for managing these changes in infrastructure. In our conversation, we talked about the necessary mindset shifts for taking a research project and turning it into a highly successful product. We also talked about the newer trends in infrastructure–why enterprises will want multicloud deployments and how serverless APIs and backends will make the lives of developers much easier.