Software Engineering Daily

Facebook was built using PHP, a programming language that was used widely in the late 90s and early 2000s. PHP allows developers to get web applications built quickly and easily, although PHP has a reputation for being difficult to scale. In the early days of Facebook, the company was scaling rapidly on every dimension. New users were piling into Facebook. Existing users were increasing their interactions and developing new patterns of usage. The Facebook application was rolling out new features quickly, adding them into the Facebook PHP codebase. A common pattern for scaling a large software application is to use a microservices architecture, breaking up the monolithic application into small services which can scale independently. For many applications, this pattern works well. But for some applications, microservices makes less sense. Microsoft Excel is one example. In Excel, a user is making updates to a complex data model using formulas, functions, and other in-app tools that need to be fast, performant, and integrated. The user needs to have a sense that the Excel data model will update quickly in response to changes. A software team working on a spreadsheet product such as Excel might prefer to keep all the application logic in a monolithic application. A monolith can centralize logic and make it easier to reason about. A monolith can reduce the number of network hops, cutting down on distributed systems problems. Testing and deploying a monolithic application can be less complex than doing so in a distributed, microservices system. Facebook chose to scale its PHP monolith rather than breaking it up into distributed microservices. Scaling PHP allowed Facebook to continue moving fast without going through a painful refactoring that would have slowed down the entire company. The first effort to scale PHP involved transpiling the entire PHP application into C++. This C++ version of Facebook ran faster and with a lower memory footprint. But C++ required ahead-of-time compilation: the PHP codebase had to be converted to C++ in one synchronous step. The Hip Hop Virtual Machine (HHVM) was started to allow the Facebook PHP monolith to run on the V8 JavaScript virtual machine. The advantage of running PHP on V8 was that it allowed Facebook to transpile PHP to JavaScript, an interpreted language that fit better with Facebook’s dynamic application build path. HHVM also allows Facebook to take advantage of the rapidly improving JavaScript runtime environment. Keith Adams was an engineer at Facebook for six years, where he helped develop infrastructure to scale PHP effectively. Keith is now the chief architect at Slack, which is also a scaled PHP application. Keith returns to Software Engineering Daily to discuss why and how Facebook scaled PHP.

Data engineering allows a company to take advantage of the large quantities of data that the company has generated. In many companies, new data has been produced rapidly for many years, but the company has not been able to take full advantage of it. Creating large data sets does not provide immediate value for a company. A company needs to perform data engineering and data science to take full advantage of it. When data gets generated, it is stored in a database, data lake, or API backend like Google Analytics. In order to manipulate that data, it is often pulled into a data warehouse. A data warehouse provides fast access time to large quantities of data. Pulling data from a source like a database or data lake into a data warehouse requires a process known as extract and load. Once the data is in the data warehouse, it may also undergo a transform, which enriches the data or puts it in a format that is easier to make use of. Once data is in a data warehouse, it can be used to build models, interactive dashboards, and Jupyter Notebooks. The data engineering lifecycle has many different components, which is why data engineering can often be intimidating to a company that is trying to make use of their data. Meltano is a project with the goal of providing a system of conventions for managing the data engineering lifecycle. Meltano was started by GitLab, and the Meltano project has some strategic similarities to GitLab. Danielle Morill is the general manager of Meltano at GitLab. She joins the show to discuss the world of data engineering, and the architecture of Meltano. We touch on the different components of a data engineering pipeline, and the most acute pain points for data engineers.

Software engineers have a wide variety of media to choose from, including podcasts, blogs, YouTube videos, conferences. The amount of software engineering media that is available is growing and accelerating. Eight years ago, there were not as many options for information about software. Charles Max Wood founded Devchat.tv to create a network of podcasts and other content for software engineers. Today, his podcasts include the popular shows JavaScript Jabber, Ruby Rogues, and Adventures in Angular. Chuck joins the show for a conversation about software media. This was his second time on the show, with his first episode exploring his podcast JavaScript Jabber.

Software applications running within a host operating system need to be isolated. Isolation prevents security vulnerabilities, such as one application accessing the memory of another. In modern cloud environments, a single physical host might be running multiple virtual machines on top of a hypervisor. Those virtual machines might be divided up into containers. The different virtual machines and containers might be operated by different users, or even different companies. gVisor is a container sandbox runtime open sourced by Google. gVisor runs containers in a new user-space kernel, and provides a container security system with low overhead. gVisor improves on the previous security properties of containers. Michael Pratt and Yoshi Tamura work on gVisor at Google, and they join the show to talk through the purpose of gVisor and the engineering around the project.

Creating a software company has never been easier. Software engineers are increasingly seeing entrepreneurship as a viable career path. But the path to being an independent software developer is not always clear. Most engineers spend some of their career working at a software company. Even an engineer who intends to build a company someday can thrive within the right environment. Lior Kanfi is a software engineer and the founder of Tikal, a company he started at the height of the dot com bubble in 1999. His initial vision for the company was to build a product around managing knowledge and people within a company. When the dot com bubble burst in the year 2000, it became much harder to run a product-focused business. Companies were not buying lots of experimental software, and investors were not aggressively funding new software companies. After the market collapsed, Lior shifted Tikal from a product-focused company into consulting, in order to have a more reliable income stream. Today, Tikal is a successful software consultancy based in Israel and the Bay Area. Lior describes the engineers within Tikal as “free radical” software developers: independent people who want to learn about new technologies and build experience interacting with clients. Lior joins the show to talk about his 20 year journey building Tikal and the differences between engineers in the Bay Area and those in Israel. Lior also hosted the Full Stack Tech Radar Day in Tel Aviv, which was a great conference I attended.

Edge computing allows for faster data access and computation. When your client application makes a request, that request might be routed to the edge. Edge servers are more numerous and more widely distributed than normal data centers, but an edge server might not have all of the data or the complete application logic for the backend to serve your request. Edge servers have historically been used for content delivery networks (CDN). CDNs are useful for hosting and serving media files that might otherwise be slow to access over a network. More recently, applications are also using edge servers for computation, as well as storage of resources which are smaller than the movies, music, and images that have traditionally been stored on an edge server. Steve Klabnik is an engineer with Cloudflare, and he returns to the show to discuss storage at the edge. In Steve’s previous appearances we have explored Rust and WebAssembly, and we also touch on those topics in today’s episode.

Today’s episode is a keynote I gave at Full Stack Tech Radar Day in Tel Aviv. The talk is called “You Are Not a Commodity”. This talk is also available as a YouTube video. The slides can be accessed here. The world of commodity engineering is coming to an end. Developers are becoming more productive, more flexible, and more entrepreneurial. How does this impact you as an engineer? And how does it impact large companies who need a fungible set of engineers to maintain their software?

ListenNotes allows users to find podcasts by categories, popularity, and search queries. ListenNotes is not a podcast client that you download onto your phone, but it allows users to build playlists. These playlists are automatically published to an RSS feed, so that users can curate a playlist on desktop and subscribe to that playlist from any client. Podcasts are growing in popularity. There are more podcasts than ever before, and a podcast search engine needs to refresh its index regularly enough to capture the updates to the podcast universe. A podcast search engine needs to decide what to index. There are many potential fields to choose from within a podcast: the podcast title, the description, the audio transcription, and other metadata–should all of these factor into the results of a search engine? Wenbin Fang is the founder of ListenNotes, and the engineer who has built most of the application. He joins the show to talk about the world of podcasting and his work on ListenNotes, including the business model. Today, ListenNotes makes its money from advertisements and API requests–there is a growing market for applications that want an API for querying the ListenNotes backend.

Open source software allows developers to take code from the Internet and modify it for their own use. Open source has allowed innovation to occur on a massive scale. Today, open source software powers our consumer client applications and our backend cloud server infrastructure. Linux powers single node operating systems and Kubernetes is the foundation for new distributed systems. Hadoop created an open source distributed file system, Spark gave us a computational runtime on top of it, and Kafka created a middleware platform for shuttling data from one place to another. There are numerous other examples of how open source has changed the world of software development. Open source has also reshaped the business landscape of infrastructure software companies. A common business structure for a modern infrastructure company is the “open core” model. An open core company maintains an open source project that is free to use, but also sells a product or service around that product. Companies with an open core model include Red Hat, HashiCorp, and GitLab. Many companies are building a thriving business with the open core business model. But these companies do not directly control the most important part of the infrastructure supply chain: the cloud provider. Cloud providers have a fundamental tension with open core companies because the cloud providers offer services that compete with the open core companies. In addition to the issue of cloud providers competing directly with the open core companies, some people have questioned whether Amazon Web Services is capturing an unfair portion of the value that is being created by open source. Amazon Web Services is the biggest cloud provider, and it has built a large catalog of services that are built off of open source software. But AWS has not historically contributed heavily to open source relative to the value it has captured. One example of an open core company which has lost market share to an AWS cloud-hosted offering is Elastic, the open core company which maintains the ElasticSearch open source project. Amazon ElasticSearch Service is a closed-source hosted offering built on top of the ElasticSearch. Elastic (the company) has increasingly intermingled proprietary software with their open source repository, making it less clear how that open source repository can be used by companies that want to deploy it for their commercial use. Open core companies such as MongoDB, Redis Labs, and Cockroach Labs have responded to the competitive pressures of AWS by changing their licenses and making it more expensive for cloud providers to offer a cloud-hosted offering of their open source project. The dynamics between cloud providers and open core companies will continue to evolve in the coming years. The norms around open source are up for debate. Joseph Jacks is the founder of OSS Capital, a venture firm focused on investments in commercial open source software companies. He returns to the show to discuss the changing landscape of open core companies, and the benefits of permissionless innovation.

Google Earth allows users to explore the imagery of the real world. Imagery for Google Earth is taken from satellites, cars equipped with cameras, and other sources. Google Earth renders a data intensive 3-D model of the world on a client application such as a desktop browser or virtual reality system. WebAssembly is a runtime for executing code other than JavaScript in a browser-based environment. WebAssembly is useful for data-intensive workloads, and developers can use programming languages such as Rust or C++ in the browser by compiling to WebAssembly. Jordon Mears works on Google Earth, and he joins the show to talk about the engineering behind Google Earth and how WebAssembly is being used to improve efficiency. He also discusses the state of tooling around WebAssembly today.

FoundationDB is a multi-model distributed key-value store. It is fully ACID compliant and horizontally scalable. FoundationDB is not usually used directly by an application developer–FoundationDB is a foundational building block for higher level distributed systems such as the metadata store for data warehousing tool Snowflake. Ryan Worl is a software engineer who specializes in FoundationDB. He joins the show to discuss the architecture of FoundationDB, including the roles of different server components and the read and write path of FoundationDB. We also talk about applications of FoundationDB, and how it compares to storage engines such as RocksDB and databases such as CockroachDB and Spanner.

FindCollabs is a platform for finding collaborators and building projects. Three months ago we had our first hackathon, with lots of projects being created and collaborated on. In an earlier episode, we showcased the first place winner ARhythm. Today’s show features two more interviews with winners from the first FindCollabs hackathon. Kitspace is an open source hardware registry, and Rivaly is an app for informal ranked leagues, such as ping pong games at work, or board game clubs among your friends. The second FindCollabs hackathon is going on today. Check it out by going to findcollabs.com/open.

A new database company needs to solve numerous problems in order to succeed. There are already lots of existing database companies, so a new company needs to find a way to strongly differentiate itself. Databases are core infrastructure, so a new database company must earn trust with its customers. A database is a complicated distributed system, so a database company must have strong engineering to get its product to market. Citus Data was founded in 2011. Around that time, companies were looking for solutions to both their online transactional workloads and their large-scale offline analytics workloads. Citus Data built its company around Postgres, a popular database that has been around since 1996. Products from Citus Data include a Postgres scalability extension as well as a cloud-hosted offering for spinning up Postgres instances. Citus Data was acquired by Microsoft earlier this year. In today’s episode, Umur Cubukcu joins the show to talk about the story of the company from its early days to its eventual acquisition. Umur describes the landscape of data systems in 2011 when the company started, and explains how that evolved to the current ecosystem. Umur also talks about how to make an acquisition successful and gives some perspective on the future of data platforms.

Programmers are in high demand, and software engineering is a career path that is fun, creative, and lucrative. There are many people who want to transition into a career in software and are looking for the right path toward writing code. The traditional college computer science curriculum teaches some software engineering skills, but the time and financial cost of attending a university is prohibitive for many people who want to learn to code. Over the past decade, there have been several new models for software education. Online video platforms such as Udacity and Coursera put computer science courses online to be watched at the viewer’s convenience. Online schools such as Free Code Camp allow someone to learn how to program without any experience, and with no financial payment. Boot camps with income-sharing agreements such as App Academy create an in-person education environment that mimics a university, but with better cost structure and incentive alignment. Lambda School is an education system that takes elements of other software education models and combines them with newer SaaS technologies such as Slack and Zoom videoconferencing. Lambda School is an online software engineering curriculum with an income sharing agreement. “Income sharing agreement” means that the student does not pay for their education until they get a job. With this model, the student can pay back Lambda School after their software engineering education gets them a high-paying software engineering job. Andrew Madsen works at Lambda School, and he joins the show to describe the path that a student takes through Lambda School, the school’s curriculum for software education, and how Lambda School differs from the other options for coding education.

A grocery store contains fruit, vegetables, meat, bread, and other items that can expire. In order to keep these items in stock, the store must be aware of how much food has been sold and what has gone bad. When a food item is low in stock, the store needs to order more of that food from a central distribution system. Managing food inventory is not simple. Some kinds of meat might expire faster than others. Avocados do not become ripe at the same rate as apples. In order to keep the shelves stocked, there are manual workflows for checking the inventory and ordering new inventory. Afresh is a company that builds software for grocery stores. Afresh works with grocery chains that have a central distribution center. These grocery stores already have some software. At the back of the store, inventory management systems maintain records of the items that the store has on the shelves. At the front of the store, checkout systems detect what has been sold and help to update inventory. When the inventory is running low, the store can order more inventory from the central distribution center, so that trucks can deliver more inventory. Afresh improves the operational intelligence of these stores by detecting spoilage among items that are prone to expiration, such as fruit. Volodomyr Kuleshov is the CTO and co-founder of Afresh and he joins the show to discuss the technical challenges of a grocery store, and the software that Afresh is building to make groceries more intelligent.

Apache Airflow is a system for scheduling and monitoring workflows for data engineering. Airflow can be used to schedule ETL jobs, machine learning work, and script execution. Airflow also gives a developer a high level view into the graph of dependencies for their data pipelines. Chaim Turkel is a backend data architect at Tikal. He joins the show to discuss a case study of using Airflow to rearchitect the data engineering workflow of a complex financial application. We discussed the problems that Airflow solves and the process of porting existing workflows to Airflow.

Software projects are organized and planned using project management software. Examples of project management software include JIRA, Trello, and Asana. There are hundreds of tools for managing a software project because there are infinite ways that a project could be managed. Google Docs changed project management by allowing documents to be easier to share and collaborate on. Newer SaaS tools such as Slack, Dropbox, and Notion have taken the design lessons from social networking apps to make enterprise software more engaging. As the tools improve, our project management strategies change, and new software tools emerge to fit those new management strategies. Kurt Schrader is the CEO of Clubhouse, a project management tool for software engineers. Kurt joins the show to talk about the history and future of project management tools. He also discusses the engineering challenges of improving performance on a complicated webapp. Project management tools often have to load hundreds of small objects on a page, which required performance optimizations in the Clubhouse frontend JavaScript library.

Niantic is the company behind Pokemon Go, an augmented reality game where users walk around in the real world and catch Pokemon which appear on their screen. The idea for augmented reality has existed for a long time. But the technology to bring augmented reality to the mass market has appeared only recently. Improved mobile technology makes it possible for a smartphone to display rendered 3-D images over a video stream without running out of battery. Ingress was the first game to come out of Niantic, followed by Pokemon Go, but there are other games on the way. Niantic is also working on the Niantic Real World platform, a “planet-scale” AR platform that will allow independent developers to build multiplayer augmented reality experiences that are as dynamic and entertaining as Pokemon Go. Paul Franceus is an engineer at Niantic, and he joins the show to describe his experience building and launching Pokemon Go, as well as abstracting the technology from Pokemon Go and opening up the Niantic Real World platform to developers. Niantic developer contest finalists

WebAssembly allows for web-based execution of languages other than JavaScript. Programs written in Rust or C++ can be compiled down to WebAssembly and shipped over the browser for on-the-fly execution in a safe, memory controlled environment. WebAssembly has been in development for more than two years, and is still an immature ecosystem because building the necessary tooling for WebAssembly is hard. Much of the web has been built around JavaScript and the V8 JavaScript engine, which has been tuned to optimize an interpreted language (JavaScript). WebAssembly modules are often written in C++ or Rust, which are compiled languages. There are engineering challenges at the edge between the interpreted JavaScript runtime and the precompiled WebAssembly modules. Till Schneidereit is a senior research engineering manager at Mozilla. He joins the show to discuss the compilation path of WebAssembly and the state of the ecosystem.

Sheng Liang was the lead developer on the original Java Virtual Machine. Today he works as the CEO of Rancher Labs, a company building a platform on top of Kubernetes. Sheng joins the show to discuss his experiences in the technology industry. The container orchestration wars had many victims. The competing standards for how an enterprise should manage its numerous containers caused several companies to go down a path where they were building infrastructure which eventually had to be replaced. As Sheng discusses in today’s episode, the container orchestration wars almost killed his company. Rancher was originally built on top of a different container orchestrator, and the migration to Kubernetes required a massive rewrite of the Rancher platform. The container orchestration wars were not the first technology battle that Sheng has seen in his career–and it won’t be his last. In today’s show, we discuss the nature of technology wars. Are they necessary? How can a software company minimize the damage caused by a war between competing standards? Sheng was an excellent guest and we didn’t cover nearly as many subjects as I wanted to, so we will have to do another show in the future!

Kubernetes has made distributed systems easier to deploy and manage. As Kubernetes has become reliable, engineers have started to look for higher level abstractions we can define on top of Kubernetes. An operator is a method of packaging, deploying, and managing a Kubernetes application. Operators are useful for spinning up distributed systems such as Kafka, Redis, or MongoDB. These data systems are complicated, stateful applications with lots of failure domains. The operator framework enables a developer to deploy one of these complicated applications with less fear of the system crashing, or entering an erroneous state. Rob Szumski is an engineer at Red Hat. He joins the show to discuss Kubernetes, the operator pattern, and his time at CoreOS, which was acquired by Red Hat.

Cloud computing was popularized in 2006 with the launch of Amazon Web Services. AWS allowed developers to use remote server infrastructure with a simple set of APIs. But even with AWS, it was still not simple to deploy and manage a web application. In 2007, Heroku launched a platform built on top of AWS. Heroku focused on the developer experience by optimizing for users who were deploying Ruby on Rails applications. Since then, Heroku has expanded into other forms of managed infrastructure, including other application frameworks like NodeJS, and databases like Postgres. Heroku was the first popular “layer 2” cloud provider. Twelve years later, it is probably still the most popular. But there have been many other cloud providers built on top of AWS, including Netlify, Zeit, Spotinst, and Firebase. Layer 1 cloud providers are Google Cloud, AWS, Azure, Digital Ocean, and other raw infrastructure providers. These companies provide a great service in their low cost, commodity infrastructure. But the layer 1 providers are not optimizing for developer experience. They need to cater to a broad set of developers, some of whom want to work at a low level. A layer 2 cloud provider can build an opinionated solution that serves a subset of the overall cloud market particularly well. Render is a layer 2 cloud provider that optimizes for specific developer workflows, such as deploying a NodeJS web server, a static site, or a Docker container. Anurag Goel is the founder of Render, and he joins the show to discuss the strategy and the economics of Render. Anurag was also one of the early employees at Stripe, and he discusses his experience and learnings from working at the company.

Software engineering is an art and a science. To manage engineers is to manage artists and scientists. Software companies build practical tools like payment systems, messaging products, and search engines. Software tools are the underpinnings of our modern lives. You might expect this core infrastructure which modern humans rely on to have been constructed with pure formulaic rigor. But the best software tools are not built within a totally defined process. Software is built through messy iteration. When a piece of software looks pristine, that is often a function of how many mistakes have been made, and then subsequently corrected for. There is no fixed process for how to build good software. As our tools get better, we have to update our software engineering practices to utilize those new tools. We have to rethink the style that we are working in. We have to discard old tools and procedures in order to pick up the new ones, and have higher leverage. As an organization scales, the structure of the organization needs to be modified. Team members need to be reallocated. Checks and balances need to be put in place. Rules and cultural practices need to codified, because a larger organization cannot have ties broken by an individual. Software is built by humans, and every management decision must be considered in the light of human psychology. When we change a line of code, the code does not get emotional about being altered. But the same cannot be said of humans. Even a minor conversation between an engineering manager and a direct report can have lasting implications. Will Larson is the author of An Elegant Puzzle: Systems of Engineering Management. He works on Foundation Engineering at Stripe, and has worked in engineering management at Uber, Digg, and other software companies. Elegant Puzzle provides strategies, tactics, and ruminations on software development. Will joins the show to explore the multifaceted subject of engineering management.

Machine learning allows software to improve as that software consumes more data. Machine learning is a tool that every software engineer wants to be able to use. Because machine learning is so broadly applicable, software companies want to make the tools more accessible to the developers across the organization. There are many steps that an engineer must go through to use machine learning, and each additional step inhibits the chances that the engineer will actually get their model into production. An engineer who wants to build machine learning into their application needs access to data sets. They need to join those data sets, and load them into a machine (or multiple machines) where their model can be trained. Once the model is trained, the model needs to test on additional data to ensure quality. If the initial model quality is insufficient, the engineer might need to tweak the training parameters. Once a model is accurate enough, the engineer needs to deploy that model. After deployment, the model might need to be updated with new data later on. If the model is processing sensitive or financially relevant data, a provenance process might be necessary to allow for an audit trail of decisions that have been made by the model. Rob Story and Kelley Rivoire are engineers working on machine learning infrastructure at Stripe. After recognizing the difficulties that engineers faced in creating and deploying machine learning models, Stripe engineers built out Railyard, an API for machine learning workloads within the company. Rob and Kelley join the show to discuss data engineering and machine learning at Stripe, and their work on Railyard.

The software market changes every year. As individuals and enterprises become more willing to buy software, there are new markets for entrepreneurs to sell software to. Good software has high margins and high retention, so even a niche software business can prove profitable. As software spreads across the world, developing countries are showing a willingness to buy the same software tools as the developed countries, making the niche software businesses even more successful than expected. It’s a great time to start a software business. But the fundamental challenges faced by an entrepreneur have not gone away. An entrepreneur must find the necessary capital, build an initial product, iterate on that product, market to their audience, sell to their customers, and retain their users. Hiten Shah is an entrepreneur and investor who has started several businesses, both successful and unsuccessful. He is also the host of the Startup Chat podcast, a show about strategies and tactics for software businesses. Hiten joins the show to discuss his experience in the software industry, including lessons on financing, product development, and the future of software products.

Google Cloud was started with a vision of providing Google infrastructure to the masses. In 2008, it was not obvious that Google should become a cloud provider. Amazon Web Services was finding success among startups who needed on-demand infrastructure, but the traditional enterprise market was not yet ready to buy cloud resources. Googlers liked the idea of becoming a cloud provider. But was it the right time to enter the market? Google’s advertising business was a large and growing cash cow. Executives within Google were not sure how much capital and effort should be allocated into an infrastructure business. When Google decided to go into the cloud business, Joe Beda was one of the engineers who helped lead the effort, and joins the show as today’s guest. Google’s internal server infrastructure is managed by Borg, a system for allocating resources to applications. Google Cloud runs on Borg, and there were a number of early engineering challenges to building the necessary functionality into Borg for running a cloud provider on top of it. One example of a technical challenge that Google faced was the refactoring of Borg to run Google Cloud workloads. The requirements for public infrastructure are different than those of internal Googlers. Inside of Google, developers deploy their applications to containers running on bare metal. Outside of Google, developers want to create virtual machines. Borg needed to be refactored in order to instantiate VMs. Google solved this technical problem, as well as many other challenges, and Google Cloud slowly gained momentum in the market. But AWS remained the default choice for profitable enterprise workloads. It wasn’t until the container orchestration wars that Google found an opportunity to jump on a market segment that offered strong differentiation. By open sourcing Kubernetes and presenting a clear vision for where the project was going, Google shifted the battlefield of the public cloud toward a competitive landscape where it has many advantages. Kubernetes also provided many other technology companies with an opportunity to get into the cloud market, creating a collaborative, multi-company ecosystem that has accelerated the pace of software faster than anyone expected. Joe Beda has been instrumental in the evolution of the cloud native ecosystem. In today’s episode, Joe gives his memories on Google Cloud, Kubernetes, and his Kubernetes company Heptio, which he sold to VMware.

Containers are made to fail gracefully. When your container shuts down due to a hardware or software failure, your distributed application should be able to tolerate that failure. One simple way to be able to tolerate such a failure is to make all of your application logic “stateless.” If your application does not maintain state, then shutting it down in the middle of a computation is not a problem–you can just restart the application, restart your computation, and get the same result. But applications need to maintain state. We need to use databases and in-memory systems to manage long-lived user sessions and other interactions. A database is not just an on-disk abstraction–a database requires an application server to be accepting network traffic. We can run those database applications within containers. There is a fundamental tension between stateful applications and the idea that containers are meant to tolerate failure gracefully. Saad Ali is an engineer at Google, and he returns to the show to discuss Kubernetes storage and state management. He gave a keynote at KubeCon EU, which I spoke to him about. New Software Daily app for iOS. You can become a paid subscriber for ad free episodes at softwareengineeringdaily.com/subscribe

Amazon Web Services is the leading cloud provider by a large margin. Amazon established its lead by being first to market in 2006, with Google and Microsoft taking several years to catch up to the huge business opportunity of the cloud. Since 2008, Google Cloud has been working on cloud products for developers. It started with App Engine, which is widely used internally at Google, but has not had overwhelming public adoption. Over the last eleven years, Google has refined its understanding of how customers want to buy public cloud resources. Google Cloud products like Cloud Storage, Persistent Disk, and BigTable have given Google parity with many of the AWS public cloud offerings. Although Google has caught up to AWS in terms of products, the enterprise market has continued to choose AWS as its default. AWS is widely perceived as having more experience in running enterprise workloads, and a better responsiveness to customers. In order to keep Amazon from running away with the cloud market entirely, Google needed to shift the competitive landscape to different territory. Kubernetes provided the paradigm shift that Google needed. The market for cloud providers has changed completely due to Kubernetes. When Google open sourced Kubernetes, it created a common codebase for software companies to build software for managing distributed systems. In the span of five years, Kubernetes has turned the world of cloud products into a world resembling the open source Linux ecosystem. This is a remarkable shift, and every infrastructure software vendor is still figuring out its strategy for adapting. Adam Glick is the head of modern infrastructure and serverless marketing at Google. With Craig Box, he hosts the Kubernetes Podcast from Google, an excellent show about recent developments and evergreen concepts within the world of Kubernetes. Prior to Google, Adam worked at AWS for 3 years and Microsoft for twelve years. He has seen each of the major cloud providers up close and has a deep awareness for how each company thinks. We had a great conversation about the cloud native landscape, podcasting, and developer marketing. New Software Daily app for iOS. You can become a paid subscriber for ad free episodes at softwareengineeringdaily.com/subscribe

Containers offer a lightweight abstraction for running a server. Cloud providers are able to manage billions of containers from different users, allowing for economies of scale so that each user can pay less. Today, there is a variety of ways that users can deploy containers on a cloud provider. These containers can run in managed Kubernetes clusters, in functions-as-a-service, or in long-lived standalone container instances. User preferences are getting more sophisticated, with some users showing an interest in Knative, an open source serverless system originally created at Google. Whichever container deployment system you choose, your application and its multiple servers need a way to route traffic, measure telemetry, and configure security policy. A service mesh abstraction can help serve these use cases. Lachlan Evenson has worked in containers and Kubernetes since before the container orchestration wars. He was an engineer at Deis, a company which built an open source platform-as-a-service running on top of containers and Kubernetes. Deis was acquired by Microsoft, where Lachlan now works as principal program manager of Container Compute. Lachlan joins the show to discuss containers, Kubernetes, and the Service Mesh Interface, an interoperable service mesh layer that Microsoft launched with Buoyant.

Each cloud provider offers a different set of services which are not always compatible with each other. What are the challenges of building an application that interoperates with multiple different clouds? The first issue is API compatibility. Most cloud providers have a managed SQL offering, a bucket storage system, and server abstractions like virtual machines and containers. But these tools might have different APIs on each cloud. The code that you wrote to save your application data to Amazon S3 might have to be rewritten if you decide to switch your bucket storage to a different provider. Another issue that interferes with cloud interoperability is the degree of integration on a particular cloud. If I build my application for AWS, I might be heavily integrated with Amazon’s Identity and Access Management policy system and AWS logging. Each cloud provider makes it particularly easy to connect to their integrated solutions. There is also the problem of services on one cloud that simply do not map to a service on any other cloud. Google Cloud Bigtable does not have an equivalent service on Amazon. Microsoft CosmosDB does not have an equivalent service on Digital Ocean. As developers, this irritates us. We want to be able to deploy our application to any cloud. We want to be able to move applications easily from one cloud to another. And we want to mix tools from different clouds together as easily as we import a library. Bassam Tabbara is the CEO of Upbound, a company focused on making multicloud applications easier to deploy and operate. I spoke to Bassam at KubeCon EU 2019, and he described the problems of multicloud deployments and the opportunities for the cloud native ecosystem to become more cross-compatible.

Kubernetes has evolved from a nascent project within Google to a thriving ecosystem of cloud providers, open source projects, and engineers. Tim Hockin is a principal software engineer who has been with Google for 15 years. Tim joins the show to talk about the early days of the Kubernetes projects, and the engineering efforts that are under way five years into the project. At KubeCon EU 2019, two of the prevalent subjects of discussion were service mesh and serverless–particularly the Knative project. Tim gave his perspective for how projects that are adjacent to Kubernetes are developed within the community.

Google’s cloud business was long regarded as a place where startups could build a business, but not established enterprises. For serious workloads, enterprises chose Amazon almost unanimously. This phenomenon of Amazon as the default was described by a phrase that harkened back to the days of IBM’s dominance: “nobody ever got fired for choosing AWS.” With the rise of Kubernetes, Google has established a reputation as a reliable provider of container orchestration. As enterprises look to roll out Kubernetes workloads, Google is convincing many of them to work with Google Kubernetes Engine. Kubernetes is one part of the changes which fall under the description of “digital transformation”. As these enterprises build out their digital transformation strategy, they are also looking for a strategy around multicloud, on-prem, policy management, and consulting partnerships. Anthos is a platform where enterprises manage the resources and configuration of their cloud deployments, as well as a system to partner with services integrators and independent software vendors. Aparna Sinha works on Anthos, Google Kubernetes Engine, and other projects at Google. In today’s show we discuss the process of digital transformation, as well as the tactics for how a digital transformation is actually implemented. We also talked about GKE on-prem, and the kinds of tooling needed by on-prem application deployments.

The CNCF was formed under the looming shadow of AWS. The CNCF was seeded with the donation of Kubernetes by Google. Much like the Linux community was positioned as a rebellious movement in reaction to Microsoft’s dominance, the Kubernetes community represents a fervent desire to open up the market to cloud providers beyond the tight-lipped, proprietary dominion of Amazon. With such a deep spirit of insubordination, it is no surprise that the community has rejected Istio like a set of loosely coupled organs rejecting a foreign skin attempting to layer itself across them. Even though the CNCF was founded by Google, the community was formed in spite of big centralized clouds, not as a marketing vessel for their products which may or may not be open source. Microsoft seems to understand this fact better than Google, at least in the domain of service mesh. The day after this interview with William, Microsoft announced the Service Mesh Interface (SMI), a project it partnered with Buoyant and other companies on to create a minimal spec for what a service mesh should offer to a Kubernetes deployment. The SMI presents a safe buy-in point for enterprises who want a service mesh, but do not want to get caught in the evangelistic crossfire of Istio and Linkerd. It is in this environment that we begin our next series of shows on the current cloud native ecosystem. Thanks to the Cloud Native Computing Foundation for putting together an amazing podcasting zone at KubeCon, and allowing me to conduct these interviews.

Cloud computing, open source, and mobile computing are trends that affect every organization. When a large organization adapts to these trends, it is commonly referred to as a “digital transformation”. Digital transformation causes many companies to reframe their business as a software company. A candy manufacturer now must think of itself as a software company that makes candy. An insurance company must now think of itself as a software company that issues insurance. Capital One is a bank that was started in 1988. Capital One has always had an emphasis on software, but the company’s digital transformation has affected it as much any other company. The company is migrating to the cloud, building out microservices, rolling out continuous delivery pipelines, and shifting the internal culture to be more adept at using software. Hillary McTigue is a senior director of data engineering at Capital One. She joins the show to discuss her experience implementing a digital transformation within a large company. Subjects we discuss include culture, management strategy, and the sequencing of different phases of a digital transformation. Full disclosure: Capital One is a sponsor of Software Engineering Daily. We are hiring two interns for software engineering and business development! If you are interested in either position, send an email with your resume to [email protected] with “Internship” in the subject line.

Netflix started with a DVD-by-mail product. The software infrastructure and operations practices needed for the DVD business were very different from those needed by a streaming video company. Since the early days of Netflix, CEO Reed Hastings knew that the company would evolve to becoming a streaming video platform. But he did not know when the technology would be advanced enough to support video streaming, and he did not know how users would consume it. Greg Burrell has worked at Netflix for 14 years. Greg was one of the first engineers to start working on video streaming, which Netflix first attempted to implement with a set top box that downloaded movies and played them on your television. After evolving this strategy, Netflix arrived at the current model of video streaming through apps on browsers and mobile devices. As the company pivoted from DVD-by-mail to video streaming, Netflix encountered multiple challenges across engineering, operations, and communications across the company. At the time, there was no “DevOps” movement. There were not established continuous delivery practices. The available cloud technologies were immature and low level. Greg joins the show to describe the evolutionary arc of Netflix’s engineering process. Greg also presents a model for software development that he describes as “Full Cycle Development”. At Netflix, engineering teams of full cycle developers work without dedicated operations or testing teams. It is a sophisticated approach to engineering management. I spoke to Greg at the Fullstack Tech Radar Day, a software conference in Tel-Aviv put on by Tikal, an engineering community based out of Israel and San Francisco. This was a great conference, and we’ll be airing some additional content from it in the coming weeks. We are hiring two interns for software engineering and business development! If you are interested in either position, send an email with your resume to [email protected] with “Internship” in the subject line.

Augmented reality applications can be used on smartphones and dedicated AR headsets. On smartphones, ARCore (Google) and ARKit (Apple) allow developers to build for the camera on a user’s smartphone. AR headsets such as Microsoft HoloLens and Magic Leap allow for a futuristic augmented reality headset experience. The most prominent use of augmented reality today is gaming, with a notable example being Niantic’s Pokemon Go. Tony Godar is a software engineer who works on augmented and virtual reality applications. He joins the show to talk about his day job working on virtual reality experiences, and an AR game he built called ARhythm. Tony was the winner of the FindCollabs Hackathon and we also discussed his experience working on the project through FindCollabs. RECENT UPDATES: The FindCollabs Open has started. It is our second FindCollabs hackathon, and we are giving away $2500 in prizes. The prizes will be awarded in categories such as machine learning, business plan, music, visual art, and JavaScript. If one of those areas sounds interesting to you, check out findcollabs.com/open! The FindCollabs Podcast is out! We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/

CamelCamelCamel is a tool for tracking prices on items on Amazon.com. The company was launched eleven years ago and is built off of the Amazon Product Advertising API. Daniel Green is one of the founders of CamelCamelCamel, and he joins the show to describe his experience building the product. Amazon and CamelCamelCamel (C3) have a complex relationship. C3 makes its money from referral listings. When a user tracks the price of an Amazon item using C3, that user will probably eventually click on the referral listing on C3. If the user purchases the item on Amazon, C3 gets a percentage of the purchase from Amazon. C3 has millions of product listings where they are tracking the price of items on Amazon. They have created a directory with a large subset of Amazon’s items by leveraging an API that was originally meant for advertising. Whether or not this is a proper use of the API, C3 arguably leads to more purchasing volume on Amazon–which is ostensibly why Amazon lets the company continue to operate. In today’s show, Daniel gives a history of CamelCamelCamel, including his own background as an engineer working in finance and how he wound up working on C3 for the past decade. We also talk through how C3 is architected. RECENT UPDATES: The FindCollabs Open has started. It is our second FindCollabs hackathon, and we are giving away $2500 in prizes. The prizes will be awarded in categories such as machine learning, business plan, music, visual art, and JavaScript. If one of those areas sounds interesting to you, check out findcollabs.com/open! The FindCollabs Podcast is out! We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/

Gaming is becoming mainstream. Popular multiplayer games such as Fortnite and Minecraft present players with a massive virtual world to explore, build, and compete within. Turn-based games such as Hearthstone and Magic are breeding a new generation of board game and card game aficionados. Social media networks like Twitch and YouTube have turned gaming into a voyeuristic sport that is outcompeting many physical sports games for attention. Guilded is a platform for managing gaming teams. On Guilded, there are teams for games like League of Legends, Fortnite, and World of Warcraft. These teams use Guilded to manage calendars, Discord bots, forum software, documents, statistics, and recruiting. This might sound confusing–why does a gaming team need document management, calendars, and analytics? Are we talking about a video game team or a software company? To understand Guilded, you need to understand the rapidly changing modern gaming ecosystem. Eli Brown is a founder of Guilded.gg. He joins the show to talk about the world of gaming, its intersection with social media, and the fascinating engineering problems involved in building a platform for gaming teams. RECENT UPDATES: The FindCollabs Open has started. It is our second FindCollabs hackathon, and we are giving away $2500 in prizes. The prizes will be awarded in categories such as machine learning, business plan, music, visual art, and JavaScript. If one of those areas sounds interesting to you, check out findcollabs.com/open! The FindCollabs Podcast is out! We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/

Google’s codebase is managed in a single monolithic repository. An engineer at Google can explore almost any area of the codebase within the entire company. In order to enable this, Google has built tooling to support the monolithic repo, including a virtual file system and a set of build tools. A monolithic repository is not to be confused with a monolithic deployment. Google’s infrastructure consists of thousands of small services interacting over a network, and scaling individually. But all of the code for each of these different independent modules is in the same version control system. Ciera Jaspan is a staff software engineer at Google working on developer infrastructure. She worked on an internal research project within Google to find out how engineers felt about the monolithic repository system and how it compared to a large number of small repositories. Ciera joins the show to discuss repository management, internal tooling, and Google’s approach to researching developer productivity within the company. RECENT UPDATES: The FindCollabs Open has started. It is our second FindCollabs hackathon, and we are giving away $2500 in prizes. The prizes will be awarded in categories such as machine learning, business plan, music, visual art, and JavaScript. If one of those areas sounds interesting to you, check out findcollabs.com/open! The FindCollabs Podcast is out! We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/

Alex Balazs is the Intuit Chief Architect and has been working at the company for almost twenty years. Intuit’s products include QuickBooks, TurboTax, and Mint. These applications are used to file taxes, manage business invoices, conduct personal accounting, and other critical aspects of a user’s financial life. Because the applications are managing money for users, there is not much room for error. When Intuit was started, the company made desktop software. In his time at Intuit, Alex played a key role in rearchitecting the monolithic desktop applications to be resilient, reliable web applications. Intuit originally managed this software on their own servers. Since then, Intuit has migrated to the cloud using AWS. Alex joins the show to discuss his experience scaling Intuit, his strategy for cloud migration, and his evaluation criteria for questions of build versus buy. RECENT UPDATES: The FindCollabs Open has started. It is our second FindCollabs hackathon, and we are giving away $2500 in prizes. The prizes will be awarded in categories such as machine learning, business plan, music, visual art, and JavaScript. If one of those areas sounds interesting to you, check out findcollabs.com/open! The FindCollabs Podcast is out! We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/

Africa is rapidly adopting the same software and hardware technologies that have transformed the western world over the last few decades. But access to computers and technology education is still uneven. Where there is access to computers, smartphone adoption often comes before access to laptops or desktop computers. Nelly Cheboi is the founder of TechLit Africa, an organization that works to connect schools and families in Africa with computers and software. Nelly studied computer science, and worked as a software engineer before leaving her career to focus full-time on building a scalable model to take refurbished computers and give them to Africans who can make good use of them. TechLit Africa is also building a software stack to equip schools in Africa without an Internet connection with an internal subnet including Wikipedia and other educational resources, so that people in the school can get an Internet-like experience despite a lack of access to the full Internet. RECENT UPDATES: The FindCollabs Open has started. It is our second FindCollabs hackathon, and we are giving away $2500 in prizes. The prizes will be awarded in categories such as machine learning, business plan, music, visual art, and JavaScript. If one of those areas sounds interesting to you, check out findcollabs.com/open! The FindCollabs Podcast is out! We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/

Facebook’s strategy is shaped by long term goals, short term requirements, and the available resources of the company. Long term goals are necessary for thinking through big decisions such as acquisitions, hardware product investments, and open source software ecosystems. To implement long term goals, Facebook needs to communicate the vision of the company and foster an internal culture that supports that vision. Short term requirements can affect how the company is thinking on a more immediate time horizon. When Facebook realized the importance of mobile computing, the mentality in the company quickly shifted from looking at mobile as a tax on engineering resources to a long-term source of business value. When Google started to work on Google+, Facebook engineers focused their resources on the potential competitive threat. Facebook’s strategy is implemented by the engineers, product managers, and other employees of the company. Facebook is unique in its ability to allow those employees to self-assemble into work that is meaningful to the individuals as well as to the company. As the long term goals and short term requirements of Facebook change over time, company resources are shifted to focus the company on the correct set of priorities. Some of those priorities might be speculative investments in new technologies. Other priorities might include doubling down on areas of the company that are showing promise. Mike Vernal worked as a VP of product and engineering at Facebook for 8 years. He left the company in 2016 and joined Sequoia Capital, where he now works as a partner. In his time at Facebook, he helped architect and implement strategies relating to product direction and engineering. Mike joins the show for a discussion about his time at Facebook and the strategic lessons that he learned from his time at the company. RECENT UPDATES: The FindCollabs Open has started. It is our second FindCollabs hackathon, and we are giving away $2500 in prizes. The prizes will be awarded in categories such as machine learning, business plan, music, visual art, and JavaScript. If one of those areas sounds interesting to you, check out findcollabs.com/open! The FindCollabs Podcast is out! We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/

React is a set of open source tools for building user interfaces. React was open sourced by Facebook, and includes libraries for creating interfaces on the web (ReactJS) and on mobile devices (React Native). React was released during a time when there was not a dominant frontend JavaScript library. Backbone, Angular, and other JavaScript frameworks were all popular, but there was not any consolidation across the frontend web development community. Before React came out, frontend developers were fractured into different communities for the different JavaScript frameworks. After Facebook open sourced React, web developers began to gravitate towards the framework for its one-way data flow and its unconventional style of putting JavaScript and HTML together in a format called JSX. As React has grown in popularity, the React ecosystem has developed network effects. In many cases, the easiest way to build a web application frontend is to compose together open source React components. After seeing the initial traction, Facebook invested heavily into React, creating entire teams within the company whose goal was to improve React. Dan Abramov works on the React team at Facebook and joins the show to talk about how the React project is managed and his vision for the project. RECENT UPDATES: The FindCollabs Open has started. It is our second FindCollabs hackathon, and we are giving away $2500 in prizes. The prizes will be awarded in categories such as machine learning, business plan, music, visual art, and JavaScript. If one of those areas sounds interesting to you, check out findcollabs.com/open! The FindCollabs Podcast is out! We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/

Facebook leadership was able to recognize the importance of mobile computing in time to develop high quality mobile applications, but there were numerous challenges. The Facebook desktop web app had been difficult enough to build due to the unprecedented data requirements and amount of interactivity. Mobile introduced the additional hurdles of limited bandwidth and distinct native operating systems in Android and iPhone. Facebook’s early efforts to build a mobile application involved a cross-platform HTML5 solution. HTML5 had insufficient performance for Facebook’s needs, and the company needed to develop native apps in order to deliver the desired experience. Facebook’s ability to pivot to mobile is comparable to the classic story of Intel pivoting from a memory company to a microprocessor company. To succeed at mobile application development, Facebook had to shift its focus dramatically, reallocating engineering resources and acqui-hiring small mobile companies in order to build up the domain expertise for mobile. As a side effect of this transition to mobile, Facebook developed an understanding of how dramatically software engineering was changed by the introduction of smartphones and the high bandwidth requirements of social networking. The challenges of this new paradigm led to the development of open source tools such as GraphQL and React Native, which have allowed countless projects to build applications more easily. Jocelyn Goldfein was an engineering director at Facebook for four years, from 2010 to 2014. She currently works as an investor at Zetta Venture Partners. In her time at Facebook, Jocelyn saw the shift to mobile firsthand. In today’s episode, she describes how Facebook management works, and gives her perspective on the distinguishing characteristics of the engineering organization as a whole.

Facebook is a case study in the ability for developers to self-organize into groups who are working on projects that are meaningful to the company and personally satisfying to the individual engineers. Many engineers in the software industry work under a less capable manager who has complete control over their creativity. This leads to employee churn, dissatisfaction, and burnout. Facebook’s ability to move fast is predicated on its ability to match engineers with problems that are interesting to those particular individuals. Whether you want to work on newsfeed or developer productivity tools or machine learning research, there is a path within Facebook to finding a problem that is both important and fun. Facebook’s unique set of engineering challenges required the company to develop a unique set of internal tools. Because Facebook had data and throughput requirements which were unprecedented, the available tools and best practices at the time did not satisfy Facebook’s requirements. Over the years, Facebook has developed its own databases, caching strategies, and JavaScript frameworks. Nick Schrock worked at Facebook for eight years. He is best known as a co-creator of GraphQL, a tool for efficiently fetching data through a federated request language. GraphQL was the result of years of evolution of internal tooling within Facebook. Nick has discussed the creation of GraphQL in other podcasts, and we will have a more dedicated episode around a retrospective of GraphQL in the near future. Today’s episode is about the process by which developers at Facebook self-organized, and Nick’s ideas around how to identify a need for an internal tool. Since leaving Facebook, Nick has parlayed his experience in developer tools into Dagster, a programming model for data applications.

Facebook engineering is commonly described by two words: move fast. Building products quickly has been a differentiating characteristic of the company since its inception. From the longtime engineers to the summer interns, Facebook instills a sense of immediacy and opportunity in all of its employees. The goal of Facebook is to make the world more open and transparent, with the intention of creating greater understanding and connection through Internet services. More than any other company in history, Facebook has enabled people to communicate with each other via simple user interfaces and real, authenticated human identity. Facebook must move fast, because the vision for Facebook is without precedent. It may feel like the Facebook mission is already finished, because you can already use Facebook to connect with anyone across the world with an Internet connection. But once you are connected to somebody on Facebook, there are only a small number of interactions you can take: sending a message, sharing a photo, broadcasting a video stream. There are so many more parts of our lives waiting to be digitized, and many of these require a real identity system to work properly. More than any other company, Facebook is positioned to expand our system of real-world human trust onto the Internet. The depth and breadth of the engineering problems required to accomplish this demands that Facebook move fast. To move slower would cause all of us to pay the opportunity cost of having to wait longer to interconnect our global society. Pete Hunt worked as an engineer at Facebook for three and a half years. At Facebook, he helped build React, a set of technologies that have significantly improved frontend application interface development. After the Instagram acquisition, Pete was the first engineer from Facebook to join the Instagram team to help bring the two companies together. Pete left Facebook in 2014 to start Smyte, a company that made trust and safety tools for marketplaces and social networks. Smyte was acquired by Twitter, where Pete now works on engineering problems relating to trust, safety, health, and infrastructure. Pete joins the show for the first of several episodes with Facebook engineers. In these episodes, we will explore the engineering practices of Facebook–from scaling Facebook’s PHP monolith to open sourcing React and GraphQL. Other topics will include management, onboarding, and product strategy. Our goal is to present a holistic picture of how Facebook engineering works, so that other organizations can learn to adopt practices that will allow them to move faster. We hope you enjoy this series on Facebook engineering. RECENT UPDATES: The FindCollabs Open has started. It is our second FindCollabs hackathon, and we are giving away $2500 in prizes. The prizes will be awarded in categories such as machine learning, business plan, music, visual art, and JavaScript. If one of those areas sounds interesting to you, check out findcollabs.com/open! The FindCollabs Podcast is out! We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/

Software engineering is harder than it should be. There are many people who have an app idea that they are not sure how to build. Some of these people are highly technical professionals like real estate agents, scientists, and accountants. These professionals learn to use spreadsheets in their day-to-day work. Spreadsheets are also used widely by young people such as students. Spreadsheet users vary in terms of how familiar they are with the programmability of a spreadsheet, but there are certainly more people who have built complex spreadsheets than there are people who have built complex web apps. Airtable is a tool for making application development easier and more accessible. The Airtable interface is similar to a spreadsheet and can be used for most spreadsheet applications. It can also serve as a rich backend database system to improve the productivity of software developers who are fully capable of building web applications. There are high-level programmable components called Blocks and integrations with developer APIs like Twilio and Stripe. Airtable has a permissions and collaboration system that allows interaction between engineers who might be using Airtable as a programmatic transactional database and operations members who might need to read or edit specific parts of the data on an ad hoc basis. Howie Liu is the CEO of Airtable and he joins the show to talk about his vision for the product and the engineering problems he is working on to realize that vision. Airtable has not been trivial to build, and has required its own custom database backend and its own JavaScript rendering system. Special thanks to Gareth Pronovost who is a full-time Airtable expert that I found on YouTube and who was generous enough to take some time to have a call with me and describe his experience using Airtable. The fact that there is a full profession around creating Airtable applications speaks to how unique this platform is. RECENT UPDATES: FindCollabs is a company I started recently The FindCollabs Podcast is out! FindCollabs is hiring a React developer FindCollabs Hackathon #1 has ended! Congrats to ARhythm, Kitspace, and Rivaly for winning 1st, 2nd, and 3rd place ($4,000, $1000, and a set of SE Daily hoodies, respectively). The most valuable feedback award and the most helpful community member award both go to Vynce Montgomery, who will receive both the SE Daily Towel and the SE Daily Old School Bucket Hat We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/ Podsheets is our open source set of tools for managing podcasts and podcast businesses New version of Software Daily, our app and ad-free subscription service

Relational data systems have evolved from single node instances to complex distributed systems. Almost any database can be accessed through a SQL statement, but the guarantees of these databases can vary in terms of consistency, availability, latency, durability, and financial cost. Relational database systems that explore these different sets of tradeoffs are sometimes categorized as “NewSQL”. There are also a wide variety of data systems that are not categorized as databases. Kafka is a distributed queue. HDFS is a distributed file system. Spark provides a distributed in-memory working set to process data. Cloud providers offer hosted bucket storage for your data lake and fast processing in the form of a data warehouse. Sunil Kamath is a principal PM with Microsoft. Sunil has worked on database systems for two decades, and he joins the show to give his perspective on the current data world and his predictions for how data platforms will become easier to use. Sunil is optimistic about the use of virtual data for unifying the access of data for a variety of operational use cases. RECENT UPDATES: FindCollabs is a company I started recently The FindCollabs Podcast is out! FindCollabs is hiring a React developer FindCollabs Hackathon #1 has ended! Congrats to ARhythm, Kitspace, and Rivaly for winning 1st, 2nd, and 3rd place ($4,000, $1000, and a set of SE Daily hoodies, respectively). The most valuable feedback award and the most helpful community member award both go to Vynce Montgomery, who will receive both the SE Daily Towel and the SE Daily Old School Bucket Hat We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/ Podsheets is our open source set of tools for managing podcasts and podcast businesses New version of Software Daily, our app and ad-free subscription service

WebAssembly is a binary instruction format for applications to run in a memory-constrained, stack-based virtual machine. The WebAssembly ecosystem consists of tools and projects that allow programs in a variety of languages to compile into WebAssembly and run in a safe, fast, sandboxed runtime environment. WebAssembly is a transformative technology for the Internet. Most users will experience it as a set of gradual, incremental improvements to their online experiences. Pages will load faster and become more dynamic. Applications will become more secure. Infrastructure will become cheaper, and those cost savings will eventually reach the consumer. For developers, WebAssembly opens a world of possibility. In today’s operating systems, the user can feel a big difference between applications that need a large client-side runtime (such as video editing tools, or render-heavy games such as Half Life) and applications that are more lightweight and can run entirely on the web (such as Twitter). Tyler McMullen is the CTO at Fastly. He joins the show to talk about the compilation path, the runtime, and the opportunities of WebAssembly. RECENT UPDATES: FindCollabs is a company I started recently The FindCollabs Podcast is out! FindCollabs is hiring a React developer FindCollabs Hackathon #1 has ended! Congrats to ARhythm, Kitspace, and Rivaly for winning 1st, 2nd, and 3rd place ($4,000, $1000, and a set of SE Daily hoodies, respectively). The most valuable feedback award and the most helpful community member award both go to Vynce Montgomery, who will receive both the SE Daily Towel and the SE Daily Old School Bucket Hat We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/ Podsheets is our open source set of tools for managing podcasts and podcast businesses New version of Software Daily, our app and ad-free subscription service

Modern server infrastructure usually runs in a virtualized environment. Virtual servers can exist inside of a container or inside of a virtual machine. Containers can also run on virtual machines. Kubernetes has allowed developers to manage their multiple containers, whether those containers are running in VMs or on bare metal (servers without VMs). As organizations expand their Kubernetes deployments, the overhead of those deployments is becoming a relevant concern. So-called “Kubesprawl” can occur within organizations due to a lack of best practices on when new clusters should be spun up or spun down, and when clusters should be shared by teams or shared by services. Paul Czarkowski is a principal technologist with Pivotal. He joins the show to discuss virtualization, Kubernetes, and the state of the cloud native ecosystem. RECENT UPDATES: FindCollabs is a company I started recently The FindCollabs Podcast is out! FindCollabs is hiring a React developer FindCollabs Hackathon #1 has ended! Congrats to ARhythm, Kitspace, and Rivaly for winning 1st, 2nd, and 3rd place ($4,000, $1000, and a set of SE Daily hoodies, respectively). The most valuable feedback award and the most helpful community member award both go to Vynce Montgomery, who will receive both the SE Daily Towel and the SE Daily Old School Bucket Hat We are booking sponsorships for Q3, find more details at https://softwareengineeringdaily.com/sponsor/ Podsheets is our open source set of tools for managing podcasts and podcast businesses New version of Software Daily, our app and ad-free subscription service