Software Engineering Daily

In a cloud infrastructure environment, failures happen regularly. The servers can fail, the network can fail, and software bugs can crash your software unexpectedly. The amount of failures that can occur in cloud infrastructure is one reason why storage is often separated from application logic. A developer can launch multiple instances of their application, with each instance providing a “stateless” environment for serving API requests. When the application needs to save state, it can make a call out to a managed cloud infrastructure product. Managed cloud databases provide a reliable place to manage application state. Managed object storage systems like Amazon S3 provide a reliable place to store files. The pattern of relying on remote cloud services does not work so well for on-prem and hybrid cloud environments. In these environments, companies are managing their own data centers and their own storage devices. As companies with on-prem infrastructure adopt Kubernetes, there is a need for ways to manage on-prem storage through Kubernetes. Saad Ali is a senior engineer at Google, where he works on Kubernetes. He is also a part of the Kubernetes Storage Special Interest Group. Saad joins the show talk about how Kubernetes interacts with storage, and how to manage stateful workloads on Kubernetes. We discuss the basics of Kubernetes storage, including persistent volumes and the container storage interface.

When a user makes a request to product like The New York Times, that request hits an API gateway. An API gateway is the entry point for an external request. An API gateway serves several purposes: authentication, security, routing, load balancing, and logging. API gateways have grown in popularity as applications have become more distributed, and companies offer a wider variety of services. If an API is public, and anyone can access it, you might need to apply rate limiting so that users cannot spam the API. If the API is private, the user needs to be authenticated before the request is fulfilled. Kong is a company that builds infrastructure for API management. The Kong API gateway is a widely used open source project, and Kong is a company built around supporting and building on top of the API gateway. Marco Palladino is the co-founder and CTO of Kong. He joins the show to tell the story of starting Kong eight years ago, and how the API gateway product evolved out of an API marketplace. Marco also discusses the architecture of Kong and his vision for how the product will develop in the future–including the Kong service mesh.

Augmented reality glasses will let us walk through a world where the digital blends together with the physical. 3-D objects will be rendered and superimposed onto our field of vision, creating an environment for people to build applications we can hardly dream of today. These augmented reality glasses are probably three to five years away from being ready for consumer use. But developers are already building augmented reality applications for smartphones using Apple ARKit and Android ARCore. These augmented reality toolkits use powerful smartphone processors and computer vision to give developers simple primitives for placing and manipulating 3-D objects. Most of these AR applications are made for a single phone, and AR is useful for a single phone–for example, you could hold your phone up in front of an empty room, and see on your phone how it would look if you had an IKEA couch sitting in the middle of that room. But shared augmented reality experiences are much more exciting. Shared augmented reality can allow us to play a game of virtual basketball, both controlling the game that is synchronized between us. Shared AR would let me go to a restaurant, and create a virtual billboard in front of the restaurant that only you could see when you walked up to the restaurant and held your phone in front of you. Ubiquity6 is a company with the goal of enabling shared AR experiences. Ankit Kumar is the co-founder and CTO of Ubiquity6, and he joins the show to explain why building shared AR is a challenging technical problem. It requires building a digital model of the real world, and mapping that model to coordinates in space, so that users can reliably persist augmented reality objects that each other can see. We discuss computer vision, digital mapping, the increasing power of phone processors, and the potential of shared AR.

Cloud providers created the ability for developers to easily deploy their applications to servers on data centers. In the early days of the cloud, most of the code that a developer wrote for their application could run on any cloud provider, whether it was Amazon, Google, or Microsoft. These cloud providers were giving developers the same Linux server that they would expect from an on-premise deployment. Early cloud applications such as Netflix, Airbnb, and Uber took advantage of this cloud infrastructure to quickly scale their businesses. In the process, these companies had to figure out how to manage open source distributed systems tools such as Hadoop and Kafka. Cloud servers were easy to create, but orchestrating them together to build distributed systems was still very hard. As the cloud providers matured, they developed higher level systems that solved many of the painful infrastructure problems. Managed databases, autoscaling queueing systems, machine learning APIs, and hundreds of other tools. Examples include Amazon Kinesis and Google BigQuery. These tools are invaluable because they allow a developer to quickly build applications on top of durable, resilient cloud infrastructure. With all of these managed services, developers are spending less time on infrastructure and more time on business logic. But managed services also lead to a new infrastructure problem—how do you manage resources across multiple clouds? A bucket storage system like Amazon S3 has different APIs than Google Cloud Storage. Google Cloud PubSub has different APIs than Amazon Kinesis. Since different clouds have different APIs, developers have trouble connecting cloud resources together, and it has become difficult to migrate your entire application from one cloud provider to another. Crossplane is an open source control plane for managing resources across multiple clouds. Crossplane’s goal is to provide a single API surface for interfacing with all the parts of your application, regardless of what cloud they are on. Crossplane is a project that was started by Upbound, a company with the goal of making multicloud software development easier. Bassam Tabbara is the CEO of Upbound, and he joins the show to talk about multi cloud deployments, Kubernetes federation, and his strategy for building a multi cloud API.

Originally posted on 13 September 2017. Machines understand the world through mathematical representations. In order to train a machine learning model, we need to describe everything in terms of numbers. Images, words, and sounds are too abstract for a computer. But a series of numbers is a representation that we can all agree on, whether we are a computer or a human. In recent shows, we have explored how to train machine learning models to understand images and video. Today, we explore words. You might be thinking–”isn’t a word easy to understand? Can’t you just take the dictionary definition?” A dictionary definition does not capture the richness of a word. Dictionaries do not give you a way to measure similarity between one word and all other words in a given language. Word2vec is a system for defining words in terms of the words that appear close to that word. For example, the sentence “Howard is sitting in a Starbucks cafe drinking a cup of coffee” gives an obvious indication that the words “cafe,” “cup,” and “coffee” are all related. With enough sentences like that, we can start to understand the entire language. Adrian Colyer is a venture capitalist with Accel, and blogs about technical topics such as word2vec. We talked about word2vec specifically, and the deep learning space more generally. We also explored how the rapidly improving tools around deep learning are changing the venture investment landscape.

Originally posted on 28 July 2017. Self-driving cars are here. Fully autonomous systems like Waymo are being piloted in less complex circumstances. Human-in-the-loop systems like Tesla Autopilot navigate drivers when it is safe to do so, and lets the human take control in ambiguous circumstances. Computers are great at memorization, but not yet great at reasoning. We cannot enumerate to a computer every single circumstance that a car might find itself in. The computer needs to perceive its surroundings, plan how to take action, execute control over the situation, and respond to changing circumstances inside and outside of the car. Lex Fridman has worked on autonomous vehicles with companies like Google and Tesla. He recently taught a class on deep learning for semi-autonomous vehicles at MIT, which is freely available online. There was so much ground to cover in this conversation. Most of the conversation was higher level. How do you even approach the problem? What is the hardware and software architecture of a car? I enjoyed talking to Lex, and if you want to hear more from him check out his podcast Take It Uneasy, which is about jiu jitsu, judo, wrestling, and learning.

Originally posted on 1 May 2018. Technology is pushing us rapidly toward a future that is impossible to forecast. We try to imagine what that future might look like, and we can’t help having our predictions shaped by the media we have consumed. 1984, Terminator, Gattaca, Ex Machina, Black Mirror–all of these stories present a dystopian future. But if you look around the world, the most successful technologists are mostly guided by a sense of optimism. Technologists themselves are mostly idealistic–they see the future through a utopian lens. Popular media largely tells a different story: that we are headed for a dystopian world. Why is there such a gulf in the level of idealism between technologists and the media? Mike Solana found himself asking that question on a regular basis during his work at Founder’s Fund, where he is a vice president. Founder’s Fund has a bias toward funding difficult, cutting-edge technology like gene editing, robotics, and nuclear energy. This technology that Mike was seeing made him excited about the future–which led to his creation of the podcast “Anatomy of Next.” “Anatomy of Next” has explored biology, robotics, nuclear energy, superintelligence, and the nature of reality. Soon the podcast will be exploring how our civilization will explore and settle the solar system–specifically Mars. I’ve listened through the entire first season of the show twice and enjoyed it so much because Mike explores questions that are on the border of philosophy and technology–questions about the nature of reality, and what makes us human–and nobody can give perfect answers to these questions. But Mike interviews top experts on the show, which provides us with a framework. Guests on “Anatomy of Next” include Nick Bostrom (the author of Superintelligence), George Church (a pioneer in gene editing), and Palmer Luckey (the founder of VR company Oculus). Mike joins the show to talk about why he started “Anatomy of Next,” and his own perspective on the future.

Originally posted on 16 June 2017. John Looney spent more than 10 years at Google. He started with infrastructure, and was part of the team that migrated Google File System to Colossus, the successor to GFS. Imagine migrating every piece of data on Google from one distributed file system to another. In this episode, John sheds light on the engineering culture that has made Google so successful. He has very entertaining stories about clusterops and site-reliability engineering. Google’s success in engineering is due to extremely high standards, and a culture of intellectual honesty. With the volume of data and throughput that Google responds to, 1-in-a-million events are likely to occur. There isn’t room for sloppy practices. John now works at Intercom, where he is adjusting to the modern world of Google infrastructure for everyone. This conversation made me feel quite grateful to be an engineer in a time where everything is so much cheaper, so much easier, and so much more performant than it was in the days when Google first built everything from scratch. I had a great time talking to John, and hope he comes back on the show again in the future because it felt like we were just scratching the surface of his experience.

Originally posted on 14 February 2017. Most tech companies are moving toward a highly distributed microservices architecture. In this architecture, services are decoupled from each other and communicate with a common service language, often JSON over HTTP. This provides some standardization, but these companies are finding that more standardization would come in handy. At the ridesharing company Lyft, every internal service runs a tool called Envoy. Envoy is a service proxy. Whenever a service sends or receives a request, that request goes through Envoy before meeting its destination. Matt Klein started Envoy, and he joins the show to explain why it is useful to have this layer of standardization between services. He also gives some historical context for why Envoy was so helpful to Lyft.

At Facebook, Venkat Venkataramani saw how large volumes of data were changing software infrastructure. Applications such as logging servers and advertising were creating fast moving, semi-structured data. The user base was growing, the traffic was growing, and the volume of data was growing. And the popular methods for managing this data were insufficient for the applications that developers wanted to build on top. In previous episodes about data platforms, we have covered similar difficulties as experienced by Uber and Doordash. Incoming data is often in JSON, which is hard to query. The data is transformed to a file format like Parquet, which requires an ETL job. Once it is in a Parquet file on disk in a data lake, the access time is slow. To query the data efficiently, it must be loaded into a data warehouse, which loads the data into memory, often in a columnar format that is easy to aggregate. Imagine being a developer at Facebook, Uber, or Doordash, and trying to build a simple dashboard, or a machine learning application on top of this data platform. Where do you find the right data? How do you know it is up to date? And what if you don’t know the shape of your queries ahead of time, and you haven’t defined indexes over your data? The access speed will be too slow to do exploratory analysis. There are so many steps in this process, and each of these steps creates friction for application developers that want to build on top of “big data”. Since even Facebook was having trouble managing this problem of the data platform, Venkat figured there was an opportunity to build a company around solving the data platform for other software companies. Venkat is the CEO of Rockset, a data system that is built to make it easy for developers to build data-driven apps. In Rockset, data can be ingested from data streams, data lakes, and databases. Rockset creates multiple indexes and schemas across the data. Because there are multiple models for querying, Rockset can analyze an incoming query and create an intelligent query plan for serving it. Venkat joins the show to discuss his time working on data at Facebook, the untapped opportunities of using that data, and the architecture of Rockset.

Ten years ago, there was a distinction between “backend” and “frontend” developers. A backend developer would be managing the business logic and database transactions using Ruby on Rails or Java. A frontend developer would be responsible for implementing designs and arranging buttons using raw HTML and JavaScript. Today, developers can build entire applications in JavaScript. Developers who spent their early career developing frontend JavaScript skills are finding themselves with a surprising amount of power. With NodeJS providing a backend framework and React, Vue, or Angular on the frontend, a single JavaScript developer can write all the code for a whole application—hence the rise of the “full stack developer”. At the same time, the cloud infrastructure is becoming easier to use. Backend-as-a-service simplifies the frustrations of deploying your application, and standing up a database. GraphQL improves the relationship between the frontend and the backend. And futuristic technologies like WebAssembly and web virtual reality are promising to make a JavaScript engineer’s life even more interesting. Adam Conrad is an engineer and a writer for Software Engineering Daily. In recent articles, he has documented the changing nature of the frontend, including JavaScript engines, virtual reality, and how mature corporations are using React and GraphQL. He joins the show to share his perspective on what is changing in the frontend—and how full stack JavaScript engineers can position themselves for future success in a quickly changing market.

Software products are distributed across more and more servers as they grow. With the proliferation of cloud providers like AWS, these large infrastructure deployments have become much easier to create. With the maturity of Kubernetes, these distributed applications are more reliable. Developers and operators can use a service mesh to manage the interactions between services across this distributed application. A service mesh is a layer across a distributed microservices application that consists of service proxy sidecars running alongside each service in a cluster, along with a central control plane for communicating with those sidecar proxies. A service mesh has many uses. Every request and response within the application gets routed through the service proxy, which can improve observability, traffic control to different instances, and circuit breaking in case of an instance failure. The central control plane can be used manage network policy throughout the whole system. We have done shows about each of the different components of a service mesh system, including different types of service proxies, as well as the service meshes built on top of these proxies. Linkerd, which is made by the startup Buoyant, was the first service mesh product to come to market, and it has the most production use, with customers like Expedia and Monzo bank. Istio is a more recent service mesh which uses the Envoy service proxy. Istio came out of Google and is also supported by IBM—setting up a classic competition between a startup and the large incumbents. William Morgan is the CEO of Buoyant, and he joins the show to talk about the use cases and adoption of service mesh. He also talks about the business landscape of the service mesh category, and how to compete with giant cloud providers.

Market strategy defines how a company is positioning itself to be successful. This strategy encompasses engineering, sales, marketing, recruiting, and everything else within a company. Herb Cunitz has led teams at Hortonworks, VMware, SpringSource, and several other companies over his 30 year career in software. After working as president of Hortonworks, Herb started AccelG2M. AccelG2M works with software companies to define their go-to-market strategy. Software companies require a great deal of long-term strategic thinking. Engineering, sales, marketing, and leadership must work together to build a plan that will allow the company to reach an exit: either an acquisition or an IPO. Executives at a software company must create a clear strategy and communicate it to the employees throughout the organization. The strategy must be implemented, meeting deadlines and hitting milestones. New team members must be recruited, and unsuccessful workers must be let go. In today’s show, Herb provides some invaluable strategic wisdom for anyone working in software–whether you are an engineer, salesperson, or investor.

Software companies today rely on group chat applications. The world of startups and small businesses is dominated by Slack. But for some large enterprises, regulatory constraints prevent them from using Slack. Slack is a web application that is hosted in the cloud, and regulated industries such as banking often need to run their applications on their own on-prem infrastructure. Mattermost is an open source alternative to Slack that can be self-hosted. This means that all of the networking complexities and scalability challenges that are controlled in the cloud by Slack need to be handled by open source code rather than managed services running in the cloud. Because it is open source, Mattermost can also be redesigned and customized. Uber designed their own custom version of Mattermost called uChat. Corey Hulen is a co-founder and the CTO of Mattermost. He joins the show to discuss the motivation for building Mattermost and the engineering challenges of building an open source chat system. For more episodes about building chat systems, we’ve done several shows about Slack, covering the engineering, security, and chat system.

Wes Bos has created popular courses on React, GraphQL, and JavaScript. With hundreds of thousands of students, Wes has earned a cult following for his fun, practical lessons on web development. The courses produced by Wes teach developers how to build useful applications such as a complete e-commerce store. Wes has built a career around studying and evangelizing JavaScript. His approach to education centers around practice, repetition, and hacking on fun projects. He also co-hosts a podcast called SyntaxFM, and is a frequent Twitter user. Wes is a rare mix of developer, teacher, businessman, and designer. Throughout his work, there is an artist’s sense of attention to detail, and a modern entrepreneur’s sense of pricing and marketing. His sites, such as JavaScript30 and React For Beginners have the deliberate style of someone who has been building websites for a very, very long time.In today’s episode, Wes Bos joins the show to give his perspective on JavaScript, entrepreneurship, and podcasting. To learn more about Wes’s business and his background, check out the IndieHackers podcast with him.

A bank account is a platform for apps to be built on top of. If that sounds like a weird idea, think about the features of a bank account. Most users only have a single bank account, making it a tool for identity and authentication. The series of transactions in a bank account provides a data set that can be used for analyzing payment history and issuing loans, or insurance. But there are difficulties to building a platform on top of banking. There are thousands of different banks. If you want to build an application that integrates with a user’s bank, you need to be able to integrate with any bank that the user might use–whether it’s Bank of America, Wells Fargo, or Chase. Plaid is a company that builds APIs for users to connect to banks. Applications such as Venmo, Betterment, and Coinbase use Plaid to connect with the bank accounts of their users. Jean-Denis Greze joins the show to explain how applications use Plaid, and how Plaid has scaled its infrastructure to handle a high volume of requests. He also discusses the potential of banking as a platform, and the strategy for expanding the APIs that Plaid can offer to developers. Fintech Daily is a new podcast from Software Engineering Daily covering payments, cryptocurrencies, trading, and the intersection of finance and technology. We are looking for volunteer hosts for Fintech Daily, and if you are interested in working with us to conduct interviews, send an email to [email protected]. You can find the podcast on iTunes, Google, and everywhere else, and if you are interested in hosting, don’t hesitate to reach out.

When a startup finds product market fit, the adoption of that product can grow rapidly, turning a startup into a high growth company. All of a sudden, a startup that was struggling to find its first customer is bombarded with new challenges. The startup has to hire tens of new employees. This requires raising capital, so the startup has to meet with investors and lawyers. A rapid influx of new customers puts a strain on the engineering and customer service elements of the company. There is too much to do, and there is only so much time in a day. The CEO of the high-growth company is up late into the night, answering emails and losing sleep. But these are good problems to have, and the company is in a state of exuberance. The CEO must balance psychological health with the stressful task of scaling a company. Elad Gil is an entrepreneur and author of “High Growth Handbook”, a book of lessons and guidelines about how to navigate a startup that has found product market fit, and is beginning to scale. High Growth Handbook includes interviews with experienced entrepreneurs such as Marc Andreessen and Patrick Collison, whom Elad met with as he wrote the book. Elad joins the show to discuss his book, and his own personal lessons of working with companies such as Twitter, Google, Stripe, and Coinbase. Elad has worked at several high growth companies and invested in others, and he has gathered a lot of wisdom from these different experiences.

Releasing software has inherent risk. If your users don’t like your new feature, they might stop using your product immediately. If a software bug makes it into production, it can crash your entire application. Releasing software gradually has many benefits. A slow rollout to an increasing population of users allows you to test your software in multiple real-world environments before it goes live to everyone. A system of AB testing different versions of your software lets you see how different flavors of your software perform against similar audiences. Edith Harbaugh is the CEO of LaunchDarkly, a system for feature management. LaunchDarkly allows developers to deploy new software releases in a controlled fashion. Edith joins the show to discuss how to implement feature flagging, and why an intelligent release process can lead to a more scientific, predictable environment for software development. Edith is also the host of To Be Continuous, a podcast about continuous delivery, software engineering, and DevOps.

The Berkeley AMPLab was a research lab where Apache Spark and Apache Mesos were both created. In the last five years, the Mesos and Spark projects have changed the way infrastructure is managed and improved the tools for data science. Because of its proximity to Silicon Valley, Berkeley has become a university where fundamental research is blended with a sense of industry applications. Students and professors move between business and academia, finding problems in industry and bringing them into the lab where they can be studied without the day-to-day pressures of a corporation. This makes Berkeley the perfect place for research around “serverless”. Serverless computing abstracts away the notion of a server, allowing developers to work at a higher level and be less concerned about the problems inherent in servers–such as failing instances and unpredictable network connections. With serverless functions-as-a-service, the cloud provider makes guarantees around the execution of serverless code–such as with AWS Lambda. With serverless backend services, the cloud provider makes guarantees around the reliability of a database or queueing system. The cloud provider is operating servers to power this functionality. But the user is not exposed to those servers. Today’s show centers around the serverless functions-as-a-service. This is a new paradigm of computing, and there are many open questions. How can the servers for our functions be quickly provisioned? How can we parallelize batch jobs into functions as a service? How can large numbers of serverless functions communicate with each other reliably to coordinate? In production applications, functions-as-a-service are mostly used for “event-driven” applications. But the potential for functions-as-a-service is much larger. Ion Stoica is a professor of computer science at Berkeley, where he leads the RISELab. He is the co-founder of Conviva Networks and Databricks. Databricks is the company that was born as a result of the research on Apache Spark. Ion now serves as executive chairman of Databricks. Ion joins the show to describe why serverless computing is exciting, the open research problems, and the solutions that researchers at the RISELab are exploring.

Robotics, genomics, and backend infrastructure: as an in vestor, it can be difficult to assess the viability of a startup that is on the cutting edge in any of these areas. A robotics startup requires a team with an integrated understanding of hardware and software. A genomics company will not only have to develop a successful healthcare product, but will have to bring it to market through regulation. And in the world of backend infrastructure, building a business that will be differentiated from giant cloud providers gets harder every day. Amplify Partners is a venture capital fund with an emphasis on technical investments. Their portfolio includes infrastructure companies like Datadog and Gremlin, as well as pharmaceutical and hardware companies. Sunil Dhaliwal is the founder of Amplify Partners, and joins the show to discuss the thesis of Amplify. The investments that Amplify makes are in technical companies–which makes these financing decisions complex enough to require detailed, individualized research. But there are commonalities among the founding teams. Sunil lays out a useful rubric for anyone who is looking to learn about venture capital investing.

Building software was simplified by cloud providers. With the cloud, it became much easier to deploy a server, spin up a database, and scale an application. Cloud providers like AWS gave developers access to these infrastructure primitives like storage and compute. On top of those primitives, numerous API companies have been built. An API company offers a more specific set of services. Twilio offers SMS text messaging API services. Stripe offers payment API services. These APIs give developers another level of tooling to build software out of. Developers can now think of entire applications in terms of APIs, and the number of APIs is growing rapidly. From business services such as booking a flight to machine learning models like image classification, the “API economy” has given developers a huge catalog of tools. Since developers have this additional leverage, software can be built with smaller teams. The codebase can also be smaller. But one area where the complexity is growing is the number of APIs that need to be managed. For each API, there is a different system for integrating the API into your application. Different API providers have different levels of reliability. Another area of difficulty is the discoverability of APIs. If I don’t know about a flight search API, I am never going to think of what applications I could build on top of that. There are APIs for generating memes, and APIs for easily querying what music is trending across the world. RapidAPI is a marketplace for APIs. It includes search and discovery features for the wide variety of different APIs that can be found across the internet. RapidAPI is also a system for integrating with multiple APIs through it’s API management system. Iddo Gino is the CEO and founder of RapidAPI, and he joins the show to discuss the motivation for creating an API marketplace, as well as the engineering behind RapidAPI.

The Bitcoin main chain is a large distributed ledger of transactions. Bitcoin is useful for maintaining a trusted record of payments, but is not practical for small day-to-day payments. Bitcoin payment channels allow users to issue small payments to each other without paying the high transaction cost and latency of going through the main chain. When payment channels are connected to each other, a “lightning network” is formed. Lightning network is often referred to as a “second layer” scalability solution. Alex Bosworth is a lightning infrastructure lead at Lightning Labs, a company that builds infrastructure for scaling blockchains. In today’s show, Alex explains how Bitcoin payment channels work, and provides some context on how developed the modern infrastructure is in terms of practical use cases for Bitcoin.

Demand for live streaming video over the internet is increasing. After the emergence of early live streaming platforms, like Twitch and Facebook Live, more forms of video have become accessible over live streams, such as sports. Live streaming is a harder engineering problem than delivering a static video file because the information distributed on a live stream is constantly changing. DZN, spelled D-Z-N, is a live streaming service for watching fight events, such as boxing. The workloads for live streaming can be highly bursty. When a fight is scheduled to happen, the vast majority of traffic will hop on to watch the fight 20 seconds before the fight starts. A huge number of users logs into DZN and starts watching all at the same time. This quick spike in traffic means that DZN has to have servers spun up and be ready in advance. Luca Mezzarlira and Yan Cui are engineers at DZN. Yan was previously on the show in a few amazing episodes to talk about serverless infrastructure and the complexities of real-time video game software development. Those episode links are in the show notes. I highly recommend checking them out. Today’s show is a discussion of architecting a system to handle a high bandwidth customer use case. I hope you like it.

Not every company wants to move to the public cloud. Some companies have already built data centers, and can continue to operate their business with their own servers. Some companies have compliance issues with the public cloud, and want to operate their own servers to avoid legal risk. Operating a data center is not easy. Operating systems need to be updated and security vulnerabilities need to be patched. Servers fail, and their workloads need to be automatically scheduled onto other servers to avoid downtime. In contrast to classic on-prem data center management, the cloud provides many benefits: automatic updates, an infinite pool of resources, fully programmable infrastructure as code. In the cloud, developers can provision infrastructure with an API request. Continuous delivery pipelines can be spun up at the click of a button. This tooling makes it dramatically easier for developers to move quickly, and for a business to move faster. Companies that operate their own data center want to be able to have these benefits of the cloud while still controlling their own infrastructure. Today’s guest Bob Fraser works at HPE on OneView, a tool for managing on-prem infrastructure like a cloud. Bob describes the difficulties of managing legacy on-prem infrastructure, and the advantage of building a management layer on top of data center infrastructure to make it more programmable. We’ve done lots of shows recently about Kubernetes in the context of cloud computing. Today’s show outlines how modern on-prem infrastructure can be managed like a cloud. Full disclosure: HPE is a sponsor of Software Engineering Daily.

Podcasts have surged in popularity, but the podcast ecosystem remains difficult to work with. Podcast listeners have difficulty finding episodes. Podcast creators have difficulty finding out how to get started. The advertising marketplaces for podcasts are immature, and it can be difficult to build a business as a podcaster. Podcasting is unlike almost any other media format that we consume on the Internet. There is not an algorithmic feed of podcasts–we subscribe to podcasts we like and we see everything that gets published. Podcasting originated with Apple, who has not shown much interest in the podcast medium. Anchor is a platform that makes it easy for users to publish podcasts. Today, a large percentage of the new podcasts created on the Internet are started on Anchor. Nir Zicherman is the CTO at Anchor, and he joins the show to discuss the strange world of podcast technology, and how Anchor is building a business.

Cloud computing changed the economics of running a software company. Before the cloud, a software company had to purchase physical machines which often required thousands of dollars paid up front. The cloud allowed developers to deploy their applications for free, to operate a business for cheap, and to scale without hiring a dedicated team to manage the servers. Building in the cloud is cheap, but scaling in the cloud can get expensive. A growing company can often save money by changing which cloud instances and services they use. Reducing the number of server instances, changing the size of compute instances, and changing rules around auto scaling. By using monitoring, dashboards, and regular analysis of where money is spent, a business can find thousands of dollars of wasted spend per month. There are also broad strategic decisions around cost. One area to study is the use of “managed” services like Amazon DynamoDB, Google BigQuery, and Amazon Lambda. These services are proprietary, and can lead to lock-in. Sometimes they can be quite expensive. But they can save developers hours of time because they are easy to use, and provide high uptime guarantees. Ran Rothschild works at DoIT International, a company that helps businesses figure out how to save money on their cloud infrastructure. He joins the show to discuss the places where the most money is wasted and how startups can manage their infrastructure in a cost-effective manner. He also tells some stories about significant overspend. Full disclosure: DoIT International is a sponsor of Software Engineering Daily.

Slack is a real-time messaging system for work communication. On Slack, chat rooms as big as 100,000 people have productive conversations. This might sound like the same problem solved by social networks like Facebook, where billions of users communicate over a newsfeed. But the engineering constraints of a messaging system are different than that of a social network. On a newsfeed, the order in which events appear is not chronological. Events can be out of order. You can miss events. When a user posts a message to a social network, there are not strict guarantees around when other people will see that message. On Slack, messages have strong guarantees around arrival. When I send a message, everyone else who is in the room and connected should quickly receive that message as well. The messages need to be ordered and delivered exactly once. All messages on Slack are persisted. We have covered the architecture and security model of Slack in previous shows. In today’s show, Keith Adams returns to discuss how messages are processed and broadcast in Slack. The problem of Slack’s messaging system is similar to the distributed systems problem of “atomic broadcast”, in which a single process broadcasts a message which needs to be received by all other processes correctly–or else received by none of them. In Keith’s last show, he talked through the benefits of building a large system on PHP. He worked on infrastructure at Facebook, which was also a PHP application. It’s worth noting that both Slack and Facebook have scaled a monolithic architecture.

Many people have saved some money which they want to invest for the future. Some people are happy investing their money in a roboadviser, which programmatically puts money into long-term investments. Other people want a more personal approach involving a certified financial planner (CFP). A CFP is a human who allocates capital for an individual based off of that individual’s preferences. A CFP spends time and effort researching the options for a client. If the client only has a small amount of money (say, $15,000), it is not worth it for the CFP to spend much time on that account. As a result, there is a type of client who has saved a little bit of money, but has not saved enough to be an important client for a CFP. Facet Wealth is a software company that makes software for CFPs to work more effectively with their client accounts. Facet Wealth has in-house CFPs who work with this software to manage client accounts. In addition, Facet Wealth buys client accounts from independent CFPs who have small accounts which they do not have time to manage. This is an innovative way to aggregate users onto the platform. Gorkem Sevinc is CTO at Facet Wealth. He joins the show to describe the business and the software architecture of the company. We touched on many different areas–from human-computer interaction to the future of investing. We recently launched a new podcast: Fintech Daily! Fintech Daily is about payments, cryptocurrencies, trading, and the intersection between finance and technology. You can find it on fintechdaily.co or Apple and Google podcasts. We are looking for other hosts who want to participate. If you are interested in becoming a host, send us an email: [email protected]

Parity is a company that builds blockchain infrastructure. Parity has built several open source projects and works with enterprises to put blockchain technology in production. Gavin Wood is the founder of Parity, and he joins the show to talk about the state of blockchain technology and what his company is currently focused on. Four years ago, Gavin helped start the Ethereum project, so he has lots of context on decentralized technology. Gavin envisions a world with many different blockchains for many different use cases. These blockchains will interact with each other to enable trusted relationships between parties. One project that Parity has created is Substrate, a technology that allows developers to quickly stand up a blockchain with the right privacy level. Another project is Polkadot, which allows blockchains to connect and interoperate with each other. Gavin and I discussed why the world needs a variety of blockchains–and whether all of these different blockchains should need their own cryptocurrency. Gavin described the use case of blockchains for mediating supply chain trust. We also talked about the technologies used to build these projects, including WebAssembly and Rust.

Originally posted on June 23, 2016. Pieter Hintjens grew up writing software by himself. The act of writing code brought him great pleasure, but the isolated creative process disconnected him from the rest of the world. As his life progressed he became involved in open source communities, and he discovered a passion for human interaction. Open source software succeeds or fails on the strength of the community. One story of success is ZeroMQ, a popular open source distributed messaging system that was started by Pieter Hintjens. In this episode, Pieter gives his thoughts on human nature, distributed systems, and death. “A Protocol For Dying” is a blog post Pieter wrote recently, where he discussed his terminal diagnosis of cancer, and how it has reframed his perspective on life.

Originally posted on May 12, 2016. When you are deciding who to marry, you are using an algorithm. The same is true when you are looking for a parking space, playing a game of poker, or deciding whether or not to organize your closet. Algorithms To Live By is a book about the computer science of human decisions. It offers strategies for how to think through everyday life like a computer scientist. Brian Christian has a background in computer science and philosophy, and is an author of Algorithms to Live By. He joins the show to explain how the same algorithms and data structures we use for our computer programs can be applied to the real world.

Originally posted on May 12, 2015. Humans have now been defeated by computers at heads up no-limit holdem poker. Some people thought this wouldn’t be possible. Sure, we can teach a computer to beat a human at Go or Chess. Those games have a smaller decision space. There is no hidden information. There is no bluffing. Poker must be different! It is too human to be automated. The game space of poker is different than that of Go. It has 10^160 different situations–which is more than the number of atoms in the universe. And the game space keeps getting bigger as the stack sizes of the two competitors gets bigger. But it is still possible for a computer to beat a human at calculating game theory optimal decisions–if you approach the problem correctly. Libratus was developed by CMU professor Tuomas Sandholm, along with my guest today Noam Brown. The Libratus team taught their AI the rules of poker, they gave it a reward function (to win as much money as possible), and they told it to optimize that reward function. Then they had Libratus train itself with simulations. After enough training, Libratus was ready to crush human competitors, which it did in hilarious, entertaining fashion. There is a video from Engadget on YouTube about the AI competing against professional humans. In this episode, Noam Brown explains how they built Libratus, what it means for poker players, and what the implications are for humanity–if we can automate poker, what can’t we automate? Stay tuned at the end of this episode for the Indeed Prime tip on hiring developers.

Featured Image Photo Credits Originally posted on July 11, 2016. Negotiation is an important skill for software engineers. The salary you negotiate at the beginning of your job could be a difference of tens of thousands of dollars over the course of an engineer’s career, but intimidating recruiters and exploding offers scare many engineers from negotiating at all. Today, Haseeb Qureshi returns to the show to discuss his epic story of salary negotiation. On a previous episode, Haseeb discussed leaving his career as a poker player to join a coding boot camp and start down the path of a software engineer. In this episode and his recent blog post, Haseeb describes his approach to the job search and salary negotiation process, which eventually landed him at Airbnb with a $250,000 annual salary–after about a year of learning to code.

Originally published on July 6, 2016. Scheduling is the method by which work is assigned to resources to complete that work. At the operating system level, this can mean scheduling of threads and processes. At the data center level, this can mean scheduling Hadoop jobs or other workflows that require the orchestration of a network of computers. Adrian Cockcroft worked on scheduling at Sun Microsystems, eBay, and Netflix. In each of these environments, the nature of what was being scheduled was different, but the goals of the scheduling algorithms were analogous–throughput, response time, and cache affinity are relevant in different ways at each layer of the stack. Adrian is well-known for helping bring Netflix onto Amazon Web Services, and I recommend watching the numerous YouTube videos of Adrian talking about that transformation.

The volume of data in the world is always increasing. The costs of storing that data is always decreasing. And the means for processing that data is always evolving. Sensors, cameras, and other small computers gather large quantities of data from the physical world around us. User analytics tools gather information about how we are interacting with the Internet. Logging servers collect terabytes of records about how our systems are performing. From the popularity of MapReduce, to the rise of open source distributed processing frameworks like Spark and Flink, to the wide variety of cloud services like BigQuery: there is an endless set of choices for how to analyze gigantic sets of data. Machine learning training and inference is another dimension of the modern data engineering stack. Whereas tools like Spark and BigQuery are great for performing ad-hoc queries, systems like TensorFlow are optimized for the model training and deployment process. Stitching together these tools allows a developer to compose workflows for how data pipelines progress through a data engineering system. One popular tool for this is Apache Airflow, which was created in 2014 and is widely used at companies like Airbnb. Over the next few years, we will see a proliferation of new tools in the world of data engineering–and for good reason. There is a wealth of opportunity for companies to leverage their data to make better decisions, and potentially to clean and offer their internal data as APIs and pre-trained machine learning models. Today, there is a vast number of enterprises who are modernizing their software development process with Kubernetes, cloud providers, and continuous delivery. Eventually, these enterprises will improve their complex software architecture, and will move from a defensive position to an offensive one. These enterprises will shift their modernization efforts from “DevOps” to “DataOps”, and thousands of software vendors will be ready to sell them new software for modernizing their data platform. There is not a consensus for the best way to build and run a “data platform”. Nearly every company we have talked to on the show has a different definition and a different architecture for their “data platform”: Doordash, Dremio, Prisma, Uber, MapR, Snowflake, Confluent, Databricks… We don’t expect to have a concise answer for how to run a data platform any time soon–but on the bright side, data infrastructure seems to be improving. Companies are increasingly able to ask questions about their data and get quick answers, in contrast to the data breadlines that were so prevalent five years ago. Today we cover yet another approach to large scale data processing. Reflow is a system for incremental data processing in the cloud. Reflow includes a functional, domain specific language for writing workflow programs, a runtime for evaluating those programs incrementally, and a scheduler for dynamically provisioning resources for those workflows. Reflow was created for large bioinformatics workloads, but should be broadly applicable to scientific and engineering computing workloads. Reflow evaluates programs incrementally. Whenever the input data or the program changes, only the outputs that depend on the changes are recomputed. This minimizes the amount of recomputation that needs to be performed across a computational graph. Marius Eriksen is the creator of Reflow and an engineer at GRAIL. He joins the show to discuss the motivation for a new data processing system–which involves explaining why workloads in bioinformatics are different than in some other domains.

The software release process is a barrier between written code and a live production environment that affects users. A software release process can involve a variety of different practices. Code might be tested for bugs using automation and manual testing. Static analysis tools can look at the code for potential memory leaks. A software release might go out to a small percentage of the total user base before it gets deployed to the entire audience. At some organizations, a software release can be slow and painful. The release might be bottlenecked by a manual approval step, which slows down developers from quickly deploying their own changes. If a consistent version history of software is not maintained, a release can be hard to roll back in the event of an error. In the case of a large, monolithic architecture, a release can be scary, because it can be hard to understand how the monolithic codebase functions. This set of challenges within the release process lowers the quality of software, and can make it frustrating to build software. The release process is just one area of software development that many organizations have a desire to smooth out. Over the past ten years, a set of technologies and philosophies have provided improvements to the software development process. DevOps, continuous delivery, microservices, cloud providers, and serverless tools all make it easier for a company to focus on its core competency and release software faster. Baruch Sadogursky is an author of Liquid Software, a book about continuous updates and DevOps. Liquid Software describes an idealized vision of what today’s architecture could aspire to. The focus of the book is continuous updates, which allow for rapidly improving, evolving software quality. Baruch joins the show to discuss how software has changed in the last twenty years, and how the future of software development could look. Full disclosure: Baruch works at JFrog, which is a sponsor of Software Engineering Daily.

Modern software consists of sprawling international networks of servers. Users contact these servers to access applications. Microservices talk to each other to fulfill complicated requests. Databases and machine learning frameworks crunch terabytes of information to provide complicated answers. Across this infrastructure, there is a lot of different activities–and a lot of vulnerabilities. Without a reliable model for security and trust, software can be easily compromised. In the past, systems were often protected by a “firewall”, which is a security system around the perimeter of the network. A problem with this model is that if the attacker is able to penetrate the firewall, they can compromise anywhere in the network. Firewalls can be penetrated, so a much better security model is to assume that your network has already been compromised, and to require every internal system to identify and authenticate with each other. “Zero-trust security” is a security model that requires internal systems to communicate with each other as if they were potentially compromised. Evan Gilman is the author of Zero Trust Networks: Building Secure Systems in Untrusted Networks. He also works on SPIFFE, a system for managing identity and trust within a zero-trust network. In a previous episode about Google BeyondCorp, Max Saltonstall talked about zero-trust networking in the context of user and device authentication. In today’s episode, Evan discusses another side of zero-trust networking: workload identity and authentication. Just as Google BeyondCorp outlines an architecture for allowing devices to communicate with the network, the SPIFFE project outlines a system for workloads to identify and authenticate themselves. Workloads can range from MapReduce jobs to microservices to frontend application servers.

Serverless computing abstracts away the idea of a server node. Serverless lets programmers treat compute resources as high-level, reliable APIs, rather than unreliable, low-level compute nodes that might fail. Serverless dramatically improves the efficiency of programmers. Instead of thinking of a database as a set of servers that need to be sharded and replicated, the programmer can think of a database as a place to read and write data. Instead of modeling an application as a large monolith running on an application server in a container or a VM, the programmer can think of their application as a decoupled set of functions. Serverless computing is a natural evolution of software engineering in a world with cloud providers. The first version of FaaS came out of AWS with their Lambda service, which allows users to run functions in the cloud. Those functions are scheduled onto a physical server somewhere in an Amazon data center. They are executed, and they return the result. With AWS Lambda, programmers got a new abstraction to model their applications with. But it requires the use of a closed-source API. Lambda is not open source, and this makes some developers reluctant to integrate with it too tightly. Fission is an open source framework for serverless functions built on Kubernetes. Fission allows developers to deploy functions-as-a-service without being locked in to any specific cloud provider. Soam Vasani is the creator of Fission and an engineer at Platform9. In a previous episode, Soam talked about the architecture for Fission and the design choices for solving the cold start and scheduling problems. Soam joins the show today to discuss how serverless applications have evolved since last spoke more than a year ago. He also talks about how Fission itself has evolved, and the features that an open source serverless platform needs to have in order to compete with a fully developed cloud provider. Full disclosure: Platform9 is a sponsor of Software Engineering Daily.

Policies define which users and applications can access and modify resources in a computer system. In a file system, a user might have permission to read or write to a file. In a cloud infrastructure deployment, a user might have the rights to deploy a new server. One microservice may or may not have the necessary permissions to talk to another microservice. All of these are use cases where a “policy” defines the behavior within a computer system. Policies in a company can be managed in a range of ways: configuration files, dashboards, and centralized permissions databases. A policy engine is a system for managing and automating the policy creation and deployment within an organization. Microservices need to verify each request that comes in to ensure that the request has the correct permissions. To check those permissions, a microservice can contact the policy engine. The policy engine has all the information from the whole organization about who is allowed to do what. However, talking to the policy engine over the network can be a slow process. Open Policy Agent is a deployable agent that can run as a sidecar next to a service, and check policies by looking inside of a cache. Torin Sandall is a core committer to the Open Policy Agent project, and he joins the show to talk about policy management, the Open Policy Agent, and the Kubernetes ecosystem (and surprisingly, WebAssembly).

TLA+ is a formal specification language. TLA+ is used to design, model, and verify concurrent systems. TLA+ allows a user to describe a system formally with simple, precise mathematics. TLA+ was designed by Leslie Lamport, a computer scientist and Turing Award winner. Leslie joins the show to talk about the purpose of TLA+. Since its creation in 1999, TLA+ has been used to discover bugs in systems such as Amazon S3, DynamoDB, Xbox, and Cosmos DB. “TLA” stands for “temporal logic of actions”, a logical system that can be used to describe the behaviours of concurrent systems. This podcast is meant as a brief introduction of TLA+. To go deeper, check out the TLA+ website and the TLA+ video course (note: these videos are highly entertaining because of Leslie’s dry, unpredictable sense of humor).

Mapillary is a company that processes high volumes of images to develop a labeled 3-D model of the physical world. Mapillary’s APIs allow developers to build applications that are aware of stop signs, buildings, streets, trees, and other physical objects in real-world space. The potential use cases for Mapillary are numerous, ranging from self-driving cars to augmented reality. We can now build a 3-D model of the real world. It’s not a perfect representation of reality, but it is much better than we had just a few years ago. What has changed? How have the tools advanced such that we are able to build an API for accessing accurate information about the physical world around us? Mapillary is possible because of a combination of modern developments. High quality smartphone cameras enable users to crowdsource images of the world around them. Cloud computing allows for cheap workload processing. Newer computer vision techniques allow 2-D images to be stitched together in a 3-D representation. Deep learning architectures improve the classification and segmentation of objects in an image. Peter Kontschieder is the head of research at Mapillary, and he joins the show to talk about the technologies and research that has enabled Mapillary to build a futuristic business–an API for accessing information about the physical world. Software Engineering Daily is looking for sponsors. If you are interested in reaching over 50,000 developers, you can go to softwareengineeringdaily.com/sponsor to find out more, and you can send us a message. We’d love to hear from you. And if you are an engineer working at a company that is marketing to developers, or hiring developers, if you tell your marketing department or your recruiting department about softwareengineeringdaily.com/sponsor, that is one way to help us out.

An instruction set defines a low level programming language for moving information throughout a computer. In the early 1970’s, the prevalent instruction set language used a large vocabulary of different instructions. One justification for a large instruction set was that it would give a programmer more freedom to express the logic of their programs. Many of these instructions were rarely used. Think of your favorite programming language (or your favorite human language). What percentage of words in the vocabulary do you need to communicate effectively? We sometimes call these language features “syntactic sugar”. They add expressivity to a language, but may not improve functionality or efficiency. These extra language features can have a cost. Dave Patterson and John Hennessy created the RISC architecture: Reduced Instruction Set Compiler architecture. RISC proposed reducing the size of the instruction set so that the important instructions could be optimized for. Programs would become more efficient, easier to analyze, and easier to debug. Dave Patterson’s first paper on RISC was rejected. He continued to research the architecture and advocate for it. Eventually RISC became widely accepted, and Dave won a Turing Award together with John Hennessy. Dave joins the show to talk about his work on RISC and his continued work in computer science research to the present. He is involved in the Berkeley RISELab and works at Google on the Tensor Processing Unit. Machine learning is an ocean of new scientific breakthroughs and applications that will change our lives. It was inspiring to hear Dave talk about the changing nature of computing, from cloud computing to security to hardware design.

Open source projects benefit from the network effects of a large audience of developers. A popular open source project will be contributed to and used by thousands of developers, who are continuously testing, deploying, and improving the software. The open source movement has created massive communities and a thriving, collaborative economy. Infrastructure software companies are increasingly built within an open source business model. Databases, queueing systems, orchestrators, operating systems, and search engines have been started as freely available open source projects, and leveraged into billion dollar businesses. In previous shows we have talked about business strategy, go-to-market tactics, and licensing of infrastructure software. There remains plenty of room for more open source infrastructure companies. We still need better databases and distributed systems management. But over time, open source will move up the stack. From Netflix to Uber to social networks to payments systems–all software verticals will become open source because the benefits of making your software open source outweigh the costs. For many software business models, the competitive advantage is not found in their source code–it’s in their data, their network effects, their sales strategy, and their brand. Therefore, it makes sense that someday the source code will be freely available, democratizing the infrastructure concerns and letting these software businesses move up the value chain and become less operationally intensive at the bottom of the stack. Rather than asking “why should we open source our code”, these companies will be asking “why shouldn’t we open source our code?” Joseph Jacks is the founder of OSS Capital, a venture capital firm that invests exclusively in commercial OSS startup companies. Joe believes that over time, open source eats everything. In today’s show, we talk about the future of open source businesses, the impact of licensing, cloud providers, and cryptocurrencies.

Open source software powers everything we do on the Internet. Google runs on Linux servers. Content sites are served by WordPress. Our data is queued in Kafka clusters and stored in MongoDB instances. The success of an open source project often leads to the creator of that open source software becoming wealthy. An open source project can be monetized through enterprise add-ons, or consultation, or simplified hosting. The creators of open source software know their domains so well, that they are usually well-suited to operate this kind of open source business. Open source business model success stories include Elastic (ElasticSearch), Cloudera (Hadoop), and Red Hat. The rise in usage of cloud providers has changed the viability of some open source business models. AWS can monetize almost any open source project more profitably than the creators. This is because AWS has established distribution channels. If I already run my application on AWS, and I am looking for someone to provide me with a hosted version of a database, I will probably choose the hosted database that AWS provides. The Commons Clause is a license that open source projects can use to protect their code from being profited from. Redis, an open source in-memory object storage system, recently licensed their code with the Commons Clause with the goal of improving the business of Redis Labs, a company built by the creators of the Redis project. Kevin Wang joins the show to discuss everything open source–from business models to security vulnerabilities to licensing. Kevin is the CEO at FOSSA, a system for managing open source licensing and security. Kevin was involved in the creation of the Commons Clause and has written about it in detail.

Matt Klein has worked for three rapidly growing Internet companies. At AWS, he worked on EC2, the compute-as-a-service product that powers a large percentage of the Internet. At Twitter, he helped scale the infrastructure in the chaotic days before Twitter’s IPO. Today he works at Lyft, building systems to allow for ride sharing infrastructure to work more safely and reliably. Hypergrowth Internet companies are faced with quickly growing demands on their software. The demands on the software expose problems with the core infrastructure. Simultaneously, the company tries to ramp up its hiring process. More engineers get hired, and the institutional knowledge within the company starts to weaken. Documentation gets out of date. Senior engineers burn out and leave the company. When a company starts growing quickly, communications can break down. A hypergrowth company can suffer from a lack of “human scalability”. Matt Klein has observed these challenges at AWS, Twitter, and Lyft. In his article “The Human Scalability of ‘DevOps’”, he explains why these problems manifest and what can be done to alleviate them. In a previous show, Matt discussed the engineering challenges at Lyft that led him to create Envoy, a service proxy. This episode covers some broad technical topics–DevOps, site reliability engineering, platform engineering–but the episode is mostly about how a hypergrowth company can manage culture, hiring, and engineering organization. Matt is a very fun guest to have because he questions some of the strange practices that have been widely adopted by successful companies. Internet companies are a very new phenomenon, and the management tactics that they have adopted are not well proven–so it is great to have someone like Matt provide a fresh perspective on ways that companies can scale their technology and their organization more effectively.

Online labor marketplaces are widely used for one-to-one transactions. On Uber, a rider hires a driver for transportation. On TaskRabbit, a homeowner hires a cleaner to come clean their kitchen. These types of marketplaces are not as widely used for one-to-many transactions, but they can be just as useful. A warehouse owner would want to hire a group of workers to help with holiday shipments. A conference organizer would want to hire a group of event staffers to help run the conference. Wonolo is an on-demand staffing platform. Businesses post jobs and workers apply for those jobs. The types of work include event staffing, warehouse operations, merchandising, and other general labor tasks. In past shows, we have covered on-demand work platforms such as Fiverr, Thumbtack, Uber, and Instacart. Wonolo presents another variation in the business model and software architecture of the gig economy. Jeremy Burton is the CTO and chief data scientist at Wonolo. He joins the show to talk about building and scaling Wonolo, and some of the key strategic decisions that Wonolo has made along the way. As with any successful marketplace business, Wonolo has solved the chicken and egg problem of how to get supply and demand on the platform simultaneously. The company has grown deliberately, setting up operations in one city at a time to make sure that they can provide a good experience in both sides of the market in each of the new geographies. Jeremy and I also talked about the broader effects that the gig economy could potentially have on the labor market. Gig economy platforms use a 5-star rating system and written reviews to judge workers, instead of a resume system. The gig economy allows for rapid job liquidity, and the potential for workers to steadily “level up” more quickly than they might be able to in a typical corporate job. These aspects of the gig economy are rarely discussed, so it was enlightening to hear Jeremy’s views on them.

Google Search allows humans to find and access information across the web. A human enters an unstructured query into the search box, the search engine provides several links as a result, and the human clicks on one of those links. That link brings up a web page, which is a set of unstructured data. Humans can read and understand news articles, videos, and Wikipedia pages. Google Search solves the problem of organizing and distributing all of the unstructured data across the web, for humans to consume. Diffbot is a company with a goal of solving a related, but distinctly different problem: how to derive structure from the unstructured web, understand relationships within that structure, and allow machines to utilize those relationships through APIs. Mike Tung is the founder of Diffbot. He joins the show to talk about the last decade that he has spent building artificial intelligence applications, from his research at Stanford to a mature, widely used product in Diffbot. I have built a few applications with Diffbot, and I encourage anyone who is a tinkerer or prototype builder to play around with it. It’s an API for accessing web pages as structured data. Diffbot crawls the entire web, parsing websites, using NLP and NLU to comprehend those pages, and using probabilistic estimations to draw relationships between entities. It’s an ambitious product, and Mike has been working on it for a long time. I enjoyed our conversation. We recently launched a new podcast: Fintech Daily! Fintech Daily is about payments, cryptocurrencies, trading, and the intersection between finance and technology. You can find it on fintechdaily.co or Apple and Google podcasts. We are looking for other hosts who want to participate. If you are interested in becoming a host, send us an email: [email protected]

David Cancel has started five companies, most recently Drift. Drift is a conversational marketing and sales platform. David has a depth of engineering skills and a breadth of business experience that make him an amazing source of knowledge. In today’s episode, David discusses topics ranging from the technical details of making a machine learning-driven sales platform to the battle scars from his early career, when he spent a lot of time building products that people did not want. He has found success by focusing on building software that the market has shown a desire for. Chatbots were a popular, trendy subject a few years ago. The success of chatbots manifested in them fading into the background, and becoming a subtle, increasing part of our everyday interactions. Not every online interaction can be replaced by a chatbot, but many online interactions can be made more efficient by using chatbots. Chatbots can serve well-defined information, like product features, or the hours of operation of a business. When a chatbot gets a question that it cannot answer, the bot can route the conversation to a human. When a customer lands on a web page of a company using Drift, they see a chat box appear in the corner of the screen. The customer is able to communicate through that chat box with a bot that represents the company. The customer can learn about the product, schedule a call with a salesperson, and get other useful utilities from the Drift sales bot. The Drift chatbot messaging system is handled by Elixir/Erlang. Erlang is widely known as the messaging language that was used to scale WhatsApp while maintaining high availability. On the backend, Java services take the interactions from the Driftbot and pull it into a CRM, which allows sales and marketing people to manage information about the customers that are interacting with the chatbot. David gives lots more detail around the engineering stack, the deployment model, and his thoughts on the business and modern engineering. We recently launched a new podcast: Fintech Daily! Fintech Daily is about payments, cryptocurrencies, trading, and the intersection between finance and technology. You can find it on fintechdaily.co or Apple and Google podcasts. We are looking for other hosts who want to participate. If you are interested in becoming a host, send us an email: [email protected]

Engineers who start companies often find themselves building something they have no experience with: a hiring process. Hiring engineers today is not as systematic as building software. We don’t have lots of data that tells us what makes for an effective programming interview question. The smartest tech companies in the world are still making hiring mistakes–often through the “false negative” of rejecting candidates who did not do well in their interview process or through the “false positive” of hiring candidates who did well in the interview, but were not a good fit for the job. If you are a hiring manager or a company founder, you will eventually have to build a hiring process. If you don’t treat that hiring process scientifically, you will likely make some mistakes. Ammon Bartram has conducted more than 1000 interviews with engineers, accumulating a vast amount of data. This data was gathered deliberately and scientifically, through closely tracked interview questions and a consistent end-to-end process for the job candidate. Ammon joins the show to talk about the data set he has accumulated, the conclusions from all of these interviews, and how engineering organizations can use this data to develop a smart, data-driven hiring process. Ammon is co-founder of Triplebyte, a company that helps match engineers and tech companies. Triplebyte also publishes lots of research and blog articles about conducting good interviews, developer salaries, and bootcamps vs. computer science degrees. Full disclosure: Triplebyte is a sponsor of Software Engineering Daily. (However, Ammon has been a guest several times before on the show, since before Triplebyte was a sponsor, and I always enjoy getting to talk to him.)

I like to write music, and a year ago I started working on an album called “Gig Economy”. The plan for the album was to hire musicians from gig economy platforms like Fiverr and Upwork to perform on songs that I produced. The album is finished and I’m happy with the result, so I’m sharing it on the podcast today as an extra episode released on a Sunday. We’ll be back tomorrow with content about software engineering. Writing this album made me think about the future of work. I love music, but I am not a professional musician. I take the craft seriously, but perhaps not as seriously as someone who has made music their full-time career. I’ve never taken the time to network with musicians and develop collaborative relationships. The gig economy allowed me to pay creative people to collaborate with me. The transactional nature of my relationship with the collaborators meant that both of us had the incentive structure to work effectively. This improved quality and kept the pace of the project moving quickly. Today, the gig economy is widely regarded as a mode of work that removes individuality. Critics of the gig economy say that it turns workers into a commodity. If you use Uber, you aren’t thinking about the human who is driving you. As long as your driver has a high rating, you are satisfied. My experience producing this album with gig economy musicians showed a different side of our new employment systems. I found artists who I deeply enjoyed working with. There was no ambiguity about our relationship. Nobody was flaky. I paid these collaborators well, and in return they helped me fulfill an artistic vision. Without gig economy platforms, I could not have written this album. The gig economy is a playground for creativity. If you want to get paid to work as an artist, you can do so as long as you have a laptop. If you are a corporate worker making a good salary, but you spend your weekend producing art, you can pay artists to help you complete your vision. The greatest works of art are often the result of a talented workforce directed by an established leader. The gig economy lets you become a leader and recruit a team of creative, proven artists in a single day. I hope you enjoy “Gig Economy”. If you would rather listen on Spotify or YouTube, links are below. If you like the album, please share it on Twitter or Facebook. If you listened all the way through and have feedback for me, I’d love to know your thoughts. You can send me a tweet @the_prion or an email: [email protected]