Software Engineering Daily

A new software product usually starts with a single database. That database manages the tables for user accounts and basic transactions. When a product becomes popular, the database grows in size. There are more transactions and more users. A company grows around that product, and the company starts to accumulate more data in different sources. Analytics systems, time series databases, and logging tools start to generate data. Moving this data around between systems starts to become complicated. Apache Kafka is often used as a system for moving data between these different systems, performing transformations, and generating aggregations and summaries of these large quantities of data. Robin Moffatt works at Confluent, and has written numerous articles about how to move data between systems and design effective workflows for data pipelines. Robin joins the show to talk about modern data platforms and databases, and the patterns for using Kafka to connect those systems to each other. If you are interested in learning more about how companies are using Kafka, the Kafka Summit in San Francisco is September 30th – October 1st. Companies like LinkedIn, Uber, and Netflix will be talking about how they use Kafka. Full disclosure: Confluent (the company where Tim works) is a sponsor of Software Engineering Daily.

A software company needs to get many things right in order to be successful. Having a useful product with solid engineering is only the beginning. ReadMe was started five years ago. The company solved a seemingly simple problem: documentation for software products. If you have worked as a software engineer, you have looked at documentation. You know that there is a wide range of quality among the documentation that exists for different software products. ReadMe solved the problem of documentation-as-a-service, and it solved the problem better than any other company in the market. But after building a great business around documentation, the direction in which to take the business was unclear. How do you expand a documentation software business? Should you start building API management systems? Should you build lead generation tools? Should you try to sell products to developer evangelists? Greg Koberger is the CEO of ReadMe, and he joins the show to talk about the business, the strategy, and the fundraising process for ReadMe. Greg was previously on the show very early on, and he stood out as a guest that was remarkably friendly and willing to talk about a wide range of topics. In the previous show with Greg, he was one year into his business. Today, it has been five years since ReadMe was started, and we catch up on how his business has evolved since our first interview. ReadMe had become a profitable business very early in its life. In Silicon Valley, a quickly profitable business is the exception. Rather than structuring the finances of ReadMe with the expectation of successive funding rounds every 18-24 months, Greg took a slower approach. His company grew at a rate that was affordable while maintaining that profitability. While ReadMe maintained profitability from its core product of documentation software, Greg patiently thought about what product to build next within the company. A software company will typically expand into adjacent markets, or offer products that their current users are willing to pay additional money for. Eventually, ReadMe found its second product: developer metrics. Greg and his team figured out that gathering metrics around how APIs are being consumed has synergies with the core ReadMe product. With a second product, ReadMe was now able to forecast significant growth. The company also gained an incentive to raise money. With additional money, ReadMe would be able to deepen the developer metrics product, hire a bigger team, and think much bigger than they would have been able to as a single-product company with a smaller total addressable market. ReadMe raised a series A from Accel, one of the most respected venture capital firms in the business, and the company is now entering a new period in its development. ReadMe is a fascinating case study in balancing ambition with financial discipline at a software company.

Open source software is very new. Open source has existed for less than 30-40 years, depending on who you ask. The idea of open source was popularized by Linux, and open source software started to get heavily commercialized in the 1990s. By the early 2000s, open source was used by nearly every large software company. In recent years, most of the new databases, web frameworks, user interface libraries, and other software primitives are being built in the open. Dirk Hohndel is VP and chief open source officer at VMware, where he works on strategies around Linux, containers, Kubernetes, and other open source software. Dirk joins the show to talk about his long history with open source and the current state of the open source ecosystem.

Modern databases consist of multiple servers that host the data in a distributed fashion. Using multiple servers allows a database to be resilient to the failure of any one database node, because copies of the data are shared to other servers. A multi-node setup also lets the database grow beyond the size of data that could be hosted on a single node. Although a distributed database gains in scalability and resiliency, a database that runs across multiple nodes has a variety of problems that are not faced by a database running on a single node. Every operation with a distributed database becomes more complex than the single-node database. For example, if you make a query to your distributed database, you might not be able to rely on the answer that you get from a single database node, because the other database nodes might have been involved in transactions that have not propagated to all of the nodes. Aly Cabral is a lead product manager at MongoDB and co-author of a paper on causal consistency in MongoDB. Aly joins the show to discuss the engineering of distributed databases and her experience architecting MongoDB. Full disclosure: MongoDB is a sponsor of Software Engineering Daily.

Ever since Apache Kafka was open sourced from LinkedIn, it has been used to solve a wide variety of problems in distributed systems and data engineering. Kafka is a distributed messaging queue that is used by developers to publish messages and subscribe to topics with a certain message type. Kafka allows information to flow throughout a company such that multiple systems can consume the messages from a single sender. In previous shows, we have covered design patterns within Kafka, Kafka streams, event sourcing with Kafka, and many other subjects relating to the technology. Kafka is broadly useful, and new strategies for using Kafka continue to emerge as the open source project develops new functionality and becomes a platform for data applications. In today’s episode, Tim Berglund returns to Software Engineering Daily for a discussion of how applications are built today using Kafka–including systems that are undergoing a refactoring, data engineering applications, and systems with a large number of communicating services. If you are interested in learning more about how companies are using Kafka, the Kafka Summit in San Francisco is September 30th – October 1st. Companies like LinkedIn, Uber, and Netflix will be talking about how they use Kafka. Full disclosure: Confluent (the company where Tim works) is a sponsor of Software Engineering Daily.

A new employee at a software company needs access to a variety of tools. In order to get started working, the employee might need Slack, email, Google Docs, and Amazon Web Services, and all of these require an account with a username and password. Setting up all of these accounts can be time consuming, because the company needs to go into their admin portal and create the accounts. The accounts need to have the right security policies and configuration settings. And when the employee leaves the company, all of these accounts need to be shut down. Okta is a company that builds identity and access management software, such as an “SSO (single-sign on)” tool that allows users to log into all of these different types of accounts using only an Okta login. Okta was started in 2009 and has grown steadily since then, going public in 2017. Hector Aguilar is the president of technology at Okta and he joins the show to talk about the software stack of Okta and how the company has evolved over time as it has become a core infrastructure provider and hired a large engineering team.

Amazon Web Services first came out in 2006. It took several years before the software industry realized that cloud computing was a transformative piece of technology. Initially, the common perspective around cloud computing was that it was a useful tool for startups, but would not be a smart option for large, established businesses. Cloud computing was not considered economical nor secure. Today, that has changed. Every company that writes software is figuring out how to utilize the cloud. Software companies with on-prem servers are migrating old applications to the cloud, and most companies that have started in the last decade do not even have physical servers. Applications that are started on the cloud are referred to as “cloud-native.” The architecture of cloud-native applications is a newer topic of discussion, and some software patterns that became established in the pre-cloud era make less sense today. Cornelia Davis is VP of technology at Pivotal and the author of Cloud Native Patterns, a book about developing applications in the distributed, virtual world of the cloud. Cornelia was previously on the show to discuss Cloud Foundry. In today’s episode, our conversation centers on her book, and her perspective on the emerging patterns of cloud native software.

Python is one of the most popular programming languages in the software world. After working with Python and developing a love for the language, Michael Kennedy started to wonder why there was not a high quality podcast dedicated to covering the community and new technologies of the Python ecosystem. Michael started Talk Python To Me as a podcast with the goal of telling stories from the world of Python. Today, Talk Python To Me is the most popular podcast dedicated to Python and related technologies. Subjects on the podcast include web frameworks, compilers, career development, and cloud services. One of the more recent developments in the world of Python is the prevalence of data science, which Michael also covers in great detail. Michael has also spun up a second podcast called Python Bytes, which offers timely updates to the Python community. Michael joins the show to share his thoughts on several topics related to Python, including compilers, data science workflows, and web frameworks. Michael also gives his perspective on the world of software podcasting, which he has been doing for more than four years.

A new software application has simple requirements for a database. The database needs to be written to and read from. The database fulfills simple needs such as storing user information and providing the application frontend with the necessary data to render a simple webpage of financial transactions or blog posts. As an application becomes successful, the database grows in size. The complexity of queries increases, requiring more sophisticated logic in order to maintain performance. New databases need to be added to the overall system, as users begin to have demands for advanced features such as search, or analytics. Over time, the requirements for a database expand into a need for a data platform. A data platform might include multiple databases such as a NoSQL database, a relational database, a data warehouse, and a search index. The relationships between these different databases can vary in terms of consistency requirements, latency, and scalability. Andrew Davidson is the Director of Cloud Products at MongoDB. In a previous episode, Andrew discussed the tradeoffs of scaling databases while maintaining high performance indexing. Andrew returns to the show to discuss the emerging subject of “data platform.” As a growing number of companies have data requirements beyond that of a simple transactional database, Andrew’s work has increasingly involved figuring out the best ways for developers to adapt those transactional systems to providing a wider set of functionality. Full disclosure: MongoDB is a sponsor of Software Engineering Daily.

Spanner is a globally distributed, transactionally consistent database. Spanner initially emerged as a paper that came out of Google in 2012. Around this time, database scalability was difficult to solve, even for Google. The Spanner paper offered some breakthroughs in distributed systems which allowed Google to take some of the learnings from BigTable’s eventual consistency and construct a novel system that was transactionally consistent. Deepti Srivastava has worked on Spanner for more than six years. When she initially started working on Spanner, she was involved in the internally facing Spanner system that Google engineers used to spin up instances of the database. Today, she works as program manager for Cloud Spanner, a product within Google Cloud. Deepti joins the show to describe the breakthroughs of Spanner, and how her work has evolved since the product has gone from an internal system to an externally facing piece of cloud infrastructure.

Open source plays a key role in today’s world of technology businesses. Today, the impact of open source seems obvious. From Kubernetes to distributed databases to cloud providers, so much of our software is powered by open source. But it was not always this way. Bruce Perens was one of the earliest figures in the world of open source. He collaborated with Eric S. Raymond, the author of Cathedral and the Bazaar. He developed a differing belief set from Richard Stallman, who was another foundational advocate of open source. Bruce worked as an engineer at Pixar from 1987 to the year 2000. Bruce joins the show to give a history of open source and his experience in the software industry.

The browser has become the central application of the consumer operating system. Every piece of client software, from email to document management, has become usable through the browser. Even modern desktop software such as Slack is built using Electron, a tool for building client applications that essentially run inside of a browser without an address bar. One activity that still takes place largely outside of the browser is the process of writing and deploying code. A developer often uses an IDE such as Eclipse to write their code, then switches over to a terminal where they can build and deploy their code to a remote server running in the cloud. For a developer who has been writing code for a long time, this process feels completely intuitive. But for a new developer, it can be totally disorienting. New developers sometimes have trouble understanding the difference between a local and remote environment, or how to use repository management software like Git. This is in addition to all the other problems a new developer might be dealing with, such as language installation, syntax, and package management. Repl.it is a browser-based coding environment, compute engine, and collaborative workspace. Repl.it has found significant traction among new programmers who begin their programming journey within Repl.it and then stay in the environment, even as their applications become more richly featured and complicated. Repl.it is an amazing piece of software, and the story behind it is remarkable. Amjad Masad had the idea for Repl.it many years before he started the company for it, but he needed to first build up the money and confidence in order to go after the business with full force. Amjad joins the show to talk about his long journey towards building Repl.it, and to discuss the thriving Repl.it platform in its current form.

Modern software is built with cloud services, APIs, and other high level tools. Technical software development is moving beyond the realm of writing code. Individuals who do not have a background in computer science or programming can create increasingly complex tools. Higher level APIs include Twilio for managing phone communications, and Stripe for managing financial workflows. Platforms such as Shopify can be used as the core system for building an e-commerce business. Low-code tools such as Wix can be used for building web sites. Monday.com was started in 2012 with the goal of making software for managing business workflows. Business workflows link together different teams and processes within a company. Monday allows non-engineers to automate workflows that are traditionally automated by engineers. For example, Monday can be used to build workflows that link together SalesForce and Twilio, or to trigger actions in MailChimp based on how a user interacted with a Shopify web site. One of the biggest developing trends in technology is how non-engineers are becoming empowered to have more impact within an organization. The advances in these kinds of tools present us with an opportunity to rethink team structures, and invent entirely new roles for a software company. Eran Zinman is the CTO and co-founder of Monday.com and he joins the show to describe the vision for Monday and how the product fits into the changing enterprise software trends around APIs and “low code” tools. Monday has grown exponentially over the last 7 years, with the most recent growth curve looking almost vertical. Eran’s experience scaling the product and improving performance makes for excellent storytelling. Full disclosure: Monday is a sponsor of Software Engineering Daily.

Cassandra was initially released in 2008 as a project out of Facebook. Cassandra offered an open source solution to database scalability issues that were being tackled internally by large companies like Amazon, Google and Facebook. 2008 was a golden age of new infrastructure, with systems such as Hadoop and Kafka gaining traction around the same time. Jonathan Ellis started working with Cassandra, and became intrigued by the system. With the help of investors, Jonathan began working on a company based around Cassandra called Datastax. Today, Datastax is a company with more than 450 employees and a large valuation. Jonathan joins the show to discuss his experience working with Cassandra, and his reflections on the becoming an entrepreneurial founder of a highly successful database company. There is a wealth of useful knowledge for both software engineers and entrepreneurs in this episode.

Airlines have always had an emphasis on new technology. Over the years, airlines have needed to develop more and more software. Digital transformation is causing every large company to adopt the tools and practices of software companies, and that includes Delta Airlines. Delta Airlines has existed for more than 90 years. Over that period of time, the company has developed new systems in every generation of software, from the days of mainframe computers to a modern Java-based backend. When the DevOps movement started to take shape, Delta Airlines started to take a more focused approach on continuous integration, version control, and an organizational structure that removed silos between teams. Jasmine James is a manager at Delta Airlines where she focuses on improving the software practices of the company. She joins the show to talk through the process of changing the developer culture within Delta, as well as what it is like to build software for an airline. Jasmine is speaking at GitLab Commit in Brooklyn on September 17, 2019, and GitLab is a sponsor of Software Engineering Daily, so if you are thinking about attending GitLab Commit, you can go to softwareengineeringdaily.com/commit and enter code COMMITSED to save 30% on your ticket, while also supporting Software Engineering Daily.

Ilya Sukhar founded Parse in 2011 to make cloud services for mobile developers. Parse was a newer kind of cloud provider called “backend-as-a-service”, built to simplify the complexities of Amazon Web Services and the complexities of the mobile ecosystem. During this time, Facebook was in the middle of its shift toward becoming a mobile application company. Ads on the smartphone were not yet a proven business model. Facebook was exploring other business lines, and decided to purchase Parse for $85 million, with the intention of building a cloud developer platform. Shortly after the acquisition, Facebook’s mobile ads business started to see considerable success. With the mobile ads business finding traction, Facebook shifted all available resources towards supporting that business model. In 2017, Parse was shut down. Ilya joins the show to give his experience starting Parse, selling the company to Facebook, and then seeing the company he had built get shut down as it became an unfortunate casualty of Facebook’s advertising success. We talked a lot about the experience of building a backend-as-a-service company, as well as what makes Facebook special as an organization, and how the success of Facebook’s mobile business happened to coincide with the Parse acquisition.

Parse was a backend-as-a-service company that started in 2011 with the initial focus of making the cloud easier to use for mobile developers. Parse had several novel engineering challenges. In 2011, it was not easy to build on top of AWS, nor was it easy to build within the young mobile ecosystem. Charity Majors was an early engineer at Parse, and stayed with the company through its acquisition by Facebook. Charity joins the show to discuss her experience at Parse and her reflections on being part of Facebook after the acquisition. Parse was shut down in 2017, which was a disappointing outcome for Charity. Charity currently works as the CTO of Honeycomb. In a previous episode, Charity gave some background on her career in software, as well as her perspective on modern problems in observability and DevOps. In this episode, Charity tells the story of joining Parse, working through novel engineering problems, and the feeling of being acquired by a company where she did not feel like a great cultural fit. It is a useful conversation for anyone interested in engineering culture, company philosophy, and the realities of acquisitions–which can be both painful and profitable for the acquired employees.

Kent Beck is a legendary figure in the world of software engineering. Kent was an early advocate of Test-Driven Development (TDD), and popularized the idea of writing unit tests before writing code that would satisfy those unit tests. A unit test isolates and tests a small piece of functionality within a large piece of software. Practitioners of Test-Driven Development write tens or hundreds of tests in order to cover a large variety of cases that could potentially occur within their software. When Kent Beck joined Facebook in 2011, he was 50 years old and thought he had seen everything in the software industry. During Facebook Boot Camp, Kent started to realize that Facebook was very different than any other company he had seen. Facebook Boot Camp is the six-week onboarding process that every new hire learns about the software practices of the company. After graduating Facebook Boot Camp, Kent began to explore Facebook’s codebase and culture. He found himself rethinking many of the tenets of software engineering that he had previously thought were immutable. Kent joins the show to discuss his time at Facebook, and how the company’s approach to building and scaling products thoroughly reshaped his beliefs about software engineering.

When Chuck Rossi joined Facebook in 2008, he was one of the most experienced release engineers at the company. As he began to explore the engineering practices of the organization, he was surprised, confused, and impressed by the release engineering system that he encountered. Release engineering is the process by which software is released to users. As software is being developed, it moves through a series of testing environments. In these test environments, the software can be studied using simulated inputs that can help developers discover software bugs. Chuck had come to Facebook from Google. At Google, the crown jewel was Google web search which had a regimented release process. At Facebook, the crown jewel was facebook.com. Chuck found that the release process for facebook.com was much different than Google web search. Chuck joins the show to talk about release engineering at Facebook, and how the company constantly evolved its code deployment process. Chuck also describes Facebook’s pivot to mobile, and how the bottlenecks in the mobile app release process threatened Facebook’s ability to iterate and release quickly. This show provided some amazing perspective on continuous delivery, and will be useful to anyone who is working on figuring out their “DevOps” process. Chuck has a wealth of knowledge and context about the modern software industry.

Facebook is a large multiuser application. Scaling Facebook was different than scaling a single-user application such as an ecommerce store or a search engine. A social network is faced with unique infrastructure scalability challenges, as well as subjective questions around user communications, privacy, and content. Pedram Keyani worked at Google before joining Facebook in 2007. In his years at Facebook, he worked on infrastructure, internal tools, and management. He became deeply familiar with the company culture and its operations. Pedram joins the show to talk about how Facebook has scaled and the lessons he took away from his time there. After his time at Facebook, Pedram joined Uber, where he worked as a director of engineering for four years. Uber is another multi-user application, with a very different set of constraints. At Uber, Pedram worked on several projects, including Uber’s push into China, which he describes as an intense, competitive experience. Pedram is able to contrast the culture and scaling processes of Uber, Facebook, and Google which made this a rare opportunity to see how three different high performing companies build software differently.

Cryptocurrencies are decentralized monetary systems built on open source software. The open source software movement has evolved from the world of Linux, MySQL, and Apache to a thriving ecosystem of commercial enterprises built around open source software. This ecosystem includes projects such as Kubernetes, MongoDB, and ReactJS. It includes large organizations such as Amazon Web Services, Elastic, and Facebook. In a parallel software universe, the crypto ecosystem has built revolutionary new financial tools, creating billions of dollars of value, but not very many massive commercial companies. In the world of cryptocurrencies, many of the same rules apply to the classic open source world. But other rules do not. How do these two worlds differ from each other? How are they the same? And how might they end up colliding? Haseeb Qureshi, Joseph Jacks, and Alok Vasudev join the show for a spirited discussion of cryptocurrencies and open source. Haseeb is a cryptocurrency investor, JJ is the founder of OSS Capital, and Alok is an engineer and the founder of crypto venture capital firm Standard Crypto.

When a new employee joins a software company, it is often unclear where that employee should begin. Do they have a mentor? What are they working on? What are the expectations for how fast that employee should be contributing? The early period of employment is often referred to as “onboarding.” During the onboarding period, an employee will learn the basics of the company they have just joined. These basics include the work procedures, meeting schedules, expectations, and technical tools that an employee needs to become productive. A poor onboarding process can slow an employee down, and even cause them to quit the company entirely. Conversely, a good onboarding process can accelerate an employee towards rapid contribution within the organization. A scalable onboarding process can be a difference in millions of dollars in productivity per year across the organization. Kristen Gallagher is the founder of Edify, a company that helps organizations define and implement their onboarding process. Kristen joins the show to talk about the ingredients of a successful onboarding process and what she is doing with her company Edify.

A time series database is optimized for the storage of high volumes of sequential data across time. Time series databases are often organized as columnar data stores that can write large volumes of data quickly. These systems can sometimes tolerate data loss, because the data they are gathering is used for monitoring and other applications that require aggregated data sets rather than highly important individual transactions. The demand for time series databases has grown over the last decade with the rise of mobile devices and the decreasing cost of cloud storage. There has been an increase in the number of systems that require monitoring, and some of those systems produce an incredibly large amount of data, requiring compression, downsampling, and garbage collection. Rob Skillington is an engineer at Uber, where he helped create M3DB, a time series database. In a previous show, Rob described the basics of M3DB and how it helps Uber with storing data from Prometheus, a monitoring system. In today’s show we discuss the field of time series databases, and Rob’s approach to building M3.

Insurance is an old business. Individuals and businesses have been buying insurance policies for decades. These insurance policies can cost hundreds, thousands, or tens of thousands of dollars per year. Software is remaking the insurance industry. One way new software can improve the insurance industry is through better brokerage technology. Insurance involves carriers and brokers, who work together on delivering insurance solutions to customers. The initiation and closing of an insurance transaction often involves lots of emails, PDF files, and antiquated software systems. Technology improvements will smooth out this process and reduce manual overhead. Another way insurance can improve thanks to technology is through smarter pricing. The price of an insurance policy is offered to a customer based on how risky the insurance policy is, how large the customer pool is, and how much the insurance company could lose in the event that it would have to pay out an insurance policy. These risk profiles are calculated based on historical data. Gordon Wintrob is the co-founder and CTO of Newfront Insurance. Gordon joins the show to discuss the insurance industry and how his company got started.

Open source software has evolved into a thriving, multifaceted ecosystem. Open source encompasses operating systems and databases. Open source embodies both altruism and self-interest. And open source enables thriving businesses from WordPress blogs to hundred billion dollar cloud providers. There is a large set of business models that can be built around a successful open source project. Some of these business models are more defensible than others. A company based around open source must make a deliberate set of tradeoffs around how their private company will relate to the public open source ecosystem. Today’s episode is a discussion of open source business models with Karthik Ranganathan, Heather Meeker, and Matt Asay. Karthik is the CTO of YugaByte, an open source distributed SQL database. Heather Meeker is an open source licensing specialist and a founding partner of OSS Capital. Matt Asay is a longtime executive who has worked in several open source based companies including MongoDB–he has also written at length about open source. Full disclosure: YugaByte, where Karthik works, is a sponsor of SE Daily.

Andreas Antonopoulos is the author of several books about cryptocurrency engineering, including Mastering Bitcoin and Mastering Ethereum. In these books, Andreas lays out the systems of economics and computer science that underpin the two most mature decentralized monetary systems. When Andreas originally discovered the Bitcoin whitepaper, he had witnessed the repeated mismanagement of government-backed fiat currencies. Andreas has a Greek background, and the financial collapse of 2008 had led to an economic crisis in Greece. His firsthand observation of the weaknesses of centrally planned government currencies, together with a degree in computer science and distributed systems have made him a dedicated evangelist for Bitcoin. Andreas joins the show to discuss the Bitcoin ecosystem, and the relationship between decentralized cryptoeconomic systems and centralized corporations. Facebook has recently announced a cryptocurrency project called Libra, and Andreas suggests that Libra changes everything–not necessarily because Libra will make it to production, or because Libra itself will upend the world of finance–but because it allows us to further call into question the very nature of what makes a modern currency valuable and valid. After all, if a large government has the right to back a currency, why shouldn’t a large corporation have that same privilege? Andreas is also a co-host of one of my favorite Bitcoin podcasts, Let’s Talk Bitcoin.

Software engineers often work as a contractor for some duration of their career. A contractor earns a fixed hourly salary for a defined period of weeks, months, or years. Contract work can be more flexible than full-time work, and often pays more than full-time software engineering, because contract jobs can end at any time, and they do not have the added employee benefits such as health insurance and stock options. Online contracting platforms such as Upwork and Fiverr have expanded the number of software contracting engagements that take place. Developers on Upwork and Fiverr have a wide range of skills and experience levels. The clients who come on Upwork and Fiverr looking for developers are sometimes unsophisticated at managing software projects, and in some cases they defraud the software developers who are making their living online, and trick these software developers into delivering free work. Moonlight is a contracting platform for software engineers. Today’s guests Emma Lawson and Philip Thomas are the founders of Moonlight, and they join the show to explain why they started the company, and the gaps that exist in the world of software contracting. Moonlight’s model is different than most other contracting platforms, in that Moonlight requires clients to pay a $300 subscription fee to recruit engineers on the platform. This $300 subscription price lowers the rate at which clients take advantage of software engineers. This dynamic causes the software engineers to take their work more seriously and act more professionally. Full disclosure: I am an investor in Moonlight. I have also been a paying client of the service, and our newest version of the Software Daily app was built by Mostafa Gazar, who is a talented Android engineer I met on Moonlight.

The service mesh abstraction allows for a consistent model for managing and monitoring the different components of a microservices architecture. In the service mesh pattern, each service is deployed with a sidecar container that contains a service proxy. These sidecars are collectively referred to as the “data plane.” Each sidecar provides the service that it is deployed next to with a set of features such as security policy, rate limiting, and monitoring instrumentation. The sidecars in the data plane communicate with a central module called a control plane. In the control plane, an engineer can operate across these individual services at scale, by pushing out updates to them. Kubernetes has made it easier to manage large fleets of microservices, and has led to wider adoption of service mesh. Istio is one of the most popular service mesh products. In today’s show, Varun Talwar returns to the show to describe the state of the Istio project and the process of deploying Istio to a cluster. Varun is the CEO of Tetrate, a company building an enterprise-ready service mesh. Prior to Tetrate, Varun was at Google, where he helped found the gRPC and Istio projects.

The PlayStation is a line of game consoles created by Sony. PlayStation devices include the PS2, PS3, PS4, and the PSP mobile system. Tony Godar worked as an engineer in the PlayStation ecosystem for 15 years, and he joins the show to give a retrospective on his time in the console industry. Developing hardware and software for game consoles differs significantly from the world of web development. Tony describes the culture of the game development world, and the challenges involved in the domains of software tooling, custom operating systems, and streaming media. In 2010, the PS3 was hacked by notorious tinkerer George Hotz, a previous guest on the show, an event which Tony also discusses. We also discuss the world of modern gaming and VR technology. Tony currently works as an engineer at MelodyVR, a company that makes virtual reality live music experiences.

Large software companies have become a target for criticism. Google, Facebook, Amazon and other prominent technology giants find themselves under a kind of scrutiny that is reminiscent of banks in 2008 and oil companies in the early 1900s. Across the planet, there is a growing sentiment that “big tech” has too much power, and that they are abusing that power in order to dominate markets, censor speech, and manipulate politics. Tyler Cowen is the author of Big Business: A Love Letter to an American Anti-Hero. He is also the host of the popular podcast Conversations with Tyler and an economist at George Mason University. In Big Business, Tyler explores the modern relationships between consumers and businesses, including the large technology companies. Tyler joins the show to discuss his new book. In previous episodes with Tyler, we also talked about his previous books as well as the effects of technology on American society and the philosophical approaches software engineers can bring to their careers.

In this episode Will Larson, author of An Elegant Puzzle: Systems of Engineering Management, speaks with Uma Chingunde of Stripe and Jeff Meyerson of Software Engineering Daily about engineering management. Will was also featured on SE Daily recently. An Elegant Puzzle is an excellent resource on management techniques and strategies for scaling software organizations. Will’s writing draws from his experiences working at Uber and Stripe.

The a16z Podcast is a show that is produced by Andreessen Horowitz, an investment fund based in Silicon Valley. The a16z Podcast covers topics including software engineering, biology, media, cryptocurrencies and entrepreneurship. A16z is one of the most popular podcasts about technology. Sonal Chokshi is the editor in chief at Andreessen Horowitz and the showrunner for the a16z podcast. For five years, she has been interviewing entrepreneurs, engineers, artists, and investors, exploring how software has increasingly impacted our lives and transformed society. The success of the a16z Podcast is largely a result of Sonal’s high editorial standards and her ability to ask the right questions and drive conversations in fruitful directions. Much of the content of Software Engineering Daily has been shaped by a16z, and I have listened to every single episode. Sonal Chokshi joins today’s show for a conversation about podcasting and technology. Sonal shares her beliefs for why the podcast medium has taken off, and describes how her background in education, ethnography, and technology has shaped the completely distinct voice and flavor of the a16z Podcast.

Data-as-a-service businesses offer paid access to data sets. These data sets can be useful for building products or training machine learning models. There has been steady growth in the tools and practices around processing and storing data. But access to data sets remains a bottleneck for widespread development of machine learning applications in a large set of domains. SafeGraph is a company focused on the problem of data-as-a-service. Today, SafeGraph’s primary product is reliable, up-to-date location information on places. The data for these points-of-interest needs be acquired, verified, cleaned, made accessible through an API, and intelligently priced. In previous episodes with SafeGraph, we have explored the basic premise of data businesses and why they are important platforms for building futuristic data products that are impossible for entrepreneurs to build today. SafeGraph CEO Auren Hoffman returns to the show to discuss the data as a service business model. Software-as-a-service has existed as a category for more than a decade. Infrastructure-as-a-service has existed for just as long. Data-as-a-service is much more undeveloped. Auren recently published the “Data-As-A-Service Bible: Everything You Wanted To Know About Running A DaaS Business”. This was a very useful article, as it breaks down a category of software that is almost entirely unexplored.

A large sales organization has hundreds of sales people. Each of those sales people manages a set of accounts who they are trying to close sales deals on. Sales people are overseen by managers who ensure that the sales people are performing well. Directors and VPs ensure the scalability and health of the overall sales organization. The sales lifecycle mostly takes place within a piece of software called a CRM: customer relationship management. This tool documents the interactions between sales people and accounts. CRMs have been around for many years, and although CRM software is a useful repository of data, it does not fulfill all the needs of a salesperson. People.ai is a system of machine learning tools built around the sales tooling ecosystem. People.ai helps a sales organization avoid manual data entry, understand areas of potential improvement, and decide on who the highest value sales lead to pursue might be. Andrey Akselrod is the CTO At People.ai and he joins the show to discuss the potential applications of machine learning in the domain of sales, and the engineering work that his company has done.

During 2015, Uber was going through rapid scalability. The internal engineering systems were constantly tested by the growing user base. Over the next two years, the number of internal services at Uber would grow from 500 to 2000, and standardizing the monitoring of all these different services became a priority. After working with a variety of available tools, Uber’s engineering team decided that something new needed to be built internally. Jaeger is an open source distributed tracing tool that provides observability features throughout Uber’s microservices architecture. Yuri Shkuro is an engineer at Uber, where he works on Jaeger and other infrastructure projects. He joins the show to discuss the history of engineering at Uber, the architecture of Jaeger, and the requirements for building and scaling a distributed tracing tool.

A knowledge base assembles information from a wide variety of sources into a central platform. The most popular knowledge base is Wikipedia, which covers a wide variety of concepts through a system that attempts to remain authoritative and impartial. Other open knowledge platforms include Stack Overflow, which focuses on programming concepts, and Quora, which adds a social element to the process of information accumulation. Golden.com is a knowledge base that indexes, categorizes, and surfaces information. Golden has information about software, genetics, world history, social media, sports, and all kinds of other subjects. The company monetizes with a paid knowledge base product for enterprises, which allows for easy easy querying of the open knowledge base and a private knowledge base for internal information. Jude Gomila is the founder of Golden, and joins the show to discuss the process of building a universal knowledge base, and the engineering problems that he is working on to improve Golden.

Coding bootcamps allow anyone to become a programmer at a faster pace than the traditional computer science education system. In the last five years, coding bootcamps have grown rapidly in popularity, with thousands of people gaining the necessary skills to work as a software engineer. Career Karma is a platform that allows individuals to find the best coding bootcamp. There are many coding bootcamps, and they are not all the same. Much like different schools have different cultures and focus on different disciplines, coding bootcamps vary widely in their teaching styles and acceptance path. Reuben Harris and Artur Meyster are co-founders of Career Karma, and they join the show to discuss the changing nature of software engineering education and the frictions that new programmers encounter as they navigate the world of coding bootcamps. They also describe their journey to entrepreneurship and their own personal experience with coding bootcamps.

A hackathon is an organized event where participants work together to build a product or tool. Hackathons are about creativity, learning, and exploration. A developer that is participating in a hackathon is often working on something that is outside of their normal day-to-day focus. Hackathons can provide significant value to the participants. Hackathons have led to friendships, new companies, and newly developed confidence that can be critical to a developer who feels uncertain in their ability to launch their own projects. Jonathan Gottfried is a co-founder of Major League Hacking, an official student hackathon league that powers invention competitions. Major League Hacking is a B corporation that is focused on improving the education and community of technology leaders, entrepreneurs, and young hackers. Jon joins the show to discuss hackathons–and how he has built and scaled an organization that is devoted to creating systematically successful hackathon experiences.

Videos play a key role in the continuing education of a software engineer. Video can capture many different types of content that is useful for engineers: conference talks, tutorial videos, and podcast-style interviews are all popular formats of online video. YouTube has become the predominant source for video content about software engineering. The open nature of YouTube’s format allows for a tremendous range of content. No matter what your preference is for how you like to learn and be entertained, YouTube has something for you. TechPrimers is a media channel that is dedicated to sharing technical knowledge in the form of videos, GitHub repositories, and a thriving community. TechPrimers has over 300 videos on YouTube, and more than 48,000 subscribers who regularly watch content about subjects like AWS, Spring, and Kubernetes. TechPrimers was founded by Ajay Kumar, a software engineer and vice president at JP Morgan. Ajay has been in the software industry for fifteen years, and has been working in enterprise banking systems for seven years. He has deep experience in modern technologies and engineering practices. Ajay joins the show to discuss the modern world of software engineering, and his experience building a media platform. He also talks about the technology culture of India. Ajay is based in Bengalaru, and it was exciting to learn how much our different societies have in common thanks to technology. Ajay also had me on his YouTube channel for an interview, which was a lot of fun.

“Internet of Things” is a term used to describe the increasing connectivity and intelligence of physical objects within our lives. IoT has manifested within enterprises under the term “Industrial IoT,” as wireless connectivity and machine learning have started to improve devices such as centrifuges, conveyor belts, and factory robotics. In the consumer space, IoT has moved slower than many people expected, and it remains to be seen when we will have widespread computation within consumer devices such as microwaves, washing machines, and lightswitches. IoT computers have different constraints than general purpose computers. Security, reliability, battery life, power consumption, and cost structures are very different in IoT devices than in your laptop or smartphone. One technology that could solve some of the problems within IoT is WebAssembly, a newer binary instruction format for executable programs. Jonathan Beri is a software engineer and the organizer of the San Francisco WebAssembly Meetup. He has significant experience in the IoT industry, and joins the show to discuss the state of WebAssembly, the surrounding technologies, and their impact on IoT.

Data engineering involves numerous tools–a data lake, databases, data warehouses, numerous APIs, streaming systems, and microservices. There is no shortage of ways to interact with data and manage data, but many companies are struggling to figure out design patterns and best practices for how to manage data and build data infrastructure. Zhamak Dehgani is a principal consultant and portfolio director with ThoughtWorks, where she works with enterprises to improve their software systems and workflows. She is the author of an article called How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh and she joins the show to discuss her perspective on data infrastructure, as well as the “data mesh” concept that she has coined. Data mesh represents the movement away from having a centralized data lake that all teams interact with, and towards having different data products and individual data management systems for individual domain teams.

Software companies such as Slack, Zoom, and Uber have recently gone public. When a company goes public, they issue a document called an S-1. Within the S-1, there is a wealth of information about the company, providing a detailed story about the company’s business model, economics, and future prospects. The S-1 describes the operating model and the philosophy of a newly public company. Going public serves several purposes. Being public allows a company to gain access to the public capital markets. It allows previous investors to have a liquidity event, by selling the shares that they purchased from the company in private markets. Being public also puts some constraints and visibility on a company, which can be useful for a company that is trying to develop internal discipline. In the software industry, it is useful for most people to understand the dynamics of going public. A technology worker who is earning equity at a private company needs to understand the roadmap to their company going public, or potentially getting acquired. Anyone who invests in public technology stocks is evaluating the different available options for investment, and considering the best software companies to place a bet on. Tom Tunguz is a venture investor at Redpoint, and the author of a popular blog at tomasztonguz.com. In a recent series of posts, Tom has evaluated the S-1s and compared the growth dynamics between a variety of newer public software companies. Tom joins today’s show to discuss his writing, and offer reflections on what can be learned about company building from the recent series of IPOs and direct listings.

Envoy is an open source edge and service proxy that was originally developed at Lyft. Envoy is often deployed as a sidecar application that runs alongside a service and helps that service by providing features such as routing, rate limiting, telemetry, and security policy. Envoy has gained significant traction in the open source community, and has formed the backbone of popular service mesh projects such as Istio. Envoy has been mostly used as a backend technology, but the potential applications of Envoy include frontend client applications as well. The goal of Envoy is to make the network easier to work with–and the network includes client applications such as mobile apps running on a phone. Envoy Mobile is a network proxy for mobile applications. Envoy Mobile brings many of the benefits of Envoy to the mobile client ecosystem. It provides mobile developers with a library that can simplify or abstract away many of the modern advances that have been made in networking in recent years, such as HTTP2, gRPC, and QUIC. Matt Klein is the creator of Envoy, and he joins the show to discuss Envoy Mobile. Matt describes how the networking challenges of mobile applications are similar to those of backend systems and cloud infrastructure. We discuss the advances in networking technology that Envoy Mobile helps bring to the mobile ecosystem, and also touch on the scalability challenges that Matt is seeing at Lyft.

In the developed world, it is easy to take for granted that we grew up with computers. Technology is so pervasive in the United States that we have debates about how early in child development a human should be given a smartphone. Across much of Africa, there is a shortage of access to computers. Children grow up without much exposure to computers at all. Smartphones are starting to proliferate the continent, but the bandwidth limitations prevent the sort of unrestricted mobile Internet usage that many of us have in the west. Tyler Cinnamon is a software engineer and the co-founder of TechLitAfrica, an organization dedicated to improving technology literacy and reducing poverty in Africa. TechLitAfrica takes old computers from the United States which are no longer in use and repurposes those computers with educational software and a downloaded subset of the Internet. Then, TechLitAfrica takes those computers to Africa and sets them up as computer labs. Tyler joins the show to talk through the technical and cultural challenges of building TechLitAfrica.

The Hadoop ecosystem provided every company with the tools to store and query large amounts of data at a low cost. Since 2005, that ecosystem has expanded with more and more open source applications for data infrastructure. Data infrastructure includes databases, data lakes, distributed queues, data warehouses, query engines, web applications, on-prem software, closed source, open source, cloud platforms, and software-as-a-service. There is so much software, and it is difficult to keep track of everything in the data engineering ecosystem. Tobias Macey is the host of the Data Engineering Podcast, a show about technologies and practices in the world of data engineering. Tobias joins today’s episode as a guest to give his perspective on the evolving landscape of data engineering, and the trends he is seeing on the data engineering podcast.

From Africa to India to Asia to South America–computer science and programming are rising in popularity in every emerging market. Each of these markets has regional needs for technology. Just like every culture develops its own food and television, every culture needs different types of applications to run their lives. In Vietnam, the day-to-day life of a citizen is different than it is in the United States. Yes, everyone needs Google and YouTube and Instagram. But the trends in messaging, food delivery, gig economy, and other B2C technology sectors are considerably different than the west. Charles Lee is the founder of CoderSchool a coding school in Vietnam. Before moving to Vietnam to start CoderSchool, he worked as a software engineer in San Francisco for several years. In today’s show, Charles describes the difference between the US and Vietnamese technology sectors, from consumer applications to business software to coding education.

In 2011, Facebook had begun to focus its efforts on mobile development. Mobile phones did not have access to reliable, high bandwidth connections, and the Facebook engineering team needed to find a solution to improve the request latency between mobile clients and the backend Facebook infrastructure. One source of latency was recursive data fetching. If a mobile application client made a request to the backend for newsfeed, the backend API would return the newsfeed, but some components of that feed would require additional requests to the backend. In practice, this might result in a newsfeed loading partially on a phone, but having a delayed loading time for the comments of a newsfeed item. GraphQL is a solution that came out of this problem of recursive data fetching. A GraphQL server provides middleware to aggregate all of the necessary information to serve a complete request. GraphQL connects to backend data sources and federates the frontend request across these different data sources. GraphQL was open sourced in 2015, and has found many use cases in addition to simplifying backend data fetching for mobile clients. Today, GraphQL is used by PayPal, Shopify, Twitter, and hundreds of other companies. Lee Byron is the co-creator of GraphQL and he joins the show to tell the story of GraphQL, and how it fit into Facebook’s shift to mobile.

Facebook has released open source software projects that have changed the industry. The most impactful projects to date are the React frontend user interface tools: ReactJS and React Native. Before React became popular, there were multiple competing solutions for the dominant frontend JavaScript framework. React became the most prominent because of its invention of JSX, its one-way data flow, and the strength of its community. The React community is led by Facebook engineers. Facebook has a full-time team dedicated to improving React. Facebook also has the benefit of seeing the hardest problems in web development ahead of the rest of the industry, since Facebook is always pushing the leading edge of what software can do. As React evolves, Facebook itself is the first user of updates to the libraries. Engineers on the React team will proactively update the applications on other teams with new React code. In this way, Facebook’s React team is similar to a platform engineering team, except the open source community benefits from the improvements as well. Tom Occhino is an engineering director at Facebook. He joins the show to discuss his ten years of experience at the company and his role today as a senior leader in charge of open source developers.

Facebook generates high volumes of data at a rapid pace. Dhruba Borthakur joined Facebook in 2008 to work on data infrastructure. His early projects at Facebook were around Hadoop, the distributed file system and MapReduce computation platform that laid the foundation for the “big data” movement. At the time, Facebook was generating as much data as any other startup, and the company needed to stay at the leading edge of scalability techniques for its Hadoop Distributed File System (HDFS) cluster. Traditionally, Hadoop managed its file system by synchronizing the coordination of the different data nodes with the help of a single master node. At Facebook, the scale of the data was such that the HDFS cluster had thousands of data nodes, which was too much volume for a single master node to handle. Dhruba helped implement redundancy at the master node to create a more resilient system. The early days of the big data movement was focused on batch processing. A company like Facebook would gather large amounts of data into databases and HDFS, and run offline analytics workloads to gather reports on an hourly, daily, or weekly basis. Over time, data infrastructure has moved closer to a “real time” processing model. Data infrastructure does not only support batch offline reporting–it also supports machine learning jobs that need to be run on a more frequent basis. These jobs have lower latency requirements, and have driven the adoption of in-memory stream processing systems like Spark and Flink. Dhruba joins the show to discuss his time at Facebook building data infrastructure. He takes us through the major projects he worked on, including the early Hadoop infrastructure, the refactoring of online user workloads to be more “pull” based than “push” based, and the creation of RocksDB, a storage engine he helped create at Facebook. Today, Dhruba is the CTO and co-founder of Rockset, a company that builds data infrastructure and database APIs on top of RocksDB. Rockset is building infrastructure for modern technology companies–many of which are facing problems that bear significant resemblance to the ones Facebook encountered as it scaled.

Facebook moves fast because of vision, collaboration, and trust. The fast pace of development is enabled by constantly improving infrastructure and a sense of unity throughout the company. In Facebook’s early days, there was an emphasis on rapidly deploying new code to drive constant improvement and experimentation. Product quality was maintained by engineers closely checking each other’s code reviews, rather than writing detailed unit testing suites. Facebook engineers had a sense for how the product should operate, and they were able to evaluate whether a new feature was working properly by testing a live version of that feature. At Facebook, the vision of the company is clearly communicated to the employees. Every employee within Facebook can articulate the vision for the company, and will use similar language in describing that vision. Since the employees are aligned on strategy, they can also align in their implementation of product features. This reduces conflicts across roles and between teams. Facebook has also shown a willingness to trust its engineers. Trust was exemplified by Facebook’s tolerance for failures in the early days. When an engineer broke a build, or shipped a feature that failed to gain traction, that engineer was usually not punished. They may have even been rewarded, if the company could learn significantly from such an error. Raylene Yung was an engineer at Facebook from 2009 to 2015. As she moved from individual contributor to manager to engineering director, Raylene worked on products including News Feed, Timeline, Privacy, and Sharing. Raylene joins the show to give her reflections on the Facebook product and engineering environment. She explained how Facebook’s culture of collaboration, vision, and trust drive fast product development and minimizes conflict. Raylene left Facebook and joined Stripe, where she worked on payments systems and international expansion for almost four years.