Software Engineering Daily

GraphQL is a system that allows frontend engineers to make requests across multiple data sources using a simple query format. In GraphQL, a frontend developer does not have to worry about the request logic for individual backend services. The frontend developer only needs to know how to issue GraphQL requests from the client, and these requests are handled by a GraphQL server. GraphQL is mostly used to issue queries across internal databases and services. But many of the data sources that a company needs to query in modern infrastructure are not databases–they are APIs like Salesforce, Zendesk, and Stripe. These API companies might store a large percentage of the data that a given company needs to query, and executing queries, subscriptions, and joins against these APIs is not a simple task. OneGraph is a company that builds integrations with third-party services and exposes them through a GraphQL interface. Sean Grove is a founder of OneGraph, and he joins the show to explain the problem that OneGraph solves, how OneGraph is built, and some of the difficult engineering challenges required to design OneGraph.

A data warehouse serves the purpose of providing low latency queries for high volumes of data. A data warehouse is often part of a data pipeline, which moves data through different areas of infrastructure in order to build applications such as machine learning models, dashboards, and reports. Modern data pipelines are often associated with the term “ELT” or Extract, Load, Transform. In the “ELT” workflow, data is taken out of a source such as a data lake, loaded into a data warehouse, and then transformed within the data warehouse to create materialized views on the data. Data warehouse queries are usually written in SQL, and for the last 50 years, SQL has been the primary language for executing these kinds of queries. DBT is a system for data modeling that allows the user to write queries that involve a mix of SQL and a templating language called Jinja. Jinja allows the analyst to blend imperative code along with the declarative SQL. Tristan Handy is the CEO of Fishtown Analytics, the company that created DBT, and he joins the show to discuss how DBT works, and the role it plays in modern data infrastructure.

ReactJS developers have lots of options for building their applications, and those options are not easy to work through. State management, concurrency, networking, and testing all have elements of complexity and a wide range of available tools. Take a look at any specific area of JavaScript application development, and you can find highly varied opinions. Kent Dodds is a JavaScript teacher who focuses on React, JavaScript, and testing. In today’s episode, Kent provides best practices for building JavaScript applications, specifically React. He provides a great deal of advice on testing, which is unsurprising considering he owns TestingJavaScript.com. Kent is an excellent speaker who has taught thousands of people about JavaScript, so it was a pleasure to have him on the show. Kent is also speaking at Reactathon, a San Francisco JavaScript conference taking place March 30th and 31st in San Francisco. This week we will be interviewing speakers from Reactathon, and if you are interested in JavaScript and the React ecosystem then stay tuned, and if you hear something you like, you can check out the Reactathon conference in person.

JavaScript fatigue. This phrase has been used to describe the confusion and exhaustion around the volume of different tools required to be productive as a JavaScript developer. Frameworks, package managers, typing systems, state management, GraphQL, and deployment systems–there are so many decisions to make. In addition to the present-day tooling choices, a JavaScript developer needs to watch the emerging developments in the ecosystem. ReactJS is evolving at a rapid clip, and newer primitives such as React Hooks and React Suspense allow developers to handle concurrency and networking more robustly. Tejas Kumar works with G2i, a company that connects React developers with organizations that are looking for high-quality engineers. His role at G2i is head of vetting, which requires him to assess engineers for their competency in JavaScript-related technologies. Tejas joins the show to discuss the modern stack of technologies that a React developer uses to build an application. Full disclosure: G2i, where Tejas works, is a sponsor of Software Engineering Daily. Tejas is also speaking at Reactathon, a San Francisco JavaScript conference taking place March 30th and 31st in San Francisco. This week we will be interviewing speakers from Reactathon, and if you are interested in JavaScript and the React ecosystem then stay tuned, and if you hear something you like, you can check out the Reactathon conference in person.

Full-stack JavaScript applications have been possible since the creation of NodeJS in 2009. Since then, the best practices for building and deploying these applications have steadily evolved with the technology. ReactJS created consolidation around the view layer. The emergence of AWS Lambda created a new paradigm for backend execution. Serverless tools such as DynamoDB offer autoscaling abstractions. CDNs such as Cloudflare and Fastly can now do processing on the edge. Brian LeRoux is the founder of Begin.com, a hosting and deployment company built on serverless tools. He’s also the primary committer to Architect, a framework for defining applications to be deployed to serverless infrastructure. Brian joins the show to talk about his work in the JavaScript ecosystem and his vision for Begin.com. Brian is also speaking at Reactathon, a San Francisco JavaScript conference taking place March 30th and 31st in San Francisco. This week we will be interviewing speakers from Reactathon, and if you are interested in JavaScript and the React ecosystem then stay tuned, and if you hear something you like, you can check out the Reactathon conference in person.

ReactJS began to standardize frontend web development around 2015. The core ideas around one-way data binding, JSX, and components caused many developers to embrace React with open arms. There has been a large number of educators that have emerged to help train developers wanting to learn React. A new developer learning React has numerous questions around frameworks, state management, rendering, and other best practices. In today’s episode, those questions are answered by Ryan Florence, a co-founder of React Training. React Training is a company devoted to helping developers learn React that trains large companies like Google and Netflix how to use React. Ryan has a strong understanding of how to be productive with React, and in today’s episode, he explains some of the fundamentals that commonly confuse new students of React. Ryan is also speaking at Reactathon, a San Francisco JavaScript conference taking place March 30th and 31st in San Francisco. This week we will be interviewing speakers from Reactathon, and if you are interested in JavaScript and the React ecosystem then stay tuned, and if you hear something you like, you can check out the Reactathon conference in person.

When ReactJS became popular, frontend web development became easier. But React is just a view layer. Developers who came to React expecting a full web development framework like Ruby on Rails or Django were required to put together a set of tools to satisfy that purpose. A full-stack JavaScript framework has numerous requirements. How does it scale? How does it handle server-side rendering versus client-side rendering? Should GraphQL be included by default? How should package management work? Guillermo Rauch is the creator of NextJS, a popular framework for building React applications. He is also the CEO of ZEIT, a cloud hosting company. Guillermo joins the show to discuss NextJS, and his vision for how the React ecosystem will evolve in the near future, as features such as React Suspense and Concurrent Mode impact the developer experience. Guillermo is also speaking at Reactathon, a San Francisco JavaScript conference taking place March 30th and 31st in San Francisco. This week we will be interviewing speakers from Reactathon, and if you are interested in JavaScript and the React ecosystem then stay tuned, and if you hear something you like, you can check out the Reactathon conference in person.

Slack is a messaging application with millions of users. The desktop application is an Electron app, which is effectively a web browser dedicated to running Slack. This frontend is built with ReactJS and other JavaScript code, and the application is incredibly smooth and reliable, despite its complexity. When a user boots up Slack, the application needs to figure out what data to fetch and where to fetch it from. Companies that use Slack heavily have thousands of messages in their history, and Slack needs to determine which of those should be pulled into the client. There are profile images, and logos, and custom emojis, all of which are used to define the user’s custom workspace experience. Anuj Nair joined Slack in late 2017. In the years since he has been with the company, Anuj helped rewrite the Slack frontend client, including work on the bootup experience, the caching infrastructure, and the role of service workers. Anuj joins the show to discuss his work on the Slack frontend architecture and the canonical view layer problems that Slack faces.

Low code tools can be used to build an increasing number of applications. Knowledge workers within a large corporation can use low code tools to augment their usage of spreadsheets. Entrepreneurs can use low code tools to start businesses even without knowing how to code. Modern low code tools have benefited from steady improvements in cloud infrastructure, front-end frameworks like ReactJS, and browser technology such as the V8 JavaScript engine. These building blocks led to the popular low code products such as Webflow, Bubble, Retool, and Airtable. The low code products are supported by a broad selection of domain-specific APIs such as Stripe, Twilio, and Zapier. Ben Tossell runs Makerpad, a site devoted to low-code and no-code applications. Makerpad describes how to use these tools to design sophisticated applications that don’t require you to write code. But they do require a different kind of software engineering. To create applications intelligently with low-code tools, you need to know how the tools fit together, and you need to be willing to persist through a process of iteration and debugging that is similar to traditional software engineering. Ben joins the show to talk about his experience building low-code tools, the use cases for these tools, and his predictions for how they will impact the future of software.

Every company has a large number of routine data workflows. These data workflows involve spreadsheets, CSV files, and tedious manual work to be done by a knowledge worker. For example, data might need to be taken from Salesforce, filtered for new customers, and piped into Mailchimp. Or perhaps you need to sort all your customers to find only the ones who have spent more than $50. These data workflows might require some basic knowledge of SQL, or an understanding of how to make an API request. Not everyone knows how to execute these technical commands. A software company can be slowed down due to a shortage of technical analysts who have the necessary programming skills to build these data workflows. Parabola is a low-code tool for building data workflows. Parabola lets the user drag and drop different components together to build an application without using a programming language. Parabola lowers the technical barrier for knowledge workers who want to build these kinds of data workflows. Alex Yaseen is the CEO of Parabola, and he joins the show to talk about the ideas behind Parabola and his goals with the company. parabola.io parabola.io/careers Twitter: @alexyaseen

Cryptocurrencies today serve two purposes: store of value and speculation. The application infrastructure that has been built around cryptocurrency is mostly to support these use cases. At some point in the future, perhaps cryptocurrencies can be used as a global medium of exchange that is accepted at the grocery store. Perhaps we will use the blockchain for supply chain management, and as a universal ledger for real estate ownership. But today, cryptocurrencies are mostly used for speculative trading. Users buy and sell different cryptocurrencies and stablecoins, looking to make short-term profits. And the markets for trading cryptocurrencies have evolved to have a sophistication that looks like the centralized markets of derivatives and leverage-based day trading. The term “decentralized finance” refers to this phenomenon of cryptocurrency lending markets. Decentralized finance increases the volume of speculated capital by providing liquidity through smart contracts. This short-term liquidity is often collateralized by a volatile cryptocurrency such as Ethereum, creating an opportunity for a type of market participant called a “liquidator.” Tom Schmidt is an investor with Dragonfly Capital, a cryptoasset investment firm. Tom joins the show to describe the dynamics of decentralized finance.

At Airbnb, infrastructure management is standardized across the organization. Platform engineering teams build tools that allow the other teams throughout the organization to work more effectively. A platform engineering team handles problems such as continuous integration, observability, and service discovery. Other teams throughout a company use the tools that a platform engineering team builds. For example, there is a team at Airbnb that builds the search and discovery system that is used by customers who are looking for a place to stay. That team does not want to have to worry about how they are deploying, how their service is being logged, and how to scale up. All of that should be taken care of by the platform engineering team. At a large company like Airbnb, there is so much happening across the infrastructure. Services are being deployed, services are having outages, databases are being resharded. With all of this change occurring, it can be difficult for a team to pinpoint the cause of a service outage. Digging through logs and dashboards is often insufficient. Joey Parsons is the founder of Effx, a company that is building a platform for observing and managing the changes across the infrastructure. Effx is like a newsfeed for a service. An application instrumented with Effx gives the engineers a single endpoint that they can navigate to for understanding the history of their service. Joey joins the show to talk about his experience as an infrastructure engineer at Airbnb, and how that experience informs the work of his new company, Effx.

A gig economy application generates lots of notifications. There is SMS, mobile phone updates, emails, and native application updates. If you order a ride from Uber, you might receive a text message and a push notification at the same time. If an app overloads the user with notifications, the user might end up annoyed and delete the app from their phone. But perhaps all of these notifications are necessary. You would rather get three simultaneous notifications from your food delivery app than fail to get your food on time. If you are the mobile application developer building the food delivery app, what other choice do you have? At large companies such as Linkedin, there are entire teams devoted to figuring out how to optimize the notifications that they send you. It has a surprisingly large impact on the usability of a mobile application. Troy Goode is the founder of Courier, a company that provides notification optimization. This might sound like a small, trivial problem. But it actually has a large impact on the usage of apps. And it is not an easy engineering problem. Troy joins the show to talk about the problem that Courier solves and the backend infrastructure that powers it. Courier is built entirely on serverless APIs. This is a great case study in how to build a completely scalable infrastructure product based on serverless tools.

In a modern data platform, distributed streaming systems are used to read data coming off of an application in real-time. There are a wide variety of streaming systems, including Kafka Streams, Apache Samza, Apache Flink, Spark Streaming, and more. When Eric Anderson joined the show back in 2016, he was working at Google on Google Cloud Dataflow, a managed service for handling streaming data. Today, he works as an investor at Scale Venture Partners. In his current job, he analyzes companies built around data infrastructure, developer tooling, and other enterprise engineering domains. Eric also hosts the podcast Contributor, which explores open source maintainers and the stories of their projects. His podcast has featured the creators of projects such as Envoy, Alluxio, and Chef. In today’s episode, Eric returns to the show to discuss data infrastructure, investing, and the evolving world of open source.

Distributed stream processing frameworks are used to rapidly ingest and aggregate large volumes of incoming data. These frameworks often require the application developer to write imperative logic describing how that data should be processed. For example, a high volume of clickstream data that is getting buffered to Kafka needs to have a stream processing system evaluate that data to prepare it for a data warehouse, Spark, or some other queryable environment. In practice, many developers simply want to have the high volume of data become queryable in the fewest number of steps possible. Materialize is a streaming SQL materialized view engine that provides materialized views over streaming data. The materialized views are incrementally updated over time and reconciled with new data that may have come in out of order. Arjun Narayan and Frank McSherry are the co-founders of Materialize, a company whose technology is based on the Naiad paper, which was written at Microsoft Research. Arjun and Frank join the show to talk about modern streaming systems and their strategy for taking an academic paper and productizing it.

A cloud provider gives developers access to virtualized server infrastructure. When a developer rents this infrastructure via an API call, a virtual server is instantiated on physical machines. That virtual server needs to be made addressable through the allocation of an IP address to make it reachable from the open Internet. When the virtual server starts to receive too much traffic, that traffic needs to be load balanced with another virtual server. The backend networking code that runs a cloud provider needs to be fast, secure, and memory-efficient. Languages that fit that description include C++, Rust, and Go. Digital Ocean’s low-level networking code is mostly written in Go. Sneha Inguva is an engineer with Digital Ocean who has written and spoken about writing networking applications using Go. She joins the show to talk about her work at Digital Ocean, including the implementation of a DHCP server, a network server that assigns IP addresses and other parameters to devices that sit on that network.

A data pipeline is a series of steps that takes large data sets and creates usable results from them. At the beginning of a data pipeline, a data set might be pulled from a database, a distributed file system, or a Kafka topic. Throughout a data pipeline, different data sets are joined, filtered, and statistically analyzed. At the end of a data pipeline, data might be put into a data warehouse or Apache Spark for ad-hoc analysis and data science. At this point, the end-user of the data set expects that data to be clean and accurate. But how do we have any guarantees about the correctness? Abe Gong is the creator of Great Expectations, a system for data pipeline testing. In Great Expectations, the developer creates tests called “expectations”, which verify certain characteristics of the data set at different phases in a data pipeline. This helps ensure that the end result of a multi-stage data pipeline is correct. Abe joins the show to discuss the architecture of a data pipeline and the use cases of Great Expectations.

A data warehouse provides low latency access to large volumes of data. A data warehouse is a crucial piece of infrastructure for a large company, because it can be used to answer complex questions involving a large number of data points. But a data warehouse usually cannot hold all of a company’s data at any given time. Users need to move a subset of the data into the data warehouse by reading large files from a data lake on disk and putting that data into the data warehouse. The process of moving data from one place into another is broken down into three sequential steps, often called “ETL” (extract, transform, load) or “ELT” (extract, load, transform). In ETL, the data is extracted from a source such as a data lake, transformed into a schema that is customized for the data warehouse application, and then loaded into the data warehouse. In ELT, the last two steps are reversed, because modern systems can often leave the necessary schema transformation until after the data has been loaded into the data warehouse. Matthew Scullion is the CEO of Matillion, a company that specializes in building tools for data transformations. Matthew joins the show to talk about the problem of data transformation, and how that problem has evolved over the nine years since he started Matillion. If you enjoy the show, you can find all of our past episodes about data infrastructure by going to SoftwareDaily.com and searching for the technologies or companies mentioned. And if there is a subject that you want to hear covered, feel free to leave a comment on the episode, or send us a tweet @software_daily.

Machine learning applications are widely deployed across the software industry. Most of these applications used supervised learning, a process in which labeled data sets are used to find correlations between the labels and the trends in that underlying data. But supervised learning is only one application of machine learning. Another broad set of machine learning methods is described by the term “reinforcement learning.” Reinforcement learning involves an agent interacting with its environment. As the model interacts with the environment, it learns to make better decisions over time based on a reward function. Newer AI applications will need to operate in increasingly dynamic environments, and react to changes in those environments, which makes reinforcement learning a useful technique. Reinforcement learning has several attributes that make it a distinctly different engineering problem than supervised learning. Reinforcement learning relies on simulation and distributed training to rapidly examine how different model parameters could affect the performance of a model in different scenarios. Ray is an open source project for distributed applications. Although Ray was designed with reinforcement learning in mind, the potential use cases go beyond machine learning, and could be as influential and broadly applicable as distributed systems projects like Apache Spark or Kubernetes. Ray is a project from the Berkeley RISE Lab, the same place that gave rise to Spark, Mesos, and Alluxio. The RISE Lab is led by Ion Stoica, a professor of computer science at Berkeley. He is also the co-founder of Anyscale, a company started to commercialize Ray by offering tools and services for enterprises looking to adopt Ray. Ion Stoica returns to the show to discuss reinforcement learning, distributed computing, and the Ray project. If you enjoy the show, you can find all of our past episodes about machine learning, data, and the RISE Lab by going to SoftwareDaily.com and searching for the technologies or companies you are curious about . And if there is a subject that you want to hear covered, feel free to leave a comment on the episode, or send us a tweet @software_daily.

Distributed stream processing systems are used to read large volumes of data and perform operations across those data streams. These stream processing systems often build off of the MapReduce algorithm for collecting and aggregating large volumes of data, but instead of processing a calculation over a single large batch of data, they process data on an ongoing basis. There are so many different stream processing system for this same use case–Storm, Spark, Flink, Heron, and many others. Why is that? When there seems to be much more consolidation around the Hadoop MapReduce batch processing technology, why are there so many stream processing systems? One explanation is that aggregating the results of a continuous stream of data is a process that very much depends on time. At any given point in time, you can take a snapshot of the stream of data, and any calculation based on that data is going to be out of date by the time that your calculation is finished. There is a latency between when you start calculating something, and when you finish calculating it. There are other design decisions for a distributed stream processing system. What data do you keep in memory? What do you keep on disk? How often do you snapshot your data to disk? What is the method for fault tolerance? What are the APIs for consuming and processing this data? Maximilian Michels has worked on the Apache Flink and Apache BEAM stream processing systems, and currently works on data infrastructure at Lyft. Max joins the show to discuss the tradeoffs of different stream processing systems and his experiences in the world of data processing. You can find all of our past episodes about data infrastructure by going to SoftwareDaily.com and searching for the technologies or companies mentioned. And if there is a subject that you want to hear covered, feel free to leave a comment on the episode, or send us a tweet @software_daily.

Large companies generate large volumes of data. This data gets dumped into a data lake for long-term storage, then pulled into memory for processing and analysis. Once it is in memory, it is often read into a dashboard, which presents a human with a visualization of the data. The end-user who is consuming this data is often a data scientist who is looking at the data to find trends and design new machine learning models. Another kind of user is the operational analyst. An operational analyst is creating complex queries across this data to find latencies in the infrastructure, or perhaps slicing and dicing clickstream data that is coming from online advertisements, in order to figure out how to tweak those advertising algorithms and spend money more effectively. For an operational analyst, a key use case for a data warehouse is fast, interactive querying. The operational analyst needs to be able to query the data to quickly create a dashboard, make judgments based on that dashboard, and then change the query slightly to look at a slightly different dashboard. Druid is a high-performance database that is used for these kinds of queries. Druid is used for ad-hoc queries and operational analytics. Imply Data is a company that builds visualization, monitoring, and security around Druid. Jad Naous is vice president of R&D for Imply, and he joins the show to talk about the use case for Druid, the architecture, and the business model of Imply. If you enjoy the show, you can find all of our past episodes about data infrastructure by going to SoftwareDaily.com and searching for the technologies or companies mentioned. And if there is a subject that you want to hear covered, feel free to leave a comment on the episode, or send us a tweet @software_daily.

Data infrastructure has been transformed over the last fifteen years. The open source Hadoop project led to the creation of multiple companies based around commercializing the MapReduce algorithm and Hadoop distributed file system. Cheap cloud storage popularized the usage of data lakes. Cheap cloud servers led to wide experimentation for data tools. Apache Spark emerged from academia, and Apache Kafka came out of the corporate challenges faced by LinkedIn. Over these 15 years, Ben Lorica has been following the world of data engineering as an engineer, a conference organizer, and a podcaster. When he was host of the O’Reilly Data Show, his material served as inspiration for some of the episodes of this podcast. Today he hosts The Data Exchange podcast and writes The Data Exchange newsletter. Ben joins the show to talk about modern data engineering, and his opinion on the past and future of data infrastructure. If you enjoy the show, you can find all of our past episodes about data infrastructure by going to SoftwareDaily.com and searching for the technologies or companies mentioned. And if there is a subject that you want to hear covered, feel free to leave a comment on the episode, or send us a tweet @software_daily.

A data platform contains all of the data that a company has accumulated over the years. Across a data platform, there is a multitude of data sources: databases, a data lake, data warehouses, a distributed queue like Kafka, and external data sources like Salesforce and Zendesk. A user of the data platform often has a question that requires multiple data sources to answer. How does this user join two data sources from a data lake? How does this user join data across a transactional database and a data lake? How does the user join data from two different data warehouse technologies? Presto is an open source tool originally developed at Facebook. Presto allows a user to query a data platform with a SQL statement. That query gets parsed and executed across the data platform to read from any heterogeneous data source. For some use cases, Presto is replacing the technology Hadoop MapReduce-based technology Hive. For other use cases, Presto is solving a problem in a completely novel way. Justin Borgman joins the show to discuss the motivation for Presto, the problems it solves, and the architecture of Presto. He also talks about the company he started, Starburst Data, which sells and supports technologies built around Presto. If you enjoy the show, you can find all of our past episodes about data infrastructure by going to SoftwareDaily.com and searching for the technologies or companies mentioned. And if there is a subject that you want to hear covered, feel free to leave a comment on the episode, or send us a tweet @software_daily.

Nubank is a popular bank that is based in Brazil. Nubank has more than 20 million customers, and has accumulated a high volume of data over the six years since it was started. Mobile computing and cloud computing have given rise to “challenger banks” that operate more like software companies. When a software company reaches the size that Nubank is at today, it needs a data platform. A data platform is a collection of different technologies that move data into different storage formats and applications, so that different members of an organization can access that data. New data often enters an organization through an OLTP database, which supports user transactions. That data is copied into a data lake, which provides cheap bulk storage. From the data lake, the data is moved into a data warehouse system for fast access. Along the way, tools like Kafka, Spark, and S3 are used to implement the needs of the data platform. Data platform architecture is not an exact science. Different companies build their data platform based on their own unique requirements. Previous shows have covered the data infrastructure companies like Lyft, Uber, and Facebook. Today’s show is another case study in data infrastructure, with a modern bank. In a previous episode, we covered the engineering of Nubank. Sujith Nair from Nubank joins today’s show to talk about the data infrastructure of the company. If you enjoy the show, you can find all of our past episodes about data infrastructure by going to SoftwareDaily.com and searching for the technologies or companies mentioned. And if there is a subject that you want to hear covered, feel free to leave a comment on the episode, or send us a tweet @software_daily.

The Changelog is a podcast about the world of open source. As open source has become closely tied with the entire software development lifecycle, The Changelog has expanded its coverage to the broader software industry. Since starting the podcast ten years ago, Adam Stacoviak and Jerod Santo have become full-time podcasters, and they have started several other podcasts within the Changelog network, including Go Time, JS Party, and Practical AI. Throughout all of their shows, there is a consistent theme of technical, entertaining conversations about software. In the last decade, so much has changed within open source: GitHub became the de facto social network for open source; Kubernetes created a widely used platform for distributed systems; React has given frontend developers a component system to consolidate around. Adam and Jerod return to the show to discuss their perspective on the past and future of open source, and their learnings from interviewing influential software professionals for 10 years.

Animations can be used to create games, app tutorials, and user interface components. Animations can be seen in messaging apps, where animated reactions can convey rich feelings over a text interface. Loading screens can become less boring through animation, and voice assistant products can feel more alive through animation. But we still don’t see much animation in our everyday applications. This is partly because animation tooling is difficult to use. To make an animation, the typical workflow is to go into a tool like After Effects, render your animation, and then export that animation in a movie format. This format is not dynamic enough to be easily used on the wide variety of development platforms. The animation library Lottie did improve this tooling by creating a system for exporting animations to JSON and allowing them to easily scale up and down as vectors. But the animations still were simple and unidirectional. The developer did not have much freedom for how to move an animation in response to user input. Rive is a system for creating dynamic, movable animated objects. Rive allows for the creation of animated elements that respond to user input. Rive has a tool that runs in the browser and allows the user to define the animation. The animations in Rive use a bone system that allows animators and designers to define the points of the animated sprite that the developer can then manipulate with code. This improves the painful handoff process that exists between animators and developers, and gives the developer some programmatic control. Guido and Luigi Rosso are the founders of Rive and they join the show to talk about the frictions of animation tooling, and what they have built to improve

Robotics has changed modern agriculture. Autonomous systems are powering the tractors, cotton pickers, and corn cutters that yield plants at industrial scale. John Deere is a company that has been making farm equipment for 183 years. Over that period, the planting and harvesting process has become increasingly mechanized, and John Deere has been at the forefront. Over the last few decades, software has played an increasingly important role at John Deere. Today, there is software inside the vehicles. These vehicles can operate autonomously, they collect large amounts of data, and they are supported by a large system of cloud services. The teams within John Deere who create the software have an elaborate testing workflow that allows them to deploy the software to the vehicles and drive them in the field. Ryan Bergman is a software engineer at John Deere and he joins the show to talk about the software engineering, management, and DevOps practices within the company.

Venture capital investing requires an understanding of market dynamics, technology, and finance. There is also an element of human nature. Consumer trends can make or break the viability of a new product. And early stage venture investing is always a bet on a small team or individual founder. Early stage investments are usually into companies that have not found perfect traction with their product. Judging the worth of an early stage investment means judging the likelihood that the founders can make their vision a reality. Venture Stories is a podcast that explores the wide spectrum of ideas that go into venture investing. Shows include two person interviews on economics, social networking, food technology, cryptocurrencies, and consumer psychology. Erik Torenberg is a co-founder and partner of Village Global, an early stage venture capital firm. He is also the host of Venture Stories. Erik joins the show to discuss investing, media, and the kinds of new technology companies that are being created today.

Stock trading takes place across a variety of software platforms. Etrade and Schwab have allowed individual traders to buy securities for decades. Robinhood built a business around a similar model, but also removed the commission. Wealthfront and Betterment provide “roboadvisor” services that abstract away the underlying securities and focus on managing a risk profile. Each of these services has a programmatic execution system for managing assets. In order for a developer to build a product like Robinhood or Wealthfront, that developer needs access to an API that can execute trades. Alpaca is an API for stock trading. Alpaca can be used to build financial products, apps, and algorithmic trading programs. Yoshi Yokokawa is the founder of Alpaca, and he joins the show to talk about why he built an API for trading and the potential applications of Alpaca. Yoshi’s background includes work in finance at Lehman Brothers, a period spent as an individual day trader, and the previous company he started selling custom trading algorithms to enterprises.

Large software companies have lots of users, and the activity from those users results in high volumes of traffic. These companies also have a large surface area across the enterprise. There are hundreds of services and databases that are fulfilling user requests. As these requests enter the infrastructure of the enterprise, the requests travel through the different services and result in database queries, payments, and other transactions. These transactions result in the generation of log messages. The log messages tell the story of what is happening across the entire company. Log messages can provide valuable data for security and site reliability engineering. But analyzing a high volume of log data requires a scalable system that can account for that high volume. Jack Naglieri is the CEO of Panther Security. He previously worked at Airbnb, where he helped develop a system called StreamAlert. At Airbnb, log messages are buffered into distributed queueing systems like Kafka or Kinesis, and they are written to bucket storage systems like S3. Those logs are processed by AWS Lambda functions that test the log messages for rules defined by a system operator. Jack left Airbnb and started Panther Security to generalize the tools he built within Airbnb and build a company around the same ideas. Jack joins the show to discuss modern logging infrastructure, his work at Airbnb, and his experience building Panther.

Distributed systems are required to run most modern enterprise software. Application services need multiple instances for scalability and failover. Large databases are sharded onto multiple nodes. Logging services, streaming frameworks, and continuous integration tools all require the orchestration of more than one server. Deploying a distributed system has historically been difficult because the nodes of the system must be managed by the underlying infrastructure. If I have a distributed database that I want to deploy, the complexity of that deployment is going to be different depending on whether I am running on AWS, or VMware, or my own bare metal server infrastructure. Heterogeneous server infrastructure makes it hard to sell distributed applications that get deployed to that infrastructure. A vendor that is selling a distributed database would need to figure out how to make their database work on the infrastructure of any given customer. Kubernetes has simplified the process of deploying a distributed application. Kubernetes is a container orchestration system that has steadily grown in popularity, to the point where the ecosystem is mature and the software is stable. Now that the software industry has a reliable, portable means of deploying a distributed application, the enterprise software market is becoming easier to enter for companies that sell a distributed application. Replicated is a company that builds products for software delivery. Replicated allows for the distribution and updating of applications that would have been hard to deploy in the past. Grant Miller and Marc Campbell are the CEO and CTO of Replicated, and they join the show to talk about the modern enterprise software market, and the process of delivering software to companies that might otherwise have trouble consuming it. Full disclosure: Replicated is a sponsor of Software Engineering Daily.

Chat systems have been a part of software development for decades. Older systems like Pidgin and Yammer were surpassed by newer systems like HipChat. And when Slack was created, it quickly became a part of most software companies. But Slack does not fulfill the needs of every company. Mattermost is an open-source chat system. Mattermost can be configured to work within enterprises that have strong constraints around compliance and data governance. Whereas Slack is a SaaS product that requires users to send their data to the cloud servers managed by Slack, Mattermost allows the enterprise to decide how data moves through services, and where the databases are hosted. Ian Tien is the CEO of Mattermost, and he joins the show to talk about why many companies need their chat system to be hosted in a private cloud or on-premises. In a previous episode with CTO Corey Hulen we discussed the engineering behind the company. In today’s episode, we explore the management and strategy of the business, as well as some additional engineering, since Ian Tien’s background is as a software engineer and computer scientist.

The word “DevOps” has a different definition depending on who you ask. For some people, it is about the process of managing and releasing code. It can involve container management and server orchestration. It can involve infrastructure-as-code, and safer configuration management. In addition to a set of technologies, DevOps can be seen as a management concept that describes agile practices, and breaking down communication barriers between different teams. One thing that most software companies have decided is that whatever DevOps is, we want it. We want to release more software, we want to do it faster, and we want to do it safer. We want streamlined communication between management and engineering. We want a full understanding of the “value chain” of software. Despite the elusiveness of a single description for what DevOps is, GitLab can credibly describe itself as a tool that satisfies the DevOps needs of most enterprises. GitLab started as an open source version control management system based on Git. It has expanded into products that include continuous integration, security, issue tracking, and monitoring. The trajectory of GitLab into such a large platform is something that nobody anticipated. The best explanation for how it happened is that it is the downstream result of an engineer within GitLab deciding that the code hosting product needed to have a continuous integration product bundled with it as an option for a tightly coupled, unified workflow. Today, there are many enterprises trying to make a big set of changes to their development practices. The world is consolidating around Git for version control and Kubernetes for container management. Almost every enterprise is figuring out a “cloud strategy”. Every team wants to have continuous integration, and they want it to have some security products paired with that release workflow in a popular, vaguely defined set of practices known as “DevSecOps”. With so many changes coming to enterprises, it turns out that many of these enterprises just want some sane defaults. When GitLab came to market with a bundled CI and code hosting product, the company discovered that the customers were very happy to have integrated tools that worked well out of the box. This was in stark contrast to the years of NxN tooling integration that an enterprise would have to make to stitch together their broad range of carefully selected tools. Sid Sibrandij is the CEO of GitLab, and he joins the show for a conversation about how GitLab arrived at its product development strategy. In a previous episode, Sid discussed some of the core features and history of GitLab. Today’s show expands on many of the subjects we explored previously. We also had a spirited discussion of the modern nature of work, and how GitLab’s unique culture and fully remote team have evolved as the company has scaled.

The ridesharing infrastructure of Lyft has a high volume of traffic that is mostly handled by servers on AWS. When Vicki Cheung joined Lyft in 2018, the company was managing containers with an internally built container scheduler. One of her primary goals at the company was to move Lyft to Kubernetes. In today’s episode, Vicki gives an overview of Lyft infrastructure and the core engineering problems within the company. One subject she touched on was the network communications between the user on a mobile phone and the cloud backend. This was a topic we explored in detail on a previous episode about Envoy Mobile with Matt Klein. Vicki also discussed the broader Kubernetes ecosystem, as well as her time at OpenAI, where she managed infrastructure deployments for scheduling large machine learning jobs.

If the Internet was reimagined with the software and hardware infrastructure we have today, what would it look like? That is the question that DFINITY is working on answering. DFINITY’s goal is to build a decentralized, secure Internet computer. DFINITY takes concepts from the cryptocurrency world, but it is focused on computation, not financial products. DFINITY can be thought of as a decentralized cloud provider, with redundancy and scalability properties that are achieved by operating on data centers across the world. DFINITY wants to host web applications such as the ones that we use today on centralized servers. A developer who wants to run their application on DFINITY compiles their code to WebAssembly and deploys it to the DFINITY decentralized runtime. Transactions across DFINITY applications are processed through a collateralized proof-of-stake system to ensure reliable, decentralized computation. DFINITY is an ambitious project, and it would seem nearly impossible to bring to market if not for the quality of the team. DFINITY has hired Andreas Rossburg, a co-designer of WebAssembly, as well as talented engineers across security, web development, and backend infrastructure. Dominic Williams is the president and chief scientist of DFINITY, and he joins the show to talk about the vision for DFINITY and the roadmap to making it a reality.

Webflow is a visual programming tool used by designers, developers, and other technical users. Webflow is a leader in the “low code” or “no code” category of software tools that has become prominent in the last few years. Webflow has been years in the making. In a previous show with Webflow CEO Vlad Magdalin, he told the story of being heads down on Webflow, steadily working through the engineering problems that stood between him and his vision of a visual programming environment. In the early days, it was unclear who would even want to use Webflow. A critic of Webflow might have said that Webflow was too high level for developers, and too technical for designers. But Webflow caters to a large subset of developers and designers who want the kind of low code experience that Webflow provides. The product has also helped define a new category of knowledge worker: the “visual programmer.” Bryant Chou is a co-founder of Webflow and was the CTO in the early days. He joins the show to discuss the engineering problems that the company has had to work through, and his perspective on how Webflow fits into the software market going forward.

Software has changed the way the world functions. The rapid pace of change has made it difficult to know how to navigate the new world. Knowledge workers who want to keep advancing in their careers develop a strategy of continuous learning in order to adapt to these changes. O’Reilly Media has existed for almost 40 years, providing resources for the technical consumer. As O’Reilly has expanded its product line from books to conferences to online learning, the business has grown slowly but steadily. That business trajectory stands in contrast to many of the software companies that are financially structured to either grow rapidly or die. Today, O’Reilly has a large impact on the software ecosystem. Software professionals congregate at O’Reilly conferences. Enterprises pay O’Reilly to educate their employees. And O’Reilly continues to grow into new product lines, recently acquiring the interactive learning platform Katacoda, which can be used to learn about Kubernetes and other popular technologies. In a previous episode, we discussed Tim O’Reilly’s book “What’s The Future”. In today’s show, Tim returns to the show to discuss his experience building O’Reilly, and how his business philosophy contrasts with much of the assumed wisdom of software company building.

GraphQL has become a core piece of infrastructure for many software applications. GraphQL is used to make requests that are structured as GraphQL queries and responded to through a GraphQL server. The GraphQL server processes the query and fetches the response from the necessary databases, APIs, and backend services. Around 2016, when GraphQL was becoming popular, a company called Meteor was deciding what to do with its business. Meteor was started off of the popular framework MeteorJS, a system for building real-time JavaScript applications. MeteorJS was loved by many developers, but Meteor needed to decide if it was the most viable opportunity that it could be pursuing with its resources. From the vantage point within the Meteor company, there were some trends in the frontend ecosystem that were potentially disruptive to the viability of Meteor. There were also some large potential opportunities. The dramatic change to the frontend was largely coming as a downstream effect of Facebook’s open source technologies: React and GraphQL. Amidst these changes, Meteor shifted the company’s efforts entirely towards GraphQL and renamed the company Apollo. Geoff Schmidt is the CEO of Apollo, and he joins the show to talk about the GraphQL ecosystem, the business opportunities around GraphQL, and the process of pivoting from Meteor to Apollo. If you are planning a hackathon, check out FindCollabs Hackathons. Whether you are running an internal hackathon for your company, or you are running an open hackathon so that users can try out your product, FindCollabs Hackathons are a tool for people to build projects and collaborate with each other. You can create your own hackathon at FindCollabs.com.

The JavaScript ecosystem stretches across frontend, backend, and middleware. There are newer tools such as GraphQL, Gatsby, and WebAssembly. There are frameworks like React, Vue, and Angular. There is complex data handling with streams, caches, and TensorFlow.js. JavaScript is unlike any other ecosystem, because a single language can be used to construct every part of an application. Because JavaScript is used for such a broad spectrum of use cases, the amount of tooling available can be intimidating to someone new to the ecosystem. Kevin Ball is a host of JS Party, a podcast on The Changelog network. Kevin joins the show to give his perspective on the JavaScript ecosystem. We discussed ES Modules, the JAM Stack, and the growing number of tools, libraries, and workflows used by JavaScript developers.

Cloud infrastructure is usually consumed in the form of virtual machines or containers. These VMs or containers are running on a physical host machine that is also running other VMs and containers. This is called multitenancy. Servers across cloud providers such as AWS have a high utilization because there are multiple virtual instances running on each physical server host. Cloud computing has led to a low cost of compute infrastructure. But in some cases, this low cost comes at the price of not being able to control the underlying hardware with as much precision as the user would want. Some users want specific types of hardware. Other users want to be using dedicated hardware that does not risk the “noisy neighbor” problem of sharing a physical server with some other application that is using most of the resources. Packet is a company that provides remote access to baremetal infrastructure. The user experience is similar to that of a cloud provider, but with more control over how a given physical host will be used. Zachary Smith is the CEO of Packet and Nathan Goulding is the chief architect. Zach and Nathan join the show to talk about the business and the engineering behind Packet, as well as the future goals for where they want to take the company.

Edge computing is the usage of servers that are geographically close to the client device. The first common use case for edge computing was CDNs: content-delivery networks. A content delivery network placed media files such as images and videos on multiple servers throughout the world. These are big files, and they take lots of bandwidth to transfer. By placing them at CDNs, the files would be closer to any user around the world. These early use cases for edge computing were purely about storing large files. But the vast majority of compute still took place at the central application servers. Over time, users have required faster and faster application experiences. Today, an increasing amount of compute has been moved to the edge, in addition to the existing storage applications. More user data is being cached at the edge to make for quicker transactional processing. Machine learning model training and hosting at the edge makes for a faster, more responsive machine learning feedback loop. Jaromir Coufal is an engineer with Red Hat. He joins the show to talk about modern applications of edge computing, and how the demand for edge computing is creating a market opportunity for companies that have lots of servers at the edge, such as telecoms. These telecoms can repurpose their widely distributed telecom infrastructure as edge servers that they can sell usage on.

Every large company generates large amounts of data. Data engineering is the process of storing, transforming, and leveraging that data. Data infrastructure companies provide tools and platforms for performing data engineering. The last fifteen years has seen a rise in modern data management companies built in a time of decreasing storage costs, an increased volume of data, and the prevalence of cloud computing. Modern data companies include Hadoop vendors, cloud providers, and a wide variety of individual software companies offering products such as databases, ETL tools, and open source tooling. The go-to-market strategy for a data infrastructure company requires a deep understanding of the data engineering landscape. A company must build something useful, sell it to customers, and eventually build a replicable strategy. Sean Knapp is the CEO of Ascend, a company that builds Apache Spark-based data pipelines that connect APIs, data lakes, and data warehouses together to enable data applications. Sean joins the show to talk about the process of building a data infrastructure company, and his lessons building Ascend.

Slack is a messaging platform for organizations. Since its creation in 2013, Slack has quickly become a core piece of technology used by a wide variety of technology companies, groups, and small teams. The messages that are sent on Slack are generated at a very high volume, and are extremely sensitive. These messages must be stored on Slack’s servers in a way that does not risk a message from one company accidentally being accessible to another company. The messages must be highly available, and they also must be indexed for search. When Slack was scaling, the company started to encounter limitations in its data infrastructure that the company was unsure how to solve. During this time, Josh Wills was the director of data engineering at Slack, and he joins the show to retell the history of his time at Slack, and why the problem of searching messages was so hard. Josh also provides a great deal of industry context around how engineers from Facebook and Google differ from one another. When Slack was starting to become popular, the company quickly began to attract engineers from both of those companies. Facebook and Google have distinct solutions for how they have tackled the problems of data engineering.

NoSQL databases provide an interface for storing and accessing data that allows the user to work with data in an “unstructured” fashion. SQL databases require the data in the database to be “normalized,” meaning that each object in the entire database has an entry (or a null value) for each field. One advantage of NoSQL is that the different objects are “denormalized,” meaning that different objects in the database can have unique fields. There is a widely held belief that NoSQL databases do not scale, or that there is some significant penalty that a developer will pay for using a NoSQL database as soon as their app becomes popular. The truth is much more subtle than that. NoSQL databases can perform as well as or better than SQL databases if the developers know the query patterns that their applications make. SQL databases will be a better choice in the condition where the database has a very wide spectrum of access patterns. But in many cases, an application has a narrow range of different requests for the database, and a NoSQL database can perform very well if the database is structured and optimized for these requests. Rick Houlihan is an executive with Amazon Web Services who works with database teams and engineers to optimize their products and database infrastructure. Rick joins the show to discuss the tenets of NoSQL and describe the fundamental contrast between NoSQL and SQL database limitations.

Amazon EC2 (Elastic Compute Cloud) is a virtualized server product that provides the user with scalable compute infrastructure. EC2 was created in 2006 as one of the first three AWS services along with S3 and Simple Queueing Service. Since then, EC2 has provided the core server infrastructure for many of the companies that have been built in the cloud. A large scale virtualization product requires its engineers to have a deep understanding of scheduling and multitenancy. In previous shows, we have touched on subjects such as hypervisors, the noisy neighbor problem, the cold start problem, and other aspects of multitenant infrastructure. To make EC2 successful, these issues must be continuously revisited and resolved at different areas of the stack. Dave Brown joined the EC2 team in 2007, and now leads the EC2 Compute, Networking, and Load Balancing teams as a Vice President. Dave joins the show to discuss the history of EC2 and the canonical problems of virtualized server infrastructure.

Amazon’s container offerings include ECS (Elastic Container Service), EKS (Elastic Kubernetes Service), and Fargate. Through these different offerings, Amazon provides a variety of ways that a user can manage Kubernetes clusters and standalone container instances. The choice of which containerization system to choose depends on the needs of the user, and the tradeoffs they want to make on control and portability. Amazon’s container products have been designed in the context of a shifting competitive landscape. Kubernetes presents a potential long-term threat to Amazon’s status as the most popular cloud provider. Properly responding to this threat has required Amazon to extend itself into the world of open source, contributing to Kubernetes and having more conversations with customers who want their products to have the high quality user experience of AWS along with the open characteristics of Kubernetes. Abby Fuller is a principal technologist with Amazon who works on containers and Linux. Abby joins the show to describe Amazon’s perspective on containers and Kubernetes.

When the Kubernetes project was started, Amazon Web Services was the dominant cloud provider. Most of the code that runs AWS is closed source, which prevents an open ecosystem from developing around AWS. Developers who deploy their application onto AWS are opting into a closed, controlled ecosystem, which has both costs and benefits. The software industry has a history of closed and open ecosystems existing at the same time. AWS represented a huge closed ecosystem. With the amount of money at stake in the cloud business, it was only a matter of time before a more open ecosystem emerged. Since the creation of Kubernetes, the world of cloud computing has evolved rapidly. Google and Microsoft have both invested heavily into Kubernetes, and Amazon itself has adapted to the newer competitive landscape with a Kubernetes offering of its own. Amazon has also made efforts to become more publicly involved in open source projects, including Kubernetes. Kelsey Hightower has been a part of the Kubernetes ecosystem since the project was started. He is one of the most recognizable faces in the world of Kubernetes, delivering keynotes, appearing on podcasts, and co-authoring the popular Kubernetes Up and Running. Kelsey joins the show to discuss the progress in the Kubernetes ecosystem, and the competitive dynamics between Kubernetes and AWS.

freeCodeCamp was started five years ago with the goal of providing free coding education to anyone on the Internet. freeCodeCamp has become the best place to begin learning how to write software. There are many other places that a software engineer should visit on their educational journey, but freeCodeCamp is the best place to start, because it is free, and there are no advertisements. For most people learning to code, the price of that education is important, because they are learning to code to build a new career. It’s also important that a new programmer learns from an unbiased source of information, because an ad-supported environment will educate the new programmer towards products that they might not need. freeCodeCamp has not been easy to build. Building freeCodeCamp has required expertise in software engineering, business, media, and community development. The donation-based business model of freeCodeCamp doesn’t collect very much money. Why would somebody build a non-profit when they could spend their time building a highly profitable software company? Quincy Larson is the founder of freeCodeCamp, and he joins the show for a special episode about his backstory and the journey to building the best place on the Internet for a new programmer to begin. Support freeCodeCamp

The software category known as “no-code” describes a set of tools that can be used to build software without writing large amounts of code in a programming language. No-code tools use visual interfaces such as spreadsheets and web based drag-and-drop systems. In previous shows, we have covered some of the prominent no-code products such as Airtable, Webflow, and Bubble. It is clear that no-code tools can be used to build core software infrastructure in a manner that is more abstract than the typical software engineering model of writing code. No-code tools do not solve everything. You can’t use a no-code tool to build a high performance distributed database, or a real-time multiplayer video game. But they are certainly useful for building internal tools and basic CRUD applications. We know that no-code tools can create value. But how do they fit into the overall workflow of a software company? How should teams be arranged now that knowledge workers can build certain kinds of software without writing code? And how should no-code systems interface with the monoliths, microservices, and APIs that we have building for years? Shawn Wang is an engineer with Netlify, a cloud provider that is focused on delivering high-quality development and deployment experience. Netlify is not a no-code platform, but Shawn has explored and written about the potential of no-code systems. Since he comes from a code-heavy background, he is well-positioned to give a realistic perspective on how no-code systems might evolve to play a role in the typical software development lifecycle.

Roblox is a gaming platform with a large ecosystem of players, creators, game designers, and entrepreneurs. The world of Roblox is a three-dimensional environment where characters and objects interact through a physics engine. Roblox is multiplayer, and users can interact with each other over the Internet. Roblox is not one single game—it is a system where anyone can design and monetize their own games within Roblox. Over the last 14 years, Roblox has grown to be massively popular. As the product has grown, the software has evolved to meet changes in consumer demands and engineering constraints. Client devices include mobile phones, desktop computers, and virtual reality. All of these clients must have a consistent experience in graphics and functionality. The backend platform has to support a high volume of concurrent players who are accessing a high volume of content. The networking needs to support multiple players operating in an environment that demands high bandwidth. Claus Moberg is a vice president of engineering at Roblox. He joins the show to discuss the engineering of Roblox and the future of gaming.