Software Engineering Daily

A sample of the human voice is a rich piece of unstructured data. Voice recordings can be turned into visualizations called spectrograms. Machine learning models can be trained to identify features of these spectrograms. Using this kind of analytic strategy, breakthroughs in voice analysis are happening at an amazing pace. Rita Singh researches voice at Carnegie Mellon University. Her work studies the high volume of latent data that is available in the human voice. As she explains, just a small fragment of a human voice can be used to identify who a speaker is. Your voice is as distinctive as your fingerprint. Your voice can also reveal medical conditions. Features of the human voice can be strongly correlated with psychiatric symptom severity, and potentially heart disease, cancer, and other illnesses. The human voice can even suggest a person’s physique–your height, weight, and facial features. In this episode, Rita explains the machine learning techniques that she uses to uncover the hidden richness of the human voice. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Data Skeptic is a podcast about machine learning, data science, and how software affects our lives. The first guest on today’s episode is Kyle Polich, the host of Data Skeptic. Kyle is one of the best explainers of machine learning concepts I have met, and for this episode, he presented some material that is perfect for this audience: machine learning for software engineers. Second Spectrum is a company that analyzes data from professional sports, turning that data into visualizations, reports, and futuristic sports viewing experiences. We had a previous show about Second Spectrum where we went into the company in detail–it was an excellent show, so I wanted to have Kevin Squire, an engineer from Spectrum, come on the show to talk about how the company builds machine learning tools to analyze sports data. If you have not seen any of the visualizations from Second Spectrum, stop what you are doing and watch a video on it! This year we have had three Software Engineering Daily Meetups: in New York, Boston, and Los Angeles. At each of these Meetups, listeners from the SE Daily community got to meet each other and talk about software–what they are building and what they are excited about. I was happy to be in attendance at each of these, and I am posting the talks given by our presenters. The audio quality is not perfect on these, but there are also no ads. Thanks to Telesign for graciously providing a space and some delicious food for our Meetup. Telesign has beautiful offices in Los Angeles, and they make SMS, voice, and data solutions. If you are looking for secure and reliable communications APIs, check them out. We’d love to have you as part of our community. We will have more Meetups eventually, and you can be notified of these by signing up for our newsletter. Come to SoftwareDaily.com and get involved with the discussion of episodes and software projects. You can also check out our open source projects–the mobile apps, and our website.

Voice interfaces are a newer form of communicating with computers. Alexa is a voice interface platform from Amazon. Alexa powers the Amazon Echo, as well as Alexa-enabled cars, refrigerators, and dishwashers. Any developer can build a device with a voice interface using a Raspberry Pi. Paul Cutsinger works on Echo and Alexa at Amazon. He’s focused on growing the market of developers who are building voice interfaces. In this episode, Paul describes how to design and implement a voice application for the Amazon Alexa platform. The market for voice powered apps is so new, and there has yet to be a “killer app.” If you like to tinker on new platforms, you will like this episode–and I was surprised by how easy it sounds to build a voice app. Personally I use voice interfaces all the time–to set timers, to find out how to tell if a cucumber has gone bad, to ask what temperature to cook a potato at. Sometimes, when I am lying in bed trying to get to sleep, I will ask my nearest device to read me a Wikipedia article. These are great use cases, but I’m sure we will see something much more groundbreaking in the future. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Message broker systems decouple the consumers and producers of a message channel. In previous shows, we have explored ZeroMQ, PubNub, Apache Kafka, and NATS. In this episode, we talk about another message broker: Apache Pulsar. Pulsar is an open source distributed pub-sub message system originally created at Yahoo. It was used to scale products with high volumes of users–such as Yahoo Mail. There are three components of a Pulsar deployment: the Pulsar broker (which handles the message brokering), Apache Bookkeeper (which handles the durable storage of the messages), and Apache Zookeeper, which manages the distributed coordination. Lewis Kaneshiro joins the show to describe how Apache Pulsar works, and how it compares to other messaging systems like Apache Kafka. Lewis is the CEO of Streamlio, a company that builds messaging and stream processing systems for enterprises, and uses Pulsar in its core product. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Gloo is a function gateway built on top of the popular open source project Envoy. The goal of Gloo is to decouple client-facing APIs from upstream APIs. Gloo is similar to an API gateway, which is a tool that software companies can use to collect all their APIs and one place and impose security, monitoring, and other standards around those APIs. The goal of Gloo is to provide all the tools necessary to glue together traditional and cloud-native applications. Idit Levine is the CEO of Solo.io, a company that is building Gloo and several other projects. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

YouTube runs a large MySQL database to hold the metadata about its videos. As YouTube scaled, the database was sharded, and applications within YouTube had to write queries that were aware of the sharding layout of that database. This is problematic, because it pushes complexity to the application developer. An application developer shouldn’t have to be aware of how a database is laid out among different nodes. The developer should be able to issue a query, and have the cluster simply return the data. Vitess is an open source system for scaling large MySQL databases. Sugu Sougoumarane co-created Vitess at YouTube. Since YouTube is owned by Google, Vitess was able to leverage the Borg cluster manager developed at Google. Once Kubernetes came to market, it became more viable to make Vitess accessible to open source developers. Sugu joins the show to talk about the scalability problems that YouTube’s database infrastructure encountered and the motivations for building Vitess. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

The Kubernetes ecosystem consists of enterprises, vendors, open source projects, and individual engineers. The Cloud Native Computing Foundation was created to balance the interests of all the different groups within the cloud native community. CNCF has similarities to the Linux Foundation and the Apache Foundation. CNCF helps to guide open source projects in the Kubernetes ecosystem–including Prometheus, Fluentd, and Envoy. With the help of the CNCF, these projects can find common ground where possible. KubeCon is a conference organized by the Cloud Native Computing Foundation. I attended the most recent KubeCon in Copenhagen. KubeCon was a remarkably well-run conference–and the attendees were excited and optimistic. As much traction as Kubernetes has, it is still very early days and it was fun to talk to people and forecast what the future might bring. At KubeCon, I sat down with Chris Aniszczyk and Dan Kohn, who are the COO and director of the CNCF. I was curious about how to scale an organization like the CNCF. In some ways, it is like scaling a government. Kubernetes is growing faster than Linux grew, and the applications of Kubernetes are as numerous as those of Linux. Different constituencies want different things out of Kubernetes–and as those constituencies rapidly grow in number, how do you maintain diplomacy among competing interests? It’s not an easy task, and that diplomacy has been established by keeping in mind lessons from previous open source projects. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Mesos is a system for managing distributed systems. The goal of Mesos is to help engineers orchestrate resources among multi-node applications like Spark. Mesos can also manage lower level schedulers like Kubernetes. A common misconception is that Mesos aims to solve the same problem as Kubernetes, but Mesos is a higher level abstraction. Ben Hindman co-founded Mesosphere to bring the Mesos project to market. Large enterprises like Uber, Netflix, and Yelp use Mesosphere for resource management. Before he started the company, Ben worked in the Berkeley AMP Lab, a research program where the Spark and Tachyon projects were also born. At this point, he has spent significant time in both academia and industry. This conversation spans distributed systems theory, history, and practice. Ben and I spoke at KubeCon 2018 in Copenhagen–which was an amazing conference. We were both amazed at how big the audience for Kubernetes has gotten, and the pace at which the technology is advancing. Today, Kubernetes is mostly used for scheduling containerized applications that engineers have built themselves. But there will be higher level tools that use Kubernetes as a building block. Much like Zookeeper was used as a building block for Hadoop, Kubernetes will be used to build serverless applications and distributed databases. Once you are using a distributed database built on Kubernetes, you don’t want to think about the container orchestration–you want to think about the raw storage and CPU requirements for that database. This is one reason why Mesos is so compelling. Since Kubernetes creates an increased cardinality of distributed systems, it’s good to know that there is a framework built to manage those higher level applications. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Algorithms for building neural networks have existed for decades. For a long time, neural networks were not widely used. Recent changes to the cost of compute and the size of our data have made neural networks extremely useful. Our smart phones generate terabytes of useful data. Lower storage costs make it economical to keep that data. Cloud computing democratized the ability to do large scale machine learning across GPUs. Over the last few years, these trends have been driving widespread use of deep learning, in which neural nets with a large series of layers are used to create powerful results in various fields of classification and prediction. Neural networks are a tool for making sense of unstructured data–text, images, sound waves, and videos. “Unstructured” data is data with high volume or high dimensionality. For example, an image has a huge collection of pixels, and each pixel has a color value. One way to think about image classification is that you are finding correlations between those pixels. A certain cluster of pixels might represent an edge. After doing edge detection on pixels, you have a collection of edges. Then you can find correlations between those edges, and build up higher levels of abstraction. Yinyin Liu is a principal engineer and head of data science at the Intel AI products group. She studies techniques for building neural networks. Each different configuration of a neural network for a given problem is called a “topology.” Engineers are always looking at new topologies for solving a deep learning application–such as natural language processing. In this episode, Yinyin describes what a deep learning topology is and describes topologies for natural language processing. We also talk about the opportunities and the bottlenecks in deep learning–including why the tools are so immature, and what it will take to make the tooling better. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Cloud computing lowered the cost and improved accessibility to tools for storing large volumes of data. In the early 2000s, Hadoop caused a revolution in large scale batch processing. Since then, companies have been building ways to store and access their data faster and more efficiently. At the same time, the sheer volume of data has increased and machine learning has given rise to methods of extracting signal from seemingly inconsequential data points. This confluence of factors gave rise to the role of the data engineer. A data engineer defines the data pipeline and supports data scientists and machine learning engineers. Tobias Macey hosts the “Data Engineering Podcast,” where he covers the fast moving world of data engineering–including databases, cloud providers, and open source tools. Tobias and I covered a range of topics in the data engineering space and also spent significant time discussing the world of software engineering podcasting. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Starting an Internet business is harder than it should be. You need to incorporate, create an operating agreement, set up a system to accept payments, and many other straightforward tasks. In the 1990s, this was how it felt to set up anything on the Internet. You always had to stand up a web server on your own infrastructure, before you could get to the interesting part–which was building an actual product. With the popularization of cloud computing, it became massively easier to stand up a server. Because of that lower activation energy, millions of applications and thousands of software businesses got started. But the activation energy required to start a business remains higher than necessary. It feels like standing up a web server in the 90s–lots of tedium and reinventing the wheel that has been done by people before you. This is the motivation behind Stripe Atlas, a project to simplify the process of starting an Internet business. Patrick McKenzie works on Atlas at Stripe. He was previously on the show to discuss his experience leaving a large corporation to work on his own small software companies. And his name has become synonymous with the modern phenomenon of the small software company–he has been writing about this topic for over a decade at Kalzumeus.com. It was great to talk to Patrick once again about Internet businesses, and I’m excited to see Stripe Atlas become something huge. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

When I buy a mattress online, I pay for it with my credit card. Behind the scenes, a complex series of transactions occur between a payment gateway, the credit card company, and a few banks. There are problems with this process–it is slow, complex, and involves the synchronization of several different parties. Some consumers will not want to purchase the mattress because they do not have cash, and the lending rates they get offered are higher than they are willing to spend. If these consumers were presented with more intelligent loan rates, the lender could still make money, the mattress company could still make money, and the consumer would get a new mattress. It’s a missed opportunity all around. Affirm is a consumer financial services company. Their first product offers loans to consumers making purchases. In today’s episode, Affirm CTO Libor Michalek explains how Affirm decided to build this product, and what they have done to scale it. The conversation took me by surprise. Affirm was started by Max Levchin, who was a co-founder of PayPal. I assumed that when Affirm was created, they already knew exactly what they were going to build–because Affirm is a payments company and Max has had knowledge of the payments industry going back two decades. In reality, Affirm started out with more vague ideas around what they were building. They spent some time running small experiments as they looked for product/market fit–just like a bootstrapped startup would have. It was inspiring to know that even an experienced team is willing to go through the humble process of searching for a product within a space they are deeply familiar with. We didn’t get to all the questions I was planning to explore, but I hope to do another show about Affirm in the future. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Superpedestrian is a robotic bicycle wheel that learns how you pedal and personalizes your bicycle ride. The engineering challenges of Superpedestrian are at the intersection of robotics, software, and real-time analytics. The first half of today’s show is about Superpedestrian. Goss Nuzzo Jones and Matt Cole are engineers at Superpedestrian. The slides for their presentation are also in the show notes. The second half of today’s show is about HubSpot, a massive business with lots of infrastructure challenges. Thomas Petr explained how HubSpot’s engineering has matured, and some of the scaling problems they have tackled. Last month, we had three Software Engineering Daily Meetups: in New York, Boston, and Los Angeles. At each of these Meetups, listeners from the SE Daily community got to meet each other and talk about software–what they are building and what they are excited about. I was happy to be in attendance at each of these, and I am posting the talks given by our presenters. The audio quality is not perfect on these, but there are also no ads. Thank you to HubSpot for hosting this Meetup–they have beautiful offices and if you are looking for a job (or if you want to host a technology Meetup in the Boston area) I strongly recommend checking them out. We’d love to have you as part of our community. We will have more Meetups eventually, and you can be notified of these by signing up for our newsletter. Come to SoftwareDaily.com and get involved with the discussion of episodes and software projects. You can also check out our open source projects–the mobile apps, and our website. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Phones are constantly tracking the location of a user in space. Devices like cars, smart watches, and drones are also picking up high volumes of location data. This location data is also called “geospatial data.” The amount of geospatial data is rapidly increasing, and there is a growing demand for software to perform operations over that data. Geospatial data sets are often massive–so it is non-trivial to perform operations over this data. Geospatial data can consist of something as simple as a set of latitude/longitude data points. A single lat/long coordinate pair can be enriched with information about what ZIP code it is in, how far that data point is from the other data points in the set, and where the nearest coffee shop is in relation to that data point. Ram Sriharsha created Magellan, a geospatial analytics library for Spark. In today’s show, Ram describes the set of problems within the domain of geospatial analytics engineering. Ram also works as a product manager for Apache Spark at Databricks. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Last year, the WannaCry ransomware attack shut down hospitals, public transportation systems, and governments, demanding payment to unlock key computer systems. A programmer named Marcus Hutchins was able to stop WannaCry by registering a DNS entry buried in the WannaCry code. Not long after he stopped the WannaCry attack, Marcus Hutchins was arrested at a security conference in Las Vegas. Marcus’s arrest was due to actions that were unrelated to WannaCry. He is accused of writing a piece of malware called Kronos. Marcus volunteered his time to help stop WannaCry–a piece of ransomware that threatened to cause billions of dollars in damages. Whether or not he was a black hat in the past, perhaps Marcus should be absolved of his past actions. Reeves Wiedeman is a journalist with New York Magazine, and he joins the show to tell the story of WannaCry’s Gray Hat: Marcus Hutchins. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Alexis Le-Quoc started Datadog in 2010, after living through the Internet boom and bust cycle of the late 90s and early 2000s. In 2010, cloud was just starting to become popular. There was a gap in the market for infrastructure monitoring tools, which Alexis helped fill with the first version of Datadog. Since 2010, the number of different cloud infrastructure products has proliferated. There were new databases, queueing systems, virtualization and containerization tools. Web 2.0 took off, and thousands of new Internet businesses got started. Many of these businesses used Datadog to monitor their increasingly wide range of infrastructure configurations–and Datadog began to scale. On today’s show, Alexis tells the story of how Datadog grew from its first product into a variety of tools–infrastructure monitoring, logging, and application performance monitoring. Monitoring is a unique challenge–there is a ton of data, the data is latency sensitive, and the data is operationally important. These engineering constraints provide for a great conversation. Alexis is the CTO of Datadog, and we talked about cloud providers, building a business, infrastructure, and how to scale engineering management. Full disclosure: Datadog is a sponsor of Software Engineering Daily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Technology is pushing us rapidly toward a future that is impossible to forecast. We try to imagine what that future might look like, and we can’t help having our predictions shaped by the media we have consumed. 1984, Terminator, Gattaca, Ex Machina, Black Mirror–all of these stories present a dystopian future. But if you look around the world, the most successful technologists are mostly guided by a sense of optimism. Technologists themselves are mostly idealistic–they see the future through a utopian lens. Popular media largely tells a different story: that we are headed for a dystopian world. Why is there such a gulf in the level of idealism between technologists and the media? Mike Solana found himself asking that question on a regular basis during his work at Founder’s Fund, where he is a vice president. Founder’s Fund has a bias toward funding difficult, cutting-edge technology like gene editing, robotics, and nuclear energy. This technology that Mike was seeing made him excited about the future–which led to his creation of the podcast “Anatomy of Next.” “Anatomy of Next” has explored biology, robotics, nuclear energy, superintelligence, and the nature of reality. Soon the podcast will be exploring how our civilization will explore and settle the solar system–specifically Mars. I’ve listened through the entire first season of the show twice and enjoyed it so much because Mike explores questions that are on the border of philosophy and technology–questions about the nature of reality, and what makes us human–and nobody can give perfect answers to these questions. But Mike interviews top experts on the show, which provides us with a framework. Guests on “Anatomy of Next” include Nick Bostrom (the author of Superintelligence), George Church (a pioneer in gene editing), and Palmer Luckey (the founder of VR company Oculus). Mike joins the show to talk about why he started “Anatomy of Next,” and his own perspective on the future. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Podcasting about cryptocurrencies is a strange occupation. You get emails all the time from companies doing a token sale that you would never want to be affiliated with. You get angry tweets from anonymous Twitter accounts that are on one side of the Bitcoin scaling debate. You get to interview extreme personalities, and the technical discussions can be highly educational. Brian Fabian Crain started the Epicenter podcast four years ago. Podcasting about cryptocurrencies allows a podcaster to report on a wide range of areas: economics, software, philosophy–and the stories within the blockchain world itself. Epicenter is one of my favorite podcasts about cryptocurrencies because Brian is always prepared enough to ask sophisticated questions. In this episode, we talked about ICOs–when does an ICO make sense? It seems that many token economies could function just as well without a token involved. We discussed the scaling approaches of Bitcoin and Ethereum–why are these two blockchains taking very different approaches to their scaling plans? And we talked about Chorus, the company that Brian founded to build infrastructure for proof-of-stake cryptocurrencies. I enjoyed talking to Brian about all these different subjects, and look forward to having him on again in the future. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Keybase is a platform for managing public key infrastructure. Keybase’s products simplify the complicated process of associating your identity with a public key. Keybase is the subject of the first half of today’s show. Michael Maxim, an engineer from Keybase gives an overview for how the technology works and what kinds of applications Keybase unlocks. The second half of today’s show is about Clarifai. Clarifai is an AI platform that provides image recognition APIs as a service. Habib Talavati explains how Clarifai’s infrastructure processes requests, and the opportunities for improving the efficiency of that infrastructure. Last month, we had three Software Engineering Daily Meetups: in New York, Boston, and Los Angeles. At each of these Meetups, listeners from the SE Daily community got to meet each other and talk about software–what they are building and what they are excited about. I was happy to be in attendance at each of these, and I am posting the talks given by our presenters. The audio quality is not perfect on these, but there are also no ads. Thanks to Datadog for graciously providing a space for our Meetup, and for being a sponsor of SE Daily. You can sign up for Datadog and get a free t-shirt by going to softwareengineeringdaily.com/datadog. We’d love to have you as part of our community. We will have more Meetups eventually, and you can be notified of these by signing up for our newsletter. Come to SoftwareDaily.com and get involved with the discussion of episodes and software projects. You can also check out our open source projects–the mobile apps, and our website. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Google’s central system for managing compute resources is called Borg. On Borg, millions of Linux containers process a wide variety of workloads. When a new application is spun up, Borg provides that application with the resources it needs. Workloads at Google usually fall into one of two distinct categories: long-running application workloads (such as Gmail) and batch workloads (such as a MapReduce job). In the early days of Google, the long-lived workloads were scheduled onto a system called “BabySitter” and the batch workloads were scheduled onto a system called “Global Work Queue.” Borg was the first cluster manager at Google designed to service both long-running and batch workloads from a single system. The second cluster manager at Google was Omega, a project that was created to improve the engineering behind Borg. The innovations of Omega improved efficiency and architecture of Borg. More recently, Kubernetes was created as an open source implementation of the ideas pioneered in Borg and Omega. Google has also built a Kubernetes as a service offering that companies use to run their infrastructure in the same way that Google does. Brian Grant is an engineer at Google who has seen the iteration of all three cluster management systems that have come out of Google. He joins the show to discuss how the workloads at Google have changed over time, and how his perspective on how to build and architect distributed systems has evolved. Full disclosure: Google is a sponsor of Software Engineering Daily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Rajat Monga is a director of engineering at Google where he works on TensorFlow. TensorFlow is a framework for numerical computation developed at Google. The majority of TensorFlow users are building machine learning applications such as image recognition, recommendation systems, and natural language processing–but TensorFlow is actually applicable to a broader range of scientific computation than just machine learning. TensorFlow has APIs for decision trees, support vector machines, and linear algebra libraries. The current focus of the TensorFlow team is usability. There are thousands of engineers building data intensive applications with TensorFlow, but Rajat and the rest of the TensorFlow team would like to see millions more. In today’s show, Rajat and I discussed how TensorFlow is becoming more usable, as well as some of the developments in TensorFlow around edge computing, TensorFlow Hub, and TensorFlow.js, which allows TensorFlow to run in the browser. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

There are hundreds of different databases. There are tens of continuous delivery products. There is an ocean of cloud providers and CRM systems and monitoring platforms and sales prospecting tools. The range of available software products is so diverse that it can be overwhelming to figure out which products to buy. Siftery is a company that was started to index software products and help buyers make decisions. Siftery can build a data set from your web site or from your Google account, assess your software stack, and compare those software products to others on the market. In a previous show with Ayan Barua, we discussed how engineers should explore the question of build vs. buy. In today’s episode, Ayan joins the show to discuss how Siftery has evolved, and the engineering behind Siftery products. A newer Siftery product called Track can ingest banking transactions, QuickBooks records, or other transaction histories and use that information to compile the cost structure of your software company, and we spent the latter part of our conversation discussing why and how they built it. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

A message broker is an architectural component that sends messages between different nodes in a distributed system. Message brokers are useful because the sender of a message does not always know who might want to receive that message. Message brokers can be used to implement the “publish/subscribe” pattern, and by centralizing the message workloads within the pub/sub system, it lets system operators scale the performance of the messaging infrastructure by simply scaling that pub/sub system. Derek Collison has worked on messaging infrastructure for 25 years. He started at TIBCO, then spent time at Google and VMWare. When he was at VMWare, he architected the open source platform Cloud Foundry. While working on Cloud Foundry, Derek developed NATS, a messaging control plane. Since that time, Derek has started two companies–Apcera and Synadia Communications. In our conversation, Derek and I discussed the history of message brokers, how NATS compares to Kafka, and his ideas for how NATS could scale in the future to become something much more than a centralized message bus. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Stripe processes payments for thousands of businesses. A single payment could involve 10 different networked services. If a payment fails, engineers need to be able to diagnose what happened. The root cause could lie in any of those services. Distributed tracing is used to find the causes of failures and latency within networked services. In a distributed trace, each period of time associated with a request is recorded as a span. The spans can be connected together because they share a trace ID. The spans of a distributed trace are one element of observability. Others include metrics and logs. Each of these components of observability make their way into services like Lightstep and Datadog. The path traveled by different elements of observability is called the observability pipeline. In an episode last year, Cory Watson explained how observability works at Stripe. In today’s episode, Cory describes how observability is created and aggregated. It’s a useful discussion for anyone working at a company that is figuring out how to instrument their systems for better monitoring. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Bitcoin and Bitcoin Cash are two cryptocurrencies with similar properties. But the supporters of each of these Bitcoin versions have strongly divergent opinions on the direction of the Bitcoin project. At the center of this debate is the subject of block size. Bitcoin’s block size determines how many transactions fit into each block that is mined. A larger block size leads to faster transactions and lower fees, but creates higher demands on mining hardware. A smaller block size leads to a slower on-chain network and higher fees, but allows the full nodes on the network to be run on low performance hardware like Raspberry Pi. Bitcoin Cash has a large block size. Bitcoin Core has a smaller block size. Proponents of the smaller block size argue that Bitcoin’s scaling can be achieved by the off-chain “lightning network” solution. Roger Ver is a Bitcoin entrepreneur and investor. Since he discovered the currency, he has been buying it and evangelizing it. More recently, Roger has become an ardent supporter of Bitcoin Cash–emphasizing that Bitcoin Cash is Bitcoin. In this episode, Roger describes his economic ideology, and explains why Bitcoin is so important to him. We explore how vested interests can shape the narrative and the direction of Bitcoin, and talk about the future of how corporations, governments, and individuals might be using cryptocurrencies. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Most new frontend webapps today use ReactJS. An increasing number of mobile apps are created using the cross-platform components of React Native. GraphQL, Facebook’s open source data-fetching middleware tool is being used by more and more companies, who are finding that it simplifies their development. Facebook’s open source suite of technologies created a new developer ecosystem. There is an increased demand for engineers who know how to build software with React, ReactJS, and GraphQL. This was the reasoning behind Gabe Greenberg starting G2i, a developer marketplace of engineers who write ReactJS, React Native, and GraphQL applications. In this episode, Gabe, Lee Johnson, and Chris Severns from G2i join the show to discuss React and the other Facebook open source technologies–as well as the ecosystem around them. We explored the architecture of a developer marketplace business, and how to scale a consulting company. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Machine learning tools are rapidly maturing. TensorFlow gave developers an open source version of Google’s internal machine learning framework. Cloud computing provides a cost effective, accessible way of training models. Edge computing allows for low latency deployments of models. But even if you are a kid with a laptop who has learned all the machine learning algorithms, read all of the deep learning textbooks, and figured out how to use AWS, all of the tooling and education in the world doesn’t change the fact that you still need data to build models. This illustrates why we need data-as-a-service. A kid with a laptop has access to infrastructure-as-a-service, platform-as-a-service, and software-as-a-service. As these tools build on each other, there has been an explosion of high-leverage software products. But the world of data sets remains crude and underdeveloped. Think about some data sets you could take advantage of: the number of emergency room patients that come into a hospital with chest pain; the size of the average coffee mug; the principal component breakdown of sidewalk concrete in San Francisco. SafeGraph is a company that offers data sets as a service. Auren Hoffman is the CEO of SafeGraph, and he joins the show to discuss why he started building SafeGraph and how he thinks about the state of publicly accessible data. Auren was previously on the podcast, and I always enjoy talking to him–this was a great episode and I think you will like it as well. Full disclosure: LiveRamp is a sponsor of Software Engineering Daily, LiveRamp being the company that Auren created prior to SafeGraph. Raj Chetty economic papers Paul Graham “Keep Your Identity Small” Auren Hoffman on Quora Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Let’s Talk Bitcoin is one of the most popular podcasts about cryptocurrencies. Adam B. Levine started it after three other podcasts he started did not get the traction he had hoped for. Adam parlayed the success of Let’s Talk Bitcoin into a network of podcasts–the Let’s Talk Bitcoin Network–which also includes one of my favorite shows, Epicenter. Adam joins me on today’s episode for a discussion of so many topics: the culture around cryptocurrencies, the art of podcasting, blockchain scalability, and ICOs. The conversation around ICOs was particularly exciting–if you have been listening to recent episodes, you have heard interviews with companies who have done ICOs. Some ICO companies are now facing legal ramifications for their token sales–and Adam and I have some disagreement over whether these ICO companies deserve much sympathy. It was a debate that I enjoyed and I hope to have Adam back on the show in the future for more debates. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Monitoring a Kubernetes cluster allows operators to track the resource utilization of the containers within that cluster. In today’s episode, Ilan Rabinovitch joins the show to explore the different options for setting up monitoring, and some common design patterns around Kubernetes logging and metrics gathering. Ilan is the VP of product and community at Datadog. Earlier in his career, Ilan spent much of his time working with Linux and taking part in the Linux community. We discussed the similarities and differences between the evolution of Linux and that of Kubernetes. In previous episodes, we have explored some common open source solutions for monitoring Kubernetes–including Prometheus and the EFK stack. Since Ilan works at Datadog, we explored how hosted solutions compare to self-managed monitoring. We also talked about how to assess different hosted solutions–such as those from a large cloud provider like AWS versus vendors that are specifically focused on monitoring. Full disclosure: Datadog is a sponsor of Software Engineering Daily. 8 Surprising Facts About Real Docker Adoption – Datadog Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Laura Shin is the host of Unchained, a podcast about cryptocurrencies and decentralized technology. For every episode, Laura does significant research and preparation, so the content turns out polished and high quality. Her enthusiasm for the subject of cryptocurrencies comes through in her reporting. Podcasting about cryptocurrencies requires walking a fine line. Cryptocurrencies have a mixture of drama and exciting technology–which are both great for a journalist. But you can’t get too deep in the drama, because the podcast will feel like a tabloid. And you can’t get too deep in the technical weeds, because the listener will fall asleep. Laura joins the show to discuss how she got into reporting on cryptocurrencies, why she got so obsessed with the subject, and her experience as a solo entrepreneurial journalist. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Social networks can make you feel connected to a global society. But those social networks are controlled by a corporate entity. The profit motivations of the corporation are not directly aligned with the experience of the users. Mastodon is an open source, decentralized social network. Eugen Rochko started building Mastodon in response to his dissatisfaction with centralized social networks like Facebook and Twitter. In the Mastodon model, users can run their own nodes, and other users can connect to them. You can follow users whose accounts reside in other nodes. Eugen joins the show to discuss how Mastodon works, and how its thousands of users interact on the platform. We explore the open source community that is building Mastodon, and speculate on the future of social networks. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Go is a language designed to improve systems programming. Go includes abstractions that simplify aspects of low level engineering that are historically difficult—concurrency, resource allocation, and dependency management. In that light, it makes sense that the Kubernetes container orchestration system was written in Go. Erik St. Martin is a cloud developer advocate at Microsoft, where he focuses on Go and Kubernetes. He also hosts the podcast “Go Time,” and has written a book on Go called Go In Action. Recently, Erik helped build the virtual Kubelet project, which allows Kubernetes nodes to be backed by services outside of that cluster. If you want your Kubernetes cluster to leverage abstractions such as serverless functions and standalone container instances, you can use Virtual Kubelet to treat these other abstractions as nodes. Erik also discussed his experience using Kubernetes at Comcast—which was a great case study. Near the end of the show, he also talked about organizing Gophercon, a popular conference for Go programmers—if you are organizing a conference or thinking about organizing one, it will be useful information to you. Full disclosure: Microsoft, where Erik works, is a sponsor of Software Engineering Daily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Tammy Butow has worked at Digital Ocean and Dropbox, where she built out infrastructure and managed engineering teams. At both of these companies, the customer base was at a massive scale. At Dropbox, Tammy worked on the database that holds metadata used by Dropbox users to access their files. To call this metadata system simply a “database” is an understatement–it is actually a multi-tiered system of caches and databases. This metadata is extremely sensitive–this is metadata that tells you where the objects across Dropbox are located–so it has to be highly available. To encourage that reliability, Tammy helped institute chaos engineering–inducing random failures across the Dropbox infrastructure, and making sure that the Dropbox systems could automatically respond to those failures. If you are unfamiliar with the topic, we have covered chaos engineering in two previous episodes of Software Engineering Daily. Tammy now works at Gremlin, a company that does chaos engineering as a service. In this show we talked about her experiences at Dropbox, and how to institute chaos engineering across databases. We also explored how her work at Gremlin–a smaller startup–compares to Dropbox and Digital Ocean, which are larger companies. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Software engineers have interacted with operations teams since software was being written. In the 1990s, most operations teams worked with physical infrastructure. They made sure that servers were provisioned correctly and installed with the proper software. When software engineers shipped bad code that took down a software company, the operations teams had to help recover the system—which often meant dealing with the physical servers. During the 90s and early 2000s, these operations engineers were often called “sysadmins,” “database admins” (if they worked on databases), or “infrastructure engineers.” Over the last decade, virtualization has led to many more logical servers across a company. Cloud computing has made infrastructure remote and programmable. The progression of infrastructure led to a change in how operations engineers work. Since infrastructure can be interacted with through code, operations engineers are now writing a lot more code. The “DevOps” movement can be seen through this lens. Operations teams were now writing software—and this meant that software engineers could now work on operations. Both software engineers and operators could create deployment pipelines, monitor application health, and improve the system scalability—all through written code. Site reliability engineering (or SRE) is a newer point along the evolutionary timeline of operations. Web applications can be unstable sometimes, and SRE is focused on making a site work more reliably. This is especially important for a company that makes business applications which other companies rely on. Mike Hiraga is the head of site reliability engineering at Atlassian. Atlassian makes several products that many businesses rely on—such as JIRA, Confluence, HipChat, and Bitbucket. Since the infrastructure is at a massive scale, Mike has a broad set of experiences from his work managing SRE at Atlassian. One particularly interesting topic is Atlassian’s migration to the cloud. Atlassian was started in 2002, before the cloud was widely used, and they have more recently made a push to move applications into the cloud. Full disclosure: Atlassian is a sponsor of Software Engineering Daily—and they are hiring, so if you are looking for a job, check out Atlassian jobs, or send me an email directly and I’m happy to introduce you to the team. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

The Interplanetary File System (IPFS) is a decentralized global, peer-to-peer file system. IPFS combines ideas from BitTorrent, Git, and Bitcoin, creating a new way to store and access objects across the Internet. When you access an object on almost any website, you are accessing the object via a location address—a URL. The URL tells you where to find the object. If the object is a photo on Facebook that you are linking to, the URL will have an address of somewhere on Facebook. Other objects that we access through URLs include web pages, videos, and JavaScript import packages. URLs seem natural to us. You look up an object based on where that object is being stored. Why would you do anything differently? A downside of location addressing is that if the location disappears, you can no longer access that object. If a government decides to censor a website that I wanted to visit, the government can shut down access to the server where that website sits, and my link will break. This happened in Turkey—where Wikipedia was shut down last year. Objects in IPFS are content addressed—you access an object by giving IPFS a cryptographic hash of the object, and IPFS will find someone on the network who has a copy of that object, and give you access to it. To look up a webpage in an IPFS browser, you put the content address in the address bar. When the HTML for that page is received, that page might have lots of other content-addressed files referred to on the page. Your browser can also grab all of those content-addressed files from the IPFS P2P network. In this episode, David Dias explains how IPFS is designed. David is an engineer at Protocol Labs, the company building out IPFS. This episode is a great companion to our previous show with Juan Benet, the creator of IPFS. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

The Ethereum community started as a small group of dedicated engineers. It has ballooned to thousands of engineers, entrepreneurs and investors, all of whom have a stake in the direction of Ethereum. Ethereum is an open source project, and the direction of a popular open source project can get complex. Ethereum is figuring out how to govern itself. It’s not clear what the perfect model is, but there are a few historical examples to think about: namely Linux and Bitcoin. Linux is similar to Ethereum in that there is a clear leader—Linux has Linus Torvalds and Ethereum has Vitalik Buterin. Linux is massively successful, and the Linux development team does have a top-down, hierarchical approach. But does a hierarchy with clear leadership make sense for a project like Ethereum, which has decentralization at its core? Bitcoin is headless—Satoshi disappeared in 2010, and there is not an official leader. Bitcoin has succeeded without a well-defined hierarchy–depending on what your definition of success is. Bitcoin development does not move as fast as Ethereum (this is by design)—but there is more widespread trust that the integrity of the system cannot be compromised by its creator. Hudson Jameson is an Ethereum developer and entrepreneur who has been part of the community since the early days. He works on Ethereum governance, which defines how changes to the Ethereum project are proposed, accepted, and implemented. Hudson joins the show today to talk about Ethereum governance, smart contracts, and the DAO hack. We did not discuss on-chain vs. off-chain governance, but I am hoping to cover that in a future episode. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

The pubsub pattern allows a developer to create channels, which messages can be written to and read from. Pubsub messaging is useful for multicast messaging–when you want to publish messages from a producer, and have multiple consumers who are subscribed to the publisher receive those messages. Almost any application that reaches a high level of complexity will need a pubsub system of some kind. The pubsub system itself can be complex. A pubsub system needs to scale up and down to handle different numbers of consumers and producers, and different volumes of messages. Back in 2010, the growth of mobile and cloud was leading to many new applications with high throughput, multi-user interactions. Developers were standing up their own instances of open source pubsub message queueing systems like RabbitMQ and ZeroMQ. Once the MQ systems needed to scale, the developer would need to handle the scaling. Stephen Blum started his company PubNub around this time, to create automatically scaling APIs for messaging. Stephen joins the show to discuss the infrastructure choices around building a large scale pubsub service, and how the company has scaled over time. He also talks about the management, product development, and business side of running the company. PubNub has built several additional technologies on top of the core infrastructure that was originally for messaging. Full disclosure: PubNub is a sponsor of Software Engineering Daily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Most technology companies rely on open source software projects. But open source software projects are often maintained by a group of people that is not affiliated with any particular company. When an open source project develops too much technical debt, it can become a tragedy of the commons. Who is responsible for maintaining these open source projects? This is the motivation for open source bounties. Companies and individuals who rely on open source create bounties, which are financial incentives for developers to solve problems within the open source project. Kevin Owocki is the creator of Gitcoin, a platform for open source bounties that is mediated by an Ethereum smart contract. Kevin joins the show to discuss his experience building Gitcoin–as well as some of the problems with the blockchain space, such as rampant ICOs. Gitcoin is NOT a cryptocurrency or token itself–it is a platform for open source software to be built more efficiently. Kevin was an awesome guest and you will enjoy the conversation. Gitcoin is a nice example of a real-world Ethereum use case–it uses Ethereum for escrow: if I post a $25 bounty for someone to fix a bug in my open source project, I will lock up ether in a smart contract. When the bug is fixed, the programmer who fixed it will submit a pull request on Github, and I will release the ether from the smart contract to pay them. We would love for you to fill out our listener survey at softwareengineeringdaily.com/survey. This will help us decide what other content to focus on. Of course–you can also send me an email at any time, [email protected]. And in the meantime, if you are completely sick of cryptocurrencies, check out our back catalog of episodes at softwaredaily.com, or by downloading our Software Engineering Daily apps, which have all of our episodes including our Greatest Hits, which is a curated set of the most popular shows. The apps will soon have offline downloads and bookmarking. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Zcash is a payment and consensus system that allows users to transfer money to each other with strong guarantees of privacy. Zcash implements the same core features of Bitcoin, with the added functionality of shielded payments. Shielded payments are private, and they are enabled by a novel cryptographic technique called zk-SNARKS: zero knowledge succinct non-interactive argument of knowledge. A zk-SNARK allows for the proof that a certain piece of information is valid without revealing any information other than the validity of that information itself. Before you listen to this episode, it might be useful to go back to our previous episode about Zcash with Nathan Wilcox, in which he gives an overview of the technology. This episode is a deeper dive into how Zcash transactions work, and why zk-SNARKS are important. We would love for you to fill out our listener survey at softwareengineeringdaily.com/survey. This will help us decide what other content to focus on. Of course–you can also send me an email at any time, [email protected]. And in the meantime, if you are completely sick of cryptocurrencies, check out our back catalog of episodes at softwaredaily.com, or by downloading our apps, which have all of our episodes including our Greatest Hits, which is a curated set of the most popular shows. The apps will soon have offline downloads and bookmarking. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

“The Federal Reserve System is fraudulent. Whatever its stated purpose, its effective purpose is to create a mechanism of deficit spending by politicians, through the insidious invisible taxation of monetary debasement (aka inflation).” These are the words of Erik Voorhees, the CEO of crypto financial exchange ShapeShift. Long before he started ShapeShift, Erik was opposed to some of the core principles of the global financial system, in which he sees the US dollar as a means of control. As an early adopter of Bitcoin, he saw a way to make financial transactions without using fiat currency. Erik’s company ShapeShift allows users to convert different digital currencies between each other. Because it only makes exchanges of currencies and does not hold much currency at any time, ShapeShift is resilient to hacking. In this episode, Erik and I discussed his economic philosophy, and how that informs his affinity for cryptocurrencies. Erik also describes the architecture of ShapeShift, and gives some advice for how to think about building businesses around cryptocurrencies. ShapeShift has had a few near-death experiences, like any startup, and there is a useful story in this episode about how to survive and recover from a serious business setback. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

We sign many different types of contracts throughout our lives. We sign a mortgage to get a loan for a house. When we go to the hospital, we sign a piece of paper that defines how our medical data can be shared between organizations. These pieces of paper represent our opting into an agreement that will be mediated and enforced by computer interactions. We can’t see the code behind those computer interactions, and we can’t verify that it is abiding by the contract we agreed to. Smart contracts allow for programmatic execution of contractual agreements. Code is law, and there is less ambiguity. The most widely used smart contract platform is the Ethereum blockchain–but several large enterprises are creating their own smart contracts. Should all smart contracts be decentralized, or do enterprise consortium blockchains make sense? In this episode, Marley Gray from Microsoft joins the show to discuss enterprise smart contracts–why you would want to use them and how they can be architected. Marley has worked on banking and financial technology for over a decade, and makes some strong arguments for why banks will adopt smart contracts, and the timeline for how that might take place. We would love for you to fill out our listener survey at softwareengineeringdaily.com/survey. This will help us decide what other content to focus on. Of course–you can also send me an email at any time, [email protected]. And in the meantime, if you are completely sick of cryptocurrencies, check out our back catalog of episodes at softwaredaily.com, or by downloading our apps, which have all of our episodes including our Greatest Hits, which is a curated set of the most popular shows. The apps will soon have offline downloads and bookmarking. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Ethereum is a system for running decentralized smart contracts. In the current implementation of Ethereum, every smart contract gets deployed to every full node. Whenever a user wants to call a smart contract, that smart contract gets executed on each full node–across the entire network. The current model for smart contract execution needs to be made more scalable. In today’s episode, Christian Reitwiessner joins the show to describe Plasma–a system for scaling smart contracts. Christian is a developer who has worked extensively on Solidity, the most popular smart contract programming language in Ethereum. For the last month, we have focused on blockchain related topics, and we will soon be shifting to other subjects. Some of the listeners have not enjoyed the blockchain focus, other people have loved it–for everyone listening, we would love for you to fill out our listener survey at softwareengineeringdaily.com/survey. This will help us decide what other content to focus on. Of course–you can also send me an email at any time, [email protected]. And join our Slack at softwareengineeringdaily.com/slack. And if you are completely sick of cryptocurrencies, check out our back catalog of episodes at softwaredaily.com, or by downloading our apps, which have all of our episodes including our Greatest Hits, which is a curated set of the most popular shows. The apps will soon have offline downloads and bookmarking. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Soumya Basu is a PhD student at Cornell, where he studies distributed systems problems associated with cryptocurrencies. Soumya is advised by Emin Gun Sirer, a Cornell professor who previously appeared on the show to discuss smart contract security. Soumya joins the show today to talk about a variety of issues in the cryptocurrency space. We first explored the degree to which Bitcoin and Ethereum are actually decentralized–which might be less than you think. Because of the centralization of mining pools, much of the transaction processing is also centralized. After talking about decentralization, we got into Soumya’s research focus–cryptocurrency networking and block propagation. Bitcoin transactions are collected into blocks. When a Bitcoin full node solves the cryptographic puzzle associated with a block of transactions, that full node broadcasts the new block to all the other nodes in the network. It is important for that block broadcast to be fast and efficient, so that the other full nodes in the network can be made aware of the new block as soon as possible, and they can start working from the updated chain. The problem of making all nodes in the network aware of a new block is known as “block propagation.” Block propagation can be accelerated through the use of relay nodes. A relay node is a node that is dedicated to communicating these new blocks throughout the blockchain. Soumya is working on a relay node architecture called Falcon–and in this episode we talk all about what Falcon is. If you are looking for all 700 episodes of Software Engineering Daily, check out our apps on the iOS or Android app store. We’ve got tons of episodes on blockchains, business, distributed systems, and tons of other topics. If you want to become a paid subscriber to Software Engineering Daily, you can hear all of our episodes without ads–you can subscribe at softwaredaily.com. And all of the code for our apps is open source. If you are looking for an open source community to be a part of, come check out github.com/softwareengineeringdaily. Meetups for Software Engineering Daily are being planned! Go to softwareengineeringdaily.com/meetup if you want to register for an upcoming Meetup. In March, I’ll be visiting Datadog in New York and Hubspot in Boston, and in April I’ll be at Telesign in LA. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Consensus protocols are used to allow computers to work together. A consensus protocol lets different servers agree on the state of a system. For decades, these protocols have been used to establish consensus among database nodes, application servers, and other infrastructure that runs within an enterprise. More recently, new consensus protocols have been invented to allow cryptoeconomic systems to agree on the state of a financial system. The first cryptoeconomic consensus protocol to reach wide adoption was Nakamoto consensus–the proof-of-work system used for consensus of Bitcoin. Since then, other systems have been developed, with different tradeoffs in security, speed, and formal verifiability. Ethan Buchman is the CTO at Tendermint, a consensus system for blockchains. In addition to working on Tendermint, Ethan works on Cosmos, a network of blockchains. In this episode, we talk about different consensus systems–for centralized, trustworthy systems as well as for trustless systems like currencies. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

The DAO was a system of smart contracts on the Ethereum blockchain that investors put millions of dollars into. Back in May 2016, it was the largest crowdfunding event in history, and we discussed it in detail in a previous episode with Matt Leising. The DAO was hacked due to a security vulnerability, and this event led to a hard fork of Ethereum. The DAO was organized by a company called Slock.it. Slock.it’s original goal was to allow people to connect devices to the Ethereum blockchain. If you could connect smart locks, cars, and electricity systems to the blockchain, it could create decentralized systems for sharing these devices. To raise money, Slock.it created the DAO. Although the initial scope of the DAO was to raise money for Slock.it, over time it expanded in scope to become a decentralized system for venture capital. When the DAO was hacked, the events that followed shook the Ethereum community. The hard fork lowered the financial damage inflicted on the investors–but there was still outrage within the community. How was it possible for an open source crowdfunding project to launch with a security vulnerability? As the Ethereum world looked for someone to blame, they turned to Slock.it. Thus began a very difficult period in the life of Christoph Jentzsch. Christoph is the CEO of Slock.it, and he has been involved in the Ethereum community since the early days. When people think of Slock.it, they might imagine a group of people that move fast and break things. But in fact, Christoph’s early work on Ethereum was around rigorous unit testing of different Ethereum clients. He was obsessed with testing, and consistency between the different Ethereum interfaces. In today’s episode, Christoph and I talk about his early experiences with Ethereum, his reflections on the events of the DAO, and the direction that Slock.it is going today. Since the events of the DAO, the company has refocused its efforts on the original mission–to connect devices to the Ethereum blockchain. Meetups for Software Engineering Daily are being planned! Go to softwareengineeringdaily.com/meetup if you want to register for an upcoming Meetup. In March, I’ll be visiting Datadog in New York and Hubspot in Boston, and in April I’ll be at Telesign in LA. If you are looking for all 700 episodes of Software Engineering Daily, check out our apps on the iOS or Android app store. We’ve got tons of episodes on blockchains, business, distributed systems, and tons of other topics. If you want to become a paid subscriber to Software Engineering Daily, you can hear all of our episodes without ads–you can subscribe at softwaredaily.com. And all of the code for our apps is open source. If you are looking for an open source community to be a part of, come check out github.com/softwareengineeringdaily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Data streams about the weather can be used to predict how soybean futures are going to change in price. Satellite data streams can take pictures of the number of cars on the road, and judge how traffic patterns are changing. Search engines can aggregate data from different queries and determine what people are most interested in. Data streams define how the world is changing over time. Technology companies process these data streams and make decisions based on that stream. The most direct example of this might be financial trading companies, which use all kinds of data streams to predict economic price changes. When Henri Pihkala worked on algorithmic trading systems, he saw how useful these data streams are, and decided to build products around data streaming. Eventually, Henri started working on Streamr, a platform for data streams to be bought and sold on top of the Ethereum network. Streamr is an adaptation of technology that Henri worked on before he started working on the decentralized version. The original technology is a user interface for connecting data streams and building applications on top of them, and he acquired several customers for that platform. Today, the Streamr platform is still mostly centralized, but Henri and his team are working on building out the decentralized infrastructure. Streamr raised an ICO worth ~25 million Euros. Most startups would not raise this amount of money before series B, much less before they have a product with a large user base. In this episode, Henri discusses why they raised so much money, and explains why ICOs are different than equity raises. The investors who participated in the Streamr ICO received the DATAcoin token. Henri also explained why it makes sense for this ecosystem to have its own token. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

To use a web application, you probably open a web browser or a mobile app. To access an Ethereum application, many people use an Ethereum browser. In previous episodes, we explored Metamask and Mist, which are Ethereum browsers for the desktop. In today’s episode, we explore Status, a mobile Ethereum browser. Status founders Jarrad Hope and Oskar Thoren join the show to talk about the engineering of Status. How Status connects to the Ethereum blockchain, what people want from Ethereum applications, and the engineering of the Status app itself. Status is built using React Native–which is working out quite well for them. We also talked some about the mechanics of an ICO. Status has raised $100m in their ICO for the Status Network Token. An ICO differs from raising equity in several ways. Rather than representing a direct stake in the business, a token represents a stake in the ecosystem that is being built. Through their ICO, Status raised much more than a startup at a similar stage in company development would have–and the vesting schedule for the founders is 2 years. After two years, their stake will be liquid. This illustrates another way that the ICO can contrast with a traditional startup equity offering. In a traditional startup, there is not a liquid open market for equity prior to the company going public. This can be good, as it forces the founders to maintain their skin in the game until they have proven the business. But it can also be bad–founders should arguably be able to take some money off of the table even if their business model is not completely worked out. In the interview, Jarrad explained that he anticipates the open source community around Status to be contributing more to the Status app over time, because the community has a stake in the app by purchasing the Status token. I hope this is the case–it would be very cool to see more consumer-facing open source applications. Status is a consumer facing app–and it did make me think that it is strange that there is so much open source software for building applications (think about React Native, Kubernetes, Kafka), but there are fewer consumer-facing open source apps. There’s not an open source Uber, an open source Facebook, or an open source Google. Why is that? Maybe that’s because we are still in the days where someone has to pay for the backend compute layer. In other words–open source code is free to host, but running the actual application infrastructure still requires the owner to pay–so it makes sense that consumer applications are still developed and maintained by central actors. With Ethereum, maybe that will change and we will see more consumer facing, open source, decentralized applications. That is certainly the world that Status.im is hoping for. Speaking of consumer facing open source applications: check out our Software Engineering Daily apps on the iOS or Android app store. All 700 episodes of Software Engineering Daily are in the app–we’ve got tons of episodes on blockchains, business, distributed systems, and tons of other topics. If you want to become a paid subscriber to Software Engineering Daily, you can hear all of our episodes without ads–you can subscribe at softwaredaily.com. And all of the code for our apps is open source. If you are looking for an open source community to be a part of, come check out github.com/softwareengineeringdaily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Anthony Diiorio was involved with Ethereum since the earliest days. He was one of the first people to see the Ethereum ideas presented by Vitalik Buterin, and he invested deeply in Ethereum–both financially and by helping to establish the early Ethereum community. Anthony started Decentral in 2014, which is a hub for his projects in the cryptocurrency space, the most impactful project being Jaxx. Jaxx is a blockchain wallet that can hold multiple different cryptocurrencies. It works by connecting a small client-side application to remote full nodes. The user interface is simple, and Jaxx maintains the full node instances that the small client-side application connects to. We discuss the architecture of Jaxx in more detail during this episode. We also talk about Anthony’s background–which includes a wide range of businesses: marketing, patio door manufacturing, real estate, and eventually blockchains. Anthony had a wealth of information to provide around entrepreneurship–both inside and outside of the blockchain space. If you are looking for all 700 episodes of Software Engineering Daily, check out our apps on the iOS or Android app store. We’ve got tons of episodes on blockchains, business, distributed systems, and tons of other topics. If you want to become a paid subscriber to Software Engineering Daily, you can hear all of our episodes without ads–you can subscribe at softwaredaily.com. And all of the code for our apps is open source. If you are looking for an open source community to be a part of, come check out github.com/softwareengineeringdaily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

A financial exchange is an operationally intensive business. You have customers making a high volume of transactions, your service has to be low latency and highly available, and you are dealing with a lot of money. A cryptocurrency exchange has all of the complexity of a typical financial exchange–and then some additional complexity. Shapeshift is a cryptocurrency exchange that allows users to buy and sell digital assets–Bitcoin, Ethereum, Litecoin, and lots of other currencies. Shapeshift also has a set of tools and APIs that allow developers to build higher level applications that transact in cryptocurrencies. Shapeshift’s CEO is an early cryptocurrency entrepreneur named Erik Voorhees, who will appear on the show in the near future. Today’s guest Jon is the COO of Shapeshift–he handles the operations of the company. He prefers not to use his last name, because Shapeshift is particularly sensitive to social engineering attacks. We’ll get into why that is in the episode–and explore lots of other topics too. How to scale a cryptocurrency exchange, the products Shapeshift offers, and some of the near-death experiences that Shapeshift has had. After all–it is a startup, and every startup has moments where it seems like the company will die. Meetups for Software Engineering Daily are being planned! Go to softwareengineeringdaily.com/meetup if you want to register for an upcoming Meetup. In March, I’ll be visiting Datadog in New York and Hubspot in Boston, and in April I’ll be at Telesign in LA. If you are looking for all 700 episodes of Software Engineering Daily, check out our apps on the iOS or Android app store. We’ve got tons of episodes on blockchains, business, distributed systems, and tons of other topics. If you want to become a paid subscriber to Software Engineering Daily, you can hear all of our episodes without ads–you can subscribe at softwaredaily.com. And all of the code for our apps is open source. If you are looking for an open source community to be a part of, come check out github.com/softwareengineeringdaily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Cryptocurrency speculation has pulled in a large population of people who do not know what they are investing in. If you hear about an investment of $1000 turning into $1M, it’s tempting to get sucked in yourself. For most of these everyday people, the game is completely rigged. A large percentage of market activity is driven by “pump and dumps.” A pump and dump is a conspiracy to trick investors into buying a currency. An insider group commits the pump and dump. This is accomplished by purchasing the currency ahead of time, then promoting it via Twitter, Telegram, and reddit. The outsiders fall victim to the promotion of the currency, and buy it after the fast run-up in value. The currency then crashes, and the outsiders are left “holding the bag.” Pump and dumps are not a new phenomenon—they have happened with worthless penny stocks. One thing that is new is the ease with which new cryptocurrencies are being created. Launching an ICO is easy. Marketing it is cheap. Pumping and dumping has never been more accessible. And buying them is quite easy as well. This has led to a perfect storm of naive investment capital. Bruno Skvorc is the CEO and owner of Bitfalls, a site with blog posts, news, and information about cryptocurrencies. He wrote a post called “The Anatomy of a Pump and Dump Group,” which details how cryptocurrency pump and dumps have been used to swindle investors out of millions of dollars. Meetups for Software Engineering Daily are being planned! Go to softwareengineeringdaily.com/meetup if you want to register for an upcoming Meetup. In March, I’ll be visiting Datadog in New York and Hubspot in Boston, and in April I’ll be at Telesign in LA. If you are looking for an internship, apply to the Software Engineering Daily internship, at softwaredaily.com/jobs. And if you are looking to recruit engineers, you can post jobs for your company there as well–it’s completely free to post jobs and to apply. We are hoping to find interns to contribute to the Software Daily open source project–and if you want to see what we are building, go to SoftwareDaily.com or check out our apps in the iOS or Android app store. They have all 650 of our episodes, with recommendations, related links, discussions and more. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.