Software Engineering Daily

Zcash is a payment and consensus system that allows users to transfer money to each other with strong guarantees of privacy. Zcash implements the same core features of Bitcoin, with the added functionality of shielded payments. Shielded payments are private, and they are enabled by a novel cryptographic technique called zk-SNARKS: zero knowledge succinct non-interactive argument of knowledge. A zk-SNARK allows for the proof that a certain piece of information is valid without revealing any information other than the validity of that information itself. Before you listen to this episode, it might be useful to go back to our previous episode about Zcash with Nathan Wilcox, in which he gives an overview of the technology. This episode is a deeper dive into how Zcash transactions work, and why zk-SNARKS are important. We would love for you to fill out our listener survey at softwareengineeringdaily.com/survey. This will help us decide what other content to focus on. Of course–you can also send me an email at any time, [email protected]. And in the meantime, if you are completely sick of cryptocurrencies, check out our back catalog of episodes at softwaredaily.com, or by downloading our apps, which have all of our episodes including our Greatest Hits, which is a curated set of the most popular shows. The apps will soon have offline downloads and bookmarking. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

“The Federal Reserve System is fraudulent. Whatever its stated purpose, its effective purpose is to create a mechanism of deficit spending by politicians, through the insidious invisible taxation of monetary debasement (aka inflation).” These are the words of Erik Voorhees, the CEO of crypto financial exchange ShapeShift. Long before he started ShapeShift, Erik was opposed to some of the core principles of the global financial system, in which he sees the US dollar as a means of control. As an early adopter of Bitcoin, he saw a way to make financial transactions without using fiat currency. Erik’s company ShapeShift allows users to convert different digital currencies between each other. Because it only makes exchanges of currencies and does not hold much currency at any time, ShapeShift is resilient to hacking. In this episode, Erik and I discussed his economic philosophy, and how that informs his affinity for cryptocurrencies. Erik also describes the architecture of ShapeShift, and gives some advice for how to think about building businesses around cryptocurrencies. ShapeShift has had a few near-death experiences, like any startup, and there is a useful story in this episode about how to survive and recover from a serious business setback. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

We sign many different types of contracts throughout our lives. We sign a mortgage to get a loan for a house. When we go to the hospital, we sign a piece of paper that defines how our medical data can be shared between organizations. These pieces of paper represent our opting into an agreement that will be mediated and enforced by computer interactions. We can’t see the code behind those computer interactions, and we can’t verify that it is abiding by the contract we agreed to. Smart contracts allow for programmatic execution of contractual agreements. Code is law, and there is less ambiguity. The most widely used smart contract platform is the Ethereum blockchain–but several large enterprises are creating their own smart contracts. Should all smart contracts be decentralized, or do enterprise consortium blockchains make sense? In this episode, Marley Gray from Microsoft joins the show to discuss enterprise smart contracts–why you would want to use them and how they can be architected. Marley has worked on banking and financial technology for over a decade, and makes some strong arguments for why banks will adopt smart contracts, and the timeline for how that might take place. We would love for you to fill out our listener survey at softwareengineeringdaily.com/survey. This will help us decide what other content to focus on. Of course–you can also send me an email at any time, [email protected]. And in the meantime, if you are completely sick of cryptocurrencies, check out our back catalog of episodes at softwaredaily.com, or by downloading our apps, which have all of our episodes including our Greatest Hits, which is a curated set of the most popular shows. The apps will soon have offline downloads and bookmarking. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Ethereum is a system for running decentralized smart contracts. In the current implementation of Ethereum, every smart contract gets deployed to every full node. Whenever a user wants to call a smart contract, that smart contract gets executed on each full node–across the entire network. The current model for smart contract execution needs to be made more scalable. In today’s episode, Christian Reitwiessner joins the show to describe Plasma–a system for scaling smart contracts. Christian is a developer who has worked extensively on Solidity, the most popular smart contract programming language in Ethereum. For the last month, we have focused on blockchain related topics, and we will soon be shifting to other subjects. Some of the listeners have not enjoyed the blockchain focus, other people have loved it–for everyone listening, we would love for you to fill out our listener survey at softwareengineeringdaily.com/survey. This will help us decide what other content to focus on. Of course–you can also send me an email at any time, [email protected]. And join our Slack at softwareengineeringdaily.com/slack. And if you are completely sick of cryptocurrencies, check out our back catalog of episodes at softwaredaily.com, or by downloading our apps, which have all of our episodes including our Greatest Hits, which is a curated set of the most popular shows. The apps will soon have offline downloads and bookmarking. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Soumya Basu is a PhD student at Cornell, where he studies distributed systems problems associated with cryptocurrencies. Soumya is advised by Emin Gun Sirer, a Cornell professor who previously appeared on the show to discuss smart contract security. Soumya joins the show today to talk about a variety of issues in the cryptocurrency space. We first explored the degree to which Bitcoin and Ethereum are actually decentralized–which might be less than you think. Because of the centralization of mining pools, much of the transaction processing is also centralized. After talking about decentralization, we got into Soumya’s research focus–cryptocurrency networking and block propagation. Bitcoin transactions are collected into blocks. When a Bitcoin full node solves the cryptographic puzzle associated with a block of transactions, that full node broadcasts the new block to all the other nodes in the network. It is important for that block broadcast to be fast and efficient, so that the other full nodes in the network can be made aware of the new block as soon as possible, and they can start working from the updated chain. The problem of making all nodes in the network aware of a new block is known as “block propagation.” Block propagation can be accelerated through the use of relay nodes. A relay node is a node that is dedicated to communicating these new blocks throughout the blockchain. Soumya is working on a relay node architecture called Falcon–and in this episode we talk all about what Falcon is. If you are looking for all 700 episodes of Software Engineering Daily, check out our apps on the iOS or Android app store. We’ve got tons of episodes on blockchains, business, distributed systems, and tons of other topics. If you want to become a paid subscriber to Software Engineering Daily, you can hear all of our episodes without ads–you can subscribe at softwaredaily.com. And all of the code for our apps is open source. If you are looking for an open source community to be a part of, come check out github.com/softwareengineeringdaily. Meetups for Software Engineering Daily are being planned! Go to softwareengineeringdaily.com/meetup if you want to register for an upcoming Meetup. In March, I’ll be visiting Datadog in New York and Hubspot in Boston, and in April I’ll be at Telesign in LA. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Consensus protocols are used to allow computers to work together. A consensus protocol lets different servers agree on the state of a system. For decades, these protocols have been used to establish consensus among database nodes, application servers, and other infrastructure that runs within an enterprise. More recently, new consensus protocols have been invented to allow cryptoeconomic systems to agree on the state of a financial system. The first cryptoeconomic consensus protocol to reach wide adoption was Nakamoto consensus–the proof-of-work system used for consensus of Bitcoin. Since then, other systems have been developed, with different tradeoffs in security, speed, and formal verifiability. Ethan Buchman is the CTO at Tendermint, a consensus system for blockchains. In addition to working on Tendermint, Ethan works on Cosmos, a network of blockchains. In this episode, we talk about different consensus systems–for centralized, trustworthy systems as well as for trustless systems like currencies. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

The DAO was a system of smart contracts on the Ethereum blockchain that investors put millions of dollars into. Back in May 2016, it was the largest crowdfunding event in history, and we discussed it in detail in a previous episode with Matt Leising. The DAO was hacked due to a security vulnerability, and this event led to a hard fork of Ethereum. The DAO was organized by a company called Slock.it. Slock.it’s original goal was to allow people to connect devices to the Ethereum blockchain. If you could connect smart locks, cars, and electricity systems to the blockchain, it could create decentralized systems for sharing these devices. To raise money, Slock.it created the DAO. Although the initial scope of the DAO was to raise money for Slock.it, over time it expanded in scope to become a decentralized system for venture capital. When the DAO was hacked, the events that followed shook the Ethereum community. The hard fork lowered the financial damage inflicted on the investors–but there was still outrage within the community. How was it possible for an open source crowdfunding project to launch with a security vulnerability? As the Ethereum world looked for someone to blame, they turned to Slock.it. Thus began a very difficult period in the life of Christoph Jentzsch. Christoph is the CEO of Slock.it, and he has been involved in the Ethereum community since the early days. When people think of Slock.it, they might imagine a group of people that move fast and break things. But in fact, Christoph’s early work on Ethereum was around rigorous unit testing of different Ethereum clients. He was obsessed with testing, and consistency between the different Ethereum interfaces. In today’s episode, Christoph and I talk about his early experiences with Ethereum, his reflections on the events of the DAO, and the direction that Slock.it is going today. Since the events of the DAO, the company has refocused its efforts on the original mission–to connect devices to the Ethereum blockchain. Meetups for Software Engineering Daily are being planned! Go to softwareengineeringdaily.com/meetup if you want to register for an upcoming Meetup. In March, I’ll be visiting Datadog in New York and Hubspot in Boston, and in April I’ll be at Telesign in LA. If you are looking for all 700 episodes of Software Engineering Daily, check out our apps on the iOS or Android app store. We’ve got tons of episodes on blockchains, business, distributed systems, and tons of other topics. If you want to become a paid subscriber to Software Engineering Daily, you can hear all of our episodes without ads–you can subscribe at softwaredaily.com. And all of the code for our apps is open source. If you are looking for an open source community to be a part of, come check out github.com/softwareengineeringdaily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Data streams about the weather can be used to predict how soybean futures are going to change in price. Satellite data streams can take pictures of the number of cars on the road, and judge how traffic patterns are changing. Search engines can aggregate data from different queries and determine what people are most interested in. Data streams define how the world is changing over time. Technology companies process these data streams and make decisions based on that stream. The most direct example of this might be financial trading companies, which use all kinds of data streams to predict economic price changes. When Henri Pihkala worked on algorithmic trading systems, he saw how useful these data streams are, and decided to build products around data streaming. Eventually, Henri started working on Streamr, a platform for data streams to be bought and sold on top of the Ethereum network. Streamr is an adaptation of technology that Henri worked on before he started working on the decentralized version. The original technology is a user interface for connecting data streams and building applications on top of them, and he acquired several customers for that platform. Today, the Streamr platform is still mostly centralized, but Henri and his team are working on building out the decentralized infrastructure. Streamr raised an ICO worth ~25 million Euros. Most startups would not raise this amount of money before series B, much less before they have a product with a large user base. In this episode, Henri discusses why they raised so much money, and explains why ICOs are different than equity raises. The investors who participated in the Streamr ICO received the DATAcoin token. Henri also explained why it makes sense for this ecosystem to have its own token. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

To use a web application, you probably open a web browser or a mobile app. To access an Ethereum application, many people use an Ethereum browser. In previous episodes, we explored Metamask and Mist, which are Ethereum browsers for the desktop. In today’s episode, we explore Status, a mobile Ethereum browser. Status founders Jarrad Hope and Oskar Thoren join the show to talk about the engineering of Status. How Status connects to the Ethereum blockchain, what people want from Ethereum applications, and the engineering of the Status app itself. Status is built using React Native–which is working out quite well for them. We also talked some about the mechanics of an ICO. Status has raised $100m in their ICO for the Status Network Token. An ICO differs from raising equity in several ways. Rather than representing a direct stake in the business, a token represents a stake in the ecosystem that is being built. Through their ICO, Status raised much more than a startup at a similar stage in company development would have–and the vesting schedule for the founders is 2 years. After two years, their stake will be liquid. This illustrates another way that the ICO can contrast with a traditional startup equity offering. In a traditional startup, there is not a liquid open market for equity prior to the company going public. This can be good, as it forces the founders to maintain their skin in the game until they have proven the business. But it can also be bad–founders should arguably be able to take some money off of the table even if their business model is not completely worked out. In the interview, Jarrad explained that he anticipates the open source community around Status to be contributing more to the Status app over time, because the community has a stake in the app by purchasing the Status token. I hope this is the case–it would be very cool to see more consumer-facing open source applications. Status is a consumer facing app–and it did make me think that it is strange that there is so much open source software for building applications (think about React Native, Kubernetes, Kafka), but there are fewer consumer-facing open source apps. There’s not an open source Uber, an open source Facebook, or an open source Google. Why is that? Maybe that’s because we are still in the days where someone has to pay for the backend compute layer. In other words–open source code is free to host, but running the actual application infrastructure still requires the owner to pay–so it makes sense that consumer applications are still developed and maintained by central actors. With Ethereum, maybe that will change and we will see more consumer facing, open source, decentralized applications. That is certainly the world that Status.im is hoping for. Speaking of consumer facing open source applications: check out our Software Engineering Daily apps on the iOS or Android app store. All 700 episodes of Software Engineering Daily are in the app–we’ve got tons of episodes on blockchains, business, distributed systems, and tons of other topics. If you want to become a paid subscriber to Software Engineering Daily, you can hear all of our episodes without ads–you can subscribe at softwaredaily.com. And all of the code for our apps is open source. If you are looking for an open source community to be a part of, come check out github.com/softwareengineeringdaily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Anthony Diiorio was involved with Ethereum since the earliest days. He was one of the first people to see the Ethereum ideas presented by Vitalik Buterin, and he invested deeply in Ethereum–both financially and by helping to establish the early Ethereum community. Anthony started Decentral in 2014, which is a hub for his projects in the cryptocurrency space, the most impactful project being Jaxx. Jaxx is a blockchain wallet that can hold multiple different cryptocurrencies. It works by connecting a small client-side application to remote full nodes. The user interface is simple, and Jaxx maintains the full node instances that the small client-side application connects to. We discuss the architecture of Jaxx in more detail during this episode. We also talk about Anthony’s background–which includes a wide range of businesses: marketing, patio door manufacturing, real estate, and eventually blockchains. Anthony had a wealth of information to provide around entrepreneurship–both inside and outside of the blockchain space. If you are looking for all 700 episodes of Software Engineering Daily, check out our apps on the iOS or Android app store. We’ve got tons of episodes on blockchains, business, distributed systems, and tons of other topics. If you want to become a paid subscriber to Software Engineering Daily, you can hear all of our episodes without ads–you can subscribe at softwaredaily.com. And all of the code for our apps is open source. If you are looking for an open source community to be a part of, come check out github.com/softwareengineeringdaily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

A financial exchange is an operationally intensive business. You have customers making a high volume of transactions, your service has to be low latency and highly available, and you are dealing with a lot of money. A cryptocurrency exchange has all of the complexity of a typical financial exchange–and then some additional complexity. Shapeshift is a cryptocurrency exchange that allows users to buy and sell digital assets–Bitcoin, Ethereum, Litecoin, and lots of other currencies. Shapeshift also has a set of tools and APIs that allow developers to build higher level applications that transact in cryptocurrencies. Shapeshift’s CEO is an early cryptocurrency entrepreneur named Erik Voorhees, who will appear on the show in the near future. Today’s guest Jon is the COO of Shapeshift–he handles the operations of the company. He prefers not to use his last name, because Shapeshift is particularly sensitive to social engineering attacks. We’ll get into why that is in the episode–and explore lots of other topics too. How to scale a cryptocurrency exchange, the products Shapeshift offers, and some of the near-death experiences that Shapeshift has had. After all–it is a startup, and every startup has moments where it seems like the company will die. Meetups for Software Engineering Daily are being planned! Go to softwareengineeringdaily.com/meetup if you want to register for an upcoming Meetup. In March, I’ll be visiting Datadog in New York and Hubspot in Boston, and in April I’ll be at Telesign in LA. If you are looking for all 700 episodes of Software Engineering Daily, check out our apps on the iOS or Android app store. We’ve got tons of episodes on blockchains, business, distributed systems, and tons of other topics. If you want to become a paid subscriber to Software Engineering Daily, you can hear all of our episodes without ads–you can subscribe at softwaredaily.com. And all of the code for our apps is open source. If you are looking for an open source community to be a part of, come check out github.com/softwareengineeringdaily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Cryptocurrency speculation has pulled in a large population of people who do not know what they are investing in. If you hear about an investment of $1000 turning into $1M, it’s tempting to get sucked in yourself. For most of these everyday people, the game is completely rigged. A large percentage of market activity is driven by “pump and dumps.” A pump and dump is a conspiracy to trick investors into buying a currency. An insider group commits the pump and dump. This is accomplished by purchasing the currency ahead of time, then promoting it via Twitter, Telegram, and reddit. The outsiders fall victim to the promotion of the currency, and buy it after the fast run-up in value. The currency then crashes, and the outsiders are left “holding the bag.” Pump and dumps are not a new phenomenon—they have happened with worthless penny stocks. One thing that is new is the ease with which new cryptocurrencies are being created. Launching an ICO is easy. Marketing it is cheap. Pumping and dumping has never been more accessible. And buying them is quite easy as well. This has led to a perfect storm of naive investment capital. Bruno Skvorc is the CEO and owner of Bitfalls, a site with blog posts, news, and information about cryptocurrencies. He wrote a post called “The Anatomy of a Pump and Dump Group,” which details how cryptocurrency pump and dumps have been used to swindle investors out of millions of dollars. Meetups for Software Engineering Daily are being planned! Go to softwareengineeringdaily.com/meetup if you want to register for an upcoming Meetup. In March, I’ll be visiting Datadog in New York and Hubspot in Boston, and in April I’ll be at Telesign in LA. If you are looking for an internship, apply to the Software Engineering Daily internship, at softwaredaily.com/jobs. And if you are looking to recruit engineers, you can post jobs for your company there as well–it’s completely free to post jobs and to apply. We are hoping to find interns to contribute to the Software Daily open source project–and if you want to see what we are building, go to SoftwareDaily.com or check out our apps in the iOS or Android app store. They have all 650 of our episodes, with recommendations, related links, discussions and more. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

In the finance industry, many people have a computer on their desk called a Bloomberg terminal. A Bloomberg terminal contains news, stock prices, communication tools, and other features that make it worth a high subscription price. And people in finance can afford to pay that high subscription because their decisions can cause a gain or loss of thousands of dollars. Cryptocurrency investors have a similar set of informational problems as traditional financiers. There is a flood of information. Financial quotes are inconsistent across different exchanges. Opinions from Twitter and reddit can be tremendously useful—if they are captured and leveraged correctly. Santiment is a platform that is working to build a Bloomberg terminal for the cryptocurrency investor. Santiment has raised 45k Ether in their ICO last July, which was originally an amount equal to ~$11m. Valentin Mihov is the CTO of Santiment, and he joins the show to explain what Santiment’s product does, and how the token holders will ultimately derive value from Santiment’s ecosystem. If you are looking for an internship, apply to the Software Engineering Daily internship, at softwaredaily.com/jobs. And if you are looking to recruit engineers, you can post jobs for your company there as well–it’s completely free to post jobs and to apply. We are hoping to find interns to contribute to the Software Daily open source project–and if you want to see what we are building, go to SoftwareDaily.com or check out our apps in the iOS or Android app store. They have all 650 of our episodes, with recommendations, related links, discussions and more. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Most applications today run on a cloud provider like AWS. They are built with a framework like Ruby on Rails. They use a set of APIs like Stripe and Twilio for middleware services. This is the era of “web 2.0.” With decentralized systems, we are starting to get a feel for what “web 3.0” might feel like. The futuristic idea of “web 3.0” works off of the following idea: instead of using a centralized service owned by a single company, you might purchase your computation and storage from a network of nodes. The nodes will be running peer-to-peer software that competes on price. Fabian Vogelsteller works on Web3.js, a JavaScript library for interfacing with the Ethereum blockchain. He also works on Mist, a browser for Ethereum. Fabian joins the show to discuss the difference between decentralized apps and centralized apps—and to explain why we need to build a bridge between those two worlds. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Decentralized applications can be built on the Ethereum blockchain. Just as the Bitcoin blockchain is a distributed, append-only ledger of financial transaction history, Ethereum is a distributed, append-only ledger of computational transaction history. New kinds of applications can be built on the Ethereum blockchain—and just like every new technology, we need an interface to bridge that new technology and our existing technology. We can use a pure Ethereum browser like Mist—or we can use a Chrome extension like Metamask to turn our normal browser into an Ethereum interface. Dan Finlay is the lead developer of Metamask. In today’s episode, we explore why you would want to interface with decentralized applications, and the different ways of doing so. A few examples we explore—simple transactions like transferring Ether from one person to another; or transacting with a smart contract. My personal anecdote: I recently used Metamask for the first time to fund a GitCoin issue. GitCoin is a way to put up financial rewards for people solving open source issues. I locked up $42 in an Ethereum smart contract, and it became the bounty of that issue. The issue was solved, and I released the $42 from the smart contract to be sent to the developer who solved it. In this example, Ethereum served as a simple escrow service. To send my Ether, I used the Metamask plugin on my Chrome browser. If you are a little confused—don’t worry. We explain it all in this episode. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Decentralized applications might someday offer alternatives to modern monopolies. Uber, Airbnb, Facebook, Amazon—all of these services could be recreated on a decentralized stack of technologies like Ethereum, IPFS, and Golem. Fully decentralized services could be more transparent, cheaper, and more efficient. But let’s be realistic. Today, even the simplest applications of fully decentralized blockchains don’t work as well as we need them to. Cryptokitties offered a glimpse into how a simple viral application can limit the throughput of Ethereum. And don’t forget that these technologies are in some ways still subject to centralization in their current form. Miners form the decentralized consensus layer—and that mining activity is physically centralized in large server farms. The decentralized future is possible. In order to get there, we need to make progress on the low-level tools that such a world will be built upon. This is the realization that today’s guest Karl Floersh had. Karl is a researcher for the Ethereum Foundation. He was initially excited about the prospect of decentralized apps—such as a decentralized Uber. But as he looked more closely at the space, he realized how early we are, and how much work there is to be done on foundational technologies. Proof of Stake is the central topic of discussion in today’s conversation with Karl. Proof of Stake is a consensus mechanism that is an alternative to Proof of Work. In Proof of Work, miners race to validate blocks of transactions. This results in duplicated effort and perhaps wasted energy. In Proof of Stake, validators are chosen to approve transactions. These validators lock up an amount of currency that they are willing to “stake.” If a validator acts badly, the validator will lose their entire stake. This mechanism could be more efficient—and we will explain why that is in this episode. If Proof of Stake works, it could lead to a faster, truly decentralized Ethereum blockchain. That’s a remarkable potential outcome. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

For a decade, Bitcoin’s proof-of-work system has run without disruption. In a proof-of-work scheme, Bitcoin miners compete to solve a cryptographic puzzle associated with a block of transactions. Every ten minutes, all the Bitcoin miner nodes race to be the first to solve a block of transactions. Only one miner wins each block, meaning the other nodes’ time was ultimately wasted. There is also a massive expense of electricity. Bitcoin is a system with low transaction throughput—about 7 transactions per second. Computer scientists have wondered—is there an alternative way of doing consensus? What if we took all the wasted compute power from proof of work, and allocated it in a way that makes transactions get processed faster? But Bitcoin’s governance tends to be extremely conservative. A change to the consensus mechanism probably won’t happen any time soon in Bitcoin. Ethereum’s consensus mechanism is modeled after that of Bitcoin—proof-of-work mining. But Ethereum’s governance ethos is quite different. Ethereum is in the process of planning and implementing proof of stake, an alternative consensus mechanism in which trusted validators are chosen to validate blocks of transactions. Subhan Nadeem is a student at the University of Waterloo where he studies computer science and business. He is the author of several popular articles on Medium that explain blockchain concepts. He joins the show to talk about crypto from the point of a student—and gives us a great walk through of different consensus mechanisms. To find all of our old episodes about cryptocurrencies, check out our apps in the iOS or Android app store. They have all 700 of our episodes, with recommendations, related links, discussions and more. And it’s all open source–if you are looking for an open source project to contribute to, come check us out at github.com/softwareengineeringdaily. We welcome all kinds of contributors–new developers and experts. Engineers and non-technical people. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Humans organize into groups. There are lots of group types: religions, corporations, national governments, state governments, citizenries, clubs, musical bands. Every group has governance. Governance defines the rules, and the ways that rules change. The United States requires citizens to pay taxes. A corporation requires you to show up to work, but they have to pay you a salary. Most groups today are managed by people. If you break a law, you have to go to court and sit in front of a judge and jury, who decide how you will be punished. If you work at a corporation, and you have a problem with your manager, you go to HR to arbitrate it. These organizations are centralized. There is a governing body who sets the rules. If there is an ambiguity, the person who happens to be in power gets to decide how the ambiguity is resolved. Power is centralized in that governing body. These organizations are run by people. The governance of these organizations is enforced only to the extent that the human government carries out its duties. A decentralized autonomous organization is a group that can run with neither centralized nor human intervention. It is decentralized and autonomous. It is a DAO. Aragon is a platform for running and managing decentralized autonomous organizations. Luis Cuende is the founder of Aragon, and joins the show to explain what a DAO is and why people want to create them. We also talk about the engineering of Aragon and the structure of its ICO—which raised $25m via token sale. Meetups for Software Engineering Daily are being planned! Go to softwareengineeringdaily.com/meetup if you want to register for an upcoming Meetup. In March, I’ll be visiting Datadog in New York and Hubspot in Boston, and in April I’ll be at Telesign in LA. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Smart contracts are programs that run on the Ethereum blockchain. A smart contract developer pays Ether to deploy the contract. When a contract is deployed, every full node on the Ethereum blockchain has a copy of the contract code in that node’s address space. Every full node needs to hold a copy of every smart contract. This allows every full node to process every call to any smart contract. If you want to call a smart contract, that contract will execute on every full node. When you call a smart contract, you are initiating a transaction. Like Bitcoin transactions, these Ethereum transactions get batched into blocks. Ethereum full nodes compete to solve the cryptographic puzzle associated with a block. But instead of mere financial transactions, these are computational transactions. Raine Revere is a smart contract engineer and cofounder at Maiden and she joins the show to describe smart contract creation and deployment. It’s a great introduction to some Ethereum fundamentals. To find all of our old episodes about cryptocurrencies, check out our apps in the iOS or Android app store. They have all 700 of our episodes, with recommendations, related links, discussions and more. And it’s all open source–if you are looking for an open source project to contribute to, come check us out at github.com/softwareengineeringdaily. We welcome all kinds of contributors–new developers and experts. Engineers and non-technical people. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Joseph Bonneau is co-author of Bitcoin and Cryptocurrency Technologies, a popular textbook. At NYU, he works as an assistant professor exploring cryptography and security. His YouTube lessons teaching Bitcoin have hundreds of thousands of views. His material offers clear explanations of how Bitcoin works. Since Joseph has a clear understanding of the objective facts around Bitcoin, he is the perfect person to ask about the more subjective topics: the common misunderstandings of Bitcoin; the governance tradeoffs between Ethereum and Bitcoin; proof of work vs. proof of stake. Joseph believes that the early mainstream cryptocurrency solutions will be largely centralized—and that we are likely to move beyond Bitcoin to more efficient currencies. I enjoyed hearing his reasons behind this perspective. Meetups for Software Engineering Daily are being planned! Go to softwareengineeringdaily.com/meetup if you want to register for an upcoming Meetup. In March, I’ll be visiting Datadog in New York and Hubspot in Boston, and in April I’ll be at Telesign in LA. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Farms have lots of data. A corn farmer needs to monitor the chemical composition of soil. A soybean farmer needs to track crop yield. A chicken farmer needs to count the number of eggs produced. If this data is captured, it can be acted upon—for example, a dry farm can automatically turn up its irrigation system. Or the data can simply be studied. If you work in a pure software business, you might take for granted how easy it is to track your metrics. On the farm, you need to use sensors and drones to gather your data. Mike Prorock is the CTO of Mesur.io, a company that makes sensors and software infrastructure for agriculture. He joins the show to describe the use cases for agriculture technology, and the architecture behind it. Today’s episode is a great complement to our recent episodes on streaming data. Mesur.io offers a case study in how streaming systems can be put into practice. Mike will also be speaking at the upcoming Strata Data Conference in San Jose. Meetups for Software Engineering Daily are being planned! Go to softwareengineeringdaily.com/meetup if you want to register for an upcoming Meetup. In March, I’ll be visiting Datadog in New York and Hubspot in Boston, and in April I’ll be at Telesign in LA. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Dogecoin was started in 2013 as a joke. Jackson Palmer forked Bitcoin and created his cryptocurrency as a play off the “doge” meme. The currency became popular as a means of reddit users “tipping” each other. If I made a comment on reddit that you liked, you might send me some Dogecoin. This use case allowed people to share the idea of Dogecoin virally, and Dogecoin became valuable, even though the currency did not have any technical properties that made it significantly different than Bitcoin. As Dogecoin was becoming popular, an experienced Internet scam artist took notice and started a Dogecoin exchange called Moolah. Moolah was used to steal money from its customers and investors, and the CEO was arrested. Jackson Palmer was not involved in this scheme, but it soured his feelings about Dogecoin and the entire Bitcoin space. His coin, which had been created as a joke, had been repurposed as a weapon to steal money. Jackson left the Dogecoin community in 2015 to focus on other things. But as Bitcoin entered the mainstream conversation, Jackson has been pulled back into the world of cryptocurrency. Jackson’s YouTube channel has over 20,000 subscribers, who tune into learn about consensus protocols, new tokens, and cryptocurrency news. In today’s episode, Jackson and I discuss his experiences with Dogecoin, and how that compares with the scams around low-quality ICOs that are pulling in retail investors today. We also discuss more positive things–such as proof-of-stake and newer consensus protocols. If you are looking for an internship, apply to the Software Engineering Daily internship, at softwaredaily.com/jobs. And if you are looking to recruit engineers, you can post jobs for your company there as well–it’s completely free to post jobs and to apply. We are hoping to find interns to contribute to the Software Daily open source project–and if you want to see what we are building, go to SoftwareDaily.com or check out our apps in the iOS or Android app store. They have all 650 of our episodes, with recommendations, related links, discussions and more. Also–Meetups for Software Engineering Daily are being planned! Go to softwareengineeringdaily.com/meetup if you want to register for an upcoming Meetup. In March, I’ll be visiting Datadog in New York and Hubspot in Boston, and in April I’ll be at Telesign in LA. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

There are two factors that limit the rate at which transactions are accepted into the Bitcoin blockchain: block time and block size. Block time defines how often a new block is appended onto the blockchain. Block size defines how many transactions fit into a new block. As of March 2018, the current block time and block size allow for about 7 transactions per second to be accepted into the Bitcoin blockchain. In today’s episode, we discuss the technical limitations of the Bitcoin blockchain, and some potential solutions to scalability: SegWit and lightning network. Today’s guest is Peter Ullrich, the host of Explain Blockchain. Explain Blockchain is a podcast I have found tremendously useful as I have started to learn about blockchains. He provides thorough, technical explanations of complicated topics, and I recommend subscribing to his show, and listening to the episodes multiple times, because there is a lot of content condensed into a short amount of time. Over the next month, we will be exploring a variety of blockchain-based technologies. Some interviews will be high-level conversations that assume only a familiarity with cryptocurrencies. Some of them will be deeply technical, and assume a strong understanding of Bitcoin and Ethereum. And some episodes, like today’s episode, will be aimed at the developer who is in the process of “going down the rabbit hole.” If you are looking for an internship, apply to the Software Engineering Daily internship, at softwaredaily.com/jobs. And if you are looking to recruit engineers, you can post jobs for your company there as well–it’s completely free to post jobs and to apply. We are hoping to find interns to contribute to the Software Daily open source project–and if you want to see what we are building, go to SoftwareDaily.com or check out our apps in the iOS or Android app store. They have all 650 of our episodes, with recommendations, related links, discussions and more. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Bitcoin is an immutable, append-only blockchain ledger that reaches consensus through proof-of-work. The contents of the ledger are financial transactions–people sending and receiving Bitcoin currency to each other. Since Bitcoin, there have been other cryptocurrencies that have similar properties–like Ethereum and the IPFS/Filecoin system. Similar to Bitcoin, they use a decentralized, proof of work based system with a currency reward system–but the ledger being maintained is not purely financial. A currency is a necessary component to maintaining the blockchain’s validity. Over the next month, we will be exploring a variety of blockchain-based technologies. Some interviews will be high-level conversations that assume only a familiarity with cryptocurrencies. Some of them will be deeply technical, and assume a strong understanding of Bitcoin and Ethereum. And some episodes, like today’s episode, will be aimed at the developer who is in the process of “going down the rabbit hole.” If you are finding yourself reading about Bitcoin and Ethereum a few hours every day, but you are still struggling to grasp the basics, this episode is for you. It is meant to be a complement to other introductory resources, such as “Mastering Bitcoin,” by Andreas Antonopoulos. Cryptocurrency systems are revolutionary–they will unlock completely new applications in the very near future. It’s If you are trying to understand these decentralized currencies and applications, the best place to start is with Bitcoin. That’s why I was happy to have Daniel Van Flymen back on the show. Daniel was previously on for one of our most popular episodes–”Blockchain Building,” in which he talked about how useful it can be to build a blockchain based system for practice. Today, Daniel discusses the basics of Bitcoin transactions. What happens when you send money? How are transactions represented on the blockchain? How do full nodes and light clients interact with each other? These are difficult topics to discuss purely over audio, so this episode is best listened to as a companion resource for someone who is studying cryptocurrencies. If you are looking for an internship, apply to the Software Engineering Daily internship, at softwaredaily.com/jobs. And if you are looking to recruit engineers, you can post jobs for your company there as well–it’s completely free to post jobs and to apply. We are hoping to find interns to contribute to the Software Daily open source project–and if you want to see what we are building, go to SoftwareDaily.com or check out our apps in the iOS or Android app store. They have all 650 of our episodes, with recommendations, related links, discussions and more. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

The easiest way to train a computer to recognize a picture of cat is to show the computer a million labeled images of cats. The easiest way to train a computer to recognize a stop sign is to show the computer a million labeled stop signs. Supervised machine learning systems require labeled data. Today, most of that labeling needs to be done by humans. When a large tech company decides to “build a machine learning model,” that often requires a massive amount of effort to get labeled data. Hundreds of thousands of knowledge workers around the world earn their income from labeling tasks. An example task might be “label all of the pedestrians in this intersection.” You receive a picture of a crowded intersection, and your task is to circle all the pedestrians. You have now created a piece of labeled data. Scale API is a company that turns API requests into human tasks. Their most recent release is an API for labeling data that has been generated from sensors. As self-driving cars emerge onto our streets, the sensors on these cars generate LIDAR, radar, and camera data. The cars will interpret that data in real time using their machine learning models, and then they will send that data to the cloud so that the data can be processed offline to improve the machine learning models of every car on the road. The first step in that processing pipeline is the labeling–which is the focus of today’s conversation. Alexandr Wang is the CEO of Scale, and he joins the show to discuss self-driving cars, labeling, and the company he co-founded. A few notes before we get started. We just launched the Software Daily job board. To check it out, go to softwaredaily.com/jobs. You can post jobs, you can apply for jobs, and it’s all free. If you are looking to hire, or looking for a job, I recommend checking it out. And if you are looking for an internship, you can use the job board to apply for an internship at Software Engineering Daily. Also, Meetups for Software Engineering Daily are being planned! Go to softwareengineeringdaily.com/meetup if you want to register for an upcoming Meetup. In March, I’ll be visiting Datadog in New York and Hubspot in Boston, and in April I’ll be at Telesign in LA. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Apache Spark is a system for processing large data sets in parallel. The core abstraction of Spark is the resilient distributed dataset (RDD), a working set of data that sits in memory for fast, iterative processing. Matei Zaharia created Spark with two goals: to provide a composable, high-level set of APIs for performing distributed processing; and to provide a unified engine for running complete apps. High-level APIs like SparkSQL and MLlib enable developers to build ambitious applications quickly. A developer using SparkSQL can work interactively with a huge dataset, which is a significant improvement on batch Hive jobs running on Hadoop. A developer training a machine learning model can put the model through multiple steps in the training process without checkpointing the data to disk. The second goal of Spark–a “unified engine for running complete apps”–was the focus of my conversation with today’s guest Matei Zaharia. Matei is the CTO of Databricks, a company that was started to implement his vision for Spark and to build highly usable products on top of the technology. Databricks Delta is a project that combines a data warehouse, data lake, and streaming system–all sitting on top of Amazon S3 and using Spark for processing. In our recent episodes about streaming, we explored some common streaming architectures. A large volume of data comes into the system and is stored in Apache Kafka. Backend microservices and distributed streaming frameworks read that data and store it in databases and data lakes. A data warehouse allows for fast access to the large volumes of data–so that machine learning systems and business analysts can work with data sets interactively. The goal of Databricks Delta is to condense the streaming system, the data lake, and the data warehouse into a single system that is easy to use. If you listened to the previous episodes, you will have an idea for the level of complexity that is involved in managing these different systems. For some companies, it makes complete sense to manage a Kafka cluster, a Spark cluster, a set of S3 buckets, and a data warehouse like Amazon Redshift. But we probably don’t want all of that management to the lowest barrier to entry. Delta will hopefully reduce that barrier to entry and make it easier for enterprises to set up large systems for processing data. A few notes before we get started. We just launched the Software Daily job board. To check it out, go to softwaredaily.com/jobs. You can post jobs, you can apply for jobs, and it’s all free. If you are looking to hire, or looking for a job, I recommend checking it out. And if you are looking for an internship, you can use the job board to apply for an internship at Software Engineering Daily. Also, Meetups for Software Engineering Daily are being planned! Go to softwareengineeringdaily.com/meetup if you want to register for an upcoming Meetup. In March, I’ll be visiting Datadog in New York and Hubspot in Boston, and in April I’ll be at Telesign in LA. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Steve Herrod led engineering at VMWare as the company scaled from 30 engineers to 3,000 engineers. After 11 years, he left to become a managing director for General Catalyst, a venture capital firm. Since he has both operating experience and a wide view of the technology landscape as an investor, he is well-equipped to discuss a topic that we have been covering on Software Engineering Daily: the integration of cloud and edge computing. Today, we think of the cloud as a network of large data centers operated by big players like Google, Amazon, and Microsoft. The cloud is where most of the computation across the world takes place. My smartphone and laptop are “edge” devices. They are lightweight computers that don’t perform much complex processing. I would not be able to run a large production database or a 3 terabyte MapReduce job on my laptop. The current division of labor makes sense in this world of smart clouds and low-power, low-bandwidth devices. But the devices are getting cheaper, smarter, and more proliferate. Cars, drones, security cameras, sensors and other devices can serve as points of computation that are geographically between the edge devices and the cloud. With more devices between you and the cloud, there is an opportunity to put computation on those devices. Everyone knows that cloud and edge computing will become intermingled in the coming years. But predicting just how it will play out is nearly impossible. And as an investor, if you bet on something too early, you get the same result as someone who was wrong altogether. A good analogy for the “cloud and edge” space of investments might be the “smart home.” Everyone knows the smart home is coming eventually, but it’s very hard to tell how long it will be before smart home systems are in widespread use–so it is an open question how to invest in the space. Summer internship applications to Software Engineering Daily are also being accepted. If you are interested in working with us on the Software Engineering Daily open source project full-time this Summer, send an application to [email protected]. We’d love to hear from you. If you haven’t seen what we are building, check out softwaredaily.com, or download the Software Engineering Daily app for iOS or Android. These apps have all 650 of our episodes in a searchable format–we have recommendations, categories, related links and discussions around the episodes. It’s all free and also open source–if you are interested in getting involved in our open source community, we have lots of people working on the project and we do our best to be friendly and inviting to new people coming in looking for their first open source project. You can find that project at Github.com/softwareengineeringdaily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

On Software Engineering Daily, we have been covering the “serverless” movement in detail. For people who don’t use serverless functions, it seems like a niche. Serverless functions are stateless, auto-scaling, event driven blobs of code. You might say “serverless sounds kind of cool, but why don’t I just use a server? It’s a paradigm I’m used to.” Serverless is exciting not because of what it adds but because of what it subtracts. The potential of serverless technology is to someday not have to worry about scalability at all. Today, we take for granted that if you start a new company, you are building it on cloud infrastructure. The problem of maintaining server hardware disappeared for 99% of startups, which unlocked a wealth of innovation. The cloud also simplified scalability for most startups–but there are still plenty of companies that struggle to scale. Significant mental energy is spent on the following questions: How many database replicas do I need? How do I configure my load balancer? How many nodes should I put in my Kafka cluster? Serverless functions are important because they are an auto-scaling component that sits at a low level. This makes it easy to build auto scaling systems on top of them. Auto scaling databases, queueing systems, machine learning tools, and user applications. And since the problem is being solved at such a low level, the pricing competitions will also take place at the low level, meaning that systems built on serverless functions will probably see steep declines in costs in the coming years. Serverless compute could eventually become free or nearly free, with the major cloud providers using it as a loss leader to onboard developers to higher level services. All of this makes for an exciting topic of discussion, that we will be repeatedly covering. Today’s show is with Eduardo Laureano, the principal program manager of Azure Functions. It was a fantastic conversation and we covered applications of serverless, improvements to the “cold start problem,” and how the Azure Functions platform is built and operated. Full disclosure: Microsoft is a sponsor of Software Engineering Daily. Meetups for Software Engineering Daily are being planned! Go to softwareengineeringdaily.com/meetup if you want to register for an upcoming Meetup. In March, I’ll be visiting Datadog in New York and Hubspot in Boston, and in April I’ll be at Telesign in LA. Summer internship applications to Software Engineering Daily are also being accepted. If you are interested in working with us on the Software Engineering Daily open source project full-time this Summer, send an application to [email protected]. We’d love to hear from you. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Earlier this year we did several shows about Cloud Foundry, followed by several shows about Kubernetes. Both of these projects allow you to build scalable, multi-node applications–but they serve different types of users. Cloud Foundry encompasses a larger scope of the application experience than Kubernetes. Kubernetes is lower level, and is actually being used within newer versions of Cloud Foundry to give Cloud Foundry users access to the Kubernetes abstractions. Recording those shows gave me a wide understanding of how infrastructure is managed and how it has evolved. Today’s episode provides more context on Cloud Foundry–how the project got started, how people use it, and where Cloud Foundry is going. Today’s guest Mike Dalessio is a VP of engineering on Pivotal Cloud Foundry, and we had a great time talking about his work. Engineering leadership is a fine art, and conversations with engineering leaders are consistently interesting–this was no exception. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Kafka is at the center of modern streaming systems. Kafka serves as a database, a pubsub system, a buffer, and a data recovery tool. It’s an extremely flexible tool, and that flexibility has led to its use as a platform for a wide variety of data intensive applications. Today’s guest is Gwen Shapira, a product manager at Confluent. Confluent is a company that was started by the creators of Apache Kafka–Jay Kreps, Neha Narkhede, and Jun Rao, who have all been on the show in previous episodes. In those shows, we discussed the inner workings of Kafka. This episode is more about practical use cases and design patterns. Gwen explores a few use cases. First, reconciling data between different servers. A massive, international, multi-user game like World of Warcraft needs to keep its users in sync despite the fact that those users are pinging different server locations. Kafka can help reconcile data between the multiple servers. This discussion reminded me of the awesome show we did with Yan Cui about scalable multiplayer games. Other examples we discussed include log management, data enrichment, and large scale analytics. It was a great conversation and I think you will enjoy it as well. Summer internship applications to Software Engineering Daily are also being accepted. If you are interested in working with us on the Software Engineering Daily open source project full-time this Summer, send an application to [email protected]. We’d love to hear from you. If you haven’t seen what we are building, check out softwaredaily.com, or download the Software Engineering Daily app for iOS or Android. These apps have all 650 of our episodes in a searchable format–we have recommendations, categories, related links and discussions around the episodes. It’s all free and also open source–if you are interested in getting involved in our open source community, we have lots of people working on the project and we do our best to be friendly and inviting to new people coming in looking for their first open source project. You can find that project at Github.com/softwareengineeringdaily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Streaming architecture defines how large volumes of data make their way through an organization. Data is created at a user’s smartphone, or on a sensor inside of a conveyor belt at a factory. That data is sent to a set of backend services that aggregate the data, organizing it and making it available to business analysts, application developers, and machine learning algorithms. The velocity at which data is created has led to widespread use of the “stream” abstraction–a never ending, append-only array of data. To deal with this volume, streams need to be buffered, batched, cached, mapreduced, machine learned, and munged until they are in a state where they can provide value to the end user. There are numerous ways that data can travel this path, and in today’s episode we discuss the streaming systems, data lakes, and data warehouses that can be used to build an architecture that makes use of streaming data. Ted Dunning is a chief application architect at MapR, and he joins the show to discuss the patterns that engineering teams are using to build modern streaming architectures. Full disclosure: MapR is a sponsor of Software Engineering Daily. Meetups for Software Engineering Daily are being planned! Go to softwareengineeringdaily.com/meetup if you want to register for an upcoming Meetup. In March, I’ll be visiting Datadog in New York and Hubspot in Boston, and in April I’ll be at Telesign in LA. Summer internship applications to Software Engineering Daily are also being accepted. If you are interested in working with us on the Software Engineering Daily open source project full-time this Summer, send an application to [email protected]. We’d love to hear from you. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

When you go to a website where a video is playing, and your video lags, how does the website know that you are having a bad experience? Problems with video are often not complete failures–maybe part of the video loads, and plays just fine, and then the rest of the video is buffering. You have probably experienced sitting in front of a video, waiting for it to load as the loading wheel mysteriously spins. Since problems with video are often not complete failures, troubleshooting a problem with a user’s video playback is not as straightforward as just logging whenever a crash occurs. You need to continuously monitor the video playback on every client device and aggregate it in a centralized system for analysis. The centralized logging system will allow you to separate problems with a specific user from problems with the video service itself. A single user could have bad wifi, or have 50 tabs open with different videos. To identify problems that are caused by the video player rather than the user, you need to capture the playback from every video and every user. Scott Kidder works at Mux, where he builds a streaming analytics system for video monitoring. In this episode, Scott explains how events make it from a video player onto the backend analytics system running on Kinesis and Apache Flink. Events from the browser are constantly added to Kinesis (which is much like Kafka). Apache Flink reads those events off of Kinesis and map reduces them to discover anomalies. For example, if 100 users watch a 20 minute cat video, and the video stops playing at minute 12 for all 100 users, there is probably some data corruption in that video. You would only be able to discover that by assessing all users. Scott and I discussed the streaming infrastructure that he works on at Mux, as well as other streaming systems like Spark, Apache Beam, and Kafka. This episode is the first in a short series about streaming data infrastructure. I wanted to do some shows in preparation for Strata Data conference in March in San Jose, which I will be attending thanks to a complimentary ticket from O’Reilly. O’Reilly has been kind enough to give me free tickets since Software Engineering Daily started and did not have the money to attend any conferences. If you want to attend Strata, you can use promo code PCSED to get 20% off. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Tugdual Grall is an engineer with MapR. In today’s episode, we explore use cases and architectural patterns for streaming analytics. Full disclosure: MapR is a sponsor of Software Engineering Daily. In past shows, we have covered data engineering in detail–we’ve looked at Uber’s streaming architecture, talked to Matei Zaharia about the basics of Apache Spark, and explored the history of Hadoop. To find all of our episodes about data engineering , download the Software Engineering Daily app for iOS or Android. These apps have all 650 of our episodes in a searchable format–we have recommendations, categories, related links and discussions around the episodes. It’s all free and also open source–if you are interested in getting involved in our open source community, we have lots of people working on the project and we do our best to be friendly and inviting to new people coming in looking for their first open source project. You can find that project at Github.com/softwareengineeringdaily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Pinterest is a visual feed of ideas, products, clothing, and recipes. Millions of users browse Pinterest to find images and text that are tailored to their interests. Like most companies, Pinterest started with a large monolithic application that served all requests. As Pinterest’s engineering resources expanded, some of the architecture was broken up into microservices and Dockerized, which make the system easier to reason about. To serve users with better feeds, Pinterest built a machine learning pipeline using Kafka, Spark, and Presto. User events are generated from the frontend, logged onto Kafka, and aggregated to build machine learning models. These models are deployed into Docker containers much like the production microservices. Kinnary Jangla is a senior software engineer at Pinterest, and she joins the show to talk about her experiences at the company–breaking up the monolith, architecting a machine learning pipeline, and deploying those models into production. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Over 12 years of engineering, Box has developed a complex architecture of services. Whenever a user uploads a file to Box, that upload might cause 5 or 6 different services to react to the event. Each of these services is managed by a set of servers, and managing all of these different servers is a challenge. Sam Ghods is the cofounder and services architect of Box. In 2014, Sam was surveying the landscape of different resource managers, deciding which tool should be the underlying scheduler for deploying services at Box. He chose Kubernetes because it was based on Google’s internal Borg scheduling system. For years, engineering teams at companies like Facebook and Twitter had built internal scheduling systems modeled after Borg. When Kubernetes arrived, it provided an out-of-the-box tool for managing infrastructure like Google would. In today’s episode, Sam describes how Box began its migration to Kubernetes, and what the company has learned along the way. It’s a great case study for people who are looking at migrating their own systems to Kubernetes. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

When Box started in 2006, the small engineering team had a lot to learn. Box was one of the earliest cloud storage companies, with a product that allowed companies to securely upload files to remote storage. This was two years before Amazon Web Services introduced on-demand infrastructure, so the Box team managed their own servers, which they learned how to do as they went along. In the early days, the backup strategy was not so sophisticated. The founders did not know how to properly set up hardware in a colocated data center. The frontend interface was not the most beautiful product. But the product was so useful that eventually it started to catch on. Box’s distributed file system became the backbone of many enterprises. Employees began to use it to interact with and share data across organizations. The increase in usage raised the stakes for Box’s small engineering team. If Box’s service went down, it could cripple an enterprise’s productivity, which meant that Box needed to hire experienced engineers to build resilient systems with higher availability. And to accommodate the growth in usage, Box needed to predict how much hardware to purchase, and how much space in a data center to rent–a process known as capacity planning. As Box went from 3 engineers to 300, the different areas of the company went from being managed by individuals, to teams, to entire departments with VPs and C-level executives. Jeff Quiesser is an SVP at Box, and one of the earliest employees. He joins the show today to describe how Box changed as the company scaled. We covered engineering, management, operations, and culture. In previous shows, we have explored the stories of companies like Slack, Digital Ocean, Giphy, Uber, Tinder, and Spotify. It’s always fun to hear how a company works–from engineering the first product to enterprises with millions of users. To find all of our episodes about how companies are built, download the Software Engineering Daily app for iOS or Android. These apps have all 650 of our episodes in a searchable format–we have recommendations, categories, related links and discussions around the episodes. It’s all free and also open source–if you are interested in getting involved in our open source community, we have lots of people working on the project and we do our best to be friendly and inviting to new people coming in looking for their first open source project. You can find that project at Github.com/softwareengineeringdaily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Employees often find themselves needing to do work outside of the office. Depending on the sensitivity of your task, accessing internal systems from a remote location may or may not be OK. If you are using a corporate application that shows the menu of your company’s cafe on your smartphone, your workload is less sensitive. If you are accessing the proprietary codebase of your company’s search engine, your workload is more sensitive. As Google grew in headcount, the different cases of employees logging in from different places grew as well. Google developed a fine-grained, adaptive security model called BeyondCorp to allow for a wide variety of use cases. Whether you are an engineer logging in from a Starbucks or a human resources employee logging in from your desk, the BeyondCorp system uses the same access proxy to determine your permissions. The BeyondCorp architecture is also built around the assumption of a zero-trust network. A zero-trust network is a modern enterprise security architecture where internal servers do not trust each other. Zero-trust networks assume that the network has already been breached. If you are writing an internal application, your default assumption should be to distrust an incoming request from someone else on the network. The zero-trust model is in contrast to an outdated model of enterprise security–that of the hard outer defense of a firewall, that purports to prevent attackers from ever making their way into the vulnerable inside of a network. The firewall model assumes that all of these servers within the firewall can trust each other. Several papers have come out of Google discussing the BeyondCorp security model. These papers describe the network architecture, and the security philosophies of BeyondCorp. Since the release of these papers, an ecosystem of security providers has sprung up to provide implementation services for companies that want BeyondCorp security in their enterprise. Google has also productized its BeyondCorp system with an identity-aware proxy that is tied into their Google Cloud product. Max Saltonstall is the technical director of information technology in the office of the CTO at Google, where he has helped to facilitate the widespread adoption of the BeyondCorp program. In this episode, we talk about enterprise security–from remote employee access to zero-trust networks. We also talk about implementing the BeyondCorp model–why enterprises should consider it, and how to do it. We have done lots of past shows about security–from car hacking to smart contract vulnerabilities to discussions with luminaries like Bruce Schneier and Peter Warren Singer. To find all of our episodes about security, download the Software Engineering Daily app for iOS or Android. These apps have all 650 of our episodes in a searchable format–we have recommendations, categories, related links and discussions around the episodes. It’s all free and also open source–if you are interested in getting involved in our open source community, we have lots of people working on the project and we do our best to be friendly and inviting to new people coming in looking for their first open source project. You can find that project at Github.com/softwareengineeringdaily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Applications need to be ready to scale in response to high-load events. With mobile applications, this can be even more important. People rely on mobile applications such as banking, ride sharing, and GPS. During Black Friday, a popular ecommerce application could be bombarded by user requests–you might not be able to complete a request to buy an item at the Black Friday discount. If you attend the Superbowl, and then try to catch an Uber after leaving, all the other people around you might be summoning a car at the same time, and the system might not scale. In order to prepare infrastructure for high volume, mobile development teams often create end-to-end load tests. After recording incoming mobile traffic, that mobile traffic can be replicated and replayed, to measure a backend’s response to the mobile workload. Paulo Costa and Rodrigo Coutinho are engineers at OutSystems, a company that makes a platform for building low-code mobile applications. In this episode, Paulo and Rodrigo discuss the process of performing end-to-end scalability testing for mobile applications backed by cloud infrastructure. We talked about the high level process of architecting the load test, and explored the tools used to implement it. Full disclosure: OutSystems is a sponsor of Software Engineering Daily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Matt Leising is a journalist at Bloomberg who has covered financial markets for 15 years. Today, his reporting has been completely engulfed by cryptocurrencies. There are so many dramatic stories, it’s hard to pick what to focus on. Today, we discuss two topics he has covered recently: Ripple and Tether. Ripple is a company that makes enterprise blockchain solutions for global payments. That sounds like the future, and it is no surprise that people would want to buy into Ripple if possible. Ripple has been around for 7 years, and they have a strong team, and relationships with major financial institutions. One of Ripple’s early projects was a currency called XRP. The goal of XRP was to make a fast, scalable digital asset that would facilitate currency exchange among banks. We covered Ripple and XRP in previous episodes with David Schwartz and Greg Kidd. XRP remains in circulation, but Ripple the company has shifted development resources away from XRP, and towards RippleNet, which seeks to replace the aging SWIFT code system for banks. Today, XRP is being experimented with by several money transfer companies, but the digital currency is not widely used for anything–well, other than speculation. In the tremendous cryptocoin bull run of early 2018, XRP shot up as sharply as almost any other coin. In an article about Ripple, Matt Leising tried to get to the root explanation for why this occurred. Was it a sudden market recognition of some long term value of XRP? Was it a stampeding herd of people who did not know the state of XRP? Was it a pump and dump? A few days after publishing his article about Ripple, Matt wrote about Tether. Tether purports to be a “stablecoin”–a digital currency which is pegged to the value of something less volatile. Stablecoins are useful in that they can reduce friction of exchange between tokens. Without a stablecoin, you might have to transfer from one cryptocurrency to USD, which probably involves the US banking system. There’s a good discussion of stablecoins in our episode with Vlad Zamfir and Haseeb Qureshi on cryptoeconomics. This was Matt’s second appearance on the show, and it was a blast to have him back on. In his last episode, he discussed the infamous DAO hack, which led to an Ethereum fork. To find that episode as well as links to learn more about the topics described in the show, download the Software Engineering Daily app for iOS or Android. These apps have all 650 of our episodes in a searchable format–we have recommendations, categories, related links and discussions around the episodes. It’s all free and also open source–if you are interested in getting involved in our open source community, we have lots of people working on the project and we do our best to be friendly and inviting to new people coming in looking for their first open source project. You can find that project at Github.com/softwareengineeringdaily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Over the last decade, computation and storage has moved from on-premise hardware into the cloud data center. Instead of having large servers “on premise,” companies started to outsource their server workloads to cloud service providers. At the same time, there has been a proliferation of devices at the “edge.” The most common edge device is your smartphone, but there are many other smart devices that are growing in number–drones, smart cars, Nest thermostats, smart refrigerators, IoT sensors, and next generation centrifuges. Each of these devices contains computational hardware. Another class of edge device is the edge server. Edge servers are used to facilitate faster response times than your core application. For example, Software Engineering Daily uses a content delivery network for audio files. These audio files are distributed throughout the world on edge servers. The core application logic of Software Engineering Daily runs on a WordPress site, and that WordPress application is distributed to far fewer servers than our audio files. “Cloud computing” and “edge computing” both refer to computers that can serve requests. The “edge” is commonly used to refer to devices that are closer to the user–so they will deliver faster responses. The “cloud” refers to big, bulky servers that can do heavy duty processing workloads–such as training machine learning models, or issuing a large distributed MapReduce query. As the volume of computation and data increases, we look for better ways to utilize our resources, and we are realizing that the devices at the edge are underutilized. In today’s episode, Kenton Varda explains how and why to deploy application logic to the edge. He works at Cloudflare on a project called Cloudflare Workers, which are a way to deploy JavaScript to edge servers, such as the hundreds of data centers around the world that are used by Cloudflare for caching. Kenton was previously on the show to discuss protocol buffers, a project he led while he was at Google. To find that episode, and many other episodes about serverless, download the Software Engineering Daily app for iOS or Android. These apps have all 650 of our episodes in a searchable format–we have recommendations, categories, related links and discussions around the episodes. It’s all free and also open source–if you are interested in getting involved in our open source community, we have lots of people working on the project and we do our best to be friendly and inviting to new people coming in looking for their first open source project. You can find that project at Github.com/softwareengineeringdaily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

How do you build resilient, failure tested systems? Redundancy, backups, and testing are all important. But there is also an increasing trend towards chaos engineering–the technique of inducing controlled failures in order to prove that a system is fault tolerant in the way that you expect. In last week’s episode with Kolton Andrus, we discussed one way to build chaos engineering as a routine part of testing a distributed system. Kolton discussed his company Gremlin, which injects failures by spinning up a Gremlin container and having that container induce network failures, memory errors, and filled up disks. In this episode, we explore another insertion point for testing controlled failures, this time from the point of view of Linkedin. Linkedin is a social network for working professionals. As Linkedin has grown, the increased number of services has led to more interdependency between those services. The more dependencies a given service has, the more partial failure cases there are. That’s not to say there is anything wrong with having a lot of service dependencies–this is just the way we build modern applications. But it does suggest that we should try to test the failures that can emerge from so many dependencies. Bhaskaran Devaraj and Xiao Li are engineers at Linkedin, and are working on a project called Waterbear, with the goal of making the infrastructure more resilient. Linkedin’s backend system consists of a large distributed application with thousands of microservices communicating between each other. Most of those services communicate over Rest.li, a proxy for standardizing interactions between services. Rest.li can assist with routing, AB testing, circuit breaking, and other aspects of service-to-service communication. This proxy can also be used for executing controlled failures. As services are communicating with each other, creating a controlled failure can be as simple as telling your proxy not to send traffic to downstream services. If that sounds confusing, don’t worry, we will explain it in more detail. In this episode, Bhaskaran and Xiao describe their approach to resilience engineering at Linkedin–including the engineering projects and the cultural changes that are required to build a resilient software architecture. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

The number of ways that applications can fail are numerous. Disks fail all the time. Servers overheat. Network connections get flaky. You assume that you are prepared for such a scenario, because you have replicated your servers. You have the database backed up. Your core application is spread across multiple availability zones. But are you really sure that your system is resilient? The only way to prove that your system is resilient to failure is to experience failure, and to make swift responsiveness to failure an integral part of your software. Chaos engineering is the practice of routinely testing your system’s resilience by inducing controlled failures. Netflix was the first company to discuss chaos engineering widely, but more and more companies are starting to work it into their systems, and finding it tremendously useful. By inducing failures in your system, you can discover unknown dependencies, single points of failure, and problematic state conditions that can cause data corruption. Kolton Andrus worked on chaos engineering at Netflix and Amazon, where he designed systems that would test system resiliency through routine failures. Since then, he founded Gremlin, a company that provides chaos engineering as a service. In a previous episode, Kolton and I discussed why chaos engineering is useful, and he told some awesome war stories about working at Amazon and Netflix. In this show, we explore how to build a chaos engineering service–which involves standing up Gremlin containers that institute controlled failures. To find the previous episode I recorded with Kolton, as well as other supplementary materials described in this show, download the Software Engineering Daily app for iOS or Android. These apps have all 650 of our episodes in a searchable format–we have recommendations, categories, related links and discussions around the episodes. It’s all free and also open source–if you are interested in getting involved in our open source community, we have lots of people working on the project and we do our best to be friendly and inviting to new people coming in looking for their first open source project. You can find that project at Github.com/softwareengineeringdaily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

On this show, we spend a lot of time talking about CI/CD, data engineering, and microservices. These technologies have only been widely talked about for the last 5-10 years. That means that they are easy to adopt for startups that get founded in the last 5-10 years, but not necessarily for older enterprises. Within a large enterprise, it can be challenging to make significant changes to how technology is used. Many of the listeners might even take it for granted that your source code is in git–but if you work at an enterprise that started building software in 1981, you might be moving source code around on FTP servers or floppy disks. The difficulty of changing the technology within an enterprise gets compounded by culture. Culture develops around specific technologies. That is one interpretation of “Conway’s Law”–that the way an organization uses software informs an organization’s communication structure. This is no surprise–if your organization manages code using FTP servers and floppy disks, it will slow down your innovation. Zhamak Dehghani is an engineer at ThoughtWorks, where she consults with enterprises to modernize their software and culture. She works off of a blueprint that describes specific steps that an enterprise can take towards modernizing: continuous integration; building a data pipeline; building a system of experimentation. In some ways, this conversation fits nicely with our shows about DevOps a few years ago. Full disclosure: ThoughtWorks is a sponsor of Software Engineering Daily. To find all of our shows about DevOps, as well as links to learn more about the topics described in the show, download the Software Engineering Daily app for iOS or Android. These apps have all 650 of our episodes in a searchable format–we have recommendations, categories, related links and discussions around the episodes. It’s all free and also open source–if you are interested in getting involved in our open source community, we have lots of people working on the project and we do our best to be friendly and inviting to new people coming in looking for their first open source project. You can find that project at Github.com/softwareengineeringdaily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Sue Loh is a software engineer and author of a book with the goal of breaking developer stereotypes. Stereotyping among developers leads to bad outcomes. When incorrect assumptions are made about certain populations, those populations feel marginalized and engineering resources get misallocated. From the perspective of Sue, the main problem is about how children are socialized. Young girls in particular are discouraged from programming, leading to a steady decline in the percentage of women in the computing workforce. With her book, Sue is hoping to create a piece of literature that will expose young female and minority students to the world of technology, and help them realize how cool engineering can be. Sue created a Kickstarter around her book, and has raised more than $40,000. Sue also describes her experiences as an engineer who has risen through the ranks at Microsoft, and how her perspective on engineering has evolved as she has become a principal engineer. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Functional programming can improve the overall design of an application architecture. Runar Bjarnason has been exploring how writing in a functional style increases modularity and compositionality of software for many years. He is co-author of Functional Programming in Scala, a book that explores the relationship between functional programming and software design. In this interview with guest host Adam Bell, Runar explains how writing in a functional style involves limiting side effects, avoiding exceptions and using higher order abstractions. Writing in this style places constraints on what a module in a software system may do, but by constraining modules in this way, the software modules themselves become endlessly composable. Functional Programming In Scala Constraints Liberate, Liberties Constrain Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Training a deep learning model involves operations over tensors. A tensor is a multi-dimensional array of numbers. For several years, GPUs were used for these linear algebra calculations. That’s because graphics chips are built to efficiently process matrix operations. Tensor processing consists of linear algebra operations that are similar in some ways to graphics processing–but not identical. Deep learning workloads do not run as efficiently on these conventional GPUs as they would on specialized chips, built specifically for deep learning. In order to train deep learning models faster, new hardware needs to be designed with tensor processing in mind. Xin Wang is a data scientist with the artificial intelligence products group at Intel. He joins today’s show to discuss deep learning hardware and Flexpoint, a way to improve the efficiency of space that tensors take up on a chip. Xin presented his work at NIPS, the Neural Information Processing Systems conference, and we talked about what he saw at NIPs that excited him. Full disclosure: Intel, where Xin works, is a sponsor of Software Engineering Daily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

At KubeCon, I had the chance to catch up with Adam Stacoviak of the Changelog, a podcast that was an inspiration for starting Software Engineering Daily. Changelog has long been one of my favorite podcasts about engineering, thanks in part to Adam’s personality. This was a spontaneous conversation, but it was a good one.

A modern farm has hundreds of sensors to monitor the soil health, and robotic machinery to reap the vegetables. A modern shipping yard has hundreds of computers working together to orchestrate and analyze the freight that is coming in from overseas. A modern factory has temperature gauges and smart security cameras to ensure workplace safety. All of these devices could be considered “edge” devices. Over the last decade, these edge devices have mostly been used to gather data and save it to an on-premise server, or to the cloud. Today, as the required volumes of data and compute scale, we look for ways to better utilize our resources. We can start to deploy more application logic to these edge devices, and build a more sophisticated relationship between our powerful cloud servers and the less powerful edge devices. The soil sensors at the farm are recording long time series of chemical levels. The pressure sensors in a centrifuge are recording months and years of data. The cameras are recording terabytes of video. These huge data sets are correlated with labeled events–such as crop yields. With these large volumes of data, we can construct models for responding to future events. Deep learning can be used to improve systems over time. The models can be trained in the cloud, and deployed to devices at the edge. Aran Khanna is an AI engineer with Amazon Web Services, and he joins the show to discuss workloads at the cloud and at the edge–how work can be distributed between the two places, and the tools that can be used to build edge deep learning systems more easily. To find all of our shows about machine learning and edge computing, as well as links to learn more about the topics described in the show, download the Software Engineering Daily app for iOS or Android. These apps have all 650 of our episodes in a searchable format–we have recommendations, categories, related links and discussions around the episodes. It’s all free and also open source–if you are interested in getting involved in our open source community, we have lots of people working on the project and we do our best to be friendly and inviting to new people coming in looking for their first open source project. You can find that project at Github.com/softwareengineeringdaily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

After two weeks of episodes about Kubernetes, our in-depth coverage of container orchestration is drawing to a close. We have a few more shows on the topic before we move on to covering other aspects of software. If you have feedback on this thematic format (whether you like it or not), send me an email: [email protected] Today’s episode fits nicely into some of the themes we have covered recently–Cloud Foundry, Kubernetes, and the changing landscape of managed services. Sean McKenna works on all three of these things at Microsoft. We spent much of our time discussing the use cases of container instances versus Kubernetes. Container instances are individual managed containers–so you could spin up an application within a container instance without having to deal with the Kubernetes control plane. Container instances might be described as “serverless containers,” since you do not have to program against the underlying VM at all. This begs the question–why would you want to use a managed Kubernetes service if you could just use individual managed containers? Sean explores this question, and gives his thoughts on where this ecosystem is headed. Full disclosure: Microsoft is a sponsor of Software Engineering Daily. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.

Last month, Software Engineering Daily had our 4th Meetup at Cloudflare in San Francisco. For this Meetup, the format was short interviews with security specialists from Pinterest, Cloudflare, and Segment. Each of these companies has unique security challenges, but they also have overlap in their security strategies. Nick Sullivan, Amine Kamel, and Evan Johnson are all seasoned engineers, and it was a privilege to sit down with each of them. Some topics we discussed: cryptography, secret management, incident response, and social network security. In 2018, I am hoping to travel to several tech hubs and do Meetups. I wanted to do more of these last year, but did not plan effectively. So this year I’d like to plan them far in advance. Some locations I have in mind are New York, Los Angeles, Austin, and Seattle. If you have suggestions, or if you know of a venue that could comfortably host us, send me an email–[email protected] Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.