Software Engineering Daily

Software Engineering Daily examines the world through the lens of software engineering. In most episodes, an expert in a particular topic joins the show as a guest, and we go into deep technical detail. Occasionally we like to do episodes where we survey a collection of topics. In today’s topic roundtable, Caleb Meredith and Courtland Allen join me for a discussion of several questions: would it make sense for Facebook to build an operating system? Does online advertising work? How can you work productively on an engineering company with your brother as a co-founder? Courtland is the founder of IndieHackers.com, which was recently acquired by Stripe. Caleb Meredith is the lead JavaScript correspondent of Software Engineering Daily. It was a blast talking to both of them, and we plan to do more round table episodes in the future. We would love to get your feedback on Software Engineering Daily. Please fill out the listener survey, available on softwareengineeringdaily.com/survey. Also–Software Engineering Daily is having our third Meetup, Wednesday May 3rd at Galvanize in San Francisco. The theme of this Meetup is Fraud and Risk in Software. We will have great food, engaging speakers, and a friendly, intellectual atmosphere. To find out more, go to softwareengineeringdaily.com/meetup. Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com/sed to get 20% off the first two months of audio editing and transcription services. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to download the Topic Roundtable transcript.

Bitcoin is underappreciated even to this day. The public focus is usually on the speculative value, but Bitcoin has functional value as a technology platform. If I want to make 100 transactions with my bank for 1 cent, the bank won’t allow it. Our current financial infrastructure is not set up for micropayments. Bitcoin is built with micropayments in mind. As Bitcoin works through its governance issues and its scalability problems, we will see gradual improvement in financial liquidity between people and machines. 21 is a company that has raised $120M to make Bitcoin useful to developers. This is a long term project, and the first step of that project is to get Bitcoin in the hands of users. To fulfill that end, 21 is developing services that encourage people to make small digital transactions. The first service is the 21 messaging service, where users can pay to send messages to people who are unlikely to respond to an unsolicited email otherwise. For example, if I want to send an email to a venture capitalist pitching my company, I am more likely to get a response if I pay that venture capitalist $20 to read my message through 21, rather than if I sent a cold email from my email address. Balaji Srinivasan is the CEO of 21, and he joins me for a conversation about the potential of Bitcoin and the objectives of the company he is building. Software Engineering Daily is having our third Meetup, Wednesday May 3rd at Galvanize in San Francisco. The theme of this Meetup is Fraud and Risk in Software. We will have great food, engaging speakers, and a friendly, intellectual atmosphere. To find out more, go to softwareengineeringdaily.com/meetup.

The word “microservices” started getting used after a series of events–companies were moving to cloud virtual machines. Those VMs got broken up into containers, and the containers can fit to the size of the service. Services that are more narrowly defined take up smaller containers, and can be packed more densely into the virtual machines–hence the term “microservices.” As this change to software architecture has occurred, the DevOps movement has encouraged organizations to have better relationships between development and operations. Continuous deployment leads to fewer painful outages. Improved monitoring tools make it easier for developers to take on some of the pain that was previously centralized in operations. Several months ago, I attended the Microservices Practitioner Summit, which brings together engineers who are working with microservices at their companies. The conference was organized by Austin Gunter and Richard Li of Datawire. In this episode, they joined me for a conversation about microservices. Software Engineering Daily is having our third Meetup, Wednesday May 3rd at Galvanize in San Francisco. The theme of this Meetup is Fraud and Risk in Software. We will have great food, engaging speakers, and a friendly, intellectual atmosphere. To find out more, go to softwareengineeringdaily.com/meetup.

Swift is a language that is most commonly used to write apps for Apple client devices, such as iPhones. Since being released in 2014, Swift has become one of the most popular languages due to its high performance and developer ergonomics. In 2015, Swift was open sourced, creating the opportunity for Swift to be used outside of the Apple ecosystem. If you write an iPhone app today, your frontend is in Swift and your backend is probably in NodeJS, Java, or Ruby. Engineers are working to port Swift to the server so that the Swift developer experience is isomorphic: the same language on the backend and the frontend. Chris Bailey is an engineer at IBM working on Kitura, a Swift web framework. In this episode, we discuss the history of Swift, why it is so appealing to developers, and why Swift could become a server side language with as much popularity as Java. Software Engineering Daily is having our third Meetup, Wednesday May 3rd at Galvanize in San Francisco. The theme of this Meetup is Fraud and Risk in Software. We will have great food, engaging speakers, and a friendly, intellectual atmosphere. To find out more, go to softwareengineeringdaily.com/meetup.

Most people in Africa never had a desktop computer. The first computer they owned was a smart phone. This is why Africa is referred to as a “leap frog” place with regard to computers–Africa leapfrogged the desktop to the smart phone. The banking system in Africa also followed a trajectory that is different than the West. Westerners are used to banking on their desktop computers. African e-commerce has developed around the smartphone as the computer for banking. As a result, Africa is in a technological development phase with tremendous opportunities in the financial sector and beyond. Mymookh.com is a Kenyan end-to-end payment platform that allows small- to medium-sized businesses to easily set up online stores and sell to consumers directly from their social media pages. It is one of the many startups across Africa leveraging the accessibility of mobile payments to meet the particular needs of merchants and consumers there. Africa’s development and use of mobile payment systems is expected to grow as more countries on the continent create and adopt this technology. In this episode, George Gachui, co-founder of Mymookh, talks to Carl Mungazi about how Kenya became a leading player in the mobile payment space with its m-pesa system. He explains how it has revolutionized the way Kenyans approach commerce and payment technology and discusses the challenges of creating a unified platform system that transcends borders and currencies. Software Engineering Daily is having our third Meetup, Wednesday May 3rd at Galvanize in San Francisco. The theme of this Meetup is Fraud and Risk in Software. We will have great food, engaging speakers, and a friendly, intellectual atmosphere. To find out more, go to softwareengineeringdaily.com/meetup.

Bots on the internet can be malicious, helpful, and everything in between. A bot that responds to all of your tweets might call you a socialist–that is malicious. Google crawls the web to index Google search. That is helpful. Social media marketing bots schedule 200 Twitter posts to go out throughout the day. That is either a little annoying or a little helpful depending on who you are. Bots are being used to amplify political viewpoints. An amplified viewpoint can serve as a gravity well for like-minded individuals, and help a sparsely supported political cause find its footing. Sometimes that amplified viewpoint is completely fictional or unfalsifiable. Real people believe that Hillary Clinton is a lizard alien because they have seen that story shared by enough Twitter bots. So-called “fake news” is a topic that has been discussed on so many other podcasts. What is not reported is the connection between link bait and advertising fraud. When a botnet is able to make an article go viral, thousands of people organically click on the link to that article. That organic traffic is used to launder fake clicks. These bots that are spreading “fake news” might be controlled by conspiratorial Russians. But it doesn’t have to be that complicated. Anyone who wants to make money in online advertising fraud is incentivized to make salacious media–whether it is real or fake. Samuel Woolley is the director of research at Political Bots. He works with Jigsaw, a division of Alphabet that seeks to make the Internet safer. In today’s episode, we talk about political bots, advertising fraud, and the connection between the two. Software Engineering Daily is having our third Meetup, Wednesday May 3rd at Galvanize in San Francisco. The theme of this Meetup is Fraud and Risk in Software. We will have great food, engaging speakers, and a friendly, intellectual atmosphere. To find out more, go to softwareengineeringdaily.com/meetup.

Facebook’s open source projects include React, GraphQL, and Cassandra. These projects are key pieces of infrastructure used by thousands of developers–including engineers at Facebook itself. These projects are able to gain traction because Facebook takes time to decouple the projects from their internal infrastructure and clean up the code before releasing them into the wild. Facebook has high standards for what they are willing to release. Tom Occhino manages the React team at Facebook and works closely with engineers to determine what projects make sense to open source. In this episode, Preethi Kasireddy interviews Tom about how Facebook thinks about open source–what went right with React, why it makes sense for Facebook to continue to release new open source projects, and how full-time employees at Facebook interact with that open source codebase. We would love to get your feedback on Software Engineering Daily. Please fill out the listener survey, available on softwareengineeringdaily.com/survey. Also–Software Engineering Daily is having our third Meetup, Wednesday May 3rd at Galvanize in San Francisco. The theme of this Meetup is Fraud and Risk in Software. We will have great food, engaging speakers, and a friendly, intellectual atmosphere. To find out more, go to softwareengineeringdaily.com/meetup. Pete Hunt: React: Rethinking best practices — JSConf EU 2013

The Internet is decreasing in privacy and increasing in utility. Under some conditions, this tradeoff makes sense. We publicize our profile photo so that people know what we look like. Under other conditions, this tradeoff does not make sense. We do not want a television that costs less to purchase because it is silently recording all of the conversations that take place in the room and selling them to the highest bidder. The example of the TV that records everything you say (which is a real thing) illustrates a tradeoff of the Internet. The advertising industry pushes us towards lower marginal costs for products and services in exchange for less privacy. Someday we will live in a world where it will be easy for consumers to control the dial on the tradeoff between privacy and the price of their services. Until then, we have almost zero control over what information the advertising surveillance industrial complex knows about us. Bill Budington is a security engineer with the Electronic Frontier Foundation. In today’s episode, Bill describes some of the current techniques used by the advertising industry to track your activity through the web. Bill works on encryption tools as well as Panopticlick, a project that allows users to see what trackers they are vulnerable to. Software Engineering Daily is having our third Meetup, Wednesday May 3rd at Galvanize in San Francisco. The theme of this Meetup is Fraud and Risk in Software. We will have great food, engaging speakers, and a friendly, intellectual atmosphere. To find out more, go to softwareengineeringdaily.com/meetup.

Search is a common building block for applications. Whether we are searching Wikipedia or our log files, the behavior is similar: a query is entered and the most relevant documents are returned. The core data structure for search is an inverted index. Elasticsearch is a scalable, resilient search tool that shards and replicates a search index. Philipp Krenn from Elastic joins the show today to discuss how search works and how Elasticsearch scales. We use Wikipedia as a running example for how a query is processed and how documents are stored. If you’ve ever wondered how search works–or if your company uses Elasticsearch and you want to know more about it–this is a great episode for you.

React Native has unlocked native mobile development to web engineers who may now apply their skills to build iOS and Android applications in JavaScript. For the first time, cross platform JavaScript-based applications feel as if they were written in the native language of choice for the platforms. Businesses who choose to adopt React Native for their native app development also see great benefits such as the ability to push new JavaScript code without going through the app store review process, and the ability to share code and business behaviors across the iOS and Android platforms. Expo is building a cross-platform native runtime for React Native. Expo brings the benefits of deployment and iterative development to native without sacrificing any user experience costs. Expo plans to do this with their native SDK, custom development environment, and tools built in collaboration with Facebook like create-react-native-app. React Native has the incredible potential to revolutionize all user interface development with its core set of cross-platform UI primitives, and React’s popular declarative rendering pattern. So in this episode Brent Vatne and Adam Perry join Caleb Meredith to first discuss Expo and the future of React Native to try and answer the question: can React Native become the one UI framework to rule them all?

Every program gets compiled down to 1s and 0s before it can be executed against hardware. Before being translated to machine code, programs that are written in a language like Rust, Swift, or Java spend time in an intermediate representation. In Java, this intermediate representation is Java bytecode. Many different languages–such as Scala–translate to Java bytecode, because there has been lots of optimization written to speed up Java bytecode. Java bytecode runs on the JVM–the Java Virtual Machine. LLVM is a project that draws inspiration from the Java Virtual Machine. LLVM originally meant “low level virtual machine” but today it is just called LLVM and describes a set of compiler tools. In today’s interview with Morgan Wilde, we explore how compilers work, how different processor hardware architectures present a problem for compilers, and why LLVM’s intermediate representation creates a layer of interoperability for any language that compiles down to that intermediate representation. Whether you are new to compilers or have experience, this episode will appeal to you. Morgan is an excellent teacher and his enthusiasm for the subject comes through. He has a 30-minute YouTube video–A Brief Introduction to LLVM that I highly recommend.

Engineers in Silicon Valley see a world of constant progress. Our work is creative and intellectually challenging. We are building the future and getting compensated quite well for it. But what if we are actually achieving far less than what is possible? What if, after so many years of high margins, gourmet lunch, and self-flattery, we have lowered our standards for innovation? And if Silicon Valley has been lulled into complacency, what does that say about the rest of the United States? American exceptionalism has faltered and complacency has risen in its wake. Today’s guest Tyler Cowen is an economist and author. His new book The Complacent Class is the final book in a trilogy that describes a decline of American output and a decline in American mindset. Complacent America has lost its ability to assess risk. Children are prevented from playing tag for risk of injury. College students protest against speakers who might present challenging ideas. The number of Americans under 30 who own a business has fallen by 65% since the 1980’s–millennials are too busy going to business school to start businesses. In his books, Tyler weaves together history, philosophy, and contemporary culture. He presents hard data about many different fields, and theorizes about how the trends in those fields relate to each other. He also has a podcast, Conversations with Tyler, and in this episode I tried to mirror his interview style. If you like this episode, you should check out his show–he has interviewed people like Ezra Klein, Peter Thiel, and Kareem Abdul-Jabbar.

Cryptocurrencies are not only a financial instrument–they are a new platform for building applications. The blockchain allows for new solutions to digital property management, micropayments, hedge fund incentives, and ad fraud. The cryptocurrency platforms with the most traction are Bitcoin and Ethereum. Bitcoin has no central leader and is going through some growing pains with governance issues. Ethereum is led by the charismatic Vitalik Buterin. Bitcoin and Ethereum are not competing instruments. They fulfill different technical purposes. Under current conditions of algorithm development and network infrastructure, neither Bitcoin nor Ethereum can accomplish the dreams that will one day be realized, because of the problem of distributing transaction information across nodes in the system. If we compared cryptocurrencies to the Internet, we would not even be in the days of dial-up yet. ConsenSys is a venture production studio that is working on several projects within the blockchain space. Mike Goldin is a software developer with ConsenSys and joins the show to talk about blockchain applications in 2017–where we are and where we are going. It was a wide ranging conversation and I hope to have Mike back in the future so we can go deeper on some of the topics we glossed over.

There are various standards at play when creating and consuming Application Program Interfaces (APIs). These standards, though, are mostly technical and mostly lower-level than the content of the API. Andy Beier has experienced the broad range of API quality in his role with Domo in creating integrations with other businesses. He has made standardization of good practices in creating APIs his mission, with an emphasis on making the right information easily accessible without having to download more than necessary. He has traveled to meet with leaders in the field to promote standards and to make APIs easier to create and to consume. In this episode, Andy joins Dave Rael for a conversation about API design standards, what makes for a good API, and steps in moving the broader technical community toward more useful and secure APIs.

In this episode, I gathered questions from listeners in our Slack channel and Twitter feed. The questions I answered include: We always want more feedback and questions. Please email me, join Slack, and fill out our survey!

A hedge fund is a collection of investors that make bets on the future. The “hedge” refers to the fact that the investors often try to diversify their strategies so that the direction of their bets are less correlated, and they can be successful in a variety of future scenarios. Engineering-focused hedge funds have used what might be called “machine learning” for a long time to predict what will happen in the future. Numerai is a hedge fund that crowdsources its investment strategies by allowing anyone to train models against Numerai’s data. A model that succeeds in a simulated environment will be adopted by Numerai and used within its real money portfolio. The engineers who create the models are rewarded in proportion to how well the models perform. Xander Dunn is a software engineer at Numerai and in this episode he explains what a hedge fund is, why the traditional strategies are not optimal, and how Numerai creates the right incentive structure to crowdsource market intelligence. This interview was fun and thought provoking–Numerai is one of those companies that makes me very excited about the future.

Brendan Eich created the first version of JavaScript in 10 days. Since then JavaScript has evolved, and Brendan has watched the growth of the web give rise to new and unexpected use cases. Today Brendan Eich is still pushing the web forward across the technology stack with his involvement in the WebAssembly specification and the Brave browser. For all of its progress, JavaScript struggles to run resource-intensive programs like complex video games. With JavaScript falling short on its charge to be the “assembly language for the web” the four major browser vendors started collaborating on the WebAssembly project to allow programming languages a faster, lower level compile target when deploying to the web. Brendan is the CEO of Brave which aims to provide a faster and safer browsing experience by blocking ads and trackers by default in a new browser. The Brave browser is also helping publishers monetize in interesting new ways while also giving a share of ad revenue to its users. Caleb Meredith is the host of this show. He previously guest hosted a popular episode on Inferno, a fast, React-like JavaScript framework. As we bring on more guest hosts, please send us feedback. We want to know what every host is doing well, and what we can improve on.

Big technology companies have so much going on at any given time that a journalist can tell any type of story they want to about it. Depending on what angle you observe the company from, you can write a story depicting that company as good, evil, growing, or about to crash. The truth only becomes apparent to outsiders with time. Amazon’s culture and business strategy were detailed in The Everything Store, a 2013 book by Brad Stone. I read The Everything Store before working at Amazon, then I read it a second time after working at Amazon. The book is an accurate and balanced depiction of Amazon’s ethos. Brad’s new book The Upstarts documents the rise of Uber and Airbnb, two companies with a similar controversial valence to Amazon. It was a pleasure to speak to Brad because I admire his engrossing storytelling as an author and his strategic analysis as a business journalist. After discussing business and technology with him, we explored journalism–Brad is senior executive editor at Bloomberg.

Servers in a data center fail. Sometimes entire data centers have a power outage. Bugs in an application make it into production. Human operators make mistakes and cause data to be deleted. Failure is unavoidable. We make backups and replicate our servers so that when a failure occurs, we can quickly respond to it without making the user feel much pain. But how can we test that our response will work before an actual catastrophe occurs? Kolton Andrus is CEO of Gremlin, a company that works on failure injection as a service. Gremlin is based on ideas around planned failure that Kolton learned from his years at Amazon and Netflix. We ended up talking as much about the culture of Netflix and Amazon as we did about how and why to build failure injection. It’s always nice to share war stories with other people who have worked at Amazon because the culture is so distinct. If you want to know more about Amazon’s culture, check out the episode tomorrow with Brad Stone, author of The Everything Store.

Designers and software engineers need to communicate with each other. From Apple to Slack to Uber, the emphasis on visual design within a product is rising in importance. Much like development and operations siloes have been bridged with the DevOps movement, design and engineering teams are working more closely together to align the vision of the designers with the realities of code. InVision is a product for prototyping designs and product workflows. I categorize InVision with high-level productivity tools like Trello, Asana, and github. When teams start using InVision, it often becomes an integral part of the entire product workflow. A designer will mock up their product vision on InVision and share it with the rest of the team for criticism and commentary. Since the product is the perfect bridge between engineers and designers, InVision the company is expanding as a platform. Bjorn Freeman-Benson is the CTO at InVision and his task is to scale the 100% remote engineering team while developing new product features. Bjorn describes himself as a software psychologist. We talked about the psychology of managing engineers and the technical challenges of building a large, popular web app for technology products.

Developing a product requires careful balance between engineering, sales, design, and customer service. The founding CTO of a company often needs to take on each of these responsibilities, because when the company only has a few people there is nobody to delegate these different tasks to. Cullen Zandstra is the CTO at FloQast, a SaaS tool for accounting close management. It isn’t important what that means–we do describe it in this episode, but the actual software product that we are discussing is less important than the general discussions we have. Cullen gives detailed explanations for how to develop a software product. In the past, Cullen worked at MySpace. After we discussed his current company FloQast, we discussed some of what he learned at MySpace, and why the social network proved to be less durable than Facebook. We also had some great discussion as to why Google+ was ALSO less durable than Facebook. Also–I should mention that FloQast is hiring engineers in their Los Angeles office. And thanks to my friend Patrick Mathieson for connecting Cullen and myself.

Net neutrality is the principle that internet service providers and governments regulating the internet should treat all data on the internet the same. Debates around Net Neutrality can be as contentious as subjects like global warming, or tabs vs. spaces. To a hardcore free market economist, Net Neutrality sounds suspicious. Why would it be good for the government to regulate prices on the data that passes around wires owned by a corporation? The problem is that a large portion of the United States can only be served by wires owned by a single company. In those instances, the company that owns the wires has monopoly pricing power over the data delivered to homes in those geographic regions. Quincy Larson is the founder of Free Code Camp and a frequent author of internet-based articles on Medium. He returns to Software Engineering Daily to make the case for Net Neutrality. I was convinced by some of his arguments and less convinced by others. In any case, I think you will find this episode entertaining and informative. If you are like me, after this episode you will have at least a slightly better understanding of the issues surrounding Net Neutrality. Inside the Invisible War for the Open Internet

In bitcoin, every transaction in the shared ledger has the sender, recipient, and value. This ledger gets appended infinitely and is shared within a peer-to-peer network. Zcash is a cryptocurrency with all the features of bitcoin, plus encrypted transactions. The sender, recipient, and value fields are all encrypted. If bitcoin is HTTP, Zcash is like HTTPS, a secure transport layer. Nathan Wilcox works on Zcash, and in this episode we discuss why an encrypted version of bitcoin is useful, how mining works, and how Zcash the company is structured. Nathan also gives some context for the current state of the bitcoin community, and where Zcash fits in.

Wearables have become more accessible to the public. Snap’s spectacles, Google Glass, FitBit, and Apple Watch suggest a future in which many people will be wearing a smart device. In this episode Asta Roseway, Research Designer at Microsoft Research, gives insights into other categories of wearables like tattoos, scarves, and cosmetics. Asta talked about her work on DuoSkin, a wearable that looks like a beautiful metallic tattoo. We talked about its capabilities and how it was built, and why it is still too early for connected tattoos. We also talked about the intersection between health, wearables, technology, and fashion and how wearables might look in the near future. This episode is guest hosted by Edaena Salinas, who also hosts The Women in Tech Show. Asta Roseway’s Website Asta Roseway Microsoft Research MoodWings LightWear

Multiagent systems involve the interaction of autonomous agents that may be acting independently or in collaboration with each other. Examples of these systems include financial markets, robot soccer matches, and automated warehouses. Today’s guest Peter Stone is a professor of computer science who specializies in multiagent systems and robotics. In this episode, we discuss some of the canonical problems of multiagent systems, which have some overlap with the canonical problems of distributed systems–for example, the problems of coordinating between different agents with varying levels of trust resembles the problem of establishing consistency across servers in a database cluster. Peter has recently contributed to the 100 year study of artificial intelligence, so we also had a chance to discuss the opportunities and roadblocks for AI in the near future. And since Peter teaches computer science at my alma mater, UT Austin, I had to ask him a few questions about the curriculum.

Biology research is complex. The sample size of a biological data set is often too small to make confident judgments about the biological system being studied. During Jason Knight’s PhD research, the RNA sequence data that he was studying was not significant enough to make strong conclusions about the gene regulatory networks he was trying to understand. After working in academia, and then at Human Longevity, Inc Jason came to the conclusion that the best way to work towards biology breakthroughs was to work on the computer systems that enable those breakthroughs. He went to work at Nervana Systems on hardware and software for deep learning. Nervana was subsequently acquired by Intel. In this episode, we discuss how machine learning can be applied to biology today, and how industrial research and development is key to enabling more breakthroughs in the future. The main lesson I took away from this show is that while we have seen phenomenal breakthroughs in certain areas of health–like image recognition applied to diabetic retinopathy or skin cancer–the challenges of reverse engineering our genome to understand how nucleic acids fit together into humans are still out of reach, and improving the hardware used for deep learning will be necessary to tackle these kinds of informational challenges.

Every company that deals with payments deals with fraud. The question is not whether fraud will occur on your system, but rather how much of it you can detect and prevent. If a payments company flags too many transactions as fraudulent, then real transactions might accidentally get flagged as well. But if you don’t reject enough of the fraudulent transactions, you might not be able to make any money at all. Because fraud detection is such a difficult optimization problem, it is a good fit for machine learning. Today’s guest Michael Manapat works on machine learning fraud detection at Stripe. This conversation explores aspects of both data science and data engineering. Michael seems to benefit from having a depth of knowledge in both aspects of the data pipeline, which made me question whether data science and data engineering are roles that an engineering organization wants to separate. This is the third in a series of episodes about Stripe engineering. Throughout these episodes, we’ve tried to give a picture for how Stripe’s engineering culture works. We hope to do more experimental series like this in the future. Please give us feedback for what you think of the format by sending us email, joining the Slack group, or filling out our listener survey. All of these things are available on softwareengineeringdaily.com.

If you are building a service that processes payments, your software architecture has a lot of requirements. Not only do you need to be highly available, consistent, and fast–you need to be PCI compliant. In this episode, we explore the infrastructure of Stripe with Evan Broder, who has been with the company for five years. Stripe started as a small payments company catering to developers with a monolithic code base. Some of those aspects of Stripe have changed, and others have stayed the same. In our last episode, we covered how observability works at Stripe. In this episode, we explore what is being observed–the actual infrastructure itself, and how different engineers are organized around managing the infrastructure. In tomorrow’s episode, we’ll talk to Michael Manapat, about Stripe’s machine learning pipeline for detecting and preventing fraudulent transactions. Throughout these episodes, you will get a sense for how Stripe’s engineering culture works. We hope to do more experimental series like this in the future. Please give us feedback for what you think of the format by sending us email, joining the Slack group, or filling out our listener survey. All of these things are available on softwareengineeringdaily.com.

Observability allows engineers to understand what is going on inside their systems. In its most raw form, observability comes from log data. Modern systems have many layers of logs–virtualized cloud infrastructure, container orchestration, the container runtime itself, and the application logic running within the container. With all of these layers, it is not practical for a developer to have to sift through layers of logs every time a bug occurs in production, or a deployment fails integration tests. Higher level observability tools include charts, distributed tracing tools, and monitoring services. With proper observability, developers can save time during incident response. Day-to-day software development becomes safer and more comfortable. Stripe is a payments company for developers. This episode is the first in a series of episodes profiling different aspects of the company. Our guest Cory Watson leads the observability team at Stripe. In subsequent episodes, we will explore infrastructure and machine learning at Stripe. Throughout these episodes, you will get a sense for how Stripe’s engineering culture works. We hope to do more experimental series like this in the future. Please give us feedback for what you think of the format by sending us email, joining the Slack group, or filling out our listener survey. All of these things are available on softwareengineeringdaily.com.

Donald Trump has either alarmed or excited everyone in the engineering community. Some of the debates are based around innovation. Opponents to Trump say that his anti-immigration policies will reduce the innovation that relies on H1-B visas. Supporters of Trump say that his anti-regulation policies will unlock innovation that is restricted by bureaucracy. Other debates are around philosophy and free speech. Opponents to Trump say that his views on women are retrograde and that he promotes racism and white nationalism. Supporters of Trump say that he is brave enough to say what people are really thinking, and that he is truly independent from the polarized, ossified political system. Software Engineering Daily is a show about the world through the lens of the software engineer, and today our world is being shaped by Donald Trump. We needed to do an episode on this subject because so many of my day-to-day conversations with other engineers were becoming about Trump. Brad Taylor is a senior engineer with Optimizely who has formed Tech Stands Up, a grassroots movement giving a voice to the rapidly growing concerns about the Trump administration’s policies affecting the tech community and its users. We had a stimulating conversation about the tradeoffs of a Trump administration from an engineer’s perspective. Don’t worry–this is not becoming a show about politics. Tech Stands Up: Pi Day Rally Tickets TechStandsUp.org

Making the right engineering choices in today’s wide landscape of cloud technologies is hard. Predicting the future in order to invest in companies in this space has the same level of complexity. The cost of cloud computing is going down but the volume of total required space and processing power is going up. The open source community is growing and improving but people are increasingly willing to buy software that will save them time. Capital is flowing into Silicon Valley at a faster rate than can be sensibly absorbed by the number of quality companies. A decent product won’t have trouble raising money. But a decent investor has to choose wisely among the huge selection of available opportunities. Lenny Pruss works at Amplify Partners where he is currently focused on the cloud native space. In this episode, we talk about what cloud native means and how to navigate the complex landscape, whether you are an engineer or an investor. Cloud Native Landscape Project

Command Query Responsibility Segregation (CQRS) is a powerful concept that has the potential to make for reliable and maintainable systems. It is also broadly misunderstood and means different things to different people. Derek Comartin learned about the idea after viewing some talks by Greg Young and has since successfully applied the approach with great success and it has transformed the way he views features, business requirements, and dependencies. The result is a system that is easier to maintain and faster to enhance. Among his key lessons are that slices are better than layers, mediators improve dependency management, and cohesion is better applied to business concerns than to technical ones. In this episode, Derek joins Dave Rael for a conversation about CQRS and applying it to make your code your own and to separate it from technical concerns in order to make your software development operation work better and faster.

Machine learning has improved both in tools and accessibility. Frameworks like TensorFlow create the right abstractions for developers to work efficiently. Educational programs like Metis and Insight Data Science provide a place for developers to learn these tools. As a result, artificial intelligence is becoming easier to develop and more widespread. Rumman Chowdhury works on artificial intelligence at Accenture. Before her current role, she taught data science at Metis. In this episode, we talk about the current state of artificial intelligence, from the tools available to the long term implications–such as the robot tax recently proposed by Bill Gates.

Load testing measures performance of a system undergoing a large volume of requests. Before an application is pushed to production, engineers will often load test their software to ensure it is resilient in the face of high traffic. As web applications have changed, the requirements around load testing have changed as well. External APIs, internal undocumented APIs, and proprietary databases are black boxes that you might not be able to test reliably with unit tests or integration tests. Having an end-to-end load testing system can provide a measure of insurance against unknown unknowns before users start engaging with a production version. Mark Gilbert works on performance engineering at Apica, and he joins the show to discuss how load testing software is built and when engineers should use it. Full disclosure: Apica is a sponsor of Software Engineering Daily.

There is too much mobile engineering work to be done and not enough mobile engineers. As a result, a talented mobile engineer will often make more money than a similarly talented web developer. There are many other disconnects between the world of mobile engineering and the world of backend and web development. We have reported on web development far more than mobile on Software Engineering Daily, and, I wanted to get a holistic view of the mobile ecosystem. Nathan Esquenazi is a co-founder of CodePath, which is a continuing education program for professional engineers and designers. There was much to explore–from the economics of mobile, to the state of cross platform, to the increasingly blurry line between mobile and IoT.

Thursday February 23rd was a big day in security news: details were published about the Cloudbleed bug, which leaked tons of plaintext requests from across the Internet into plain view. On the same day, the first collision attack against SHA-1 was demonstrated by researchers at Google, foretelling the demise of SHA-1 as a safe hashing function. What does this mean for the average engineer? What are the implications for regular internet users? Haseeb Qureshi interviews Max Burkhardt, a security researcher at Airbnb, to get to the bottom of what exactly happened, what it means, and how it affects the security of web applications.

Manufacturing electronics is not a simple task and because of its complexity it is also quite expensive. Companies like Apple are able to pull their costs down through economies of scale. Since Apple is always placing huge bulk orders on chips, sensors, and other components, the company can get lower prices for those components than an individual hardware hacker that wants to build a prototype. Herein lies the opportunity for Tempo Automation, a company that allows rapid prototyping of electronics. By aggregating demand for electronics prototyping, Tempo Automation can offer competitive pricing and speed to anyone who needs a hardware prototype. I visited Tempo Automation and was given a tour of the production floor by CEO Jeff McAlvay. He showed me the huge machines that are used to build hardware prototypes and explained how they work in serial to produce prototypes. He also mentioned that anyone can schedule a tour of Tempo Automation’s factory by going to their website, and I highly recommend it. After the tour, Jeff and I sat down for an interview about what Tempo Automation is building and their plans for the future.

When an engineer is offered a job a tech company, their compensation is often partly in cash and partly in equity–shares of the company. How should an engineer evaluate that offer? How should they negotiate? In the world of equity compensation, costly and avoidable mistakes are routine, and this hurts both companies and employees. Josh Levy was on Software Engineering Daily previously to talk about the Amazon Web Services open guide, which was one of the most popular episodes we have ever had. In this episode, Josh and I are joined by Joe Wallin, a lawyer who has been involved in startups for many years. Josh and Joe are the authors of the The Open Guide to Equity Compensation, which is a resource designed to clear up the confusion around stock, options, and fundraising. It’s a tremendously useful and concise overview of what an engineer or founder needs to know when it comes to equity financing.

Parse was a backend as a service company built in 2011 before being acquired by Facebook in 2013. Building a backend as a service for developers requires walking a thin line between giving engineers lots of control and preventing those engineers from shooting themselves in the foot. While she was at Parse, Charity Majors learned about the operational burdens of managing a service with high uptime requirements and deeply technical edge cases that could take down a user’s entire system. Charity took the lessons in systems engineering that she learned at Parse and cofounded Honeycomb.io, a service for observability and monitoring. Honeycomb is described as a tool for your systems like an IDE is to your code. Parse was eventually shut down because the service did not have a place in the strategic plans of Facebook. Charity and I also discussed the lessons learned from how the Parse acquisition panned out–a useful conversation for anyone who is considering selling a company or acquiring a startup.

When an application is using all of its available resources, that application needs to be scaled. Scaling an application means giving it more resources–typically servers. Autoscaling is an engineering practice where an application is automatically given more or less resources based on how healthy the application performance is at a given time. Applications on Heroku have access to autoscaling. Heroku users don’t need to worry about provisioning new servers manually because the platform does it for them. In this episode, we explore how Heroku built autoscaling. Andrew Gwozdziewycz [@apgwoz] is an operational experience engineer with Heroku. As he describes, autoscaling requires frequent health checks of an application. Since thousands of applications are running on Heroku, a metrics pipeline using Kafka and Cassandra supports the high volume of health check data. That data feeds into the decision process for when an application needs to scale. Full disclosure: Heroku is a sponsor of Software Engineering Daily.

In the mid 90s, data warehousing might have meant “using an Oracle database.” Today, it means a wide variety of things. You could be stitching together a big data pipeline using Kafka, Hadoop, and Spark. You could be using managed tools like BigQuery from Google. How did we get from the simple days of Oracle databases to the wealth of options available today? Mark Rittman writes and podcasts about data engineering and data warehousing on his site Drill to Detail. Today, we explore the past, present, and future of data warehousing and touch on many of the trends that have been explored in recent episodes of Software Engineering Daily. Google BigQuery, and Why Big Data is About to Have Its Gmail Moment

At the first Software Engineering Daily Meetup, the speakers explored a range of topics. A few weeks ago, we published Courtland Allen’s talk about how to build a small software business. In today’s episode, we are publishing Haseeb Qureshi’s talk, called “Everything That Rises Must Converge: Why Engineers Disagree About Everything (And Why Fraudsters Do Too).” This talk explores philosophy, poker, software engineering, fraud, and a basket of other topics. These topics might sound like a random collection, but Haseeb currently works as an engineer on Airbnb’s Risk team and he is the author of How To Be a Poker Player, which is a book on the philosophy of poker. The audio is taken from the Meetup and we cleaned it up as best we could. We will do a better job with the audio from the next Meetup–which is Thursday, March 9th at Galvanize! Check out our Meetup page for more details.

Security vulnerabilities are an important concern in systems. When we specify that we want certain information hidden, for example our phone number or our date of birth, we expect the system to hide the information. However, this doesn’t always happen due to human error in the code because programmers have to write checks and filters across the program. In this episode, Edaena Salinas interviews Jean Yang, Assistant Professor at the Computer Science Department at Carnegie Mellon, who presents Jeeves, a language that allows programmers to specify security policies more intuitively, making it harder to leak information that is meant to be protected. Jean explained how Jeeves was implemented and how it can be used. We also talked about what it takes to bring research concepts from academia to the industry and at the end we had a very interesting conversation on how to educate a broader audience on the importance of security. Jean was also named one of the 35 innovators under 35 by MIT Technology Review.

Expectations for mobile apps have gone up steadily since the iPhone was released. But the choice of databases built for mobile apps has remained limited mostly to SQLite. RealmDB was created as a new option for mobile developers on iOS, Android, or any other mobile platform. Realm is not just a database. It is a database platform, offering a variety of systems such as authentication, sync, and an event framework. In this interview with RealmDB VP of engineering Brian Munkholm, he explains why we need not only a new mobile database type, but an entire database platform. We talk about the business of mobile, IoT, cloud, and cross platform. It’s a wide ranging conversation, but also deeply technical.

Online advertising is a system of transactions that involve many different players. The user visits a publisher’s website; the publisher notifies an exchange that the user is on the website; the exchange presents an opportunity to a marketplace that can buy that opportunity to show the end user the ad. And this a simple example. The transactions in online advertising are as opaque and rife with fraud as Wall Street–but less regulated. Blockchain technology presents an opportunity to bring more transparency to the advertising ecosystem using a shared ledger. Ken Brook is the CEO of VidRoll, a video advertising company. Using his experience in the adtech business, Ken is working on Adchain, a shared ledger for adtech. In this episode, we explore adtech, blockchains, and Adchain. Russian Cyberforgers Steal Millions a Day with Fake Sites

The impact of artificial intelligence on our everyday lives will be so profound that our modern institutions will change completely. Employment, government, romance, social norms–all of these things will be upended. To see the signs of this coming, you no longer have to read science fiction. Every week, there are blog posts, news stories, and videos chronicling our strange, exciting time. Rob May is an investor in artificial intelligence companies and is CEO of the AI company Talla. Every week, he puts out the Technically Sentient newsletter, a compilation of the best pieces of information about AI over the past week. Each newsletter also contains a short essay by Rob in which he gives a big picture perspective on what he is seeing in the AI space. This was an illuminating conversation with Rob about the implications of artificial intelligence and the topics he writes about in Technically Sentient. I first saw Rob speak at the Launch Scale Festival, which Jason Calacanis puts on as a free event for people who have started companies.

If you wanted to build a machine learning model to understand human health, where would you get the data? A hospital database would be useful, but privacy laws make it difficult to disclose that patient data to the public. In order to publicize the data safely, you would have to anonymize it, so that a patient’s identity could not be derived from data about that patient–and true anonymization is notoriously difficult. In every industry where privacy is a concern there is a similar challenge. If there is no place with public data sets, there is no place where the machines can go to learn. The possible machine learning applications that we can build are limited by the data sets that are available. Auren Hoffman started his company SafeGraph to unlock data sets so that machine learning algorithms can learn from that data. In this episode, we talk about the machine learning landscape in both the short- and long-term time horizons. We also discussed some of Auren’s strategies for building companies, which have been crucial for me in thinking about how to build Software Engineering Daily. Where Should Machines Go to Learn? Quoracast Episode

Machine learning frameworks like Torch and TensorFlow have made the job of a machine learning engineer much easier. But machine learning is still hard. Debugging a machine learning model is a slow, messy process. A bug in a machine learning model does not always mean a complete failure. Your model could continue to deliver usable results even in the presence of a mistaken implementation. Perhaps you made a mistake when cleaning your data, leading to an incorrectly trained model. It is a general rule in computer science that partial failures are harder to fix than complete failures. In this episode, Zayd Enam describes the different dimensions on which a machine learning model can develop an error. Zayd is a machine learning researcher at the Stanford AI Lab, so I also asked him about AI risk, job displacement, and academia versus industry. Why ML is hard

Data scientists need flexible interfaces for displaying and manipulating data sets. Data engineers need to be able to visualize how their data pipelines wire together databases and data processing frameworks. DevOps engineers need dashboards to understand their monitoring data at a high level. All of these programmers are building data applications. Data applications let us visualize and manipulate data sets effectively. In today’s episode, Dave King joins the show to describe the growing importance of data applications. Many knowledge workers use flexible tools like spreadsheets and business intelligence applications. But when you build a domain specific data application for a knowledge worker, you can unlock a higher degree of leverage for that knowledge worker. We discuss data applications and the future of knowledge work in this episode with Dave King. Full disclosure: Exaptive is a sponsor of Software Engineering Daily.

Most tech companies are moving toward a highly distributed microservices architecture. In this architecture, services are decoupled from each other and communicate with a common service language, often JSON over HTTP. This provides some standardization, but these companies are finding that more standardization would come in handy. At the ridesharing company Lyft, every internal service runs a tool called Envoy. Envoy is a service proxy. Whenever a service sends or receives a request, that request goes through Envoy before meeting its destination. Matt Klein started Envoy, and he joins the show to explain why it is useful to have this layer of standardization between services. He also gives some historical context for why Envoy was so helpful to Lyft.