Software Engineering Daily

Medical imaging is used to understand what is going on inside the human body and prescribe treatment. With new image processing and machine learning techniques, the traditional medical imaging techniques such as CT scans can be enriched to get a more sophisticated diagnosis. HeartFlow uses data from a standard CT scan to model a human heart and understand blockages of blood flow using simulations of fluid dynamics. In today’s episode, Razik Yousfi and Leo Grady from HeartFlow describe the data processing pipeline for the company and what their technology stack looks like.

Data visualization tools are required to translate the findings of data scientists into charts, graphs, and pictures. Understanding how to utilize these tools and display data is necessary for a data scientist to communicate with people in other domains. In this episode, Srini Kadamati hosts a discussion with Jake VanderPlas about the Python ecosystem for data science and the different attempts at creating a data visualization library. Jake VanderPlas is the Director of Research for Physical Sciences at the University of Washington’s eScience institute, where he also received his PhD in Astronomy. In addition to contributing to many Python data science libraries like scikit-learn, scipy, numpy, and matplotlib, he’s written multiple books that have been published by O’Reilly and has given many talks on data science tools and techniques. He’s also the co-creator of the Altair project, which is a declarative data visualization library for Python built on the Vega-Lite visualization grammar.

Column-oriented data storage allows us to access all of the entries in a database column quickly and efficiently. Columnar storage formats are mostly relevant today for performing large analytics jobs. For example, if you are a bank, and you want to get the sum of all of the financial transactions that took place on your system in the last week, you don’t want to iterate through every row in a database of transactions. It is more efficient to just look at the column for the amount of money, and ignore things like timestamp and user id. Julien Le Dem co-created Parquet, a file format for storing columnar data on disk. Jacques Nadeau is a VP of Apache Arrow, a format for in-memory columnar representation. They are both part of Dremio, and they join the show to talk about how columnar data is stored, processed, and shared between systems like Spark, Hadoop, and Python.

Free Code Camp is an online learning platform that takes people from knowing nothing about code to having enough knowledge to build software for a living. We have already done a show with Quincy Larson, the founder of Free Code Camp, in which we discussed his motivation for starting the organization. The economics of running a free interactive platform with thousands of users informs the architectural strategy. Free Code Camp does not even have ads, so it needs to be frugal. Berkeley Martinez is the CTO of Free Code Camp, and in today’s episode he discusses how Free Code Camp works from a technical perspective, including how the site’s sandboxed coding environment is built.

Software engineers are often one of the last lines of defense against potentially dangerous and unethical practices. Every software company encounters situations where the line between right and wrong is not clearly drawn. Back in the year 2000 Bill Sourour was asked to code a project to help market a pharmaceutical product in a misleading way. Reflecting on this experience, Bill recently wrote about it on Medium, suggesting that developers should avoid doing things that compromise our ethics in small ways. The more software defines our real world the more relevant the conversation around ethics becomes. In this episode, Bill and I explore the ethics of modern software development. The Code I’m Still Ashamed Of

A company like Pinterest has millions of transactional emails to send to people. The scalability challenges of sending high volumes of email mean that it makes more sense for most companies to use an email as a service product rather than building their own. Chris McFadden is the VP of engineering and cloud operations at SparkPost and he joins the show to explain the architecture of SparkPost’s email as a service product. SparkPost started as an on-premise email technology for large enterprises, and evolved into a SaaS product. In 2014, the company migrated to the cloud, which has changed its infrastructure as well as its operational model. Full disclosure: SparkPost is a sponsor of Software Engineering Daily.

Vulnerabilities exist in every computer system. As a system gets bigger, the number of vulnerabilities magnifies. The web is the biggest, most complex computer system we have–but fortunately, the steps we can take to secure our web applications are often quite simple. Jared Smith is a cyber security research scientist with Oak Ridge National Laboratory. He joined me on the show to discuss web application security, but I really wanted to know his position on some of the more grandiose topics–Stuxnet, our power grid, Russian hacking, and corporate backdoors. This was a wide ranging discussion and I enjoyed it a lot. For a presentation Jared gave at Nodevember about Web Security, check out this YouTube video.

Meetup is an online service that allows people to gather into groups and meet in person. Since 2002, the company has been growing and its technology stack has been changing. Today, they are in the process of migrating to the cloud, using both Amazon Web Services and Google Compute Platform. Yvette Pasqua is the CTO of Meetup and she joins the show to explain how Meetup’s technology stack works and how the teams are organized. The discussion of multiple clouds is particularly interesting–Yvette describes GCP and AWS as both having distinct, well-defined use cases at Meetup.

When a useful new technology comes out, companies that are in a position to adopt that new technology can gain an edge over competitors. As our industry grows and moves faster, these kinds of changes are coming faster–some recent examples are Docker, ReactJS, and Kubernetes. Evolutionary architecture supports incremental, guided change as a first principle along multiple dimensions. A company with an evolutionary architecture is structured to evolve in response to changes inside the company (such as a decision to change the product) or outside the company (such as the emergence of Docker). Neal Ford is an architect at ThoughtWorks and one the creators of the evolutionary architecture concept.

Every digital system has vulnerabilities. Cars can be hacked, locked computers can be exploited, and credit cards can be spoofed. Security researchers make a career out of finding these types of vulnerabilities. Samy Kamkar’s approach to security research is not just about dissection–it’s also about creativity. For many of the technologies he hacks on, Samy open-sources code that summarily describes the vulnerability he has been working on. For example, in his project PoisonTap, Samy open-sourced code that you can run on a $5 Raspberry Pi, and plug into a locked computer to exploit it. Our conversation covered the art of deconstructing technologies for vulnerabilities and Samy’s goals as a security researcher. We also touched on some of the broader issues of modern security.

Self-contained systems is an architectural approach that separates the functionality of a system into many independent systems. Each self-contained system is an autonomous web application, and is owned by one team. Communication with other self-contained systems or 3rd party systems is asynchronous where possible. As Eberhard Wolff explains in this episode, self-contained systems is not the same thing as microservices, but they are not mutually exclusive. Organizations often adopt a mix of architectural ideas, and it is worth understanding these different models so you can decide which of them to apply to your own projects.

Many people find themselves going down a career path that does not bring them satisfaction. A lawyer finds himself constantly working cases he doesn’t care about. A student in medical school gets completely burned out from hospital bureaucracy. An investment banker no longer finds joy in the accounting statements that she used to enthusiastically study. Startups offer a different career path. Within a startup, an employee can often find creativity, limited bureaucracy, and highly variable reward structures. These ingredients make the world of startups both refreshing and intimidating to someone who feels stuck in a career that no longer gives them joy. Breaking Into Startups is a podcast about people coming from non-traditional backgrounds and making their way into startups. Breaking Into Startups is run by Artur and Timur Meyster and Ruben Harris, who join me for a conversation about how they migrated from more traditional careers to startups–and how many other people are doing the same.

In the 1990s, the barriers to starting a company were significant. Not only did you need an idea, you needed $200,000 for servers and Oracle licenses. With cloud computing, the up-front financial costs of getting a company off the ground have been mostly eliminated–but the idea of starting a company is still perceived as risky. The process of building software has changed dramatically in the last twenty years, but many of the challenges of managing a software company remain timeless. How do you hire properly? How do you keep track of a software stack that is growing in complexity? How do you handle dissenting opinions from employees? Mike Wolfe has been building software companies since the 1990s and joins the show to discuss how to build products and manage engineering teams. It’s a wide-ranging discussion including technological trends, interpersonal skills, and startup financing.

A biologist wants to study the genetic makeup of an organism. A pharmaceutical researcher wants to test the effects of an experimental drug. These types of experiments require a deep knowledge of the scientific domain as well as the lab techniques to produce the data that will eventually yield a result. Transcriptic is a robotic biology laboratory that allows you to make requests for experiments to run remotely. Scientists who use Transcriptic do not have to perform wet lab experimentation. The robotic laboratory process has the potential to do for biology what Amazon Web Services did for computing. Max Hodak is a cofounder of Transcriptic and joins me today to explain how a robotic laboratory works. There are a wide variety of challenges, from hardware integration to software reverse engineering–but we also talk about why it is worth it to overcome these challenges, including a discussion of the “reproducibility crisis” that is undermining faith in our scientific experiments.

Application performance monitoring helps an engineer understand what is going on with an application. An application on a single machine is often monitored by inserting bytecode instructions into the application after it has been interpreted. Distributed cloud applications with functionality broken up across multiple servers often use distributed tracing. Andi Grabner from Dynatrace joins today’s show to explain how monitoring software is built, and how engineers use it to solve problems. Monitoring is core to every business, whether the goal is to understand top-level business processes of to dissect and debug a specific engineering problem. And because monitoring is important at every layer of the stack, there is a plethora of monitoring tools for sale.

Online marketplaces and social networks often have a trust and safety team. The trust and safety team helps protect the platform from scams, fraud, and malicious actors. To detect these bad actors at scale requires building a system that classifies every transaction on the platform as safe or potentially malicious. Since every social platform has to build something like this, Smyte decided to engineer trust and safety as a service. Josh Yudaken joins the show today to discuss how Smyte engineered its platform to provide machine learning models for any organization that wants to take advantage of Smyte for its trust and safety. The tools we discuss include Kubernetes, RocksDB, and Kafka, and Smyte is solving some problems that have not been solved before, so this is a great episode for anyone interested in data engineering or fraud detection–or how to use cloud services and open source tools in unique ways.

Every song you hear on the radio is written with a computer. Computer musicians mostly use synthesizers and samples to compose these songs. A sample is a snippet of recorded sound, sometimes taken from a songs, a movie, or another source. The more samples a musician has access to the better. SampleFocus is a platform where musicians upload and download samples to build songs with. Daniel Trostli is the engineer who is building SampleFocus, and he joins the show to discuss how people use samples to write music and how he built SampleFocus. I met Daniel at the Launch Scale event, a conference about building businesses, and when I was having lunch with Daniel, he told me about working on this personal project full-time, and some of the different business models he was exploring. It’s a great episode for anyone who is thinking about turning a side project into a business.

In the last five years, companies started hiring data engineers. A data engineer creates the systems that manage and access the huge volumes of data that are accumulating on cheap cloud servers. As the saying goes, “it’s more expensive to throw out the data than to store it.” Pete Soderling joins the show to discuss the rise of the data engineer, and how that role interacts with data scientists, software engineers, and machine learning experts. Pete runs Hakka Labs, a site that aggregates material for data science and data engineering. He also organizes DataEngConf, a conference for data engineering.

A huge percentage of online advertisements are never seen by humans. They are viewed by bots–automated scripts that are opening web pages in a browser and pretending to be a human. Advertising scammers set up web pages, embed advertisements on those pages, and then pay for bot traffic to come and view those advertisements. This aspect of the internet is bizarre and alarming. Think about it–how much of the Internet economy depends on online advertising? A lot! How many of those advertisements are consumed by robots? Some estimates say as much as 80%. The more time I spend looking at the online advertising industry, the more perplexed and curious I become. Augustine Fou also has this curiosity. He got a PhD at MIT in material science and engineering–but today he spends his time researching advertising fraud and working as an independent consultant.

For many years, software companies have been breaking up their applications into individual services for the purpose of isolation and maintainability. In the early 2000s, we called this pattern “service-oriented architecture”. Today we call it “microservices”. Why did we change that terminology? Did the services get smaller? Not exactly. Jonas Boner suggests that the movement towards cloud and the increased prevalence of mobile changes how we look at these services–so much that we needed to change the terminology necessary to even talk about them. And once the conversation has shifted to “microservices”, what steps do we need to take to implement them properly? The reactive manifesto is a collection of principles for how to build applications. When the reactive manifesto is applied to the idea of microservices, we get reactive microservices, which Jonas and I discuss in today’s episode.

Some tasks are simple, but cannot be performed by a computer. Audio transcription, image recognition, survey completion–these are simple procedures that almost any human could execute, but the machine learning models have not gotten consistent enough to do them accurately. Scale is an API for human labor, created by Lucy Guo and Alexandr Wang. Similar to Amazon Mechanical Turk, Scale sends small, simple tasks to workers who can complete those tasks. Scale provides an interface that is easy for developers to use, unlike Mechanical Turk, which requires a dashboard. Similar to how Stripe allows developers to build software off of payments systems easily, Scale allows developers to build human-driven, manual tasks into their code–which unlocks a wide range of potential applications, which Lucy and Alexandr discussed with me.

Over the next few years, bots will pervade our lives more and more. These smart, conversational text interfaces provide a new way of engaging with the computer systems that we have been mostly interacting with through web and mobile app interfaces for the last decade. Bots are a necessary complement to the voice interfaces of the future, because we don’t always want to talk to the computer, and natural language processing is not yet good enough to always translate our vocal request accurately. Bots are not toys, they aren’t trivial, and they aren’t going away any time soon. Jon Bruner is the host of the O’Reilly Bots Podcast, a show that has quickly become one of my favorites because it covers both theoretical and applied artificial intelligence, while remaining approachable to listen to. Jon hosts it with Pete Skomoroch, who works on a bot startup full time. Jon and Pete were also organizers of O’Reilly Bot Day, which I enjoyed tremendously and got a lot of knowledge from.

Mesos and Kubernetes are tools for distributed systems management. Kubernetes is built with an emphasis on running services, whereas Mesos is commonly used for a wider variety of workloads, including data infrastructure like Spark and Kafka. Mesos can also be used as a platform to provide resource management for Kubernetes. Dharmesh Kakadia is the author of Apache Mesos Essentials, and has spent time studying both Mesos and Kubernetes. He joins the show to explain the strengths of both platforms, and how they can be used together. In addition to dissecting Mesos and Kubernetes, we had a wide-ranging discussion, from cloud providers to software architecture to business strategy. *** If you are in the Bay Area on January 11th, come to the Software Engineering Daily meetup. There will be awesome speakers, food, and a positive atmosphere. Find more information on softwareengineeringdaily.com or at the Software Engineering Daily meetup page.

Advertising fraud is easy, legal, and extremely profitable. A fraudster can set up a website, scrape content from the internet, and run programmatic advertisements against that website. The fraudster can then purchase bot traffic. Those bots will visit the page, consume advertisements, and return profit to the owner of the page. In a past life, Shailin Dhar worked for a company that set up these types of advertising fraud schemes. He was fascinated by the industry in the same way that plenty of people get fascinated with the fast-moving market dynamics that are enabled by modern software. But over time, the novelty wore off and Shailin realized how big the fraud problem is and how much it was hurting people. In today’s episode Shailin and I discuss how bots and poorly aligned incentives lead to systemic failures and significant financial loss for brands who purchase advertising.

Cloud Foundry is an open-source platform as a service. Companies use Cloud Foundry as a control plane to deploy and manage applications. It provides abstractions for microservices and continuous integration. Cornelia Davis joins the show to discuss Cloud Foundry. I interviewed her at the DevOps Enterprise Summit, where enterprises share their stories of improving their culture and their technology stack. Cornelia explained how and why a large enterprise adopts Cloud Foundry, and why they often hire a company like Pivotal to help them as they migrate their infrastructure and change their processes.

Caching is a fundamental concept of computer science. When data is accessed frequently, we put that data in a place where it can be accessed more quickly–we put the data in a cache. When data is accessed less often, we leave it in a place where the access time is slow or expensive. Netflix has a huge variety of data, and a huge variety of access patterns for how that data gets retrieved from storage. In today’s episode, Scott Mansfield gives an overview of Netflix’s caching architecture, including EVCache, the ephemeral, volatile cache built for Netflix’s cloud architecture. As with other episodes about Netflix architecture, this show is a deeply technical case study.

Software Engineering Daily was started a year and a half ago, based on what I learned from my podcasting experience on Software Engineering Radio. Last week, I interviewed Robert Blumen, the editor of Software Engineering Radio, about how that podcast is produced. In today’s episode, Robert interviews me about this podcast. If you are thinking about starting a podcast about software engineering, this podcast will be useful to you. We discuss the processes behind preparing for a show, how to conduct an interview, why people listen to podcasts, and advertisers. I also give some of the history behind Software Engineering Daily–why I started it and what my goals are.

When you are working on a program, a lot of things are going through your head. In some sense, you become part machine when you are programming. Learnable Programming is a concept that facilitates this, by showing developers what the computer is doing in real time, before compiling. In this episode, Josh Varty, co-founder of Code Connect Inc., talks to Edaena Salinas about incorporating concepts from Learnable Programming into Visual Studio and C# by showing developers what the computer is thinking while they are typing. Code Connect lets a developer immediately see what value a variable is taking, and provide test values to troubleshoot a portion of code to understand how different code paths are going to work. This is instead of having to compile and run the full program. If you have ever thought about making developer tools for an IDE or a programming language, this episode provides some great information.

When a human passes away, we create a tombstone as a memorial. Friends and family visit a grave to remember the times they had with that person while they were still alive. Memorial bots are another way to celebrate the life of someone who has passed away. A memorial bot is created by taking the messages sent by a deceased person and passing it through a machine learning model in order to make a bot that replicates the deceased person. Eugenia Kuyda is the CEO of Luka, a company that builds AI products. When her friend Roman Mazurenko suddenly died, she worked with her team to make a bot that replicates his speech patterns. In our interview, we discussed memorial bots, deep learning, and the product Luka is working on–Replika, a personal AI friend for anyone.

When you are deciding who to marry, you are using an algorithm. The same is true when you are looking for a parking space, playing a game of poker, or deciding whether or not to organize your closet. Algorithms To Live By is a book about the computer science of human decisions. It offers strategies for how to think through everyday life like a computer scientist. Brian Christian has a background in computer science and philosophy, and is an author of Algorithms to Live By. He joins the show to explain how the same algorithms and data structures we use for our computer programs can be applied to the real world.

There are multiple paths to constructing a piece of software from start to finish. An individual programmer can build an entire product from scratch in a couple days. A giant corporation can commission a project and delegate responsibilities to hundreds of people. An open source community can use the wisdom of the crowds to efficiently build an operating system. Today’s episode is about another path for building software, and my own experience traveling that path. I designed and contracted an app called Adforprize that is now available on the iOS app store. Designing an app and contracting it out to an engineer to build is a worthwhile exercise for most people in the tech industry–whether you are an engineer or not. This post has two main arguments: As with the previous monologue episodes such as 10 Philosophies for Engineers and You Are Not a Commodity, there are no advertisements on this episode. Please email me any feedback you have–[email protected] Everyone has an app idea, and many of them are pretty good. You have probably had a conversation with a friend where one of you says “hey, wouldn’t it be great if there was a 2-sided marketplace for bacon?” Or “hey, wouldn’t it be great if there was an app where users could take a selfie and instantly see what celebrity they resemble the most?” These things would be cool! But most ideas end with the ideation. Who has the time to implement these things? Most of us are busy at work, where we spend our time and mental energy dealing with the technical complexity of our jobs. If you are a “smart creative”, when you get home from your job, you probably don’t want to spend your leisure hours reading iOS documentation and slowly learning to write the necessary code for your bacon marketplace app. Smart creatives who have a busy full-time job often have a hobby with more immediate gratification, like playing an instrument or blogging. Some of us also write apps on the side, but if you don’t already have all the requisite skills for writing an app, the prospect of reading tutorials and documentation can make the feeling of ever publishing an app seem so distant. There are many people listening to this who are better at sales, marketing, design, or product management than they are at programming–they want to build an app. One common narrative is that if these people want to see something built, they need to learn to program, and that’s not true. There is a more fun, fulfilling way to build an app than learning to code in Swift or React Native–contracting out the programming to someone else. Designing an app is a fun exercise. Building a prototype with tools like Sketch, InVision, and Keynote is free and frictionless. Most people listening to this probably spent some time drawing in KidPix or Microsoft Paint or PhotoShop when they were a kid. Most of us know how to draw at some basic level, because we spend all day seeing and understanding images–drawing is second nature to anyone who can understand shapes. Similarly, we all know how to design apps because we spend all day using them. We understand the purpose of a button or a slider or a swipe. But most of us have no idea how to configure the code around this user experience–the buttons and sliders and TextViews. Along with the design process, the business ideation process is also fun. It’s fun to imagine business models and viral loops and ways to drive user engagement. But without the capability to build the app that embodies your business ideas, it is less fun to think about those ideas. And again–most of us don’t relish the idea of struggling through the compiler errors and Stack Overflow posts that an app implementation requires. We’ve all become product designers. We are using our phones all day–most of us have naturally developed a good intuition for how an app should function. Product design ability is naturally acquired by being steeped in technology all day. If you are an employe...

Eight years ago, MongoDB was an internal project at 10gen, a company that was trying to build a platform-as-a-service out of open-source components. The team at 10gen realized that the platform-as-a-service play would be too complex, and difficult to build. Since MongoDB was the most valuable component of that project, they narrowed their focus to this new document-oriented database. In today’s episode, MongoDB CTO Eliot Horowitz describes the history of MongoDB the open source project, as well as the company–which recently released a managed cloud service. Eliot explained how the company has architected the MongoDB Atlas service on top of AWS, and why developers often want a managed service as their database rather than managing database servers themselves. Full disclosure–MongoDB is a sponsor of Software Engineering Daily.

Four years ago, I started volunteering for a popular podcast about software–Software Engineering Radio. For the next two years, I learned about the process of making a quality podcast about engineering. With its emphasis on preparation, timeless engineering principles, and attention to the listener, Software Engineering Radio continues to be one of the most popular shows about engineering. Software Engineering Daily is my effort to bring the quality of SE Radio on a daily basis. In today’s show, I talk to Robert Blumen, the editor of Software Engineering Radio. Robert is responsible for guiding SE Radio, and without my experience producing shows alongside him, Software Engineering Daily probably wouldn’t exist. If you listen to SE Daily, you should definitely subscribe to SE Radio. I still produce content for SE Radio because the process and the community help me get internal feedback.

C# .NET is the framework that is most often used to write software for the Microsoft Windows operating system. For many years, the C# .NET framework was closed source, but Microsoft’s recent push towards open source has led to the creation of .NET Core, a fork of C# .NET composed of a small subset of features from the original C# .NET stack. This episode takes us through a history of .NET, with two program managers who have worked on .NET for many years–Immo Landwerth and Lee Coward. We also explored the present and future of .NET–discussing .NET Core and .NET usage on operating systems other than Microsoft Windows, which is another recent development.

Biology research at Microsoft is focused on three main areas: molecular programming, synthetic biology, and stem cell biology. At the intersection of biology and computing there are implications for health, medicine, and efficient computing techniques. The field of Biological Computation is in its early days, and there is still lots of work to be done. Colin Gravill works in the computational science group at Microsoft Research, and today he explains how Microsoft is investigating biology with the same curiosity and pragmatism that Microsoft investigates computer science. I’d love to do more shows at the border between software and biology–if you have suggestions, please send me an email. [email protected]

Rust is a systems level language that is built to prevent crashes and eliminate data races. A language like C++ gives you high speed and lots of control, but it is easy to have segfaults, data races, and other problems if you aren’t careful. On this spectrum of control versus safety, we can plot other languages like Java, Go, and Haskell–but none of these languages have the unique feature set of Rust. Rust’s concurrency model is built on the foundation of ownership–a piece of data can only be owned by a single thing at a time. In today’s interview, Alex Crichton explains ownership, and the other abstractions that give Rust its unique model of concurrency.

Microservices are a widely adopted pattern for breaking an application up into pieces that can be well-understood by the individual teams within the company. Microservices also allow these individual pieces to be scaled independently and updated in isolation. Past Software Engineering Daily episodes have covered the microservice architectures of Twitter, Netflix, Google, Uber and other companies. In today’s episode, I sit down with Rafael Schloming, who is building tools for microservices at Datawire.

There is a hierarchy of ways to access and store data in a computer system. The cheapest, slowest way to store and retrieve data is disk. On the faster end, we have memory. As we architect systems with increasing complexity, we have additional considerations–network latency, transient compute nodes, and numerous caching layers. Apache Geode is a distributed, in-memory system for the JVM. It provides fast data storage and retrieval. Swapnil Bawaskar is an engineer who works on Geode, and he joins the show today to explain the architecture of Geode, and how Geode is different from other in-memory systems that get used for caching, such as memcache or Redis.

Many programmers listening to this podcast are working at a big company, and they would prefer to be running their own software business. Patrick McKenzie has been writing about this topic for several years on his blog Kalzumeus.com. Almost a decade ago, he was working as an enterprise developer at a large company in Japan. Over time, his side projects started making enough money to justify his leaving to work on them full-time. Starting a small successful software business is not about luck, or taking big risks, or having your muse deliver some brilliant insight. It is about studying the markets where you can build software that can deliver value. Patrick and I discuss this topic, as well as Stripe–where he now works today. He sold his small businesses to go work full time on Stripe Atlas, so his story of transitioning back to a big company is fascinating as well.

Botnets have a massive influence on the Internet. As we have seen recently with the Mirai Botnet, IOT bots can take down companies as big as Netflix. In our recent episodes about advertising fraud, we’ve talked about how bots are being used to take billions of dollars of revenue from advertisers. Derek Muller is one of those advertisers who has spent money on ads and gotten nothing but fake traffic in return. Two years ago, he posted a video on YouTube about his experience purchasing advertising traffic on Facebook, and getting Likes from accounts that were clearly from fake accounts. Derek is the host of Veritasium, an awesome YouTube channel about science, truth, and technology–so we also talked some about how he built a successful YouTube business.

Slack is a chat client that has reached wide adoption. The rise of Slack has coincided with the rise of chatbots. A chatbot is a simple, conversational interface into a computer program that may have simple functionality, like telling you some simple statistics, or more complex functionality, like helping you manage your continuous integration pipeline. Bot design and engineering is a new field, and a vast array of resources and techniques are available for developers looking to hack on it. Amir Shevat is the director of developer relations at Slack, and is responsible for communicating with developers about the best ways to build bots for Slack. After seeing his talk at O’Reilly Bot Day, I had a number of questions about where we are with bots today and where we are going. I enjoyed this conversation with Amir, and if you aren’t already convinced that bots are an important platform for engineers to understand, this conversation will convince you.

Large technology companies have no shortage of data. But raw data itself does not provide a competitive advantage. Many companies are bottlenecked by a shortage of data scientists who can query that data effectively. This results in an organizational dysfunction where people lining up to ask questions of the data science team are unable to move as fast as they want to. Tomasz Tunguz saw this when he was working at Google, and he continues to see it today from his position at Redpoint, where he works as a venture capitalist looking at companies. The problems with the “data supply chain” led him to write “Winning With Data”, a book about how technology companies can successfully operationalize, explore, and act on their data. We’ve done many shows about the interactions between software engineers, data engineers, and data scientists. This episode is a great complement to previous episodes, providing a holistic view into the way data moves through an organization–and I highly recommend Tomasz’s book “Winning with Data”.

Amazon Web Services changed the economics of building an internet application. Instead of having to invest tens of thousands of dollars up front for hardware, developers can pay for services over time as their application scales. As AWS has grown to be a gigantic platform, the documentation about how to use cloud infrastructure has become insufficient. As an answer to this, Joshua Levy initiated The Open Guide to Amazon Web Services, an open-source collection of resources available on github. Joshua has experience at a variety of companies, including Viv, the conversational interface that was recently acquired by Samsung. In our conversation, Josh brought his years of experience to the table to explain the risks, benefits, and alternatives of Amazon Web Services. Josh has become a friend over the last year, and if you get the chance to have a conversation with him, I highly recommend it. He will be at the re:Invent conference in Vegas at the end of this month (November) and if you are interested in talking to him about the AWS Open Guide, or anything else, reach out to him.

Bot Day was an O’Reilly conference featuring talks from some of the leading figures in the bot industry. Before I attended Bot Day, I knew there were lots of applications for chatbots, but I didn’t realize how good the tooling has gotten–it’s very easy to get started with chatbots today so if you are a developer and you have a basic idea, I suggest checking out the videos from Bot Day, as they are a great resource for how to get started. When I was at the conference, I had a number of conversations with Edaena Salinas, who hosts The Women In Tech Show. We also caught up a few weeks after the conference to discuss our reflections on Bot Day.

The blueprint for a typical startup involves investing heavily in cloud services–either from Amazon, Google, or Microsoft. The high costs can quickly eat away at all of the money that startup has raised. In today’s episode, Avi Freedman outlines some of the infrastructure mistakes that can set back a company severely–cloud jail, hipster tools, and lack of monitorability. Avi is the CEO of Kentik, a network traffic, performance, and security visibility company, so he gets lots of perspective on the infrastructure of the average company. We also discussed the business of running a software company, contrasted with the life of a poker player. Avi and I both have some experience playing high stakes poker, so it was a great opportunity to get his perspective on the parallels between the two fields.

When Facebook acquired Instagram, one of the first systems Instagram plugged into was Facebook’s internal spam and fraud prevention system. Pete Hunt was the first Facebook engineer to join the Instagram team. When he joined, the big problems at Instagram were around fake accounts, harassment, and large volumes of spammy comments. After seeing the internal Facebook spam prevention tools clean up Instagram, Pete decided to start a company to build products that would allow this type of spam and fraud prevention as a service. Smyte provides protection against bad actors on the Internet. Complex marketplaces like Tilt, Meetup, and TaskRabbit plug into Smyte to filter their transactions for suspicious behavior. In this episode, we talk about how this type of behavior manifests, and the event-driven software architecture that the team at Smyte has built.

Ben Northrop was sitting at a tech conference, listening to a presentation about a new piece of technology, when he was struck by the sense that history was repeating itself. For the twenty years that Ben has worked as a software engineer, he has been hearing about new technologies that claim they will be able to disrupt everything, and he has relentlessly been keeping up as best he can with the latest trends. Of course, Ben is all of us. Nobody can keep up. As a daily software podcaster, I think I am as well positioned as anyone to be tuned into the breadth of the software engineering ecosystem–but I am constantly behind as well. In his blog post “Reflections of an Old Programmer”, Ben theorized about the causes of this feeling of fatigue, and the constant struggle to keep up with the world of software engineering. His post contains enduring lessons that will be of value to new programmers and will feel familiar to programmers who have been around for awhile.

Jeff Bezanson’s university thesis described the motivation for a new programming language. He discussed the shortcomings of “array based programming environments” and his desire to create a more performant language with the best qualities of Lisp, Python, Ruby, Perl, Mathematica, R, and C. The Julia Language is a high performance language designed to suit technical users that crave the flexibility to pick their own notation. The language has support for a wide variety of operators, allowing scientists to create domain specific equations that can be elegantly expressed as code. Jeff Bezanson joins the show to discuss his motivations for Julia, and how the language has evolved since he started working on it.

Legacy code is code without automated tests. Most companies have lots of legacy code, and most developers don’t like working on legacy code. Why is that? What is it that makes legacy code so difficult to work with? And why does a large amount of legacy code slow down an organization so severely? Andrea Goulet is the CEO of Corgibytes, a consultancy that specializes in remodeling legacy codebases. Andrea’s company developed out of an observation that there are developers who actually love working on legacy code. Since founding Corgibytes, she has become an expert on legacy code and published a recent article about Building Technical Wealth. The article presents ways that companies can confront the realities of legacy code.

Indie Hackers is a website that profiles independent developers who have made profitable software projects, usually without raising any money. These projects make anywhere from a few hundred dollars a month to more than $100,000 as in the case with park.io, one of the services profiled by Indie Hackers. Courtland Allen is the creator, engineer, and interviewer behind Indie Hackers. For each business that is profiled by Indie Hackers, Courtland conducts a short interview with the founder. Courtland joins the show to discuss the changing trends that are making it easier to bootstrap a software business if you are a capable developer–or even if you are a nontechnical person who understands how software works. Since Courtland and I are both in the business of interviewing engineers, we had a lot to talk about, and this is a fantastic episode.