Software Engineering Daily

When you visit a web page, that web page can write data to a file on your computer, known as a cookie. Scripts on that page can also read from your cookie file to understand where you have been in the past. All of this data about you is getting shared between advertising companies like Google, Facebook, and AppNexus. Ghostery is a browser extension that allows you to limit what you share with these online tracking companies. Larry Furr develops products at Ghostery, and on this episode he takes us through the process of how we are tracked through the Internet. We also explore the topic of ad fraud, which is a theme we will continue to explore on SE Daily. If you have any information on ad fraud, or have recommendations for guests, please email me at [email protected]

Chat bots are your newest co-worker. Slack, HipChat, and other chat clients allow developers and other team members to communicate more dynamically than the limits of email. Companies have started to add bots to their chat rooms. These bots can give you technical information, restart a server, or notify you that a build has finished. Jason Hand is the author of ChatOps: Managing Infrastructure in Group Chat. He joins the show today to discuss how ChatOps improves development and operations by centralizing lots of functionality in group chat. Edaena Salinas is the host for today’s show. She also hosts the excellent Women In Tech Show–a podcast we highly recommend. Since we are on the subject of bots–we want to thank O’Reilly Media for recently providing Software Engineering Daily a ticket to Bot Day.

Online advertising is heavily affected by a set of delusions and fraudulent practices that few people in the adtech industry have an interest in stopping. This is the curious, perverse nature of the world of online media. Some of the online advertising we see fulfills its job, when an ad successfully conveys a meaningful message from a marketer to a consumer. But there are reasons to be extremely skeptical of the way that online advertising works. In this episode, and several more to come, Software Engineering Daily will be diving deep into the adtech business to understand the bot fraud and mass psychosis that bolsters our tech media giants. Bob Hoffman is the author of The Ad Contrarian, a long-running blog about advertising. After selling the advertising agency he was running, he left the industry. In today’s episode, Bob discusses his experience in the ad industry, and the delusions he saw when he was working in it.

Music collaboration software that works over the Internet is a software challenge that has not been fully tackled. On today’s Internet, users collaborate intensively on programming projects, journalism, and other projects, but the tools for collaborating on music online have not yet become popular. Blend.io is a social music collaboration tool–a github for musicians. I have been using it myself and enjoying it tremendously. Today’s guest Alan Grow joins the show to talk about how to build a social version control system for musicians, as well as where music is headed.

Advertising fraud takes billions of dollars out of the economy every year. We don’t know exactly how much money is being lost, because we don’t know what percentage of Internet users are bots. Are You A Human is a company designed to solve that exact problem, and provide a service for verifying whether a user is real or automated. Ben Trenda is the CEO of Are You A Human. In this episode, we discuss some of the ways that bots pose as humans–clicking on ads, pretending to read ads, and defrauding marketers and publishers. We also talk about how Are You A Human combats this. This episode is part of a series of episodes about advertising fraud. If you know anything about the area, send me an email: [email protected]

Blockchain technologies like Bitcoin and Ethereum have not impacted the lives of most consumers today. The theoretical breakthroughs that blockchain enables will eventually happen–I will be able to pay 1 cent to a knowledge worker in Africa without having to pay a 5 cent transaction fee. My servers will be able to pay other servers for small compute jobs. We will have decentralized versions of sharing economy systems like Airbnb and Uber. But first there are many infrastructure and policy issues to sort through. The technology is not quite there yet, and people running even simple mining operations are threatened by our current legal framework. Coin Center is an organization dedicated to education of policymakers and media about blockchain technology. Peter Van Valkenburgh directs research at Coin Center, and joins the show to talk about the issues that stand in the way of that utopian dream of micropayments, IoT, and other blockchain miracles.

Kafka is a distributed log for producers and consumers to publish messages to each other. We’ve done many shows about Kafka as a key building block for distributed systems, but we often leave out the discussion of the complexities of setting up Kafka and monitoring it. Kafka deployments can be a complex piece of software to manage. Tom Crayford is an engineer at Heroku, where he helped engineer the recent Heroku Kafka product, which is a managed version of Apache Kafka. Our conversation explored the use cases of Kafka and how to build Kafka as a cloud service at scale. For more information about Heroku Kafka, check out an upcoming webinar. Full disclosure: Heroku is a sponsor of Software Engineering Daily. That said, this is a topic I am genuinely interested in–it is often difficult to get cloud providers to talk in detail about how they are architecting their services.

Netflix has a highly interactive user interface. As I move my mouse around the page, hovering over titles and inspecting movie descriptions, there is a lot going on under the hood. One component of this UI is RxJS, a library for building reactive JavaScript. Reactive programming uses the observer pattern to create objects that emit streams of events. We can compose these streams together to create elegant abstractions. Reactive programming may seem confusing at first, but it can simplify certain patterns that may be hard to describe with imperative programming. Ben Lesh, a senior software engineer at Netflix, joins the show to explain why reactive programming is useful, and how RxJS is used at Netflix.

Bot fraud, the New York tech scene, RethinkDB and open source; these topics and more are discussed in today’s episode. Two of the most popular guests return to the show to explore a variety of topics. Ben Halpern is the creator of The Practical Dev, a massively popular Twitter account and blog that you may recognize from its parody O’Reilly book covers. Haseeb Qureshi is an engineer at Airbnb, and a blogger who is well known for his writings on salary negotiation and coding boot camps. This episode was an experiment–if you like it, please let me know what you think.

Google Compute Engine is the public cloud built by Google. It provides infrastructure- and platform-as-a-service capabilities that rival Amazon Web Services. Today’s guest Joe Beda was there from the beginning of GCE, and he was also one of the early engineers on the Kubernetes project. Google’s internal systems have made it easy for employees to spin up compute resources, but it was not a simple task to make this internal cloud consumable by the public–not to mention competitive with AWS. In order for a cloud provider to be successful, it needs to offer self-healing, self-managing infrastructure that can run microservices.

Cloud computing was something much different in 2011, when Brian Gracely and Aaron Delp started The Cloudcast, a podcast I listen to on a regular basis. The Cloudcast features technical discussions about cloud infrastructure technology, and one of the most recent shows was a monologue by Brian Gracely where he explained his perspective on the industry rumblings about a Docker fork. The utility of a container for so many different purposes leads to different organizations having differing preferences for what use case is optimized for. The impression that I took away from this conversation, as well as the next episode that will air with Joe Beda, is that the diverse opinions and products in the container and orchestration ecosystem is quite healthy. Brian does a great job explaining his perspective on The Cloudcast, and he discusses his beliefs further in this episode.

The Simpsons is a classic, beloved television show. The scripts of The Simpsons have been made publicly available, and include dialogue, location, and character information. Todd Schneider used these scripts and other information sources as a corpus to analyze The Simpsons and find interesting statistics–such as who the most important supporting characters were, and how the ratings of the show have trended over time relative to other TV shows that have declined in ratings. Todd works at Genius in New York, and I took the opportunity to ask him other questions about pop culture, given that Genius is a place where pop culture and data collide. If you haven’t seen the site, it started out as a wiki for rap music, and quickly grew to more aspects of pop culture. Todd also has a variety of other side projects, involving betting markets, analysis of taxis vs Lyft and Uber, and a systematic study of the wedding section of the New York Times.

Data engineering is the software engineering that enables data scientists to work effectively. In today’s episode, we explore the different sides of data engineering–the data science algorithms that need to be processed and the implementation of software architectures that enable those algorithms to run smoothly. The PANCAKE STACK is a 12-letter acronym that Chris Fregly gave to a collection of data engineering technologies including Presto, Cassandra, Kafka, Elastic Search, and Spark. In his current life, Chris travels around the world giving workshops on how to deploy and use the PANCAKE STACK. Before that, he was an engineer at Netflix, where he received an Emmy for Streaming Engineering Excellence.

When a user of a social network updates her profile, that profile update needs to propagate to several databases that want to know about such an update–search indexes, user databases, caches, and other services. When Neha Narkhede was at LinkedIn, she helped develop Kafka, which was deployed at LinkedIn to help solve this very problem. Using Kafka as an event queue, LinkedIn adopted the CQRS architectural pattern together with event sourcing. Event sourcing is an architectural pattern that allows changes to our application model to be represented as events. Each event is published to an event queue, and is pulled off of the queue by each of the various services that need to consume that event. Event sourcing and the related architectural pattern CQRS allow for a flow of information through an application that is easy to reason about, and has several other desirable properties. In today’s episode, Neha explains how to use Kafka for event sourcing and how related software patterns are improving the architectures of companies like Netflix and Uber. For more information, check out this Confluent blog entry.

The intent of the DevOps movement is to get organizations moving faster and more effectively by breaking down siloes, and improving communication. Gene Kim’s book The Phoenix Project illustrated this by telling the fictional story of a company adopting a DevOps mentality. Although that book was fiction, Gene is an experienced engineer, having worked as founder and CTO of Tripwire, a software company that makes security and compliance automation software. In his new book The DevOps Handbook, Gene presents a practical companion to The Phoenix Project. Together with his co-authors, Gene has written a guide for how to move an organization toward DevOps, and in this episode we explore some of the topics from his book. Check out an excerpt from Gene’s book here: DevOps Handbook

At Netflix, developers write applications with a variety of requirements–from simple requests for a list of movies to more resource-intensive requests like a complex machine learning workflow. Netflix wants developers to be able to request the resources they need from a compute cluster and receive those resources on-demand, without thinking too much about the state of that pool of resources they are drawing from. At the cluster level, this means scheduling the application requests intelligently. Sharma Podila, a distributed systems software architect at Netflix, explains how Netflix has built a cloud native scheduling system on top of Mesos.

Building a monitoring system is a complex distributed systems problem. Events are produced from different points in an application and must be aggregated in order to form metrics. These events are often ingested by a time series database, which forms the backbone of our monitoring system. Theo Schlossnagle is the CEO of Circonus, where he has been working on architecting the company’s monitoring software for six years. In this episode, we talk about how to build a monitoring system and the requirements for the underlying time-series database, as well as what monitoring even is.

In order to move software updates from the development team to production, companies do a variety of things. Some teams might email files to each other or use FTP or even floppy disks. Most companies today at least use version control systems like Git together with separate servers for development and production. When code is ready to move to production, a build that is on the development server gets copied over to the production servers, and the production servers begin serving real users. This process is known as deployment, and over the last few decades companies have started deploying more rapidly (even “continuously”), leading to faster iterations and better feedback loops between the software development team and the users of the product. A particularly effective version of this workflow is known as continuous delivery. In today’s episode, David Rice from ThoughtWorks joins the show to give a short history of continuous delivery, and how continuous delivery actually looks in practice.

Kafka Streams is a library for building streaming applications that transform input Kafka topics into output Kafka topics. In a time when there are numerous streaming frameworks already out there, why do we need yet another? To quote today’s guest Jay Kreps “the gap we see Kafka Streams filling is less the analytics-focused domain these frameworks focus on and more building core applications and microservices that process data streams.” Jay is the CEO of Confluent, a company that is building Kafka technology, and he is one of the original authors of Kafka. We began with a brief history of why Kafka evolved to be the message broker of choice for so many data engineering stacks, then we got into a discussion of streaming–what is streaming? How does Kafka communicate with streaming frameworks? Our conversation concluded with a discussion of Kafka Streams and the future of the Kafka data platform.

Scala is a functional programming language built on the JVM. For more than a decade, this didn’t mean anything to front end web developers. More recently, ScalaJS has brought Scala to the front end. ScalaJS is a project that compiles any Scala program down to JavaScript–so that all of your Scala programs can run on the browser. Haoyi Li has worked on ScalaJS extensively and has written an online book about ScalaJS. If you are a front end developer looking for a safer way to write your web apps, you will like this episode–or if you are a fan of functional programming.

Software teams are traditionally composed of roles such as project manager, developer, QA, and manager. What happens if you throw out all of those titles, hire mostly engineers, and ask them to do whatever they think is best? That is the core idea behind Fred George’s idea of Developer Anarchy. In today’s episode, David Curry guest hosts an interview with Fred George. They talk about the structure of teams and the idea of developer anarchy. For a great complement to this episode, check out the Software Engineering Radio episode about developer anarchy.

Platform as a service can mean different things to different people. The most prominent feature of a PaaS is the ability to abstract away issues that every developer within an organization has to deal with. As an example, developers today don’t need to fear scalability and load balancing issues as much as engineers of the 90’s and early 2000s. We can develop our applications without worrying about the scaling that is going on under the covers. Sinclair Schuller is the CEO of Apprenda and he joins the show today to discuss what is involved in building a PaaS, particularly one targeted at enterprises. His company was started in 2007, just a few years after AWS got started. We also talked about the potential Docker Fork, for which Sinclair had useful business and engineering perspectives.

Docker containers changed how engineers think about building software, and the company most responsible for the widespread adoption of containers is Docker itself. Since containerization has caught on in the mainstream, companies like RedHat, Google, Huawei, and many other big players have built platform products that utilize Docker containers. Docker containers are the unit that many engineers use to deploy their applications, but the servers that run those containers are usually on infrastructure providers like Amazon, Google, or Rackspace. This raises the question–how will Docker the company make money? Having raised $180M in equity, Docker has plenty of time to find something that works, but there are suspicions in the open source community that aggressive changes to the open source Docker project are being made by Docker the company with the long-term objective of monetization. These changes–namely the default support for Docker Swarm–are not nefarious, but they might be less likely to occur of Docker were just an open source project, as opposed to an open source project being tended by a well-financed company. The New Stack broke the story about the potential Docker Fork, and Alex Williams and Joab Jackson of The New Stack join me on today’s episode. We go deep into the facts and speculations about why a fork might occur, the motivations of different key players, and the disputes within the community that have led us to this moment.

Africa has a huge reservoir of untapped technical talent. Tech companies have a huge amount of work they need skilled employees for. Andela recruits the most talented technologists across Africa, shapes them into world-class software developers, and places them with tech companies worldwide such as Google and Microsoft. Brice Nkengsa is the director of engineering at Andela, and he joins the show to discuss Andela. This episode touches on many of the themes of Software Engineering Daily–boot camps, scaling an organization, microservices, and the future of work.

After nine years at Microsoft, Scott Berkun left to become an author. One of his books on project management was read by Matt Mullenweg, the creator of the WordPress blogging tool that runs a large percentage of the internet (including Software Engineering Daily). Scott became friends with the WordPress founder, who is also the CEO of Automattic, a company that sells WordPress hosting and support. Around that time, Automattic was run as a completely flat company. Everyone reported to Matt. This was unsustainable, and eventually Matt decided that Automattic should try management, and he convinced Scott to take a break from his writing career to return to the world of software engineering as Automattic’s first manager. Scott accepted the role under the condition that he could write a book about the experience, and the result of that agreement is “The Year Without Pants: WordPress.com and the Future of Work”, which is a brilliant depiction of what it is like to work at a completely remote software company. Scott joins the show today to talk about his book and his career journey–from Microsoft, to writing, to WordPress, and back to writing. He also shares his thoughts on the workplace dynamics that we spend our days in at software companies.

Mark Zuckerberg may be the most powerful person in the world. At no other time in history has a single human had such fine-grained control over the most influential tool for media. Today’s guests are Michael Zimmer and Nick Proferes, the creators of The Zuckerberg Files, an index of every recorded word that Mark Zuckerberg has said in text, video, or audio. Why would someone create this? I was partly expecting Michael and Nick to be anti-Facebook, because The Zuckerberg Files sounds like a tool to hold the Facebook CEO accountable. In reality, Michael and Nick have a nuanced view of Facebook and Zuckerberg–they are Facebook users themselves, and they initially built The Zuckerberg Files as a way to portent product directions that Facebook might lean towards–particularly those related to privacy.

Scikit-learn is a set of machine learning tools in Python that provides easy-to-use interfaces for building predictive models. In a previous episode with Per Harald Borgen about Machine Learning For Sales, he illustrated how easy it is to get up and running and productive with scikit-learn, even if you are not a machine learning expert. Srini Kadamati hosts today’s show and interviews Andreas Mueller, a core committer to scikit-learn. Srini and Andreas discuss the background and implementation of scikit-learn and walk through some prototypical workflows for using it.

Containers have become the unit of infrastructure that many technology stacks deploy to. With the shift to containers, the attack surface of an application has changed, and we need to reconsider our security models; the resource allocation of our containers, the interactions between different containers on a single machine, and the big picture–how the external web may interact with our containers. Phil Estes joins the show to discuss container security, as well as the OCI, container orchestration, as well as other container related topics.

Security for the popular chat application Slack is a major focus for the company. A corporate Slack account is as valuable to a hacker as a corporate email account. In today’s episode, Ryan Huber and I talk through Slack’s approach to security–from philosophical discussions of how to company approaches security to the technical practices of logging and monitoring, and why Slack has a separate Amazon Web Services account just for the security team. Ryan works on security at Slack and has written a Medium post called Distributed Security Alerting, that I recommend checking out. If you haven’t heard it, you might also like the previous episode we did about Slack’s Architecture, with Keith Adams from Slack.

The role of “tech lead” is a combination of a software engineer, a project manager, and an architect. A tech lead might spend 30% of her time coding and 70% managing a project, resolving conflicts, and planning. In today’s episode, we discuss the structure of software teams, and when it might make sense to have a tech lead on your team, in contrast to the more common team structure of project managers, engineers, and engineering managers. Jeff Norris, an engineer from ThoughtWorks joins the show to discuss the idea of the tech lead. This is a great show for anyone in management who is looking for alternative team structures, and this episode might also appeal to engineers who are looking to transition to a role with more variety.

Devoxx4Kids is an organization that is inspiring children through robotics, programming, and engineering. Kids who attend a Devoxx4Kids workshop are exposed to entertaining and educational activities, such as writing a Minecraft mod or programming an Arduino. Arun Gupta is a software engineer and the president of the board for Devoxx4Kids USA. In this episode we discuss coding education for kids, and what happens at a Devoxx4Kids workshop. If you have a kid who you want to expose to technology, this episode has some great suggestions for how to do that.

Google builds cloud services for developers, such as PubSub, Cloud Storage, BigQuery, and Cloud DataStore. On Software Engineering Daily, we’ve done lots of shows about how these types of services are built. In this episode, we are zooming in on the interaction between the developer using a cloud service and the design and engineering of the client APIs. To build a useful cloud service, Google has to make a common way of interacting with that service from any programming language on any device. Jon Skeet is a longtime engineer at Google, and he joins the show to explain how Google uses generated code to make the creation of those APIs more streamlined. We talk about gRPC, protocol buffers, C#, and lots of other topics.

When the US government hacks its own citizens, The Electronic Frontier Foundation is often the best source of reporting to find out what laws the government has broken. When a change to the privacy policy of Google or Facebook is made, the Electronic Frontier Foundation is the best place to find out how that change in privacy exploits users. The Electronic Frontier Foundation provides legal defense and editorialism at the intersection of law and technology. Nate Cardozo joins the show today to discuss the mission of the EFF and his work at the organization. We have a wide ranging conversation, ranging from governments to corporations to Stuxnet to how the internal discussions at the EFF lead to the stances taken by the EFF.

Batch and stream processing systems have been evolving for the past decade. From MapReduce to Apache Storm to Dataflow, the best practices for large volume data processing have become more sophisticated as the industry and open source communities have iterated on them. Dataflow and Apache Beam are projects that present a unified batch and stream processing system. A previous episode with Frances Perry discussed how they work in detail. In today’s episode, Eric Anderson discusses Cloud Dataflow, a service from Google that lets users manage their data processing pipelines without having to spin up individual servers to run them on. Cloud Dataflow–like the “serverless” movement we have done several shows on–represents a growing shift towards cloud providers offering services that abstract away the operational challenges of managing compute nodes. If you have suggestions for topics in this area, please do send me an email.

A previous episode of Software Engineering Daily called “You Are Not A Commodity” received a lot of feedback, both negative and positive. The episode was a monologue I wrote about why engineers should build products on their own as a default career path, rather than work at a large corporation as a default career path. A reddit thread about the episode was almost entirely negative. Most of the emails I received about the episode were very positive. Other regular listeners wrote in and said they had mixed feelings. One of those listeners was Preethi Kasireddy, a previous guest of the show. In this episode, Preethi gives some counterarguments and questions the ideas that were presented in the “You Are Not A Commodity” episode. This will be a useful discussion for anyone who is thinking about their career path, and whether to go work for a company or do something on their own.

Unikernels allow us to specify the minimum features of an operating system we need to deploy our applications. We’ve had many shows about containers, which allow you to deploy your application on top of a segregated portion of an operating system. Unikernels are different because they can be deployed directly to bare metal or to a hypervisor. Idit Levine works on Unikernels at EMC. We had an interview a few weeks ago with her colleague Scott Weiss, and today’s conversation goes deeper into the motivations for Unikernels, the problems they solve, and the project she is leading called Unik, which seeks to add the degree of usability and adoption to Unikernels that Docker brought to containers.

Microsoft was the dominant technology company in the 1990’s, until it came under fire for anticompetitive practices. Internet Explorer was tightly coupled to the Windows operating system, which prevented Netscape Navigator–a competing browser–from reaching users on the dominant platform. This episode is about antitrust–what businesses can and cannot do in the name of competition, what the impact of Microsoft’s legal battles in the late 90’s was, and how the law might respond to potential technology monopolies in the near future–such as Facebook, Google, and Uber. Our guest Harry First is a professor of law at New York University School of Law. He wrote a textbook called “The Microsoft Antitrust Cases: Competition Policy for the Twenty-First Century”.Sponsors

Slack is a chat application that is rapidly growing in popularity. The focus of Slack is to create a polished, responsive tool for productivity that cuts down on the emailing, context switching, and useless meetings that take place at a typical enterprise. Keith Adams, the chief architect at Slack, joins the show to explain how those high level principles translate into engineering decisions. Keith previously worked for 7 years at Facebook, contributing heavily to the tools that make PHP easier to develop with at scale. Slack’s core product is also built with PHP, and Keith discusses the similarities and differences between scaling Facebook and Slack.

When a company switches the relational database it uses, you wouldn’t expect the news of the switch to go viral. Most engineers are not interested in the subtle differences between MySQL and Postgres, right? Uber recently switched from having Postgres as its main relational database to using MySQL. Evan Klitzke wrote a detailed blog post about the migration, and post got very popular for at least three reasons: If you are even slightly interested in distributed systems or databases, I recommend reading Evan’s blog post in detail.

Relational databases are used by most applications. MySQL, Postgres, Microsoft SQL Server, and other products implement the core features of a relational database in different ways. A developer who has never studied this space in detail may not know the differences between these databases, and in this episode we describe some tradeoffs that relational databases can make. Craig Kerstiens is an engineer at Citus Data, a company that makes scalable Postgres. We talk about the requirements for a relational database, ACID compliance, how different databases handle different distributed systems problems, and the recent blog post from Uber about the company’s switch from Postgres to MySQL.

Learning to program is about self-driven exploration. Universities help guide you, coding boot camps provide a rigorous environment to work in, and online coding courses provide content for you to study. But none of this will turn you into a great programmer unless you have the drive to improve and the curiosity to explore. The Recurse Center is a place where people can come to become better programmers. Nick Bergson-Shilcock is a founder of The Recurse Center and he joins the show to discuss how it works and why he started it. This episode is a continuation of our exploration of coding boot camps, online courses, and universities, which are the pillars of programming education. The Recurse Center presents another unique model for improving as a programmer.

Facebook users provide lots of information about the structure of their relationship graph. Facebook uses that information to provide content and services that are expected to be important to users. If Facebook knows who the most important people in my life are, Facebook can use that knowledge to serve me content that is more relevant to me. Jon Kleinberg studied Facebook network structures together with Lars Backstrom, creating a paper called “Romantic Partnerships and the Dispersion of Social Ties: A Network Analysis of Relationship Status on Facebook”. The goal of this study was to find a metric that could help rank the strength of relationships on Facebook, and the results have implications for sociology as well as the way that we think about building social networks. Jon is a professor of Computer Science at Cornell, and wrote the textbook “Algorithm Design” which I used in college, so it was a privilege to get to talk to him.

Drones will become a central part of our lives. Drones are delivering packages, surveying cell phone towers, providing wi-fi, or fertilizing crops. Drones are assisting humans in dangerous work, and serving as an entirely new computing platform, providing services that were previously nonexistent. Airware is a company that is building a full-stack drone platform. In this episode, Buddy Michini takes us through the software architecture of a drone. Airware’s drones have two operating systems–one for the real-time flight critical aspects and one for application developers who want to build their own software for drones.

Machine learning can be used to generate music. In the case of Feynman Liang’s research project BachBot, the machine learning model is seeded with the music of famous composer Bach. The music that BachBot creates sounds remarkably similar to Bach, although it has been generated by an algorithm, not by a human. BachBot is a research project on computational creativity. Feynman Liang created BachBot using Python machine learning tools to build a long-short term memory model. Our conversation explores artificial intelligence, music, and his approach to this research project.

You have probably read a news article that was written by a machine. When earnings reports come out, or a series of sports events like the Olympics occurs, there are so many small stories that need to be written that a news organization like the Associated Press would have to use all of its resources to write enough content to cover it all. Wordsmith is a tool for automated content generation, and today’s guest Robbie Allen is the CEO of Automated Insights, the company that makes Wordsmith. He talks today about the wide range of uses for automated content, as well as how to engineer a product that takes data from a spreadsheet and turns it into a human-readable sentence. Robbie is also speaking at the O’Reilly Artificial Intelligence Conference in New York, September 26-27.

The Haskell programming language is often thought of as an academic tool, useful for teaching students about monads and functors, but not much else. But there are advantages to using Haskell as a production backend language. Better is a company built with Haskell on the backend, and Carl Baatz wrote a blog post detailing his experiences using Haskell. He joins the show to give a detailed explanation of why a company might want to use Haskell on the backend, from software architecture, to testing, to hiring.

Google’s infrastructure has been the source inspiration for research papers, software projects, and entire companies. Google pioneered the idea that we care less about the individual machines we are running our applications on, and more about the applications themselves. Containers are the abstraction we use to separate the concerns of the application from those of the underlying hardware. CoreOS is an operating system built with this paradigm shift in mind. In a data center, the main job of the operating system is to be a platform for containers to run smoothly on. Brandon Philips is the CTO of CoreOS and he joins the show to explain what CoreOS does differently to power the applications that get deployed on top of it.

Research in artificial intelligence takes place mostly at universities and large corporations, but both of these types of institutions have constraints that cause the research to proceed a certain way. In a university, basic research might be hindered by lack of funding. At a big corporation, the researcher might be encouraged to study a domain that is not squarely in the interest of public good–such as targeted advertising. Oren Etzioni is the CEO of the Allen Institute for Artificial Intelligence, and in this episode we discuss AI research–from the doomful premonitions of Nick Bostrom to the unbridled optimism of Ray Kurzweil, as well as the realities of how AI research actually proceeds. Projects at the Allen Institute are defined and structured to solve problems in an intelligent, scalable fashion, so that engineering can proceed steadily from the local maxima of a problem domain to the global maxima. The Allen Institute seeks to bridge the gap by providing ample funding for open source AI research for the common good. Oren is also speaking at the O’Reilly Artificial Intelligence Conference in New York, September 26-27.

Uber has a software architecture with unique requirements. Uber does not have the firehose of user engagement data that Twitter or Facebook has, but each transaction on Uber is both high value and time-sensitive. Users are paying for transportation that they expect to be available and reasonably close by. When Uber’s system is trying to match a rider with a driver, availability is favored over consistency. It is important that the rider can always get some driver, even if it is not the best driver. Ringpop is a system built at Uber to provide scalable, fault-tolerant, application layer sharding. Ringpop consists of a membership protocol, consistent hashing, and forwarding capabilities. Jeff Wolski is a software engineer at Uber working in Ringpop, and he joins the show to explain how Ringpop brings coordination to distributed applications.

Kubernetes is a cluster management tool open sourced by Google. On Software Engineering Daily, we’ve done numerous shows on how Kubernetes works in theory. Today’s episode is a case study in how to deploy Kubernetes to production at a company with existing infrastructure. GolfNow is a fifteen year-old application written in C# .NET. It is a successful, growing business that is a division of NBC Sports. As GolfNow has grown, it has encountered scalability issues, and the engineering team at GolfNow decided to move its entire monolithic infrastructure to microservices running in Docker containers, managed by Kubernetes. Sheriff Mohamed joins the show today to discuss migrating his company’s application to Kubernetes. It’s a great show for anyone who is moving a large team to Kubernetes, or considering the technology for their application.