Software Engineering Daily

The Internet is decreasing in privacy and increasing in utility. Under some conditions, this tradeoff makes sense. We publicize our profile photo so that people know what we look like. Under other conditions, this tradeoff does not make sense. We do not want a television that costs less to purchase because it is silently recording all of the conversations that take place in the room and selling them to the highest bidder. The example of the TV that records everything you say (which is a real thing) illustrates a tradeoff of the Internet. The advertising industry pushes us towards lower marginal costs for products and services in exchange for less privacy. Someday we will live in a world where it will be easy for consumers to control the dial on the tradeoff between privacy and the price of their services. Until then, we have almost zero control over what information the advertising surveillance industrial complex knows about us. Bill Budington is a security engineer with the Electronic Frontier Foundation. In today’s episode, Bill describes some of the current techniques used by the advertising industry to track your activity through the web. Bill works on encryption tools as well as Panopticlick, a project that allows users to see what trackers they are vulnerable to. Software Engineering Daily is having our third Meetup, Wednesday May 3rd at Galvanize in San Francisco. The theme of this Meetup is Fraud and Risk in Software. We will have great food, engaging speakers, and a friendly, intellectual atmosphere. To find out more, go to softwareengineeringdaily.com/meetup.

Search is a common building block for applications. Whether we are searching Wikipedia or our log files, the behavior is similar: a query is entered and the most relevant documents are returned. The core data structure for search is an inverted index. Elasticsearch is a scalable, resilient search tool that shards and replicates a search index. Philipp Krenn from Elastic joins the show today to discuss how search works and how Elasticsearch scales. We use Wikipedia as a running example for how a query is processed and how documents are stored. If you’ve ever wondered how search works–or if your company uses Elasticsearch and you want to know more about it–this is a great episode for you.

React Native has unlocked native mobile development to web engineers who may now apply their skills to build iOS and Android applications in JavaScript. For the first time, cross platform JavaScript-based applications feel as if they were written in the native language of choice for the platforms. Businesses who choose to adopt React Native for their native app development also see great benefits such as the ability to push new JavaScript code without going through the app store review process, and the ability to share code and business behaviors across the iOS and Android platforms. Expo is building a cross-platform native runtime for React Native. Expo brings the benefits of deployment and iterative development to native without sacrificing any user experience costs. Expo plans to do this with their native SDK, custom development environment, and tools built in collaboration with Facebook like create-react-native-app. React Native has the incredible potential to revolutionize all user interface development with its core set of cross-platform UI primitives, and React’s popular declarative rendering pattern. So in this episode Brent Vatne and Adam Perry join Caleb Meredith to first discuss Expo and the future of React Native to try and answer the question: can React Native become the one UI framework to rule them all?

Every program gets compiled down to 1s and 0s before it can be executed against hardware. Before being translated to machine code, programs that are written in a language like Rust, Swift, or Java spend time in an intermediate representation. In Java, this intermediate representation is Java bytecode. Many different languages–such as Scala–translate to Java bytecode, because there has been lots of optimization written to speed up Java bytecode. Java bytecode runs on the JVM–the Java Virtual Machine. LLVM is a project that draws inspiration from the Java Virtual Machine. LLVM originally meant “low level virtual machine” but today it is just called LLVM and describes a set of compiler tools. In today’s interview with Morgan Wilde, we explore how compilers work, how different processor hardware architectures present a problem for compilers, and why LLVM’s intermediate representation creates a layer of interoperability for any language that compiles down to that intermediate representation. Whether you are new to compilers or have experience, this episode will appeal to you. Morgan is an excellent teacher and his enthusiasm for the subject comes through. He has a 30-minute YouTube video–A Brief Introduction to LLVM that I highly recommend.

Engineers in Silicon Valley see a world of constant progress. Our work is creative and intellectually challenging. We are building the future and getting compensated quite well for it. But what if we are actually achieving far less than what is possible? What if, after so many years of high margins, gourmet lunch, and self-flattery, we have lowered our standards for innovation? And if Silicon Valley has been lulled into complacency, what does that say about the rest of the United States? American exceptionalism has faltered and complacency has risen in its wake. Today’s guest Tyler Cowen is an economist and author. His new book The Complacent Class is the final book in a trilogy that describes a decline of American output and a decline in American mindset. Complacent America has lost its ability to assess risk. Children are prevented from playing tag for risk of injury. College students protest against speakers who might present challenging ideas. The number of Americans under 30 who own a business has fallen by 65% since the 1980’s–millennials are too busy going to business school to start businesses. In his books, Tyler weaves together history, philosophy, and contemporary culture. He presents hard data about many different fields, and theorizes about how the trends in those fields relate to each other. He also has a podcast, Conversations with Tyler, and in this episode I tried to mirror his interview style. If you like this episode, you should check out his show–he has interviewed people like Ezra Klein, Peter Thiel, and Kareem Abdul-Jabbar.

Cryptocurrencies are not only a financial instrument–they are a new platform for building applications. The blockchain allows for new solutions to digital property management, micropayments, hedge fund incentives, and ad fraud. The cryptocurrency platforms with the most traction are Bitcoin and Ethereum. Bitcoin has no central leader and is going through some growing pains with governance issues. Ethereum is led by the charismatic Vitalik Buterin. Bitcoin and Ethereum are not competing instruments. They fulfill different technical purposes. Under current conditions of algorithm development and network infrastructure, neither Bitcoin nor Ethereum can accomplish the dreams that will one day be realized, because of the problem of distributing transaction information across nodes in the system. If we compared cryptocurrencies to the Internet, we would not even be in the days of dial-up yet. ConsenSys is a venture production studio that is working on several projects within the blockchain space. Mike Goldin is a software developer with ConsenSys and joins the show to talk about blockchain applications in 2017–where we are and where we are going. It was a wide ranging conversation and I hope to have Mike back in the future so we can go deeper on some of the topics we glossed over.

There are various standards at play when creating and consuming Application Program Interfaces (APIs). These standards, though, are mostly technical and mostly lower-level than the content of the API. Andy Beier has experienced the broad range of API quality in his role with Domo in creating integrations with other businesses. He has made standardization of good practices in creating APIs his mission, with an emphasis on making the right information easily accessible without having to download more than necessary. He has traveled to meet with leaders in the field to promote standards and to make APIs easier to create and to consume. In this episode, Andy joins Dave Rael for a conversation about API design standards, what makes for a good API, and steps in moving the broader technical community toward more useful and secure APIs.

In this episode, I gathered questions from listeners in our Slack channel and Twitter feed. The questions I answered include: We always want more feedback and questions. Please email me, join Slack, and fill out our survey!

A hedge fund is a collection of investors that make bets on the future. The “hedge” refers to the fact that the investors often try to diversify their strategies so that the direction of their bets are less correlated, and they can be successful in a variety of future scenarios. Engineering-focused hedge funds have used what might be called “machine learning” for a long time to predict what will happen in the future. Numerai is a hedge fund that crowdsources its investment strategies by allowing anyone to train models against Numerai’s data. A model that succeeds in a simulated environment will be adopted by Numerai and used within its real money portfolio. The engineers who create the models are rewarded in proportion to how well the models perform. Xander Dunn is a software engineer at Numerai and in this episode he explains what a hedge fund is, why the traditional strategies are not optimal, and how Numerai creates the right incentive structure to crowdsource market intelligence. This interview was fun and thought provoking–Numerai is one of those companies that makes me very excited about the future.

Brendan Eich created the first version of JavaScript in 10 days. Since then JavaScript has evolved, and Brendan has watched the growth of the web give rise to new and unexpected use cases. Today Brendan Eich is still pushing the web forward across the technology stack with his involvement in the WebAssembly specification and the Brave browser. For all of its progress, JavaScript struggles to run resource-intensive programs like complex video games. With JavaScript falling short on its charge to be the “assembly language for the web” the four major browser vendors started collaborating on the WebAssembly project to allow programming languages a faster, lower level compile target when deploying to the web. Brendan is the CEO of Brave which aims to provide a faster and safer browsing experience by blocking ads and trackers by default in a new browser. The Brave browser is also helping publishers monetize in interesting new ways while also giving a share of ad revenue to its users. Caleb Meredith is the host of this show. He previously guest hosted a popular episode on Inferno, a fast, React-like JavaScript framework. As we bring on more guest hosts, please send us feedback. We want to know what every host is doing well, and what we can improve on.

Big technology companies have so much going on at any given time that a journalist can tell any type of story they want to about it. Depending on what angle you observe the company from, you can write a story depicting that company as good, evil, growing, or about to crash. The truth only becomes apparent to outsiders with time. Amazon’s culture and business strategy were detailed in The Everything Store, a 2013 book by Brad Stone. I read The Everything Store before working at Amazon, then I read it a second time after working at Amazon. The book is an accurate and balanced depiction of Amazon’s ethos. Brad’s new book The Upstarts documents the rise of Uber and Airbnb, two companies with a similar controversial valence to Amazon. It was a pleasure to speak to Brad because I admire his engrossing storytelling as an author and his strategic analysis as a business journalist. After discussing business and technology with him, we explored journalism–Brad is senior executive editor at Bloomberg.

Servers in a data center fail. Sometimes entire data centers have a power outage. Bugs in an application make it into production. Human operators make mistakes and cause data to be deleted. Failure is unavoidable. We make backups and replicate our servers so that when a failure occurs, we can quickly respond to it without making the user feel much pain. But how can we test that our response will work before an actual catastrophe occurs? Kolton Andrus is CEO of Gremlin, a company that works on failure injection as a service. Gremlin is based on ideas around planned failure that Kolton learned from his years at Amazon and Netflix. We ended up talking as much about the culture of Netflix and Amazon as we did about how and why to build failure injection. It’s always nice to share war stories with other people who have worked at Amazon because the culture is so distinct. If you want to know more about Amazon’s culture, check out the episode tomorrow with Brad Stone, author of The Everything Store.

Designers and software engineers need to communicate with each other. From Apple to Slack to Uber, the emphasis on visual design within a product is rising in importance. Much like development and operations siloes have been bridged with the DevOps movement, design and engineering teams are working more closely together to align the vision of the designers with the realities of code. InVision is a product for prototyping designs and product workflows. I categorize InVision with high-level productivity tools like Trello, Asana, and github. When teams start using InVision, it often becomes an integral part of the entire product workflow. A designer will mock up their product vision on InVision and share it with the rest of the team for criticism and commentary. Since the product is the perfect bridge between engineers and designers, InVision the company is expanding as a platform. Bjorn Freeman-Benson is the CTO at InVision and his task is to scale the 100% remote engineering team while developing new product features. Bjorn describes himself as a software psychologist. We talked about the psychology of managing engineers and the technical challenges of building a large, popular web app for technology products.

Developing a product requires careful balance between engineering, sales, design, and customer service. The founding CTO of a company often needs to take on each of these responsibilities, because when the company only has a few people there is nobody to delegate these different tasks to. Cullen Zandstra is the CTO at FloQast, a SaaS tool for accounting close management. It isn’t important what that means–we do describe it in this episode, but the actual software product that we are discussing is less important than the general discussions we have. Cullen gives detailed explanations for how to develop a software product. In the past, Cullen worked at MySpace. After we discussed his current company FloQast, we discussed some of what he learned at MySpace, and why the social network proved to be less durable than Facebook. We also had some great discussion as to why Google+ was ALSO less durable than Facebook. Also–I should mention that FloQast is hiring engineers in their Los Angeles office. And thanks to my friend Patrick Mathieson for connecting Cullen and myself.

Net neutrality is the principle that internet service providers and governments regulating the internet should treat all data on the internet the same. Debates around Net Neutrality can be as contentious as subjects like global warming, or tabs vs. spaces. To a hardcore free market economist, Net Neutrality sounds suspicious. Why would it be good for the government to regulate prices on the data that passes around wires owned by a corporation? The problem is that a large portion of the United States can only be served by wires owned by a single company. In those instances, the company that owns the wires has monopoly pricing power over the data delivered to homes in those geographic regions. Quincy Larson is the founder of Free Code Camp and a frequent author of internet-based articles on Medium. He returns to Software Engineering Daily to make the case for Net Neutrality. I was convinced by some of his arguments and less convinced by others. In any case, I think you will find this episode entertaining and informative. If you are like me, after this episode you will have at least a slightly better understanding of the issues surrounding Net Neutrality. Inside the Invisible War for the Open Internet

In bitcoin, every transaction in the shared ledger has the sender, recipient, and value. This ledger gets appended infinitely and is shared within a peer-to-peer network. Zcash is a cryptocurrency with all the features of bitcoin, plus encrypted transactions. The sender, recipient, and value fields are all encrypted. If bitcoin is HTTP, Zcash is like HTTPS, a secure transport layer. Nathan Wilcox works on Zcash, and in this episode we discuss why an encrypted version of bitcoin is useful, how mining works, and how Zcash the company is structured. Nathan also gives some context for the current state of the bitcoin community, and where Zcash fits in.

Wearables have become more accessible to the public. Snap’s spectacles, Google Glass, FitBit, and Apple Watch suggest a future in which many people will be wearing a smart device. In this episode Asta Roseway, Research Designer at Microsoft Research, gives insights into other categories of wearables like tattoos, scarves, and cosmetics. Asta talked about her work on DuoSkin, a wearable that looks like a beautiful metallic tattoo. We talked about its capabilities and how it was built, and why it is still too early for connected tattoos. We also talked about the intersection between health, wearables, technology, and fashion and how wearables might look in the near future. This episode is guest hosted by Edaena Salinas, who also hosts The Women in Tech Show. Asta Roseway’s Website Asta Roseway Microsoft Research MoodWings LightWear

Multiagent systems involve the interaction of autonomous agents that may be acting independently or in collaboration with each other. Examples of these systems include financial markets, robot soccer matches, and automated warehouses. Today’s guest Peter Stone is a professor of computer science who specializies in multiagent systems and robotics. In this episode, we discuss some of the canonical problems of multiagent systems, which have some overlap with the canonical problems of distributed systems–for example, the problems of coordinating between different agents with varying levels of trust resembles the problem of establishing consistency across servers in a database cluster. Peter has recently contributed to the 100 year study of artificial intelligence, so we also had a chance to discuss the opportunities and roadblocks for AI in the near future. And since Peter teaches computer science at my alma mater, UT Austin, I had to ask him a few questions about the curriculum.

Biology research is complex. The sample size of a biological data set is often too small to make confident judgments about the biological system being studied. During Jason Knight’s PhD research, the RNA sequence data that he was studying was not significant enough to make strong conclusions about the gene regulatory networks he was trying to understand. After working in academia, and then at Human Longevity, Inc Jason came to the conclusion that the best way to work towards biology breakthroughs was to work on the computer systems that enable those breakthroughs. He went to work at Nervana Systems on hardware and software for deep learning. Nervana was subsequently acquired by Intel. In this episode, we discuss how machine learning can be applied to biology today, and how industrial research and development is key to enabling more breakthroughs in the future. The main lesson I took away from this show is that while we have seen phenomenal breakthroughs in certain areas of health–like image recognition applied to diabetic retinopathy or skin cancer–the challenges of reverse engineering our genome to understand how nucleic acids fit together into humans are still out of reach, and improving the hardware used for deep learning will be necessary to tackle these kinds of informational challenges.

Every company that deals with payments deals with fraud. The question is not whether fraud will occur on your system, but rather how much of it you can detect and prevent. If a payments company flags too many transactions as fraudulent, then real transactions might accidentally get flagged as well. But if you don’t reject enough of the fraudulent transactions, you might not be able to make any money at all. Because fraud detection is such a difficult optimization problem, it is a good fit for machine learning. Today’s guest Michael Manapat works on machine learning fraud detection at Stripe. This conversation explores aspects of both data science and data engineering. Michael seems to benefit from having a depth of knowledge in both aspects of the data pipeline, which made me question whether data science and data engineering are roles that an engineering organization wants to separate. This is the third in a series of episodes about Stripe engineering. Throughout these episodes, we’ve tried to give a picture for how Stripe’s engineering culture works. We hope to do more experimental series like this in the future. Please give us feedback for what you think of the format by sending us email, joining the Slack group, or filling out our listener survey. All of these things are available on softwareengineeringdaily.com.

If you are building a service that processes payments, your software architecture has a lot of requirements. Not only do you need to be highly available, consistent, and fast–you need to be PCI compliant. In this episode, we explore the infrastructure of Stripe with Evan Broder, who has been with the company for five years. Stripe started as a small payments company catering to developers with a monolithic code base. Some of those aspects of Stripe have changed, and others have stayed the same. In our last episode, we covered how observability works at Stripe. In this episode, we explore what is being observed–the actual infrastructure itself, and how different engineers are organized around managing the infrastructure. In tomorrow’s episode, we’ll talk to Michael Manapat, about Stripe’s machine learning pipeline for detecting and preventing fraudulent transactions. Throughout these episodes, you will get a sense for how Stripe’s engineering culture works. We hope to do more experimental series like this in the future. Please give us feedback for what you think of the format by sending us email, joining the Slack group, or filling out our listener survey. All of these things are available on softwareengineeringdaily.com.

Observability allows engineers to understand what is going on inside their systems. In its most raw form, observability comes from log data. Modern systems have many layers of logs–virtualized cloud infrastructure, container orchestration, the container runtime itself, and the application logic running within the container. With all of these layers, it is not practical for a developer to have to sift through layers of logs every time a bug occurs in production, or a deployment fails integration tests. Higher level observability tools include charts, distributed tracing tools, and monitoring services. With proper observability, developers can save time during incident response. Day-to-day software development becomes safer and more comfortable. Stripe is a payments company for developers. This episode is the first in a series of episodes profiling different aspects of the company. Our guest Cory Watson leads the observability team at Stripe. In subsequent episodes, we will explore infrastructure and machine learning at Stripe. Throughout these episodes, you will get a sense for how Stripe’s engineering culture works. We hope to do more experimental series like this in the future. Please give us feedback for what you think of the format by sending us email, joining the Slack group, or filling out our listener survey. All of these things are available on softwareengineeringdaily.com.

Donald Trump has either alarmed or excited everyone in the engineering community. Some of the debates are based around innovation. Opponents to Trump say that his anti-immigration policies will reduce the innovation that relies on H1-B visas. Supporters of Trump say that his anti-regulation policies will unlock innovation that is restricted by bureaucracy. Other debates are around philosophy and free speech. Opponents to Trump say that his views on women are retrograde and that he promotes racism and white nationalism. Supporters of Trump say that he is brave enough to say what people are really thinking, and that he is truly independent from the polarized, ossified political system. Software Engineering Daily is a show about the world through the lens of the software engineer, and today our world is being shaped by Donald Trump. We needed to do an episode on this subject because so many of my day-to-day conversations with other engineers were becoming about Trump. Brad Taylor is a senior engineer with Optimizely who has formed Tech Stands Up, a grassroots movement giving a voice to the rapidly growing concerns about the Trump administration’s policies affecting the tech community and its users. We had a stimulating conversation about the tradeoffs of a Trump administration from an engineer’s perspective. Don’t worry–this is not becoming a show about politics. Tech Stands Up: Pi Day Rally Tickets TechStandsUp.org

Making the right engineering choices in today’s wide landscape of cloud technologies is hard. Predicting the future in order to invest in companies in this space has the same level of complexity. The cost of cloud computing is going down but the volume of total required space and processing power is going up. The open source community is growing and improving but people are increasingly willing to buy software that will save them time. Capital is flowing into Silicon Valley at a faster rate than can be sensibly absorbed by the number of quality companies. A decent product won’t have trouble raising money. But a decent investor has to choose wisely among the huge selection of available opportunities. Lenny Pruss works at Amplify Partners where he is currently focused on the cloud native space. In this episode, we talk about what cloud native means and how to navigate the complex landscape, whether you are an engineer or an investor. Cloud Native Landscape Project

Command Query Responsibility Segregation (CQRS) is a powerful concept that has the potential to make for reliable and maintainable systems. It is also broadly misunderstood and means different things to different people. Derek Comartin learned about the idea after viewing some talks by Greg Young and has since successfully applied the approach with great success and it has transformed the way he views features, business requirements, and dependencies. The result is a system that is easier to maintain and faster to enhance. Among his key lessons are that slices are better than layers, mediators improve dependency management, and cohesion is better applied to business concerns than to technical ones. In this episode, Derek joins Dave Rael for a conversation about CQRS and applying it to make your code your own and to separate it from technical concerns in order to make your software development operation work better and faster.

Machine learning has improved both in tools and accessibility. Frameworks like TensorFlow create the right abstractions for developers to work efficiently. Educational programs like Metis and Insight Data Science provide a place for developers to learn these tools. As a result, artificial intelligence is becoming easier to develop and more widespread. Rumman Chowdhury works on artificial intelligence at Accenture. Before her current role, she taught data science at Metis. In this episode, we talk about the current state of artificial intelligence, from the tools available to the long term implications–such as the robot tax recently proposed by Bill Gates.

Load testing measures performance of a system undergoing a large volume of requests. Before an application is pushed to production, engineers will often load test their software to ensure it is resilient in the face of high traffic. As web applications have changed, the requirements around load testing have changed as well. External APIs, internal undocumented APIs, and proprietary databases are black boxes that you might not be able to test reliably with unit tests or integration tests. Having an end-to-end load testing system can provide a measure of insurance against unknown unknowns before users start engaging with a production version. Mark Gilbert works on performance engineering at Apica, and he joins the show to discuss how load testing software is built and when engineers should use it. Full disclosure: Apica is a sponsor of Software Engineering Daily.

There is too much mobile engineering work to be done and not enough mobile engineers. As a result, a talented mobile engineer will often make more money than a similarly talented web developer. There are many other disconnects between the world of mobile engineering and the world of backend and web development. We have reported on web development far more than mobile on Software Engineering Daily, and, I wanted to get a holistic view of the mobile ecosystem. Nathan Esquenazi is a co-founder of CodePath, which is a continuing education program for professional engineers and designers. There was much to explore–from the economics of mobile, to the state of cross platform, to the increasingly blurry line between mobile and IoT.

Thursday February 23rd was a big day in security news: details were published about the Cloudbleed bug, which leaked tons of plaintext requests from across the Internet into plain view. On the same day, the first collision attack against SHA-1 was demonstrated by researchers at Google, foretelling the demise of SHA-1 as a safe hashing function. What does this mean for the average engineer? What are the implications for regular internet users? Haseeb Qureshi interviews Max Burkhardt, a security researcher at Airbnb, to get to the bottom of what exactly happened, what it means, and how it affects the security of web applications.

Manufacturing electronics is not a simple task and because of its complexity it is also quite expensive. Companies like Apple are able to pull their costs down through economies of scale. Since Apple is always placing huge bulk orders on chips, sensors, and other components, the company can get lower prices for those components than an individual hardware hacker that wants to build a prototype. Herein lies the opportunity for Tempo Automation, a company that allows rapid prototyping of electronics. By aggregating demand for electronics prototyping, Tempo Automation can offer competitive pricing and speed to anyone who needs a hardware prototype. I visited Tempo Automation and was given a tour of the production floor by CEO Jeff McAlvay. He showed me the huge machines that are used to build hardware prototypes and explained how they work in serial to produce prototypes. He also mentioned that anyone can schedule a tour of Tempo Automation’s factory by going to their website, and I highly recommend it. After the tour, Jeff and I sat down for an interview about what Tempo Automation is building and their plans for the future.

When an engineer is offered a job a tech company, their compensation is often partly in cash and partly in equity–shares of the company. How should an engineer evaluate that offer? How should they negotiate? In the world of equity compensation, costly and avoidable mistakes are routine, and this hurts both companies and employees. Josh Levy was on Software Engineering Daily previously to talk about the Amazon Web Services open guide, which was one of the most popular episodes we have ever had. In this episode, Josh and I are joined by Joe Wallin, a lawyer who has been involved in startups for many years. Josh and Joe are the authors of the The Open Guide to Equity Compensation, which is a resource designed to clear up the confusion around stock, options, and fundraising. It’s a tremendously useful and concise overview of what an engineer or founder needs to know when it comes to equity financing.

Parse was a backend as a service company built in 2011 before being acquired by Facebook in 2013. Building a backend as a service for developers requires walking a thin line between giving engineers lots of control and preventing those engineers from shooting themselves in the foot. While she was at Parse, Charity Majors learned about the operational burdens of managing a service with high uptime requirements and deeply technical edge cases that could take down a user’s entire system. Charity took the lessons in systems engineering that she learned at Parse and cofounded Honeycomb.io, a service for observability and monitoring. Honeycomb is described as a tool for your systems like an IDE is to your code. Parse was eventually shut down because the service did not have a place in the strategic plans of Facebook. Charity and I also discussed the lessons learned from how the Parse acquisition panned out–a useful conversation for anyone who is considering selling a company or acquiring a startup.

When an application is using all of its available resources, that application needs to be scaled. Scaling an application means giving it more resources–typically servers. Autoscaling is an engineering practice where an application is automatically given more or less resources based on how healthy the application performance is at a given time. Applications on Heroku have access to autoscaling. Heroku users don’t need to worry about provisioning new servers manually because the platform does it for them. In this episode, we explore how Heroku built autoscaling. Andrew Gwozdziewycz [@apgwoz] is an operational experience engineer with Heroku. As he describes, autoscaling requires frequent health checks of an application. Since thousands of applications are running on Heroku, a metrics pipeline using Kafka and Cassandra supports the high volume of health check data. That data feeds into the decision process for when an application needs to scale. Full disclosure: Heroku is a sponsor of Software Engineering Daily.

In the mid 90s, data warehousing might have meant “using an Oracle database.” Today, it means a wide variety of things. You could be stitching together a big data pipeline using Kafka, Hadoop, and Spark. You could be using managed tools like BigQuery from Google. How did we get from the simple days of Oracle databases to the wealth of options available today? Mark Rittman writes and podcasts about data engineering and data warehousing on his site Drill to Detail. Today, we explore the past, present, and future of data warehousing and touch on many of the trends that have been explored in recent episodes of Software Engineering Daily. Google BigQuery, and Why Big Data is About to Have Its Gmail Moment

At the first Software Engineering Daily Meetup, the speakers explored a range of topics. A few weeks ago, we published Courtland Allen’s talk about how to build a small software business. In today’s episode, we are publishing Haseeb Qureshi’s talk, called “Everything That Rises Must Converge: Why Engineers Disagree About Everything (And Why Fraudsters Do Too).” This talk explores philosophy, poker, software engineering, fraud, and a basket of other topics. These topics might sound like a random collection, but Haseeb currently works as an engineer on Airbnb’s Risk team and he is the author of How To Be a Poker Player, which is a book on the philosophy of poker. The audio is taken from the Meetup and we cleaned it up as best we could. We will do a better job with the audio from the next Meetup–which is Thursday, March 9th at Galvanize! Check out our Meetup page for more details.

Security vulnerabilities are an important concern in systems. When we specify that we want certain information hidden, for example our phone number or our date of birth, we expect the system to hide the information. However, this doesn’t always happen due to human error in the code because programmers have to write checks and filters across the program. In this episode, Edaena Salinas interviews Jean Yang, Assistant Professor at the Computer Science Department at Carnegie Mellon, who presents Jeeves, a language that allows programmers to specify security policies more intuitively, making it harder to leak information that is meant to be protected. Jean explained how Jeeves was implemented and how it can be used. We also talked about what it takes to bring research concepts from academia to the industry and at the end we had a very interesting conversation on how to educate a broader audience on the importance of security. Jean was also named one of the 35 innovators under 35 by MIT Technology Review.

Expectations for mobile apps have gone up steadily since the iPhone was released. But the choice of databases built for mobile apps has remained limited mostly to SQLite. RealmDB was created as a new option for mobile developers on iOS, Android, or any other mobile platform. Realm is not just a database. It is a database platform, offering a variety of systems such as authentication, sync, and an event framework. In this interview with RealmDB VP of engineering Brian Munkholm, he explains why we need not only a new mobile database type, but an entire database platform. We talk about the business of mobile, IoT, cloud, and cross platform. It’s a wide ranging conversation, but also deeply technical.

Online advertising is a system of transactions that involve many different players. The user visits a publisher’s website; the publisher notifies an exchange that the user is on the website; the exchange presents an opportunity to a marketplace that can buy that opportunity to show the end user the ad. And this a simple example. The transactions in online advertising are as opaque and rife with fraud as Wall Street–but less regulated. Blockchain technology presents an opportunity to bring more transparency to the advertising ecosystem using a shared ledger. Ken Brook is the CEO of VidRoll, a video advertising company. Using his experience in the adtech business, Ken is working on Adchain, a shared ledger for adtech. In this episode, we explore adtech, blockchains, and Adchain. Russian Cyberforgers Steal Millions a Day with Fake Sites

The impact of artificial intelligence on our everyday lives will be so profound that our modern institutions will change completely. Employment, government, romance, social norms–all of these things will be upended. To see the signs of this coming, you no longer have to read science fiction. Every week, there are blog posts, news stories, and videos chronicling our strange, exciting time. Rob May is an investor in artificial intelligence companies and is CEO of the AI company Talla. Every week, he puts out the Technically Sentient newsletter, a compilation of the best pieces of information about AI over the past week. Each newsletter also contains a short essay by Rob in which he gives a big picture perspective on what he is seeing in the AI space. This was an illuminating conversation with Rob about the implications of artificial intelligence and the topics he writes about in Technically Sentient. I first saw Rob speak at the Launch Scale Festival, which Jason Calacanis puts on as a free event for people who have started companies.

If you wanted to build a machine learning model to understand human health, where would you get the data? A hospital database would be useful, but privacy laws make it difficult to disclose that patient data to the public. In order to publicize the data safely, you would have to anonymize it, so that a patient’s identity could not be derived from data about that patient–and true anonymization is notoriously difficult. In every industry where privacy is a concern there is a similar challenge. If there is no place with public data sets, there is no place where the machines can go to learn. The possible machine learning applications that we can build are limited by the data sets that are available. Auren Hoffman started his company SafeGraph to unlock data sets so that machine learning algorithms can learn from that data. In this episode, we talk about the machine learning landscape in both the short- and long-term time horizons. We also discussed some of Auren’s strategies for building companies, which have been crucial for me in thinking about how to build Software Engineering Daily. Where Should Machines Go to Learn? Quoracast Episode

Machine learning frameworks like Torch and TensorFlow have made the job of a machine learning engineer much easier. But machine learning is still hard. Debugging a machine learning model is a slow, messy process. A bug in a machine learning model does not always mean a complete failure. Your model could continue to deliver usable results even in the presence of a mistaken implementation. Perhaps you made a mistake when cleaning your data, leading to an incorrectly trained model. It is a general rule in computer science that partial failures are harder to fix than complete failures. In this episode, Zayd Enam describes the different dimensions on which a machine learning model can develop an error. Zayd is a machine learning researcher at the Stanford AI Lab, so I also asked him about AI risk, job displacement, and academia versus industry. Why ML is hard

Data scientists need flexible interfaces for displaying and manipulating data sets. Data engineers need to be able to visualize how their data pipelines wire together databases and data processing frameworks. DevOps engineers need dashboards to understand their monitoring data at a high level. All of these programmers are building data applications. Data applications let us visualize and manipulate data sets effectively. In today’s episode, Dave King joins the show to describe the growing importance of data applications. Many knowledge workers use flexible tools like spreadsheets and business intelligence applications. But when you build a domain specific data application for a knowledge worker, you can unlock a higher degree of leverage for that knowledge worker. We discuss data applications and the future of knowledge work in this episode with Dave King. Full disclosure: Exaptive is a sponsor of Software Engineering Daily.

Most tech companies are moving toward a highly distributed microservices architecture. In this architecture, services are decoupled from each other and communicate with a common service language, often JSON over HTTP. This provides some standardization, but these companies are finding that more standardization would come in handy. At the ridesharing company Lyft, every internal service runs a tool called Envoy. Envoy is a service proxy. Whenever a service sends or receives a request, that request goes through Envoy before meeting its destination. Matt Klein started Envoy, and he joins the show to explain why it is useful to have this layer of standardization between services. He also gives some historical context for why Envoy was so helpful to Lyft.

Infrastructure is a term that can mean many different things: your physical computer, the data center of your Amazon EC2 cluster, the virtualization layer, the container layer–on and on. In today’s episode, podcasters Chris Wahl and Ethan Banks discuss the past, present, and future of infrastructure with me. Ethan and Chris host Datanauts, a podcast about infrastructure. In each episode, Datanauts goes deep on a topic such as networking, serverless, or OpenStack. As someone who hosts a similar podcast, I find it entertaining and educational to hear their points of view on a regular basis. If you like Software Engineering Daily, you might like Datanauts. And if you like Datanauts, you will love this episode of Software Engineering Daily.

Deep learning uses neural networks to identify patterns. Neural networks allow us to sequence “layers” of computing, with each layer using learning algorithms such as unsupervised learning, supervised learning, and reinforcement learning. Deep learning has taken off in the last few years, but it has been around for much longer. Adam Gibson founded Skymind, the company behind Deeplearning4j. Deeplearning4j is a distributed deep learning library for Scala and Java. It integrates with Hadoop and Spark, and is specifically designed to run in business environments on distributed GPUs and CPUs. Adam joins the show today to discuss the history and future of deep learning.

Data science is typically done by engineers writing code in Python, R, or another scripting language. Lots of engineers know these languages, and their ecosystems have great library support. But these languages have some issues around deployment, reproducibility, and other areas. The programming language Golang presents an appealing alternative for data scientists. Daniel Whitenack transitioned from doing most of his data science work in Python to writing code in Golang. In this episode, Daniel explains the workflow of a data scientist and discusses why Go is useful. We also talk about the blurry line between data science and data engineering, and how Pachyderm is useful for versioning and reproducibility. Daniel works at Pachyderm, and listeners who are more curious about it can check out the episode I did with Pachyderm founder Joe Doliner.

Engineering managers face a different set of problems than engineers themselves. Whether they are hiring new employees, firing underperformers, or guiding a team of existing engineers, engineering management is all about people. Empathy is paramount because management is like being a professional therapist for the members of your team. Mike Borozdin has been managing engineers for more than a decade and blogs about engineering management. In this episode, we cover the core responsibilities of an engineering manager and some strategies around how to do it right. This episode is useful for anyone looking to improve their relationship with their manager, move into management, or improve as a manager.

Open source software is publicly available code that is worked on in the open by large crowds of developers. Almost all new software today uses some open source software in its code. But most people never contribute to open source themselves. Some people would love to get involved in open source, but they don’t know how to get started. Shubheksha Jalan found herself in this position. In her early attempts to break into open source, she failed to get much traction and found the open source community hostile to newcomers. She would fix beginner issues, but had trouble figuring out how to make more complex contributions. Eventually she found success and wrote about it. In this episode she takes us through that journey. If you are wondering how to get into open source, this episode is for you.

Giphy is a search engine for gifs, the short animated graphics that we see around the Internet. Giphy is also a creative platform where people create new gifs. Every search engine requires the construction of a search index, which is a data structure that responds to search queries efficiently. Since Giphy is a search engine for graphics, there is almost no text inherently associated with the each document. Giphy uses a pipeline of different labeling techniques in order to make a gif indexable by the search engine. In my conversation with Giphy CTO Anthony Johnson, we discuss how to scale a search engine, why Giphy needs to build new techniques for image processing, how human labeling for machine learning is evolving, and the future of Giphy–both as a creative medium and an advertising platform. This was an exciting and wide-reaching interview.

Cloud computing has pushed computation away from our own private servers and into virtual machines running on a data center. In the world of cloud computing, processing is centralized in these data centers, and our smartphone and laptop application performance suffers from having high latency between the client and the cloud server. As machine learning proliferates, the current model of cloud computing will become too slow. A small difference in the time it takes to refresh a machine learning model for a drone or car could be the difference between life and death. Computation will move to the edge. The same drones, cars, and IoT devices that need their models updated quickly will form a peer-to-peer network with which to distribute time-sensitive tasks. One bellwether for this real time peer-to-peer network might be Uber’s Ringpop, a fault-tolerant application layer sharding system. In such an edge computing model, your device could federate a complex request out to other nearby devices that have spare processing capacity, pay for those compute cycles using Bitcoin, and receive a response without any request to a centralized cloud server. The cloud servers would still be around, but they would be responsible for doing offline computation across large data sets. This prediction was described by Peter Levine, a partner at Andreessen Horowitz in his talk “The End of Cloud Computing”. In this episode, Peter discusses the pressures that our pushing toward edge computing and away from the cloud.