Greatest Hits Archives - Software Engineering Daily

Most applications today are either deployed to on-premise environments or deployed to a single cloud provider. Developers who are deploying on-prem struggle to set up complicated open source tools like Kafka and Hadoop. Developers who are deploying to a cloud provider tend to stay within that specific cloud provider, because moving between different clouds and The post Multicloud with Ben Hindman appeared first on Software Engineering Daily.

In a cloud infrastructure environment, failures happen regularly. The servers can fail, the network can fail, and software bugs can crash your software unexpectedly. The amount of failures that can occur in cloud infrastructure is one reason why storage is often separated from application logic. A developer can launch multiple instances of their application, with The post Stateful Kubernetes with Saad Ali appeared first on Software Engineering Daily.

When a user makes a request to product like The New York Times, that request hits an API gateway. An API gateway is the entry point for an external request. An API gateway serves several purposes: authentication, security, routing, load balancing, and logging. API gateways have grown in popularity as applications have become more distributed, The post Kong API Platform with Marco Palladino appeared first on Software Engineering Daily.

An instruction set defines a low level programming language for moving information throughout a computer. In the early 1970’s, the prevalent instruction set language used a large vocabulary of different instructions. One justification for a large instruction set was that it would give a programmer more freedom to express the logic of their programs. Many The post Computer Architecture with Dave Patterson appeared first on Software Engineering Daily.

Open source software powers everything we do on the Internet. Google runs on Linux servers. Content sites are served by WordPress. Our data is queued in Kafka clusters and stored in MongoDB instances. The success of an open source project often leads to the creator of that open source software becoming wealthy. An open source The post Commons Clause with Kevin Wang appeared first on Software Engineering Daily.

Matt Klein has worked for three rapidly growing Internet companies. At AWS, he worked on EC2, the compute-as-a-service product that powers a large percentage of the Internet. At Twitter, he helped scale the infrastructure in the chaotic days before Twitter’s IPO. Today he works at Lyft, building systems to allow for ride sharing infrastructure to The post Scaling Lyft with Matt Klein appeared first on Software Engineering Daily.

Google Search allows humans to find and access information across the web. A human enters an unstructured query into the search box, the search engine provides several links as a result, and the human clicks on one of those links. That link brings up a web page, which is a set of unstructured data. Humans The post Diffbot: Knowledge Graph API with Mike Tung appeared first on Software Engineering Daily.

Google Search is a highly interactive JavaScript application. As you enter a query, results are being automatically suggested to you before you even finish typing. When you press enter, some of your search results may be widgets that represent the weather, the price of a stock, a recipe for green bean soup, or a language The post Google JavaScript with Malte Ubl appeared first on Software Engineering Daily.

Google Brain is an engineering team focused on deep learning research and applications. One growing area of interest within Google Brain is that of generative models. A generative model uses neural networks and a large data set to create new data similar to the ones that the network has seen before. One approach to making The post Generative Models with Doug Eck appeared first on Software Engineering Daily.

Airbnb began in 2008 as a monolithic Rails application serving the simple purpose of listing homes for rental. Over time, the number of listings increased dramatically, as did the number of people who were renting. With that scale, the Rails app had to be broken into different services, and entire teams were built out to The post Airbnb Engineering with Surabhi Gupta appeared first on Software Engineering Daily.

GraphQL allows developers to communicate with all of their different data backends through a consistent query interface. A GraphQL query can be translated into queries to MySQL, MongoDB, ElasticSearch, or whatever kind of API or backend is needed to fulfill the GraphQL query. GraphQL users need to set up a GraphQL server to fulfill this The post Prisma: GraphQL Infrastructure with Soren Bramer Schmidt appeared first on Software Engineering Daily.

Stock traders have access to high volumes of information to help them make decisions on whether to buy an asset. A trader who is considering buying a share of Google stock can find charts, reports, and statistical tools to help with their decision. There are a variety of machine learning products to help a technical The post Real Estate Machine Learning with Or Hiltch appeared first on Software Engineering Daily.

Building software today is much faster than it was just a few years ago. The tools are higher level, and abstract away tasks that would have required months of development. Much of a developer’s time used to be spent optimizing databases, load balancers, and queueing systems in order to be able to handle the load The post Build Faster with Nader Dabit appeared first on Software Engineering Daily.

In the smartphone market there are two dominant operating systems: one closed source (iPhone) and one open source (Android). The market for self-driving cars could play out the same way, with a company like Tesla becoming the closed source iPhone of cars, and a company like Comma.ai developing the open source Android of self-driving cars. The post Self-Driving Engineering with George Hotz appeared first on Software Engineering Daily.

Chad Fowler was the CTO of Wunderlist prior to its acquisition by Microsoft. Since the acquisition, Chad has become the general manager of developer advocacy at Microsoft. He also works as a venture capitalist at BlueYard Capital, an early stage investment firm. I’ve had a lot of fun talking to Chad, because he can move The post Future Architecture with Chad Fowler appeared first on Software Engineering Daily.

React Native allows developers to reuse frontend code between mobile platforms. A user interface component written in React Native can be used in both iOS and Android codebases. Since React Native allows for code reuse, this can save time for developers, in contrast to a model where completely separate teams have to create frontend logic The post React Native at Airbnb with Gabriel Peal appeared first on Software Engineering Daily.

Tim O’Reilly’s book What’s the Future? is an overview of business, technology, and society. As the founder of O’Reilly Media, Tim has been steeped in technology trends for the last 40 years. From his vantage point running conferences and publishing technical content, Tim has been able to make informed predictions about what is coming next. The post Future Projection with Tim O’Reilly appeared first on Software Engineering Daily.

Investing is an infinite game. In a game, a player can formulate a strategy based on the available resources, the apparent variance of the environment, and the metagame of the other actors involved. For an investor, the game board includes companies, currencies, and people. A successful game player can model their actions mathematically. They can The post Investment Games with Brian Singerman appeared first on Software Engineering Daily.

Moore’s Law states that the number of transistors in a dense integrated circuit double about every two years. Moore’s Law is less like a “law” and more like an observation or a prediction. Moore’s Law is ending. We can no longer fit an increasing amount of transistors in the same amount of space with a The post Future of Computing with John Hennessy appeared first on Software Engineering Daily.

When software is performing suboptimally, the programmer can use a variety of tools to diagnose problems and improve the quality of the code. A profiler is a tool for examining where a program is spending time. Every program consists of a set of different functions. These functions call each other. The total amount of time The post Profilers with Julia Evans appeared first on Software Engineering Daily.

Applications of artificial intelligence are permeating our everyday lives. We notice it in small ways–improvements to speech recognition; better quality products being recommended to us; cheaper goods and services that have dropped in price because of more intelligent production. But what can we quantitatively say about the rate at which artificial intelligence is improving? How The post OpenAI: Compute and Safety with Dario Amodei appeared first on Software Engineering Daily.

Self-driving, electric cars will someday outnumber traditional automobiles on the road. As transportation becomes autonomous, it is hard to imagine an industry that will not be affected by the downstream effects of this change. These cars will likely be managed by fleet operators like Lyft and Uber. We will need fewer cars, and the amount The post Autonomy with Frank Chen appeared first on Software Engineering Daily.

When a user takes a ride on Uber, the app on the user’s phone is communicating with Uber’s backend infrastructure, which is writing to a database that maintains the state of that user’s activity. This database is known as a transactional database or “OLTP” (online transaction processing). Every active user and driver and UberEATS restaurant The post Uber’s Data Platform with Zhenxiao Luo appeared first on Software Engineering Daily.

A sample of the human voice is a rich piece of unstructured data. Voice recordings can be turned into visualizations called spectrograms. Machine learning models can be trained to identify features of these spectrograms. Using this kind of analytic strategy, breakthroughs in voice analysis are happening at an amazing pace. Rita Singh researches voice at The post Voice with Rita Singh appeared first on Software Engineering Daily.

Mesos is a system for managing distributed systems. The goal of Mesos is to help engineers orchestrate resources among multi-node applications like Spark. Mesos can also manage lower level schedulers like Kubernetes. A common misconception is that Mesos aims to solve the same problem as Kubernetes, but Mesos is a higher level abstraction. Ben Hindman The post Cluster Schedulers with Ben Hindman appeared first on Software Engineering Daily.

Technology is pushing us rapidly toward a future that is impossible to forecast. We try to imagine what that future might look like, and we can’t help having our predictions shaped by the media we have consumed. 1984, Terminator, Gattaca, Ex Machina, Black Mirror–all of these stories present a dystopian future. But if you look The post Technology Utopia with Michael Solana appeared first on Software Engineering Daily.

Google’s central system for managing to compute resources is called Borg. On Borg, millions of Linux containers process a wide variety of workloads. When a new application is spun up, Borg provides that application with the resources it needs. Workloads at Google usually fall into one of two distinct categories: long-running application workloads (such as The post Google Cluster Evolution with Brian Grant appeared first on Software Engineering Daily.

Machine learning tools are rapidly maturing. TensorFlow gave developers an open source version of Google’s internal machine learning framework. Cloud computing provides a cost effective, accessible way of training models. Edge computing allows for low latency deployments of models. But even if you are a kid with a laptop who has learned all the machine The post SafeGraph with Auren Hoffman appeared first on Software Engineering Daily.

“The Federal Reserve System is fraudulent. Whatever its stated purpose, its effective purpose is to create a mechanism of deficit spending by politicians, through the insidious invisible taxation of monetary debasement (aka inflation).” These are the words of Erik Voorhees, the CEO of crypto financial exchange ShapeShift. Long before he started ShapeShift, Erik was opposed The post ShapeShift with Erik Voorhees appeared first on Software Engineering Daily.

Cryptocurrency speculation has pulled in a large population of people who do not know what they are investing in. If you hear about an investment of $1000 turning into $1M, it’s tempting to get sucked in yourself. For most of these everyday people, the game is completely rigged. A large percentage of market activity is The post Crypto Pump and Dumps with Bruno Skvorc appeared first on Software Engineering Daily.

Joseph Bonneau is co-author of Bitcoin and Cryptocurrency Technologies, a popular textbook. At NYU, he works as an assistant professor exploring cryptography and security. His YouTube lessons teaching Bitcoin have hundreds of thousands of views. His material offers clear explanations of how Bitcoin works. Since Joseph has a clear understanding of the objective facts around The post Bitcoin’s Future with Joseph Bonneau appeared first on Software Engineering Daily.

Dogecoin was started in 2013 as a joke. Jackson Palmer forked Bitcoin and created his cryptocurrency as a play-off of the “doge” meme. The currency became popular as a means of Reddit users “tipping” each other. If I made a comment on Reddit that you liked, you might send me some Dogecoin. This use case The post Dogecoin with Jackson Palmer appeared first on Software Engineering Daily.

Apache Spark is a system for processing large data sets in parallel. The core abstraction of Spark is the resilient distributed dataset (RDD), a working set of data that sits in memory for fast, iterative processing. Matei Zaharia created Spark with two goals: to provide a composable, high-level set of APIs for performing distributed processing; The post Spark and Streaming with Matei Zaharia appeared first on Software Engineering Daily.

When Box started in 2006, the small engineering team had a lot to learn. Box was one of the earliest cloud storage companies, with a product that allowed companies to securely upload files to remote storage. This was two years before Amazon Web Services introduced on-demand infrastructure, so the Box team managed their own servers, The post Scaling Box with Jeff Quiesser appeared first on Software Engineering Daily.

Your friends from college are asking you how to buy Bitcoin. Your mom is emailing you articles about the benefits of decentralized peer-to-peer networks. Your shoe shiner is telling you to buy XRP. It is 2018, and cryptocurrencies have become a daily part of news headlines. The general public may not understand how this technology The post Tether, Ripple, and Blockchain Reporting with Matt Leising appeared first on Software Engineering Daily.

Kubernetes has become the standard way of deploying new distributed applications. Most new internet businesses started in the foreseeable future will leverage Kubernetes (whether they realize it or not). Many old applications are migrating to Kubernetes too. Before Kubernetes, there was no standardization around a specific distributed systems platform. Just like Linux became the standard The post The Gravity of Kubernetes appeared first on Software Engineering Daily.

Kubernetes has become the standard system for deploying and managing clusters of containers. But the vision of the project goes beyond managing containers. The long-term goal is to democratize the ability to build distributed systems. Brendan Burns is a co-founder of the Kubernetes project. He recently announced an open-source project called Metaparticle, a standard library The post Kubernetes Vision with Brendan Burns appeared first on Software Engineering Daily.

You are requesting a car from a ridesharing service such as Lyft. Your request hits the Lyft servers and begins trying to get you a car. It takes your geolocation, and passes the geolocation to a service that finds cars that are nearby, and puts all those cars into a list. The list of nearby The post High Volume Distributed Tracing with Ben Sigelman appeared first on Software Engineering Daily.

Originally published January 6, 2016 “The best computer science is the kind where the theory is inspired by some practical problem, you develop a better theoretical understanding of what you want to do, and that feeds back into better practice.” Brian Kernighan is a professor of computer science at Princeton University and the author of The post Language Design with Brian Kernighan Holiday Repeat appeared first on Software Engineering Daily.

Originally published November 18, 2015 “The playing field has never ever been more leveled – that means everything you don’t build is your choice not to build it.” Seth Godin is a writer, speaker, and entrepreneur. He is the author of many books, including most recently, What To Do When It’s Your Turn. Questions How The post Software and Entrepreneurship with Seth Godin Holiday Repeat appeared first on Software Engineering Daily.

Originally published November 10, 2015 “The cloud as an environment – I had thought it was a purely utilitarian kind of thing. What I realized is that it’s a fascinating centralized repository of computation.” Wolfram Research makes computing software powered by the Wolfram language, a knowledge-based programming language that draws from symbolic and functional programming The post Knowledge-Based Programming with Stephen Wolfram Holiday Repeat appeared first on Software Engineering Daily.

Originally published November 17, 2015 “Changing anything changes everything.” Technical debt, referring to the compounding cost of changes to software architecture, can be especially challenging in machine learning systems. D. Sculley is a software engineer at Google, focusing on machine learning, data mining, and information retrieval. He recently co-authored the paper Machine Learning: The High The post Machine Learning and Technical Debt with D. Sculley Holiday Repeat appeared first on Software Engineering Daily.

There is a quote from Jeff Bezos: “70% of the work of building a business today is undifferentiated heavy lifting. Only 30% is creative work. Things will be more exciting when those numbers are inverted.” That quote is from 2006, before Amazon Web Services had built most of their managed services. In 2006, you had The post Run Less Software with Rich Archbold appeared first on Software Engineering Daily.

Automation is changing the labor market. To automate a task, someone needs to put in the work to describe the task correctly to a computer. For some tasks, the reward for automating a task is tremendous–for example, putting together mobile phones. In China, companies like FOXCONN are investing time and money into programming the instructions The post Training the Machines with Russell Smith appeared first on Software Engineering Daily.

A popular software application serves billions of user requests. These requests could be for many different things. These requests need to be routed to the correct destination, load balanced across different instances of a service, and queued for processing. Processing a request might require generating a detailed response to the user, or making a write The post High Volume Event Processing with John-Daniel Trask appeared first on Software Engineering Daily.

As the gig economy grows, that growth necessitates innovations in the online infrastructure powering these new labor markets. In our previous episodes about Uber, we explored the systems that balance server load and gather geospacial data. In our coverage of Lyft, we studied Envoy, the service proxy that standardizes communications and load balancing among services. The post Fiverr Engineering with Gil Sheinfeld appeared first on Software Engineering Daily.

Imagine that you are a lawyer. Your work involves managing files with dense, technical text. Your co-workers collaborate with you to accomplish a complex goal that can be broken down into smaller pieces. Your work has formal specifications, but there are degrees of freedom in how you express an idea. In all of these ways, The post Legal Technology with Justin Kan appeared first on Software Engineering Daily.

An engineer who wants to start a business using investment capital needs to understand the expectations of investors. The market for the business needs to be huge. The team needs to have a differentiated understanding of the market, or a differentiated product. The CEO needs to have the determination to continue operating the company even The post Early Investments with Semil Shah appeared first on Software Engineering Daily.

The Netflix API is accessed by developers who build for over 1000 device types: TVs, smartphontes, VR headsets, laptops. If it has a screen, it can probably run Netflix. On each of these different devices, the Netflix experience is different. Different screen sizes mean there is variable space to display the content. When you open The post Netflix Serverless-like Platform with Vasanth Asokan appeared first on Software Engineering Daily.

In 2003, Paul Martino co-founded Tribe.net, one of the earliest social networking sites. Tribe had significant traction, with hundreds of thousands of users. In the early 2000s, hundreds of thousands of users was enough traffic to pose a company with engineering challenges. Paul had studied computer science, and was able to use his knowledge of The post Parlaying Failure to Fortune with Paul Martino appeared first on Software Engineering Daily.