Software Engineering Daily

In this episode, Jeff and Derrick sit down to discuss the state of tech journalism, and how it has kept up with the rapid pace of technology innovation. Additionally, they discuss the evolution of the “big data” stack and where it is headed. The conversation focuses in on Apache Mesos and Mesosphere, the company building the Datacenter Operating System (DCOS). Derrick Harris is a long time tech journalist, having worked at GigaOm and Fortune as a writer covering tech topics. He currently works at Mesosphere, where he is a tech evangelist.

Incarceration in the United States is often a punitive exercise rather than a rehabilitative one. As a consequence, recidivism is the norm – according to a 2012 report by the California Department of Corrections and Rehabilitation, just over 65% of those released from California’s prison system return within three years. San Quentin State Prison in California is challenging this issue by trying to equip inmates with valuable skills while they are incarcerated. The Last Mile is a program that works with prisoners to help them build relevant skills in technology so that they can more easily transition to productive employment once they are out of prison. Specifically, it teaches inmates how to program, oftentimes starting at the basics like how to use a computer, and how hypertext works. In addition to the challenges of teaching students who may not know computer basics, the program has to work with the constraint of not having internet access. In this episode, Wes and Jeff discuss the realities of incarceration in the United States, and how teaching inmates how to code is a remarkably empowering way to combat and weaken the prison-industrial complex. Wes Bailey is the Director of Program Operations at The Last Mile, and a self-taught developer.

Governments are often laggards in the adoption of new technology. Every aspect of life, from transportation to healthcare, are being affected by the convergence of several trends including connected mobile devices, cloud storage and distributed systems. Yet federal services are in many cases still incapable of delivering the best and most affordable services to citizens. The most visible instance of the U.S. government struggling to modernize was the implementation of Healthcare.gov. What was originally a modest budget for deploying the web marketplace of the Affordable Care Act ballooned into a costly implementation that required hiring external consultants and developers to fix. In response to this dysfunction, 18F was formed with the mission to simplify the government’s digital services. Sarah joins Software Engineering Daily to explain how and why 18F was formed, and how it plans to improve government from within in the years to come. Sarah Allen is a Presidential Innovation Fellow who serves as a developer and product manager for 18F. She is also a co-founder of Bridge Foundry, an organization that helps encourage and empower people who are learning to code.

Riak is a distributed NoSQL key value data store. It was created by Basho, taking inspiration from the Amazon Dynamo paper. In this episode, Jon and Jeff discuss the benefits and different implementations of non-relational distributed data stores, and why Riak is a compelling option. Jon Meredith is the chief architect at Basho.

Apache Spark has skyrocketed in popularity lately, arguably surpassing Hadoop as the hottest big data technology as of late. Even IBM has thrown its weight behind the framework, calling it the “most important new open source project in a decade”. In this episode, Holden joins Software Engineering Daily to discuss why Spark is growing in popularity and how developers can begin learning the framework. Holden Karau is a principal engineer at IBM working with Apache Spark. She is also an author of Learning Spark, a technical guide for developers new to the data processing framework.

Apache Drill is a schema­-free SQL query engine for Hadoop, NoSQL, and cloud storage. Drill is unique in that it uses a JSON data model instead of a relational model like many traditional engines. Additionally, it supports on-the-fly schema discovery, which allows execution to begin without knowing the structure of the data. As a result, Drill can work with data that is constantly updating or changing. Tomer Shiran is the founder of the Apache Drill project and the CEO of Dremio.

Machine learning is something that many business are starting to tack onto their existing processes. Yet, to add machine learning capabilities after the fact is often a fool’s errand. Joshua argues that machine learning cannot be an afterthought, but rather must be custom developed to suit the specific problem or question that each company is trying to answer. His company, Wise.io, tackles this challenge of helping business build ground up machine learning applications that generate accurate predictions for use in an array of business processes. Joshua Bloom is the cofounder and CTO of Wise.io. He is also an astrophysicist, and a professor of astronomy at UC Berkeley.

Data science has emerged as an integral discipline alongside software engineering. In this episode, Jeff and Srini discuss the rise of the field, what it means to be a data scientist, and how the field will evolve as more businesses adopt data driven decision making. Srini also explains his thoughts on the essential qualities of a data scientist, and how educating data scientists is a growing challenge in a world that is flooded with “big data”. Srini Kadamati is a data scientist at Dataquest, where he creates content to help people learn data science.

WebTorrent is a streaming torrent client for the browser that can be used without any additional plugins, extensions, or installations. It is written entirely in JavaScript, and uses WebRTC to handle peer-to-peer communication. There are some limitations to making BitTorrent work over the web using WebRTC, including the inability for a browser-based WebTorrent client or “web peer” to communicate directly with a torrent client. However, the potential for every browser to contribute to the hosting and streaming of files is unprecedented. Imagine if large content sites like Netflix or YouTube did not have centralized hosting, and instead relied on its users to seed the content to each other using their web clients. This enables not only more efficient networks, but also the ability for each user to contribute to and be part of the content delivery network. Feross Aboukhadijeh is the creator of WebTorrent. Previously he built PeerCDN, a peer-to-peer content delivery network, which was acquired by Yahoo in 2013.

Django is an open source web development framework that was initially released in 2005, around the same time as Ruby on Rails. In many ways, the two frameworks are similar – they expedited web development by modularizing common components of web apps and enabling developers to quickly prototype and launch their products. They both encouraged the use of the MVC architectural pattern, with each respectively acting as the controller, allowing developers to use their own front end and database technologies. But the path of the two frameworks diverged in other parameters, including mainstream adoption. Ruby on Rails became the sweetheart in Silicon Valley and elsewhere, while Django was loved by a core constituency of pythonistas who had used Python in other applications and fallen in love with it. At the peak of the frameworks’ popularity, there was a common question from new developers asking whether they should first learn Django or Rails – the answer often boiled down to whether you wanted to use Python or Ruby. In this episode, Frank discusses the history and growth of Django, and explains how it evolved in comparison with other frameworks. Frank Wiles is the President of the Django Software Foundation, and partner at Revolution Systems. He worked at the Lawrence Journal-World newspaper, where Django was initially conceived and created.

Ruby on Rails is a web app framework released 10 years ago that influenced the way websites were being built. Rails skyrocketed to popularity in the late 2000’s and empowered many small companies to quickly build and maintain robust web apps. While it is still a mainstay in web development, it has been overshadowed as of late by JavaScript, and Node.js on the backend. David joins Software Engineering Daily to discuss how Ruby on Rails has evolved to keep up with newer programming paradigms and why it is still an excellent choice to build web apps with nowadays. David Heinemeier Hansson is the creator of Ruby on Rails and a partner at Basecamp. Rails was extracted from the work done at Basecamp, and open-sourced for the public to commit to and use for their applications.

Marty Haught is a director of Ruby Central, a nonprofit dedicated to the support and advocacy of the worldwide ruby community. Ruby Central also hosts the RubyConf and RailsConf software conferences.

Michael Hartl is the author of The Ruby on Rails Tutorial, a widely acclaimed guide to learning how to build Ruby on Rails webapps.

Soylent is a meal replacement beverage, available in both liquid and powdered forms. Soylent is designed with the goal of meeting all nutritional requirements for an average adult. John Coogan is the CTO of Soylent.

Druid is a column-oriented distributed database that excels as a data warehousing solution for fast queries on large data sets. Fangjin Yang is a core contributor to Druid, and is currently co-founder and CEO of Imply, which helps build interactive analytics powered by Druid.

Brian Kernighan is a professor of computer science at Princeton University and the author of several books, including “The Go Programming Language” and “The C Programming Language”, a book more commonly referred to as K&R. Professor Kernighan also worked at Bell Labs alongside Unix creators Ken Thompson and Dennis Ritchie and contributed to the development of Unix.

Kitematic is an open source project built to simplify and streamline using Docker on a Mac or Windows machine. It allows users to get up and running on Docker with a single click, and provides a user interface that makes running and managing Docker files easy. Kitematic was acquired by Docker in March 2015. Sean Li is the co-founder and Chief Product Officer of Kitematic. He is now working on product design and developer experience at Docker.

Distributed computing is becoming an increasingly important and complex field within software engineering, with numbers of new data engineering and infrastructure frameworks being released each year. For a beginner, it can be tough to know where to start, so this episode may be considered a primer to learning about distributed systems independently. Alvaro Videla is a core developer for RabbitMQ and a distributed systems blogger, and he joins Software Engineering Daily to explain some of the core concepts of distributed systems.

Michael Hausenblas is a developer and cloud advocate with Mesosphere, which builds the Datacenter Operating System (DCOS), a distributed OS that uses Apache Mesos as its kernel. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit.

Vue.js is a lightweight front-end JavaScript framework that makes it simple to begin prototyping and building web interfaces. Vue.js provides a flexible API for MVVM data bindings, and can serve as an alternative to other front-end frameworks like Angular and React. Evan You is the creator of Vue and a core developer at Meteor. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit.

Quora is a Q&A website where questions are asked, answered, edited and organized by its community of users. Shreyes Seshasai is the Director of Engineering at Quora. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit.

Triplebyte is a technical hiring platform that vets engineers using a comprehensive evaluation platform and connects them to companies that are interesting in hiring them. Triplebyte was part of the Y Combinator summer class of 2015. Ammon Bartram is the Chief Data Officer and cofounder of Triplebyte. Previously, Ammon was lead video developer at Justin.tv and also cofounded SocialCam. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit.

Eric Elliott is a JavaScript advocate, and the author of Programming JavaScript Applications. He is also an entrepreneur tackling social issues including homelessness and inequality in the tech industry. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit.

Tessel is an open-source microcontroller that is programmable in JavaScript and compatible with Node.js. Kelsey Breseman is a hardware engineer and Steering Committee Member on the Tessel Project. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit.

The stream processing paradigm is increasingly being adopted by applications that need to process and handle large volumes of data. Apache Kafka is an open-source distributed publish-subscribe messaging system that is built to support streaming data processing. Neha Narkhede is the one of the creators of Apache Kafka, which she built to address engineering challenges while working at LinkedIn. She is also the co-founder of Confluent, which builds enterprise products around Kafka. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit.

Code Fellows in-person code school offering web and mobile development training in Seattle, Portland, and Chicago. Dave Parker is the CEO of Code Fellows and organizer of Seattle Startup Week. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit.

Scott Hanselman is a technologist, teacher and podcaster who works for Microsoft on its web platform team. Scott hosts several podcasts including Hanselminutes and This Developer’s Life, which cover various topics in software. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit.

TensorFlow is an open source machine learning library intended to bring large-scale, distributed machine learning and deep learning to everyone. Google recently released the framework to the public as a second-generation API, having learned from the successes and failures of DistBelief. Greg Corrado is a senior research scientist and tech lead at Google, where he focuses on the research areas of machine intelligence, machine perception and natural language processing. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit.

Rebecca Garcia is the creator of GeekGirlWeb and founder of CoderDojo NYC. She spends much of her time educating people about the importance of STEM education, particularly in the domain of programming. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit.

Todd Bishop from The GeekWire Podcast recently interviewed Jeff for a discussion on various topics including programming languages, coding bootcamps and the philosophical question of whether programming is an art or a science. Thanks to GeekWire for the opportunity! You can find the show and more at Geekwire: http://www.geekwire.com/2015/is-software-development-art-or-science-geekwire-podcast-with-software-engineering-daily/

Spotify is a streaming music service that uses data science and machine learning to implement product features such as recommendation systems and music categorization, but also to answer internal questions. Boxun Zhang is a data scientist at Spotify where he focuses on understanding user behavior within the product. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit.

The Internet of Things era is upon us, and with it the related concerns of security, privacy and reliability of the connected systems. This episode explores these issues and how companies can prevent these problems through better development processes and lifecycle management. Anders Wallgren is the CTO of Electric Cloud, which builds products to help companies optimize their software build, test, and deployment process. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit.

John K. Paul is an engineering manager and speaker, who has given several talks on Javascript, including JavaScript, the Real Bad Parts. He is the CTO of InRhythm and also an organizer of NYCHTML5. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit.

Richard Golden is the host of Learning Machines 101, a podcast that covers artificial intelligence and machine learning topics. Dr. Golden is also a full-time Professor of Cognitive Science and Electrical Engineering at UT Dallas. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit.

Real-time stream processing of data is becoming widely adopted in the efforts to manage and process “big data”. Some of the top frameworks for processing data streams include Storm, Spark, Samza and Flink. Satish Mittal is an architect at InMobi, an ad platform that needs to deal with processing large volumes of ad data with tight time demands. He recently conducted an analysis of streaming frameworks to determine which of the tools was right for the application at InMobi. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit.

Uber is a transportation and logistics company that manages many aspects of its ride-sharing services through mobile apps and distributed technology. Uber faces unique challenges in rapidly scaling its services internationally, and at one point increased its developer headcount from 200 to over 1000 in less than a year. Matt Ranney is the Chief Systems Architect at Uber and was previously a founder and CTO of Voxer. At QCon San Francisco, he gave a talk called Scaling Uber. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit.

React Native is leading to a future where a developer can build native experiences on web, iOS and Android platforms without having to write 3x the code. Christopher Chedeau is an engineer at Facebook who works on the React team. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit.

Data engineering is a newly emerging field that exists at the intersection of software engineering and data science. Data engineers are increasingly in demand for their abilities to use the various open source big data frameworks to build efficient and robust data pipelines. David Drummond and Austin Ouyang are program directors for the Insight Data Engineering Fellows Program, where they help train data engineers that can immediately start contributing in the industry. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit.

This is episode 100 of Software Engineering Daily, and is a behind the scenes episode on the show that explores the challenges and motivations behind building a daily software podcast. Pranay Mohan is the Producer of Software Engineering Daily.

Let’s Encrypt is a free, automated, and open certificate authority developed by the Internet Security Research Group (ISRG). The ISRG is a non-profit whose mission is to reduce financial, technological, and education barriers to secure communication over the Internet. Josh Aas is the Co-founder and Executive Director of the ISRG. He is also a Senior Technology Strategist at Mozilla. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit.

Free Lossless Image Format (FLIF) is a novel lossless image format which outperforms PNG, lossless WebP, lossless BPG and lossless JPEG2000 in terms of compression ratio. Jon Sneyers is one of the core contributors to FLIF. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit.

Popcorn Time was a free, open-source application that allowed for streaming of movie and television show torrents. Niv Sardi is an activist and developer based in Argentina, who previously worked on the main fork of Popcorn Time, popcorntime.io. Niv is also the founder of the Butter Project, a spin out project from Popcorn Time. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit.

Information retrieval and search engineering are becoming more intertwined with machine learning and natural language processing, leading to a wealth of work to be done in the field. Grant Ingersoll is a founder and CTO of LucidWorks, which helps clients make sense of their data and deploy search applications built on Apache Lucene and Solr. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit.

CodeCombat is a multiplayer live coding strategy game where the players write code to control their characters. Nick Winter is the CEO and Co-founder of CodeCombat. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit.

Seth Godin is a writer, speaker, and entrepreneur. He is the author of many books, including most recently, What To Do When It’s Your Turn. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit.

Technical debt, referring to the compounding cost of changes to software architecture, can be especially challenging in machine learning systems. D. Sculley is a software engineer at Google, focusing on machine learning, data mining, and information retrieval. He recently co-authored the paper Machine Learning: The High Interest Credit Card of Technical Debt. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit.

MotherCoders is a program that helps mothers gain technical skills with the support of a community and benefits like on-site childcare. Tina Lee is the Founder of MotherCoders, having created the program to meet her own need after trying to find existing support systems while learning how to code. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit.

SQLite is an SQL database engine that can be used in applications as a library for quick read-write operations. Unlike other database management systems, SQLite is not a client-server database designed to run in a data center. D. Richard Hipp is the architect and primary author of SQLite and the Fossil SCM. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit.

Trisha Gee is a Java developer and Java Champion. She is currently a Developer Advocate at JetBrains, where she focuses on educating developers on various tools and IDEs. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit.

Apache Flink is an open-source framework for distributed stream and batch data processing. Stephan Ewen is a committer and PMC member of the Flink project, and the CTO of Data Artisans. Hired.com is the job marketplace for software engineers. Go to hired.com/softwareengineeringdaily to get a $600 bonus upon landing a job through Hired. Digital Ocean is the simplest cloud hosting provider. Use promo code SEDAILY for $10 in free credit.