Episode 201
Decoupling Systems to Get Closer to the Data
The Real Python Podcast · Real Python
April 19, 20241h 9m
Audio is streamed directly from the publisher (dts.podtrac.com) as published in their RSS feed. Play Podcasts does not host this file. Rights-holders can request removal through the copyright & takedown page.
Show Notes
<p>What are the benefits of using a decoupled data processing system? How do you write reusable queries for a variety of backend data platforms? This week on the show, Phillip Cloud, the lead maintainer of Ibis, will discuss this portable Python dataframe library.</p>
<p>Phillip contrasts Ibis’s workflow with other Python dataframe libraries. We discuss how “getting close to the data” speeds things up and conserves memory.</p>
<p>He describes the different approaches Ibis provides for querying data and how to select a specific backend. We discuss ways to get started with the library and how to access example data sets to experiment with the platform. </p>
<p>Phillip discovered Ibis while looking for a tool that allowed him to reuse SQL queries written for a specific data platform on a different one. He recounts how he got involved with the Ibis project, sharing his background in open source and learning how to contribute to a first project. </p>
<p>This episode is sponsored by Mailtrap.</p>
<div class="alert alert-primary" role="alert">
<p><strong>Course Spotlight:</strong> <a href="https://realpython.com/courses/python-folium-web-maps-from-data/">Creating Web Maps From Your Data With Python Folium</a></p>
<p>You’ll learn how to create web maps from data using Folium. The package combines Python’s data-wrangling strengths with the data-visualization power of the JavaScript library Leaflet. In this video course, you’ll create and style a choropleth world map showing the ecological footprint per country.</p>
</div>
<p>Topics:</p>
<ul>
<li>00:00:00 – Introduction</li>
<li>00:02:18 – How did you get started with Ibis?</li>
<li>00:08:10 – First contribution to open source</li>
<li>00:13:46 – Comparing Ibis to other dataframe libraries</li>
<li>00:20:09 – Sponsor: Mailtrap</li>
<li>00:20:43 – What goes into the selection of backend?</li>
<li>00:27:07 – Database connections vs SQL compilers</li>
<li>00:30:03 – Raw SQL approach</li>
<li>00:34:06 – Dataframe approach</li>
<li>00:38:31 – What does “getting close to the data” mean?</li>
<li>00:41:52 – Video Course Spotlight</li>
<li>00:43:24 – Phillip in the cloud - YouTube channel</li>
<li>00:44:56 – Access to sample data sets</li>
<li>00:50:11 – Additional resources</li>
<li>00:52:50 – What are some of the backends Ibis supports?</li>
<li>00:54:13 – Entry points to the platform</li>
<li>00:55:00 – How are you supported?</li>
<li>00:57:10 – Exporting a SQL query</li>
<li>00:59:23 – What are you excited about in the world of Python?</li>
<li>01:04:28 – What do you want to learn next?</li>
<li>01:07:12 – How can people follow your work online?</li>
<li>01:08:00 – Thanks and goodbye</li>
</ul>
<p>Show Links:</p>
<ul>
<li><a href="https://ibis-project.org/">Ibis - the portable Python dataframe library</a></li>
<li><a href="https://voltrondata.com/">The Leading Designer and Builder of Enterprise Data Systems - Voltron Data</a></li>
<li><a href="https://peps.python.org/pep-0249/">PEP 249 – Python Database API Specification v2.0</a></li>
<li><a href="https://github.com/tobymao/sqlglot">sqlglot: Python SQL Parser and Transpiler - GitHub</a></li>
<li><a href="https://ibis-project.org/tutorials/getting_started#interactive-mode">Ibis – getting_started</a></li>
<li><a href="https://github.com/ibis-project/ibis-examples?tab=readme-ov-file">ibis-examples: A repository of runnable examples using ibis</a></li>
<li><a href="https://ibis-project.org/reference/">Ibis – Reference Documentation</a></li>
<li><a href="https://pyscript.net/">PyScript - Run Python in your HTML</a></li>
<li><a href="https://prefix.dev/">pixi - Prefix.dev</a></li>
<li><a href="https://github.com/astral-sh/uv">uv: An extremely fast Python package installer and resolver, written in Rust</a></li>
<li><a href="https://us.pycon.org/2024/">PyCon US 2024</a></li>
<li><a href="https://www.learncraftspanish.com/">LearnCraft Spanish – Fluency for Serious Learners</a></li>
<li><a href="https://github.com/ibis-project/ibis">ibis: the portable Python dataframe library - GitHub</a></li>
<li><a href="https://ibis-project.org/posts">Ibis – Blog Posts</a></li>
<li><a href="https://www.youtube.com/@cpcloud">Phillip in the Cloud - YouTube</a></li>
<li><a href="https://twitter.com/cpcloudy">Phillip Cloud (@cpcloudy) / X</a></li>
<li><a href="https://github.com/cpcloud">cpcloud (Phillip Cloud) · GitHub</a></li>
</ul>
<p>Level up your Python skills with our expert-led courses:</p>
<ul>
<li><a href="https://realpython.com/courses/building-project-documentation-mkdocs/">Building Python Project Documentation With MkDocs</a></li>
<li><a href="https://realpython.com/courses/python-folium-web-maps-from-data/">Creating Web Maps From Your Data With Python Folium</a></li>
<li><a href="https://realpython.com/courses/using-raise-for-effective-exceptions/">Using raise for Effective Exceptions</a></li>
</ul> <p><a rel="payment" href="https://realpython.com/join">Support the podcast & join our community of Pythonistas</a></p>