Episode 208
Detecting Outliers in Your Data With Python
The Real Python Podcast · Real Python
June 14, 20241h 7m
Audio is streamed directly from the publisher (dts.podtrac.com) as published in their RSS feed. Play Podcasts does not host this file. Rights-holders can request removal through the copyright & takedown page.
Show Notes
<p>How do you find the most interesting or suspicious points within your data? What libraries and techniques can you use to detect these anomalies with Python? This week on the show, we speak with author Brett Kennedy about his book “Outlier Detection in Python.”</p>
<p>Brett describes initially getting involved with detecting outliers in financial data. He discusses various applications and techniques in security, manufacturing, quality assurance, and fraud. We also dig into the concept of explainable AI and the differences between supervised and unsupervised learning.</p>
<p>This episode is sponsored by APILayer.</p>
<div class="alert alert-primary" role="alert">
<p><strong>Course Spotlight:</strong> <a href="https://realpython.com/courses/knn-python/">Using k-Nearest Neighbors (kNN) in Python</a></p>
<p>In this video course, you’ll learn all about the k-nearest neighbors (kNN) algorithm in Python, including how to implement kNN from scratch. Once you understand how kNN works, you’ll use scikit-learn to facilitate your coding process.</p>
</div>
<p>Topics:</p>
<ul>
<li>00:00:00 – Introduction</li>
<li>00:01:56 – Describing the book</li>
<li>00:03:22 – How did you get involved in outlier detection?</li>
<li>00:06:50 – Initially looking at the data to spot errors</li>
<li>00:08:22 – Amount of fraud and financial errors</li>
<li>00:09:50 – Understanding the nature of the outliers</li>
<li>00:12:15 – Industries that would be interested in detection</li>
<li>00:18:21 – Sponsor: APILayer.com</li>
<li>00:19:15 – Who is the intended audience for the book?</li>
<li>00:22:16 – Differences between supervised vs unsupervised learning</li>
<li>00:25:48 – Autonomous vehicles detecting anomalous imagery</li>
<li>00:29:08 – What is explainable AI?</li>
<li>00:36:21 – Video Course Spotlight</li>
<li>00:37:43 – Detecting an outlier across multiple columns</li>
<li>00:44:32 – Detection of LLM and bot activity</li>
<li>00:49:49 – Proving you are a human checkbox</li>
<li>00:52:25 – What are Python libraries for outlier detection?</li>
<li>00:53:57 – Creating synthetic data to work through examples</li>
<li>00:57:10 – Tools developed and described in the book</li>
<li>01:01:29 – How to find the book</li>
<li>01:02:27 – What are you excited about in the world of Python?</li>
<li>01:04:55 – What do you want to learn next?</li>
<li>01:05:52 – How can people follow your work online?</li>
<li>01:06:16 – Thanks and goodbye</li>
</ul>
<p>Show Links:</p>
<ul>
<li><a href="https://www.manning.com/books/outlier-detection-in-python?utm_source=social_media&utm_medium=int_influencer&utm_campaign=book_kennedy3_outlier_4_9_24">Outlier Detection in Python</a></li>
<li><a href="https://realpython.com/podcasts/rpp/169/">Episode #169: Improving Classification Models With XGBoost – The Real Python Podcast</a></li>
<li><a href="https://xgboost.readthedocs.io/en/stable/">XGBoost Documentation — xgboost 1.7.6 documentation</a></li>
<li><a href="https://shap.readthedocs.io/en/latest/index.html">SHAP (SHapley Additive exPlanations) Documentation</a></li>
<li><a href="https://www.dailymail.co.uk/sciencetech/article-13263567/simple-way-tell-students-used-AI-teacher.html">I’m a teacher and this is the simple way I can tell if students have used AI to cheat in their essays - Daily Mail Online</a></li>
<li><a href="https://github.com/yzhao062/pyod">pyod: A Comprehensive and Scalable Python Library for Outlier Detection (Anomaly Detection)</a></li>
<li><a href="https://github.com/xuhongzuo/DeepOD">DeepOD: Deep learning-based outlier/anomaly detection</a></li>
<li><a href="https://scikit-learn.org/stable/">scikit-learn: machine learning in Python — scikit-learn 1.5.0 documentation</a></li>
<li><a href="https://github.com/Brett-Kennedy/DataConsistencyChecker">DataConsistencyChecker: A Python tool to examine datasets for consistency</a></li>
<li><a href="https://www.linkedin.com/in/wbkenn/">Brett Kennedy - LinkedIn</a></li>
<li><a href="https://github.com/Brett-Kennedy">Brett-Kennedy - GitHub</a></li>
</ul>
<p>Level up your Python skills with our expert-led courses:</p>
<ul>
<li><a href="https://realpython.com/courses/data-cleaning-with-pandas-and-numpy/">Data Cleaning With pandas and NumPy</a></li>
<li><a href="https://realpython.com/courses/knn-python/">Using k-Nearest Neighbors (kNN) in Python</a></li>
<li><a href="https://realpython.com/courses/python-linear-regression/">Starting With Linear Regression in Python</a></li>
</ul> <p><a rel="payment" href="https://realpython.com/join">Support the podcast & join our community of Pythonistas</a></p>