Data Lineage: Understanding Data Lineage at Scale with Julien Le Dem
Software Engineering Daily · softwareengineeringdaily.com
Audio is streamed directly from the publisher (traffic.megaphone.fm) as published in their RSS feed. Play Podcasts does not host this file. Rights-holders can request removal through the copyright & takedown page.
Show Notes
Big Data has exploded the past decade as cloud computing and more efficient hardware made scaling essentially limitless. Products like Uber revolve entirely around analyzing data to provide rides. According to an EMC/IDC study, there was approximately 5.2TB of data for every person in 2020. That estimate was made before the transition to remote work, which likely makes it much higher.
The term “data lineage” refers to the collection, origin, storage, transfer, and use of data over time. Given the size of the Big Data industry and related industries, maintaining a thorough data lineage, even within small companies, can be very difficult. It becomes especially challenging at scale. What innovative tools make understanding all this information possible? Can data really continue growing at this rate?
In this episode we talk with Julien Le Dem, CTO and Co-Founder at Datakin. We discuss the challenges, available tools, and future for big data and data lineage.
Sponsorship inquiries: [email protected]