/categories/data-engineering/ Recent content in Data Engineering on Kailas Venkitasubramanian Hugo en Sat, 09 Mar 2024 00:00:00 +0000 /blog/series/reproducible-research-series/2022-04-10-designing-reproducible-data-pipelines/ Sat, 09 Mar 2024 00:00:00 +0000 /blog/series/reproducible-research-series/2022-04-10-designing-reproducible-data-pipelines/ <p>In the first post of this series, I argued that reproducibility is not a technical luxury for community research institutions—it is an ethical and operational obligation. In this post, I want to move from philosophy to plumbing—because this is where reproducibility becomes real.</p> <p>Specifically: what does it mean to design <em>reproducible data pipelines</em> in a community research environment?</p> <p>At the UNC Charlotte Urban Institute, this question became concrete as we built the <strong>Quality of Life Explorer</strong>, developed deposit and extraction pipelines for the <strong>Charlotte Regional Data Trust</strong>, and began orchestrating workflows using Apache Airflow in an AWS environment.</p>