Case Study: a large healthcare organization reduces ETL runtime and computing costs with Six Feet Up

a large healthcare organization cuts ETL runtime by 20 hours with Six Feet Up

The large healthcare organization, a not-for-profit entity, faced a significant challenge with its Extract, Transform and Load (ETL) pipeline, which took 24 hours to run. This lengthy runtime left little opportunity to troubleshoot issues or rerun data without violating service-level agreements. The organization engaged Six Feet Up to improve the pipeline's codebase for reduced human error, optimize it for speed and ensure clear data visibility.

Six Feet Up consolidated the codebase, standardized libraries and eliminated repetitive code. They migrated the pipeline to Apache Airflow 2.0, upgraded to Azure Data Lake Storage Gen2 and parallelized processes. As a result, Six Feet Up reduced the pipeline's runtime by 20 hours, from 24 to 4 hours, providing more time for troubleshooting and drastically reducing computing costs for the customer.