Data Pipeline Metrics: Freshness, Throughput, Failures & Cost

What you'll build

What you will be able to do

Each lab ends with a concrete result that you build yourself and the checker confirms, cell by cell.

• Turn 20 start and end timestamps of the Kestrel Goods orders_daily job into minutes, compute the median and p95 runtime and count the runs that broke the 50-minute SLO.
• Compute the throughput of eight nightly Kestrel Goods jobs in rows per second and GB per hour, the weighted throughput of the whole batch and how long a 1.8 TB backfill would take.
• Measure how old eight Kestrel Goods warehouse tables are, compare each age with its freshness SLA, compute the hours over SLA and count the breaches.
• Compute the first-attempt failure rate and retry recovery rate of six Kestrel Goods jobs, the final success rate of all runs and how many runs would still fail with two retries.
• The Kestrel Goods event pipeline was down for 5 hours. You compute the backlog, the catch-up time with the spare capacity, the hour-by-hour backlog and chart the recovery.
• Project the Kestrel Goods events table for six months with a growing daily ingest, roll the table size forward and size the files of a daily partition.
• Price every Kestrel Goods job from runtime, cluster size and node price, total the monthly bill and compare the cost per TB processed, job by job and blended.
• Run the nightly data quality checks of seven Kestrel Goods tables: null rate, duplicate rate, row-count difference against the source, a Pass/Fail status and a chart of the null rates.