Production-grade ML pipelines
A detailed production‑grade pipeline architecture — covering upstream data quality with Kafka, Flink validation, object‑storage SLAs, feature stores and drift handling — was shared by Aurimas Griciūnas posted. Complementing that, Databricks Lakeflow's Spark Declarative Pipeline Expectations were highlighted as a way to enforce data quality in ETL pipelines noted.
Aurimas published a 13‑tweet thread [threadreaderapp.com] that enumerates an 11‑step ML lifecycle and prescribes operational choices such as loading inference results into offline batch storage for churn workflows and using Redis for low‑latency reads in recommender systems [threadreaderapp.com]. The thread instructs connecting experiment‑tracking metadata to each model artifact in a model registry [threadreaderapp.com] and names a “responsible person” who must switch a candidate model into Production state before the ML training pipeline is considered complete [threadreaderapp.com]. A separate Aurimas post captured on ThreadReader details scheduled validation of object‑storage data against additional SLAs and explicit alerts sent to producers and consumers on SLA breaches [threadreaderapp.com], arguing that SLA validation should run as part of ingestion cadence rather than downstream reporting. Databricks published a Lakeflow guide titled “Manage data quality with pipeline expectations” last updated Feb 3, 2026 [docs.databricks.com] that documents the @dp.expect syntax, requires SQL Boolean constraints for expectations, and explicitly forbids custom Python functions and external service calls inside expectation constraints [docs.databricks.com].
