Schema drift checks for Spark

A new arXiv paper shows Scala 3 macros can catch Apache Spark schema drift at compile time, blocking broken pipelines before deployment. (arxiv.org) Schema drift is when a producer’s data shape changes (columns, nullability, nesting) and consumers only notice mismatches when jobs run against real data. (arxiv.org) The paper proposes a small Scala 3 framework that proves producer-to-contract structural compatibility at compile time, derives Spark schemas from the same contract types, and re-checks the DataFrame schema at the sink before writes. (arxiv.org) Author Vittal Mirji submitted the paper to arXiv on April 18, 2026; the submission is a seven-page mechanism artifact with two figures, one table and reproducible benchmarks. (arxiv.org) The proof-of-concept code is published as vim89/compile-time-data-contracts on GitHub and targets Spark 3.5 while using Scala 3 inline macros and policy-typed contracts. (github.com) Technically, an inline macro walks case classes to build a normalized TypeShape and requires a SchemaConforms[Out,Contract,P] compile-time witness; if the proof cannot be derived, compilation aborts with a readable diff. (github.com) At runtime the artifact mirrors the selected policy with a comparator that checks nested-collection optionality and implements structural subset semantics for backward- and forward-compatibility, supplementing Spark’s comparators. (arxiv.org) The paper positions this approach between Typed-Dataset adoption (which requires wholesale uptake) and table-level enforcement (which runs at write time), aiming to “occupy the seam” between compile-time and sink-time checks. (arxiv.org) Practical adoption faces engineering hurdles: Spark historically lacked native Scala 3 spark-sql binaries, so teams have used cross-version workarounds and emerging libraries to run Scala 3 with Spark. (xebia.com) The author and companion tutorials argue compile-time contracts provide fast feedback, explicit diffs and “no surprises at midnight,” with the goal of reducing debugging time for production data engineering teams. (github.com) The paper is explicit about limits: it describes the mechanism narrowly and leaves the broader claim—that compile-time contracts deliver measurable productivity or reliability gains in the field—for future empirical study. (arxiv.org) The code and benchmarks are public for teams to evaluate now, but the author calls for further field measurements before claiming large-scale reliability improvements. (github.com)