dbt Labs Reports 64% Cut in Compute Costs
dbt Labs reports a 64% reduction in its dbt-related compute costs by combining its Fusion concurrency engine with "state-aware orchestration." This approach triggers data transformation jobs based on data changes rather than fixed schedules. The method is designed to optimize cloud spend and data freshness for large-scale analytics pipelines on platforms like Snowflake and Databricks.
- The "state-aware orchestration" feature moves dbt from a stateless to a stateful application by maintaining a real-time fingerprint of both model code and data state. This allows it to pinpoint and refresh only the models where upstream data has changed or code has been modified, eliminating unnecessary processing. - dbt Labs' new Fusion engine, a rewrite of the core dbt logic in Rust instead of Python, provides the foundation for these performance enhancements. This shift enables significantly faster parsing and compilation, which is crucial for large-scale projects, and allows for more intelligent, state-aware job orchestration. - Simply enabling state-aware orchestration can reduce compute spend by about 10% on average, with the potential for over 50% in total savings when combined with more specific configurations. For its own internal projects, dbt Labs reported that turning on this feature would save over 9% of its annual data warehouse costs and result in 25% fewer excess models being built. - The Fusion engine and its cost-saving features are available for dbt Cloud customers on the Enterprise plan running projects on Snowflake, Databricks, BigQuery, and Redshift. - This cost-saving feature is part of a broader industry trend focusing on cloud cost optimization for platforms like Snowflake and Databricks, where expenses can escalate quickly. Key strategies include rightsizing virtual warehouses, optimizing queries, and leveraging features like auto-suspend to avoid paying for idle compute. - dbt Labs was last valued at $4.2 billion after its $222 million Series D funding round in February 2022, with strategic investors including Databricks and Snowflake. The company is estimated to have reached $100 million in annual recurring revenue in 2024. - The company, founded by CEO Tristan Handy in 2016, initially started as a consulting firm, Fishtown Analytics, and built the open-source dbt (data build tool) for its own use before pivoting to focus on the software. - Beyond cost savings, the Fusion engine also introduces features aimed at improving the developer experience, such as faster error detection without needing to run a query against the warehouse and more accurate data lineage tracking.
