Delta Lake Catalog Gains Support in Apache Doris
Apache Doris now experimentally supports the Delta Lake Catalog, providing a unified metadata layer for accessing Delta Lake tables. This integration allows teams to query data using various engines, including Trino and Doris, without duplicating metadata management. The feature is aimed at supporting end-to-end lakehouse patterns for both real-time and batch analytics.
- The integration is facilitated by Apache Doris's extensible connector framework, which is compatible with Trino plugins, allowing Doris to use the Trino Delta Lake Connector without requiring a separate Trino cluster. - Apache Doris is a Massively Parallel Processing (MPP) analytical database, originally created at Baidu, that functions as both a data warehouse and a federated query engine for data lakes. This architecture allows it to compete with query engines like Trino, with some benchmarks showing Doris to be 2-3 times faster for data lake queries. - Delta Lake, originally developed by Databricks, is an open-source storage layer that adds ACID transaction capabilities, schema enforcement, and time travel to data stored in the Parquet format. It is one of the three major open table formats, alongside Apache Iceberg and Apache Hudi. - This connection allows Doris to directly query data in Delta Lake, simplifying the architecture required for a lakehouse pattern that unifies batch and real-time data processing without data duplication. - The Delta Lake catalog requires a separate metadata service, such as a Hive Metastore or AWS Glue, which Doris connects to for schema and table information. - By supporting federated queries, this feature enables users to write a single SQL statement in Doris to join data across heterogeneous sources, such as a Delta Lake table, a Hive table, and a MySQL database. - The feature is labeled as experimental in Apache Doris 3.0, indicating it is in the early stages of development and may have limitations compared to more mature integrations.
