Snowflake Introduces AI Data Ingestion
Snowflake has launched OpenFlow, a new service that uses AI-powered workflows for automated data ingestion. The tool reflects a broader industry trend toward AI-driven data pipelines. The system aims to reduce manual work and enable the creation of more intelligent, automated backend systems.
- OpenFlow is built on the open-source dataflow system Apache NiFi; Snowflake acquired Datavolo, a company founded by the co-creators of NiFi, to integrate the technology into its platform. - The service integrates with Snowflake Cortex, allowing pipelines to call on large language models (LLMs) to perform tasks like semantic chunking or summarizing unstructured data like text and images as it is ingested. - For streaming data, OpenFlow connects directly to sources like Apache Kafka and Amazon Kinesis, using a new Snowpipe Streaming integration that provides 10GB/s throughput with a five-second query latency. - It is designed to process and structure multimodal data from sources like Google Drive, SharePoint, and Box, making it usable for building AI applications such as "chat with your data" experiences. - A related feature, Document AI, uses Snowflake's proprietary Arctic-TILT large language model to extract data from documents, including from tables, logos, and even handwritten signatures. - Developers can deploy data pipelines on Snowflake's managed infrastructure or in their own cloud environment, with both options operating as managed services. - All data processing and AI enrichment occur within Snowflake's security perimeter, meaning data doesn't need to be moved to a third-party service, and it adheres to existing role-based access controls.
