Splunk & Snowflake: Federated Search Without Data Movement
▼ Summary
– Cisco and Splunk introduced federated search allowing joint queries of Splunk machine data and Snowflake business data without data movement or duplication.
– The integration enables unified searches from Splunk’s console using SPL-like syntax to correlate data like POS system failures with sales revenue impacts.
– This approach preserves Splunk’s real-time capabilities and Snowflake’s role, focusing on blending both platforms rather than replacing one with the other.
– The design supports open formats like Apache Iceberg and is expected to extend to other data platforms like Databricks and BigQuery in the future.
– Currently in the design phase, the project aims to handle massive data scales while simplifying cross-platform data access for SecOps and ITOps teams.
The new Splunk Federated Search for Snowflake enables organizations to query both Splunk machine data and Snowflake business information simultaneously, eliminating the need for data movement or complex ETL processes. This integration allows teams to perform unified searches directly from the Splunk interface while keeping data in its original location, providing a streamlined approach to data analysis without compromising security or performance.
At the recent Splunk .conf 25 event, Cisco’s Splunk and Snowflake jointly unveiled this federated search capability, designed to search across distributed data repositories without central ingestion. Snowflake executives joined the presentation to confirm immediate availability for joint customers, enabling natural cross-platform queries between machine data in Splunk and structured data in Snowflake.
Carl Perry, Snowflake’s head of analytics, explained the partnership’s objective is delivering “simple, easy, powerful” solutions that maintain data in place while creating unified visibility for security operations, IT teams, and engineering departments. “For Snowflake and Splunk customers, this can be a game changer,” Perry emphasized. “You get more insight at lower cost because you’re not moving data around – and avoiding all the complexity, confusion and errors that go with it.”
This initiative differs from previous collaborations where Snowflake utilized Splunk for security data storage. Instead, the current integration focuses on blending machine data from Splunk with business context from Snowflake through unified searches. A practical example would be correlating point-of-sale system failure data from Splunk with revenue impact information from Snowflake to understand business consequences.
Key technical capabilities include querying Snowflake using SPL-like syntax directly from Splunk, joining Snowflake tables with Splunk indexes to enrich incidents with business context, and intelligently pushing partial queries to Snowflake while completing joins in Splunk for optimal performance. Perry clarified this isn’t about transforming Snowflake into Splunk storage or using Splunk merely as a query layer. Splunk’s real-time streaming, data collectors, and comprehensive security operations capabilities remain essential components of the solution.
The architecture leverages Snowflake’s commitment to open table formats, with Apache Iceberg emerging as the preferred standard despite evaluations of Hudi and Delta formats. Splunk’s compatibility with multiple formats aligns with this vendor-neutral approach, ensuring customers aren’t constrained by specific storage technologies.
Regarding exclusivity, Perry indicated Splunk’s federated search will likely expand to other platforms including Databricks, Fabric, Redshift, and BigQuery, reflecting Splunk’s commitment to serving customers across diverse data environments. However, Snowflake represents the initial implementation, available immediately for existing joint customers.
The project currently resides in the design phase with engineering development forthcoming. Scale considerations are paramount, with Perry referencing “ludicrous data sizes” requiring specialized architecture. Partnership development has progressed smoothly due to shared vision around data accessibility without forced migrations.
During discussions, Perry also addressed Snowflake’s acquisition of Crunchy Data, noting this responded to customer demand for transactional workloads alongside analytics using PostgreSQL, which has become the leading developer database.
Ultimately, the Splunk-Snowflake federation focuses on enabling rapid cross-platform questioning rather than data relocation. If successfully executed, this collaboration could significantly reduce time spent on data management while increasing focus on connecting signals that impact risk management, system reliability, and revenue generation. This partnership represents a strategic alignment between two organizations dedicated to transforming data signals into actionable business intelligence.
(Source: ITWire Australia)