Integration with Snowflake

Snowflake is a cloud-based data platform used for data storage, processing, and analytics. This integration allows fetching data from Snowflake tables and pushing it to Kafka for downstream consumption by AI-driven applications for real-time insights, anomaly detection, and predictive analytics

This section covers:

• Use Cases

• Onboard Snowflake platform

• Deployment

Use Cases

• Application Flow Monitoring Using Time-Series Data: Monitor and analyze application flow data in real time using Snowflake and Kafka.

• Fetch Network Logs for Troubleshooting: Retrieve network logs and configuration data from Snowflake for performance analysis.

• Real-Time Insights Using Application Flow Data: Stream application flow data from Snowflake to Kafka for real-time insights.

• Anomaly Detection in Security Events: Detects security anomalies by analyzing event data stored in Snowflake.

• Predictive Maintenance Using Time-Series Data: Use Snowflake’s time-series data for predictive maintenance of network devices.

• Automated Response to System Failures: Automate responses to network failures using data stored in Snowflake and Kafka-driven alerts

Onboard Snowflake Platform

Setup Snowflake DB and Table

Before running the collector, ensure your Snowflake environment is configured correctly. Snowflake-collector is a microservice to fetches data from snowflake tables and publishes it onto kafka as a topic for downstream service consumption

1. Snowflake Database

   • Login into a Snowflake account via the Snowflake Web UI.

   • In the Navigation Bar, select Databases to view your database

   • Navigate to Worksheets, and run the SQL commands to create the required database and table.

   • Run the following commands in the Worksheets

   CREATE DATABASE FLOW_DATA_DB;

   -- Use the default PUBLIC schema and create the FLOW_DATA table

   CREATE OR REPLACE TABLE FLOW_DATA_DB.PUBLIC.FLOW_DATA (

   TYPE VARCHAR(16777216),

   MESSAGE VARIANT,

   SRC VARCHAR(16777216),

   TIME_RECEIVED TIMESTAMP_NTZ(9)

   );

2. Data Ingestion into snowflake

   • Snowflake table FLOW_DATA will look like this once populated

   
3. Discovering and Validating Snowflake Credentials

   • In Snowflake, we can retrieve the Snowflake Account information using the SHOW ACCOUNTS; command in the Snowflake Worksheets. When you execute this query, you will get details about the account, including the account_locator and account_name, which together form the Snowflake Account URL

   

   • Similarly, we can retrieve warehouses, schemas, and databases

Deployment

The Snowflake Collector is part of the NCP package, that can be invoked from CLI to create integration

• Navigate to ingestion -> snowflake inside the NCP directory

• Run the Installer ./install.sh

• Provide the installer with the information below,

  • Snowflake Account URL

  • Snowflake Username and Password

  • Snowflake Warehouse, Database, and Schema

Uninstall Snowflake Integration

To Uninstall Snowflake integration run the below script from the NCP package,

• Navigate to ingestion -> snowflake inside the NCP directory

• Run the Installer ./uninstall.sh