ASN Visualization Integrated Installer

Overview

This document describes the integration of Kibana and Elasticsearch into the ASN installation, providing visualization capabilities for ASN data.

Required Files

  • `kibana-es-install.sh`: The main installation script that sets up Elasticsearch, Kibana, and Kafka Connect integration.

  • `docker-compose.yml`: Defines the containerized visualization stack.

  • `kafka-elasticsearch-sink-connector.tar.gz`: Connector for streaming data from Kafka to Elasticsearch.

  • `es-conn.sh`: Configures the connection between Kafka and Elasticsearch.

  • `asn-demo-dashboard.ndjson`: Pre-configured visualization dashboard.

Usage Instructions

Installation

  • Execute: `sudo ./kibana-es-install.sh`

Accessing the Dashboard

  • Open the Kibana URL displayed at the end of installation.

  • Log in using the credentials provided.

  • Navigate to the Dashboard section to view the ASN data visualizations.

Credentials Management

  • Credentials are displayed at the end of installation.

Installation Process

The visualization stack is deployed after the standard ASN installation. The installation script performs these key functions:

Infrastructure Setup

  • Deploys Elasticsearch, Kibana, and Kafka Connect using Docker Compose.

  • Configures security settings with password authentication.

  • Sets up connectivity between components.

Security Configuration

  • Enables secure authentication for Elasticsearch.

  • Creates and manages access credentials.

  • Configures Kibana to authenticate with Elasticsearch.

Data Pipeline Configuration

  • Connects Kafka topics to Elasticsearch indices.

  • Configures data transformation for visualization.

Dashboard Deployment

  • Imports pre-configured dashboards into Kibana.

How does the script work?

The `kibana-es-install.sh` script performs the following steps:

  • Validates the presence of required files.

  • Detects the system IP address for proper configuration.

  • Updates Docker Compose configuration (with system ip).

  • Extracts and installs the Kafka-Elasticsearch connector in the /etc directory.

  • Starts the visualization stack containers in the correct order (zookeper > kafka > all containers).

  • Sets up secure passwords for Elasticsearch users (uses elastic search reset commands to generate passwords).

  • Configures Kibana to use these credentials (updates required .yml file in the kibana container).

  • Updates the Kafka-Elasticsearch connector configuration (IP changes to system ip in the conn script along with the authentication details before calling the curl commands).

  • Imports the pre-configured dashboard (uses curl command to save object in the Stack Management of kibana).

  • Outputs access information including URLs and credentials.

Troubleshooting

Kibana page troubleshooting

  • Execute “sudo docker compose down” and execute the install script, this is the case where already the containers are up and installation script needs to be executed

Dashboard Import Issues

  • If dashboard import fails during installation, manually import using the command shown at the end of installation output.

Connection Issues

  • Ensure all services are running with `sudo docker ps`.

Authentication Problems

  • Verify the credentials in the Kibana configuration. It is suggested to use the elastic user and the corresponding password to login to the kibana

Other Details

Data Flow

  • Kafka topics → Kafka Connect → Elasticsearch → Kibana

This integration enhances the ASN platform with visualization capabilities while maintaining a simple shell script based deployment process.

Telco Kibana Packaging Steps -

Required Files -

asn_kafka_telco_dashboard.sh - Acts as the master script to set up Elasticsearch, Kibana, and Kafka Connect integration.

es-telco-conn.sh - Sets up the es-epc-control-sink, es-user-sink, es-radius-control-sink connectors between kafka and elastic search

index-mapp.sh - Pushes index mappings for EPC-control-session-data topic to parse the time fields in the date format

ASN-TELCO-EPC.ndjson - Dashboard that holds the visualization for EPC control session data kafka records

ASN-RADIUS.ndjson - Dashboard that holds the visualization for Radius session data kafka records\

ASN-TELCO-USER.ndjson - Dashboard that holds the visualization for Telco User session data kafka export records

Usage Instructions

Installation

  • Execute: `sudo ./asn_kafka_telco_dashboard.sh`

Accessing the Dashboard

  • Open the Kibana URL displayed at the end of installation.

  • Log in using the credentials provided.

Navigate to the Dashboard section to view the ASN data visualizations

  • ASN-RADIUS-DASHBOARD

  • ASN-TELCO-UPLANE-Dashboard

  • ASN-TELCO-CONTROL

  • Open the kafka UI URL displayed at the end of installation

Navigate to the Topics section and click on the overview to get kafka statistics information

Credentials Management

  • Credentials are displayed at the end of installation.

Kibana page troubleshooting

  • Execute “sudo docker compose down” and execute the install script, this is the case where already the containers are up and installation script needs to be executed

DC Kibana Packaging Steps -

Required Files -

asn_kafka_dashboard.sh - Acts as the master script to set up Elasticsearch, Kibana, and Kafka Connect integration

es-dc-conn.sh- Sets up the es-user-sink connector between kafka and elastic search

ASN-DC-FTTH.ndjson - Dashboard that holds the visualization for DC user session data kafka records

Usage Instructions

Installation

Execute: sudo ./asn_kafka_dashboard.sh

Accessing the Dashboard

  • Open the Kibana URL displayed at the end of installation.

  • Log in using the credentials provided.

Navigate to the Dashboard section to view the ASN data visualizations.

Open the kafka UI URL displayed at the end of installation

Navigate to the Topics section and click on the overview to get kafka statistics information

Credentials Management

  • Credentials are displayed at the end of installation.

Kibana page troubleshooting

  • Execute “sudo docker compose down” and execute the install script, this is the case where already the containers are up and installation script needs to be executed

Steps to import new dashboards

TELCO Dashboard

  • Run ./import_telco_dashboard.sh

  • Save the updated elastic credentials and login again into elastic for viewing the newly updated dashboard

Note - Existing dashboards ASN-TELCO-CONTROL and ASN-TELCO-UPLANE-Dashboard will be replaced

DC FTTH Dashboard

  • Run ./import_dashboard.sh

  • Save the updated elastic credentials and login again into elastic for viewing the newly updated dashboard

Note - Existing dashboard ASN-DC-FTTH-Dashboard will be replaced

Last updated