ASN Visualization Integrated Installer
Overview
This document describes the integration of Kibana and Elasticsearch into the ASN installation, providing visualization capabilities for ASN data.
Required Files
`kibana-es-install.sh`: The main installation script that sets up Elasticsearch, Kibana, and Kafka Connect integration.
`docker-compose.yml`: Defines the containerized visualization stack.
`kafka-elasticsearch-sink-connector.tar.gz`: Connector for streaming data from Kafka to Elasticsearch.
`es-conn.sh`: Configures the connection between Kafka and Elasticsearch.
`asn-demo-dashboard.ndjson`: Pre-configured visualization dashboard.
Usage Instructions
Installation
Execute: `sudo ./kibana-es-install.sh`
Accessing the Dashboard
Open the Kibana URL displayed at the end of installation.
Log in using the credentials provided.
Navigate to the Dashboard section to view the ASN data visualizations.
Credentials Management
Credentials are displayed at the end of installation.
Installation Process
The visualization stack is deployed after the standard ASN installation. The installation script performs these key functions:
Infrastructure Setup
Deploys Elasticsearch, Kibana, and Kafka Connect using Docker Compose.
Configures security settings with password authentication.
Sets up connectivity between components.
Security Configuration
Enables secure authentication for Elasticsearch.
Creates and manages access credentials.
Configures Kibana to authenticate with Elasticsearch.
Data Pipeline Configuration
Connects Kafka topics to Elasticsearch indices.
Configures data transformation for visualization.
Dashboard Deployment
Imports pre-configured dashboards into Kibana.
How does the script work?
The `kibana-es-install.sh` script performs the following steps:
Validates the presence of required files.
Detects the system IP address for proper configuration.
Updates Docker Compose configuration (with system ip).
Extracts and installs the Kafka-Elasticsearch connector in the /etc directory.
Starts the visualization stack containers in the correct order (zookeper > kafka > all containers).
Sets up secure passwords for Elasticsearch users (uses elastic search reset commands to generate passwords).
Configures Kibana to use these credentials (updates required .yml file in the kibana container).
Updates the Kafka-Elasticsearch connector configuration (IP changes to system ip in the conn script along with the authentication details before calling the curl commands).
Imports the pre-configured dashboard (uses curl command to save object in the Stack Management of kibana).
Outputs access information including URLs and credentials.
Troubleshooting
Kibana page troubleshooting
Execute “sudo docker compose down” and execute the install script, this is the case where already the containers are up and installation script needs to be executed
Dashboard Import Issues
If dashboard import fails during installation, manually import using the command shown at the end of installation output.
Connection Issues
Ensure all services are running with `sudo docker ps`.
Authentication Problems
Verify the credentials in the Kibana configuration. It is suggested to use the elastic user and the corresponding password to login to the kibana
Other Details
Data Flow
Kafka topics → Kafka Connect → Elasticsearch → Kibana
This integration enhances the ASN platform with visualization capabilities while maintaining a simple shell script based deployment process.
Telco Kibana Packaging Steps -
Required Files -
asn_kafka_telco_dashboard.sh - Acts as the master script to set up Elasticsearch, Kibana, and Kafka Connect integration.
es-telco-conn.sh - Sets up the es-epc-control-sink, es-user-sink, es-radius-control-sink connectors between kafka and elastic search
index-mapp.sh - Pushes index mappings for EPC-control-session-data topic to parse the time fields in the date format
ASN-TELCO-EPC.ndjson - Dashboard that holds the visualization for EPC control session data kafka records
ASN-RADIUS.ndjson - Dashboard that holds the visualization for Radius session data kafka records\
ASN-TELCO-USER.ndjson - Dashboard that holds the visualization for Telco User session data kafka export records
Usage Instructions
Installation
Execute: `sudo ./asn_kafka_telco_dashboard.sh`
Accessing the Dashboard
Open the Kibana URL displayed at the end of installation.
Log in using the credentials provided.
Navigate to the Dashboard section to view the ASN data visualizations
ASN-RADIUS-DASHBOARD
ASN-TELCO-UPLANE-Dashboard
ASN-TELCO-CONTROL
Open the kafka UI URL displayed at the end of installation
Navigate to the Topics section and click on the overview to get kafka statistics information
Credentials Management
Credentials are displayed at the end of installation.
Kibana page troubleshooting
Execute “sudo docker compose down” and execute the install script, this is the case where already the containers are up and installation script needs to be executed
DC Kibana Packaging Steps -
Required Files -
asn_kafka_dashboard.sh - Acts as the master script to set up Elasticsearch, Kibana, and Kafka Connect integration
es-dc-conn.sh- Sets up the es-user-sink connector between kafka and elastic search
ASN-DC-FTTH.ndjson - Dashboard that holds the visualization for DC user session data kafka records
Usage Instructions
Installation
Execute: sudo ./asn_kafka_dashboard.sh
Accessing the Dashboard
Open the Kibana URL displayed at the end of installation.
Log in using the credentials provided.
Navigate to the Dashboard section to view the ASN data visualizations.
Open the kafka UI URL displayed at the end of installation
Navigate to the Topics section and click on the overview to get kafka statistics information
Credentials Management
Credentials are displayed at the end of installation.
Kibana page troubleshooting
Execute “sudo docker compose down” and execute the install script, this is the case where already the containers are up and installation script needs to be executed
Steps to import new dashboards
TELCO Dashboard
Run ./import_telco_dashboard.sh
Save the updated elastic credentials and login again into elastic for viewing the newly updated dashboard
Note - Existing dashboards ASN-TELCO-CONTROL and ASN-TELCO-UPLANE-Dashboard will be replaced
DC FTTH Dashboard
Run ./import_dashboard.sh
Save the updated elastic credentials and login again into elastic for viewing the newly updated dashboard
Note - Existing dashboard ASN-DC-FTTH-Dashboard will be replaced
Last updated