Components

Open Network Operating System

Open Packet Broker NOS (OPBNOS) set of containerized applications built and runs on top of the open-source SONiC NOS. The NOS provides an affordable solution to aggregate, filter, replicate and load balance network traffic from hardware TAPs.

• Filter, Replicate and Load Balance: OPBNOS provides the basic functionalities of any packet broker on commodity ASICs including filtering based on L2/L3/L4 headers, replicating traffic with unique identification and providing symmetric load balancing.

• Deep Inspection, Truncate, Tunnel: Advance functions including user-defined offset-based inspection of tunnelled packets (IP-in-IP, VXLAN, MPLS, GRE), forward monitored traffic across data centres using IP underlay. One of the unique value-added features includes the truncation of payload reducing the storage cost at tools farm.

• 5G Ready: OPBNOS supports parsing of GTP-C and GTP-U packets, providing advanced filtering and load balancing based on inner headers.

• Unified Management: Aviz FlowVision provides a single pane of glass to manage and visualize the solution. OPBNOS also supports industry-standard CLI and RESTful API for integration into customer orchestration systems.

Packet Capture and Viewer

Open-source Arkimer (Moloch) is deployed for capturing packets from the network aggregators and load balancers. The capture module can sniff packets from the NIC and also supports processing packets from pcap files. The captured packets can be visualized using a simplified viewer providing insights into the metadata including packet headers and payloads. This component also writes the metadata to a centralized storage cluster for future analysis.

Centralized Storage and Visualization

For centralized storage Elastic Storage cluster is utilized which supports distributed scalable storage along with automatic sharding. The metadata saved across the elastic cluster is visualized using Kibana which provides drilled-down information about sessions, network stats and dashboards

Benefits

Cloud Network Infrastructure Monitoring

Monitoring a Cloud network with dislocated data centres can create blindspots that might lead to security and compliance issues. The solution provides comprehensive visibility across the cloud infrastructure, providing end-to-end visibility into the application, and traffic flow and making it easier to correlate network and security events across data centres.

Advanced Monitoring

Aviz OPBNOS exposes the ASIC capabilities for advanced monitoring including deeper packet inspection for security applications, tunnelling (VXLAN) data across data centres using existing IP underlay, and 5G (GTP) packet processing for load balancing using inner headers. All these advanced capabilities support wire-speed forwarding of monitored traffic.

Distributed Solution

The architecture of the complete solution for packet monitoring and analysis is highly distributed, supporting per-site packet capture, analysis and troubleshooting. The extracted metadata from the monitored traffic is stored in a distributed clustered database for centralized analysis and visualization.

Cost Effective

The solution provides end-to-end visibility and wire-speed access to network traffic without the need for investing in new hardware. OPBNOS is built on open-source NOS SONiC which is supported by the majority of the ASIC and switch vendors, customers can easily upgrade to this solution on their existing hardware. Packet capture, analysis and visualization are performed using open-source software running on commodity x86 hardware running Linux.

Aviz OPBNOS provides a highly scalable, flexible and affordable solution to aggregate, filter and load balance network traffic from hardware or virtual TAPs to connected tools for analysis and visualization.

It provides line rate traffic forwarding using switching ASIC by configuring flow paths between two or multiple ports, By leveraging the OPBNOS solution, enterprises can quickly be scaled up or down to meet the ever-changing demands of network visibility and security tools.

• Download the latest build of OPBNOS from here

• OPBNOS can be installed by following the installation guide

• Login into the switch with default credentials

  • User: admin

  • Password: admin

• Configure Management IP

configure terminal
interface mgmt
ip address 10.4.4.96/23 gateway 10.4.4.1
end

• Configure network ports, these are the ports that connect to network-taps

configure terminal
interface ethernet Ethernet63/1
forward-error-correction rs
type network
end

• Configure tool ports, these are the ports that connect to security tools

configure terminal
interface ethernet Ethernet64/1
forward-error-correction rs
type tool
end

• Configure flow to filter and forward traffic

// Traffic Truncation (IPv4 and IPv6)

configure terminal
interface ethernet Ethernet63/1 
truncate 64

// Traffic Redirection (L2 and L3 Traffic)

configure terminal
flow flow01
network-ports Ethernet63/1
tool-ports Ethernet64/1
rule 1 permit vlan 100 counters enable
rule 2 permit vlan 512 counters enable
rule 3 permit src-ip 10.10.10.1/32 dest-ip 20.20.20.1/32 counters enable
rule 4 permit src-ip 213.22.1.2/32 dest-ip 112.23.42.2/32 protocol udp counters enable
enable
end

// LoadBalance Traffic on Multiple Tool Ports

// Traffic Distribution (IPv4 and IPv6) and Symmetric Hashing
configure terminal
port-group 10 ports Ethernet6/1, Ethernet7/1

flow flow2
network-ports Ethernet63/1
tool-ports port-channel10
rule 1 permit src-ip 1.1.1.1/32 dest-ip 2.2.2.2/32 protocol tcp counters enable
rule 2 permit src-ip 2401::1 src-netmask f::f dest-ip 2401::2 dest-netmask f::f counters enable
enable
end

• Verify configured flow

pbnoscli# show flow summary
Flow-Name       Rule-Id        Status      Counter-Value
=========================================================
flow01          2             Active          52562              
flow01          1             Active          56289              
pbnoscli#

The same rules can also be configured by using the Aviz FlowVision:

pbnoscli# show running-config          
configure terminal
interface ethernet Ethernet248
forward-error-correction rs
type network
!
interface ethernet Ethernet252
forward-error-correction rs
type tool
!
interface mgmt
ip address 10.4.4.96/23 gateway 10.4.4.1
!
port-group 10 ports Ethernet6/1, Ethernet7/1
flow flow01
enable
network-ports Ethernet63/1
tool-ports port-group10
rule 1 permit src-ip 10.10.10.1/32 dest-ip 20.20.20.1/32 counters enable
rule 2 permit src-ip 213.22.1.2/32 dest-ip 112.23.42.2/32 protocol udp counters enable
!
pbnoscli#

The packet stream to Arkime will be fed by Aviz OPBNOS, providing the capability to filter, load-balance and aggregate traffic from network taps.

Arkime is a large-scale, open-source, indexed packet capture and search system that augments your current security infrastructure to store and index network traffic in standard PCAP format, providing fast, indexed access.

Arkime uses Elasticsearch (ES) as its backend data store for storing and indexing network traffic data. Elasticsearch is designed for fast indexing and searching of large volumes of data. It also provides a flexible query language that allows users to easily search and filter data based on various criteria, such as IP addresses, ports, protocols, and time ranges while also being highly scalable and can handle large volumes of data across multiple nodes in a cluster.

Kibana is a free and open frontend application that sits on top of Elasticsearch(ES), providing search and data visualization capabilities for data indexed in Elasticsearch. Kibana is a data visualization and management tool for Elasticsearch that provides real-time histograms, line graphs, pie charts, and maps. Kibana also includes advanced applications such as Canvas, which allows users to create custom dynamic infographics based on their data, and Elastic Maps for visualizing geospatial data.

Aviz OPB Visibility solution is truly based on open networking principles using open networking switch hardware running software-defined OPB NOS built on OpenSource NOS SONiC providing aggregation, filtering and load balancing of monitored traffic. The monitoring solution is completely designed using open-source analytics software for packet processing, storage and visibility on commodity x86 nodes.

Pre-requisite

• The following OSes should work out of the box:

  • Arch

  • CentOS/RHEL 7, 8, 9

  • Amazon Linux 2

  • Ubuntu 18.04, 20.04, 22.04

• An installation of Docker Container Engine.

• Download the Kibana installation scripts from here

• Copy the tar to the Kibana server and extract using the below command

aviz@npbsrv01:~/OPB_Arkime$ tar -zxvf OPB_Kibana.tar.gz 
OPB_Kibana/
OPB_Kibana/stop.sh
OPB_Kibana/kibana.tar.gz
OPB_Kibana/start.sh
aviz@npbsrv01:~/OPB_Arkime$  

• Move to the extracted folder

aviz@npbsrv01:~/OPB_Arkime$ cd OPB_Kibana/
aviz@npbsrv01:~/OPB_Arkime/OPB_Kibana/$ ls -l
total 312496
-rw-rw-r-- 1 aviz aviz 319979929 Apr 27 12:13 kibana.tar.gz
-rwxrwxr-x 1 aviz aviz       747 Apr 28 11:31 start.sh
-rwxrwxr-x 1 aviz aviz       155 Apr 28 11:33 stop.sh
aviz@npbsrv01:~/OPB_Arkime/OPB_Kibana$ 

• Execute the ‘start.sh’ script, the script will perform the following actions

  1. Start the Kibana node on port 5601

  2. Connect to the ES Storage node on port 9200

  3. Start the Kibana container

aviz@npbsrv01:~/OPB_Arkime/OPB_Kibana$ sudo ./start.sh 
Loading docker images...
Loaded image: docker.elastic.co/kibana/kibana:7.17.3
vm.max_map_count = 262144
WARNING: Published ports are discarded when using host network mode
5dd385bed21e56393906da9a701e61f69f0e433a4c5a3e11c901860882edd189
Access Kibana Dashboard at:
http://<host>:5601

waiting for initialization...
aviz@npbsrv01:~/OPB_Arkime/OPB_Kibana$

• Open URL http://<Kibana-server-ip>:5601/ from your favourite browser

• Create Kibana Dashboards

If the UI is not accessible,

Check the docker status

Check that Arkime and ES docker are running and the status is ‘UP’ if any of the dockers is not visible. Try running the ‘start.sh’ with the correct permissions, if the issue is not resolved try

aviz@npbsrv01:~/OPB_Arkime/OPB_Analyzer$ sudo docker ps

Check that Elasticsearch is running and the status is ‘green’

aviz@npbsrv01:~/OPB_Arkime/OPB_Analyzer$ curl http://localhost:9200/_cat/health
1671295994 16:53:14 es-docker-cluster green 1 1 21 21 0 0 0 0 - 100.0%
aviz@npbsrv01:~/OPB_Arkime/OPB_Analyzer$

Try restarting the dockers

when Arkime is unable to connect correctly with Elasticsearch the Arkime UI may not be reachable

aviz@npbsrv01:~/OPB_Arkime/OPB_Analyzer$ sudo docker restart arkime
aviz@npbsrv01:~/OPB_Arkime/OPB_Analyzer$ sudo docker restart elasticsearch

Check that UI is reachable by visiting http://arkime-hostname:8005 from your browser

If ES keeps restarting after a system reboot

sysctl -w vm.max_map_count=262144

Add the vm.max_map_count setting to a sysctl configuration file to make the change persistent across reboots:

1. Open the sysctl configuration file /etc/sysctl.conf using a text editor with root privileges: bash

sudo nano /etc/sysctl.conf

1. Add the following line to the end of the file:

vm.max_map_count=262144

Save and close the file

1. To apply the changes, either reboot your system or run the following command to reload the sysctl settings:

sudo sysctl -p

Changing ES port

If port 9002 is used by another service running on the server, it can be changed in the ‘start.sh’ script before execution

Port format: “Global port:Local port”

Using your favourite text editor, change the ‘Global ports’ to any available and accessible port, also update the same port in the ‘ES_PORT’ attribute.

Changing Arkime Password

Log in to the Web GUI and navigate to User> admin_user ⚙️>Password, enter ‘admin’ as the current password and set a new password for the admin user.

Changing Kibana Port

If port 5601 is used by another service running on the server, it can be changed in the ‘start.sh’ script before execution

also if the ES port was changed, the same can be edited here with the ES node IP.

Stopping Arkime & EC

Execute the ‘stop.sh’ script to stop & delete Arkime and EC docker containers, the script will not delete the data & es_data folder and the stored PCAPs.

aviz@npbsrv01:~/OPB_Arkime/OPB_Analyzer$ sudo ./stop.sh 
[sudo] password for aviz: 
arkime
elasticsearch
aviz@npbsrv01:~/OPB_Arkime/OPB_Analyzer$ 

Stopping Kibana

Execute the ‘stop.sh’ script to stop & delete Kibana docker containers

aviz@npbsrv01:~/OPB_Arkime/OPB_Kibana$ sudo ./stop.sh 
[sudo] password for aviz: 
Stopping Kibana...
kibana
kibana
aviz@npbsrv01:~/OPB_Arkime/OPB_Kibana$

External Links

• Download the Arkime installation scripts from here

• Copy the tar to the Arkime server and extract using the below command

aviz@npbsrv01:~/OPB_Arkime$ tar -zxvf OPB_Analyzer.tar.gz 
OPB_Analyzer/
OPB_Analyzer/stop.sh
OPB_Analyzer/elasticsearch.tar
OPB_Analyzer/start.sh
OPB_Analyzer/arkime.tar
aviz@npbsrv01:~/OPB_Arkime$ 

• Move to the extracted folder

aviz@npbsrv01:~/OPB_Arkime$ cd OPB_Analyzer/
aviz@npbsrv01:~/OPB_Arkime/OPB_Analyzer$ ls -l
total 1208660
-rw------- 1 aviz aviz 620387328 Dec 23 04:15 arkime.tar
-rw------- 1 aviz aviz 617262080 Dec 23 04:16 elasticsearch.tar
-rwxrwxr-x 1 aviz aviz      1403 Dec 23 05:21 start.sh
-rwxrwxr-x 1 aviz aviz       160 Dec 23 04:43 stop.sh
aviz@npbsrv01:~/OPB_Arkime/OPB_Analyzer$ 

• Add permission to execute ‘start.sh’ and ‘stop.sh’ scripts

aviz@npbsrv01:~/OPB_Arkime/OPB_Analyzer$ sudo chmod +x start.sh
[sudo] password for aviz:
aviz@npbsrv01:~/OPB_Arkime/OPB_Analyzer$ sudo chmod +x stop.sh
aviz@npbsrv01:~/OPB_Arkime/OPB_Analyzer$

• Execute the ‘start.sh’ script, the script will perform the following actions

  1. Start Local ES storage node on port 9200

  2. Setup directories for PCAP and Log capture

  3. Ask the user for Interfaces to capture data on

  4. Start the Arkime container

aviz@npbsrv01:~/OPB_Arkime/OPB_Analyzer$ sudo ./start.sh 
Loading docker images...
Loaded image: avizdock/docker-arkime:latest
Loaded image: docker.elastic.co/elasticsearch/elasticsearch:7.17.3
Enter Semicolon ";" separated list of interfaces to listen for live traffic: enp130s0f1
vm.max_map_count = 262144
f07938f9e08183c763791733cda1bd19664675b44269fa7bc7e8d800001156b8
18f93340d929695b81915bebf8ed4a275439a25c4542a33944f093115facc17c
Access Arkime Viewer at:
http://<host>:8005
username: admin
password: admin

waiting for initialization...

aviz@npbsrv01:~/OPB_Arkime/OPB_Analyzer$

• Open URL http://<arkime-server-ip>:8005/ from your favourite browser and enter the below credentials

  • User: admin

  • Password: admin

Customer Firewall Configuration

Note: If users want to modify the above-provided ports to run on non-default ports, please refer here to make use of docker port forwarding:

Browser Requirements: Chrome, Safari

Software Requirements

• Ubuntu 22.04

• Docker needs to be installed.

• python3 for running helper scripts.

• The following installation scripts assume static IP addresses have been assigned

  to the hosts that run the docker containers, so having the IP address list of the

  hosts would be helpful.

• Make sure the capture interface for Arkime is in UP state