1 of 26

FTAS R3.1

Introduction

Fabric Test Automation Suite (FTAS) is a comprehensive collection of test cases packaged as a virtual machine. The Objective of FTAS is to verify the necessary features and functions of SONiC NOS for Fabric Deployment Readiness. The test cases are primarily focused on qualifying the fabric for functions, features, scale, day-2 operations and chaos scenarios.

What's New?

Release 3.1

QoS
Stress testing
EVPN/L2VXLAN enhancements
More SNMP coverage
Fast reboot testing
Platform CLI coverage
More verifications in resilience testcase

Release 3.0

STP
Storm control
Static Lag
Dynamically find the route scalability limits
Route scalability enhancements
Default route advertisement
Max MED verification
Runtime optimization

Release 2.3.0

MCLAG with Layer2
MCLAG with Layer3
EVPN VxLAN
IPv6 prefix route(4K, 64K, 128K) Scalability
BGP Dual Stack (64K and 128K) Scalability
Core file detection during test execution

Release 2.2.0 (10/5/2023)

ECMP Scalability
BGP Dual Stack Scalability
BGP Graceful Restart
New BGP netops usecases
Dynamic Port Breakout
Support for chaos on routed ports (no port channel configuration)
Support for chaos on routed port-channel
Debuggability enhancements
- Syslog capture for all devices for the testcase duration
- Techsupport dump for in case of test failure
- Printing DOM information if ports fail to come up
- Easier identification of Ixia sessions by appending test id to session name

Release 2.1.0 (7/7/2023)

Scalability enhancements:
- Replaced the cases with fixed values by the variant of supported range test cases.
- Parametrized scale value to the maximum supported range .(example: scale_val = 64/128/256/512 )
BGP Netops coverage:
- eBGP multi-AS config, adjacency, route convergence and data path using router interface
- eBGP Multi-AS Route Convergence and data path using loopback
- BGP Node Drain - Add route map to remove and restore SPINE nodes 1 and 2 using the Community list
- Test Link drain - Apply for Route-Map permit and for IPv6 Traffic/Prefixes and prefix lists
- Node drain: with IPv6 Traffic, Test Node Drain (Spine 1 and Spine 2) for IPv6 Traffic/Prefixes
Debuggability enhancements:
- Added device status data before and after test cases and added log messages.
- Added FTAS version display in final report log.
Platform-specific suite files:
- Added suite files for Wistron, Nvidia and EC
eBay specific suite files:
- Added suite files for EC4630, EdgeCore AS-97xx
Platform/Version compatibility check:
- Parameterized variables were added to validate the topology file and supported sonic version.
Support for auto cleanup before the test run:
- Using this variable devices can be cleaned up forcefully before the test run.
- Cleans up the ACL, routes, ipv4 interfaces, VLANs, port-channel configurations if any on the devices and brings down all the ports before the test run starts.

Installation

Host Requirements

Hardware

CPU: x86_64 8 cores or more with Virtualization enabled
Memory: 8GB or more system memory
Disk Space: 256GB or more available disk space
Network Interface: 1 GbE NIC

For FTAS with ONES integration, more disk space and RAM is needed

Memory: 16GB or more system memory Disk Space: 512GB or more available disk space

Operating System

Ubuntu 20.04 or later (64-bit)

Other flavours of Linux that support KVM should also be able to run the FTAS.

However, it's important to note that these alternative distributions have been specifically tested for compatibility with FTAS, and therefore, users who opt for non-Ubuntu Linux systems to run FTAS should be aware that they may encounter compatibility issues and may need to perform additional configuration and testing on their own.

Hypervisor Software

KVM (Kernel-based Virtual Machine) is the leading open-source virtualisation technology for Linux. It installs natively on all Linux distributions and turns underlying physical servers into hypervisors so that they can host multiple, isolated virtual machines (VMs).

We will be using KVM to serve as the hypervisor for the FTAS VM because KVM being a type-1 hypervisor, it outperforms all type-2 hypervisors, ensuring near-metal performance.

Please refer to the following steps to install it on the host machine

Ensure that the latest Ubuntu packages are installed

sudo apt-get update && sudo apt-get upgrade

Install KVM packages

sudo apt install libvirt-clients libvirt-daemon-system libvirt-daemon virtinst bridge-utils qemu qemu-kvm

Check if KVM acceleration is ready

kvm-ok

# You should see a message like "KVM acceleration can be used"
sonic@sonic-39:~$ kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used
sonic@sonic-39:~$

Add user to libvirt group

Verify if the libvirt user group is available using the below command

sudo getent group | grep libvirt

sonic@sonic-39:~$ sudo getent group | grep libvirt
libvirt:x:119:sonic,root
libvirt-qemu:x:64055:libvirt-qemu
libvirt-dnsmasq:x:120:
sonic@sonic-39:~$

If the libvirt group is not available, it can be created using the below command

sudo groupadd --system libvirt

sonic@sonic-39:~$ sudo groupadd --system libvirt
groupadd: group 'libvirt' already exists
sonic@sonic-39:~$

Then add the current user to the group

sudo usermod -a -G libvirt $(whoami)

Set user and group for qemu. Update the qemu config with your user and libvirt group

sudo vi /etc/libvirt/qemu.conf

# Some examples of valid values are:
#
#       user = "qemu"   # A user named "qemu"
#       user = "+0"     # Super user (uid=0)
#       user = "100"    # A user named "100" or a user with uid=100
#
#user = "root"
user = "<your host user>" 


# The group for QEMU processes run by the system instance. It can be
# specified in a similar way to the user.
group = "libvirt"

Restart the libvirtd service

sudo systemctl stop libvirtd
sudo systemctl start libvirtd

Check the status of the libvirtd service

sudo systemctl status libvirtd

sonic@sonic-39:~$ sudo systemctl status libvirtd
● libvirtd.service - Virtualization daemon
   Loaded: loaded (/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2023-02-18 10:16:26 UTC; 27s ago
     Docs: man:libvirtd(8)
           https://libvirt.org
 Main PID: 68774 (libvirtd)
    Tasks: 33 (limit: 32768)
   CGroup: /system.slice/libvirtd.service
           ├─54120 /usr/bin/qemu-system-x86_64 -name guest=ftas03,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-8-ftas03/master-key.aes -machine pc-i440fx-1.5,accel
           └─68774 /usr/sbin/libvirtd

Feb 18 10:16:26 sonic-39 systemd[1]: Starting Virtualization daemon...
Feb 18 10:16:26 sonic-39 systemd[1]: Started Virtualization daemon.
lines 1-13/13 (END)

If your server has a GUI desktop installed, you may want to install virt-manager. The virt-manager application is a desktop Graphical user interface for managing virtual machines through libvirt

sudo apt-get install virt-manager

Network Configuration

It is recommended that the virtual NIC on the VM should be bridged with the physical NIC on the host machine.

In this sample configuration eno1 is the physical NIC of the host machine, which is typically used for SSH(Management).

Netplan configuration for Linux bridge using DHCP

#/etc/netplan/00-installer-config.yaml
network:
  ethernets:
    enp1s0:
      dhcp4: no
  bridges:
    br0:
      interfaces: [enp1s0]
      dhcp4: yes
      mtu: 1500
      parameters:
        stp: true
        forward-delay: 4
      dhcp6: no
  version: 2

Netplan configuration for Linux bridge using static IP

#/etc/netplan/00-installer-config.yaml
network:
  ethernets:
    enp1s0:
      dhcp4: no
  bridges:
    br0:
      interfaces: [enp1s0]
      addresses: [172.16.1.100/24]
      gateway4: 172.16.1.1
      mtu: 1500
      nameservers:
        addresses: [8.8.8.8, 8.8.4.4]
      parameters:
        stp: true
        forward-delay: 4
      dhcp4: no
      dhcp6: no
  version: 2

Please assign the same static IP as on the physical Management NIC(enp1s0).

After this step, IP will be re-assigned to the bridge interface(br0) and the Physical interface(enp1s0) will act as a Layer-2 interface.

Apply the above configuration

sudo netplan apply

This step will reset the SSH connection and reassign the static IP from the physical interface(enp1s0) to the bridge interface(br0).

Supported Traffic Generators

Traffic Generator

Type

Version

Deploy the VM

The VM image is provided as a gzip file. It should be decompressed to get the qcow2 image to deploy as a VM.

The FTAS VM has Aviz ONES integrated into it and will take some time to initialise after the first boot.

sonic@sonic-39:~$ gunzip -c ftas_ones_vmi_1.1.2.qcow2.gz > ftas_ones_vmi_1.1.2.qcow2
sonic@sonic-39:~$ ls -l
total 8302936
-rw-rw-r-- 1 sonic sonic 4929683456 Feb 21 06:21 ftas_ones_vmi_1.1.2.qcow2
-rw-rw-r-- 1 sonic sonic 3572510886 Feb 21 06:20 ftas_ones_vmi_1.1.2.qcow2.gz
sonic@sonic-39:~$

You can connect to the console port of the VM to see the installation logs.

Create the VM using GUI App virt-manager

If your host server has Ubuntu Desktop and virt-manager installed you can use it to deploy the VM. Make sure you can start the Virtual Machine Manager and that it connects successfully to the local hypervisor.

Creating a VM with virt-manager is very straightforward, Use the following steps to deploy the FTAS VM

File -> New Virtual Machine -> Import existing disk image -> Forward

Browse to the FTAS disk image location and select Ubuntu as the OS name
Click "Forward" and select vCPU (min 2 cores) and Memory (4GB) for the VM

Click "Forward", give your VM a name and check "Customize configuration before install"
Select "NIC ...", in the "Network source" select the Linux bridge you created on the host machine

Apply the configuration and start the VM

Create the VM using XML configuration

Create an XML configuration file from the following template

vi ftas.xml

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>FTAS_VM01</name>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-1.5'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='writeback'/>
      <source file='/home/oper/taas_vm/taas_vm_v3.qcow2' />
      <target bus='virtio' dev='vda'/>
    </disk>
    <serial type='pty'>
      <source path='/dev/pts/3'/>
      <target port='0'/>
    </serial>
    <!-- Management interface eth0 -->
    <interface type='network'>
	<model type='e1000' />
        <source network='br0'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x00' function='0x0'/>
    </interface>
   <controller type='usb' index='0'/>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </memballoon>
  </devices>
</domain>

The below lines can be changed to customize the VM installation:

Line #2 the Name of the VM
Lines #3 & #4 the amount of System Memory for the VM
Line #5 the number of vCPU Core for the VM
Line #25 the Path to the qcow2 VM image file
Line #35 the name of the Linux bridge on the host machine

Create a Linux bridge configuration file (bridged-network.xml) for libvirt from the following template

vi bridged-network.xml

<network>
    <name>br0</name>
    <forward mode="bridge" />
    <bridge name="br0" />
</network>

Line #4 the name of the Linux bridge on the host machine

Define the Linux bridge for the VM

#Execute the below command to attach the VM to the Linux Bridge 
sonic@sonic-39:~$ virsh net-define bridged-network.xml
sonic@sonic-39:~$ virsh net-start br0
sonic@sonic-39:~$ virsh net-autostart br0
sonic@sonic-39:~$ virsh net-list
 Name                 State      Autostart     Persistent
----------------------------------------------------------
 br0                  active     yes           yes

sonic@sonic-39:~$

Start the VM

virsh create <VM XML configuration file>

#sonic@sonic-39:~$ virsh create ftas.xml 
#Domain FTAS_VM01 created from ftas.xml
#sonic@sonic-39:~$

If you see a permission error run the virsh command with sudo may fix the issue

Check the VM status

sonic@sonic-39:~$ virsh list
 Id    Name                           State
----------------------------------------------------
 8     FTAS_VM01                      running
sonic@sonic-39:~$

Configure the IP address on the VM

If there is a DHCP server on the management network the VM will obtain its IP configuration from the DHCP server
If there is no DHCP server or you want to configure the IP address statically, Follow the below steps

Enter VM console

sonic@sonic-39:~$ virsh console FTAS_VM01
Connected to domain ftas03
Escape character is ^]

ftasvm login:

The default username is 'oper' with the default password 'oper@123'

Check connections and devices

sudo nmcli con show

oper@ftasvm:~$ sudo nmcli con show
NAME                UUID                                  TYPE      DEVICE 
Wired connection 1  782de6d4-3867-3c5e-95fb-061ae39e5fae  ethernet  eth0   
oper@ftasvm:~$ 
# Capture the connection NAME of eth0 device

Release IP assigned by DHCP

sudo dhclient -v -r

oper@ftasvm:~$ sudo dhclient -v -r
Internet Systems Consortium DHCP Client 4.4.1
Copyright 2004-2018 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

Listening on LPF/veth1dcacbe/b6:bc:e5:4a:7e:1f
Sending on   LPF/veth1dcacbe/b6:bc:e5:4a:7e:1f
<..>
Sending on   Socket/fallback
oper@ftasvm:~$

Configure static IP for the connection

sudo nmcli con mod "Wired connection 1" ipv4.addresses <ip address>/<prefix>

#Example - sudo nmcli con mod "Wired connection 1" ipv4.addresses 192.168.0.37/24

Set a default Gateway address

sudo nmcli connection modify "Wired connection 1" ipv4.gateway <GW Address>

#Example - sudo nmcli connection modify "Wired connection 1" ipv4.gateway 192.168.0.1

Set the IP configuration mode to manual

sudo nmcli con mod "Wired connection 1" ipv4.method manual

Reapply the configuration to the interface

sudo nmcli device reapply <dev_name>

#Example - sudo nmcli device reapply eth0

Verify the IP address

#verify the IP address
ip a

oper@ftasvm:~$ ip a
<..>
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:37:3c:5c brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.37/25 brd 192.168.0.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::70a4:9f2e:658c:4d29/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
<..>
oper@ftasvm:~$ 


#Verify IP method
oper@ftasvm:~$ sudo nmcli -f ipv4.method con show "Wired connection 1"
ipv4.method:                            manual
oper@ftasvm:~$

Test FTAS VM reachability from outside the VM, if the VM is not reachable please check the access rule at the below location,

Host Machine

sonic@sonic-39:~$ cat /proc/sys/net/bridge/bridge-nf-call-iptables 
1
sonic@sonic-39:~$

If the above value is 1 please change it to 0 and the reachability issue should be resolved

Host Machine

sudo echo 0 > /proc/sys/net/bridge/bridge-nf-call-iptables

There are some scaling scripts that require multiple network service servers (NTP, SYSLOG, TACACS+, etc.). In order to simulate this, we can add a secondary IP address to the VM NIC.

To add a secondary IP address, use the command

FTAS VM

sudo nmcli con mod "<con_name>" +ipv4.addresses <ip address>/<prefix>
#Example - sudo nmcli con mod "Wired connection 1" ipv4.addresses 192.168.0.42/24

# Reapply config
sudo nmcli device reapply <dev_name>
#Example - sudo nmcli device reapply eth0

# Show IP address to verify
ip a
# Restart docker containers so their services can listen on new IP addresses

Network services containers

The FTAS VM has docker containers running and the following docker images installed:

DHCP container image ztp_dhcp(DHCP sevice)

oper@ftasvm:~$ docker images
REPOSITORY    TAG       IMAGE ID       CREATED        SIZE
ztp_dhcp      v1        599313a03bfb   41 hours ago   83.3MB
netservices   v1        8a9c98506637   41 hours ago   259MB
oper@ftasvm:~$ d

ztp_dhcp(DHCP) services are not run by default as it might conflict with DHCP running in the DC infra.

Net Services container image netservices:v1 (NTP, SYSLOG, TACACS+ services). This container is run with the "--network=host" option. If you need to change the configurations of the services please find them in the following configuration files.

# NTP configuration file:
/etc/ntp.conf

# SYSLOG Configuration file
/etc/syslog-ng/conf.d/syslog_ng.conf

# Log files location:
/var/log/sonic_logs/<IP address of devices>.log

# TACACS+ configuraration
/etc/tacacs+/tac_plus.conf

Dockers running by default:

oper@ftasvm:~$ docker ps -a
CONTAINER ID   IMAGE            COMMAND                  CREATED        STATUS        PORTS     NAMES
8add12060a57   netservices:v1   "/usr/bin/supervisord"   41 hours ago   Up 41 hours             net_services
oper@ftasvm:~$

User Login

Users can log in to the FTAS VM using one of the following methods

Console login

virsh console <VM_domain_name>

#Example -  
#sonic@sonic-39:~$ virsh console FTAS_VM01
#Connected to domain FTAS_VM01
#Escape character is ^]
#oper@ftasvm:~$

SSH login

ssh <username>@<mgmt ip address of the VM>

#Example -
#sonic@sonic-39:~$ ssh oper@192.168.3.37
#oper@192.168.3.37's password: 
#Welcome to Ubuntu 20.04.4 LTS (GNU/Linux 5.4.0-137-generic x86_64)

# * Documentation:  https://help.ubuntu.com
# * Management:     https://landscape.canonical.com
# * Support:        https://ubuntu.com/advantage

#This system has been minimized by removing packages and content that are
#not required on a system that users do not log into.

#To restore this content, you can run the 'unminimize' command.
#Last login: Tue Feb 21 08:24:31 2023
#oper@ftasvm:~$

The default username is 'oper' with the default password 'oper@123'

After logging in user will be dropped into the 'Bash shell'

with the following pre-defined folders and files

oper@ftasvm:~$ ls -lrth
total 44K
drwxrwxr-x  3 oper oper 4.0K Nov 18 16:54 ones
-rwxrwxr-x  1 oper oper 1.9K Jan 27 10:19 qjob.py
drwxr-xr-x  2 oper oper 4.0K Jan 31 07:26 jobs
drwxr-xr-x  2 oper oper 4.0K Feb  7 11:03 configs
-rwx------  1 oper oper 6.2K Feb  7 11:03 jobs.py
drwxrwxr-x  2 oper oper 4.0K Feb 18 12:54 __pycache__
drwxrw-rw-  2 oper oper 4.0K Feb 18 12:54 logs
drwxr-xr-x  2 oper oper 4.0K Feb 18 14:06 testbeds
drwxr-xr-x  2 oper oper 4.0K Feb 18 14:07 testsuites
drwxrwxrwx 10 oper oper 4.0K Feb 20 04:58 reports
oper@ftasvm:~$

qjob.py - Script to schedule execution jobs.
testbeds - Directory to create and maintain testbed parameter files.
jobs - Directory containing the JSON file that holds the job queue. qjob.py script controls and edits this JSON file. Please don't edit the JSON file manually.
testsuites - Directory to maintain testsuite yaml files.
reports - Directory to store HTML reports of completed jobs.
configs - Directory to store test configs
jobs.py - Script to manipulate queue jobs. It is imported by the qjob.py utility.
logs - Maintains execution logs file of all jobs. Users can clean up the files in the logs and reports folder to regain disk space when needed.

Test Setup

Before scheduling any jobs validate the physical testbed to make sure all links are connected and are operationally UP, also we need a clean configuration file with valid interface settings (breakout, speed, FEC, admin status, etc.), no IP interfaces, no BGP instances, and no QoS. The scripts will use this clean configuration file to restore the DUT to its default configuration as part of the clean-up process.

The cleanup configuration file should be created at /etc/sonic/clean_config.json. The clean configuration file follows the config_db.json format. It must include the port-related settings for lane mapping, speed, and admin status.

How to create a clean config file

For every DUT in the testbed, backup the default config(if needed)

Start with the default configuration after a fresh installation of SONiC

Alternatively, you can also create a clean config by editing the config_db.json as below
- Edit config_db.json and remove the following configuration blocks and save the file
  - VLAN
  - VLAN_MEMBER
  - PORTCHANNEL
  - PORTCHANNEL_MEMBER
  - BGP configuration
  - Loopback interfaces
Edit "DEVICE_METADATA" in /etc/sonic/config_db.json as below
- Configure "hostname" for each device (Example: Leaf01, Leaf02, Spine01, Spine02)
- Add "docker_routing_config_mode": "split" configuration

Configure the below list through config_db.json:
- The management IP address for eth0 and gateway

Port breakout (if any)
Port speed, FEC, Auto-negotiation on links connected to other devices and Ixia
Configure "admin_status": "down"

Save config_db.json
Copy config_db.json to /etc/sonic/clean_config.json

clean_config.json should be built using the text editor or SONiC CLI, but not both.

Load the configuration on the device

Add the following line to /etc/sonic/frr/vtysh.conf

Cleanup the BGP configuration from FRR

Chaos Test Setup

TestBed File

Build the Chaos testbed file using "~/testbeds/ftas_chaos_topo.py"

Disable all cleanup options

The chaos test suite loads a base configuration to all DUTs and Ixia for its test scripts. So ensure the following variables are set to False in the testbed file.

Clean up DUTs Configuration Manually

Please make sure DUTs have a clean or default configuration before the Chaos test run.

Remove the test statistics report file

Chaos suite generates a statistics report file at ~/reports/report.txt to track its execution status and metric data for all test scripts. So please make sure to remove this file before running the Chaos test to avoid Ixia library errors

Test Bed Configuration

A Testbed parameter file is a Python script which defines the testbed parameters as variables. Testbed files are available in the ~/testbeds folder.

Full Mesh 4 DUTs Topology

All test scripts except Chaos can be run with full mesh 4 DUTs topology

Use the following sample script and steps to create your own testbed file.

Update the above details in the testbed file with your DUTs - DUT connections, Management IP, Login credentials, Link Speed etc.
The "name" parameter is very important. Provide a string to identify the respective for this parameter. This name is displayed in logs for easy identification of devices.
Refer to the topology diagram to find the link variables and update their values
All testbed files can be found in the folder "~/testbeds/"

Testbed file variables

There are two variables to control the cleanup after each test run in the testbed file.

They are CLEANUP_BY_REBOOT and CLEANUP_BY_CFG_RELOAD:

CLEANUP_BY_REBOOT = True, The script will restore the switch's configuration from the /etc/sonic/clean_config.json file and then reboot the switch. This process consumes additional execution time, but it ensures that the DUTs (Devices Under Test) are consistently configured in a clean and proper manner for subsequent test scripts.
CLEANUP_BY_CFG_RELOAD = True, The script will restore the switch's configuration from the /etc/sonic/clean_config.json file and then issue the sudo config load command to load the clean config file. This method takes less time to have a clean configuration on switches but may not work correctly sometimes.
If both CLEANUP_BY_REBOOT and CLEANUP_BY_CFG_RELOAD are set to False, The scripts use the SONiC CLI procedure to un-configure whatever was configured on the switches by the scripts.
Ideally, the below three parameters should be set to False
- CLEANUP_BY_REBOOT = False
- CLEANUP_BY_CFG_RELOAD = False
- CFG_RELOAD_BY_REBOOT = False
CLEANUP_BEFORE_TEST_RUN = False, It can be used to do the forceful clean-up of all the devices before the test run
- The following configuration will be cleaned up.

Apart from these, the following variables are specified in the testbed file:

CFG_RELOAD_BY_REBOOT = True, The scripts initiate device reboots in instances where the config reload command fails on the DUT due to certain reasons. This measure is taken as a workaround for such situations.
REBOOT_WAIT_TIME = 0, Maximum wait time for the device to reboot
NTP_SERVER = <FTAS VM IP>, FTAS VM serves as NTP server.
SYSLOG_SRVS = {"Servers": ["<FTAS VM IP>", "10.4.5.6"], "Log_Folder": "/var/log/sonic_logs"}, first list member should be set to the FTAS VM IP
MAX_V4_ACL , Maximum IPv4 ACL rules supported on the platform
MAX_V6_ACL , Maximum IPv6 ACL rules supported on the platform
MAX_SECONDARY_SUBNET, Maximum secondary subnets supported for SVI interface
MAX_IPV4_HOST_ROUTES, Maximum IPv4 host routes supported on the platform
MAX_IPV6_HOST_ROUTES, Maximum IPv6 host routes supported on the platform
MAX_IPV4_PREFIX_ROUTES, Maximum IPv4 prefix routes supported on the platform
MAX_IPV6_PREFIX_ROUTES, Maximum IPv6 prefix routes supported on the platform
TECHSUPPORT = True, Takes techsupport dump (If True) for the DUTs in case of failures
TECHSUPPORT_SINCE = "hour ago", Specifies the argument to the show techsupport command, in case TECHSUPPORT = True
TECHSUPPORT_TIMEOUT = 300, specifies the worst case timeout value for techsupport dump generation, in case TECHSUPPORT = True
ACCEPTABLE_DELTA = 0.5, Threshold value for acceptable packet/frame loss percentage.
CPU_MEM_THRESHOLD = 5 , Threshold value for acceptable change in CPU/memory utilization percentage
STRESS_AVAIL_CORES = 2 , Number of CPU cores reserved for system use; rest other CPU cores will undergo stress testing
STRESS_MEM_UTIL = 85 , Targeted percentage of total system memory to allocate for stress testing
SERVER_IP = "x.x.x.x" , IP address of the server hosting the stress-ng Docker image
SERVER_USER_ID = "oper" , User ID for SCP access to the server hosting the stress-ng Docker image
SERVER_PASSWORD = "oper@123" , Password for SCP access to securely transfer the stress-ng Docker image

Configuring Ixia Variables

For UHD Chassis:

Specify the port connected to the DUT like this: "s1_p1": "8" , where the port number 8 is connected to Spine01_Port01 for Ixia traffic.
For example, Ixia port 8 can be configured like this: "localuhd/8", where localuhd refers to the Ixia chassis and 8 is the UHD port number.

For Novus Chassis:

Specify the port connected to the DUT like this: "s1_p1": "1;8" , where the port number 8 on Ixia card 1 is connected to Spine01_Port01 for Ixia traffic.
For example, Ixia port 1;8 can be configured like this: "<chassis_ip>;<card_no>;<Port_no>", where <chassis_ip> refers to the Ixia chassis IP and 1;8 is the port with the card number.

The FTAS VM has been extensively tested with UHD version 1.3.3003.118, It is recommended to use UHD with the same version for smooth operations.

Ixia port format varies and depends on the Ixia Chassis type

Control Ixia traffic rate globally

For the testbeds where links between DUTs have less bandwidth than Ixia ports traffic tests may fail due to traffic drops on the least bandwidth links (i.e. DUTs may be connected with 1G ports and the Ixia link maybe 100G).

To avoid this situation a global parameter for the ixia traffic rate "global_traffic_rate" in percentage could be used to enforce traffic rate in all tests.

When the global_traffic_rate parameter is defined as a sub-parameter in the "Ixia_Ports" all traffic streams in all test scripts would override their own rate value with global_traffic_rate value configured.

When global_traffic_rate is not defined all traffic streams in all test scripts would use their own static values for the traffic rate.

Here the "global_traffic_rate": 25 referrers to 25% speed for all Ixia ports

CLI_TIMEOUT, set the integer value for this variable when the command execution is slow or taking more than 30 sec to respond. Set the default value to 0 in the testbed file.

2 DUTs Topology

If you don't have 4 DUTs topology and want to run scripts that require only 2 DUTs, You can use 2dut_topo.py testbed parameter file

There are two physical DUTs in the topology but the script might pick the name Spine and Leaf interchangeably. So in the testbed file, we should define parameters for both but they point to the same physical Spine and Leaf DUTs.

Full Mesh 4 DUTs Topology for Chaos

ftas_chaos_topo.py (sample topology file) the testbed parameter file is used for Chaos test scripts only.

A new parameter PCH_CONFIGURATION = False, has been added. This parameter determines if the tests are to be run on routed port (if False) or routed PortChannel (if True) configuration for the interfaces connecting DUTs.

Test Suite Configuration

Test suite configuration is a text file where you can list all test functions (each test function is a test case) for a batch run. Test suite files are stored in the "~/testsuites" folder

Below is an example of how a test suite file is structured:

TEST***_FOLDER - Path to the test artefacts including testbeds, testsuites and reports
TEST_CONTACT - Email address of the test owner. This information is included in the test report
TESTSUITES - Define the sub-test suites in a key-value pair for execution
- "./essential/taas_platform_Interface_test.py" - Location of the test script files.
- SKIP - Defines whether a particular test script will be included or excluded from the execution. If set to "SKIP": true, then the tests will be excluded from execution.
- COMMON_TESTBED - The testbed file all test cases use.
- TESTCASES - A list of test functions (test cases) with the structure of "TESTCASES": [{"test_syslog_004": ""},{"test_syslog_002": ""}].

Customer suites

FTAS comes with the following predefined suites which can be run directly on the applicable platform and release combinations. These suite files are present in ~/testsuites directory

Test Case Execution

In the home folder of the logged-in user, there is a Python script name "qjob.py". This script handles test scheduling.

Following is a brief usage of the script:

The script can take the following actions:

Show the current queue

When no tests are scheduled the script will show an empty queue.

Adding test suite to the queue

we can add multiple test suits to the queue at any time to be executed.

Removing the test suite from the queue

Changing queue status

There are two statuses in the job queue:

When the job queue is in "paused" status, the test_runner service does not pick any job in the queue for execution.

When the job queue is in "running" status, the test_runner service picks the oldest job in the queue for execution. After the test case execution is complete the queue test_runner service changes the queue status to "paused".

Changing the queue status from running to paused when there is a running job, won't stop the running job but the test_runner service won't pick the next job for execution.

Kill or terminate a running job

After terminating the running job test_runner service pauses the queue.

Test Case Results Reporting

The FTAS test_runner service collects the logs from the test case execution, captures and saves them to "~/logs/jobs.log" file.

The FTAS also creates an HTML version of the reports available "~/reports/test_report_20230218_**/"

To view the HTML test report, Visit the URL at http://<VM IP addr>:8090/

Deleting Test Reports

Test Cases

No.

Description

Test Case ID

Topology

PD(Platform dependent) means that FTAS is designed to work with specific hardware and software configurations and may not be compatible with other platforms.

Support

How to contact Aviz Networks Support?

Contact Support

The Aviz Network Support team can be reached by

Sending an email to
Submitting a Ticket at
Live Chat on

Submitting a Ticket

A ticket can be submitted with or without an account at the

Mandatory Fields:

Subject
Issue Type (Post Deployment, Pre-Deployment, General Query, RMA)
Priority (Low, Normal, High, Urgent)
Description

Optional Fields:

External ID (Community Request ID or Past Case Number)
Hardware (Switch Model)
ASIC vendor (chipset)
Serial Number
Host Name
Attachments (Tech Support Dump, Screenshots, Logs)

For Technical Issues, we recommend the description include the following:

Repro steps, if the issue is reproducible
The sequence of events that lead to the failure state
Artefacts - Tech Support dump (tar.gz file), Logs, Command Outputs, Topology Diagrams etc...

Deploy the VM

The VM image is provided as a gzip file. It should be decompressed to get the qcow2 image to deploy as a VM.

The FTAS VM has Aviz ONES integrated into it and will take some time to initialise after the first boot.

sonic@sonic-39:~$ gunzip -c ftas_ones_vmi_1.1.2.qcow2.gz > ftas_ones_vmi_1.1.2.qcow2
sonic@sonic-39:~$ ls -l
total 8302936
-rw-rw-r-- 1 sonic sonic 4929683456 Feb 21 06:21 ftas_ones_vmi_1.1.2.qcow2
-rw-rw-r-- 1 sonic sonic 3572510886 Feb 21 06:20 ftas_ones_vmi_1.1.2.qcow2.gz
sonic@sonic-39:~$

You can connect to the console port of the VM to see the installation logs.

Create the VM using GUI App virt-manager

Creating a VM with virt-manager is very straightforward, Use the following steps to deploy the FTAS VM

File -> New Virtual Machine -> Import existing disk image -> Forward

Browse to the FTAS disk image location and select Ubuntu as the OS name
Click "Forward" and select vCPU (min 2 cores) and Memory (4GB) for the VM

Click "Forward", give your VM a name and check "Customize configuration before install"
Select "NIC ...", in the "Network source" select the Linux bridge you created on the host machine

Apply the configuration and start the VM

Create the VM using XML configuration

Create an XML configuration file from the following template

vi ftas.xml

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>FTAS_VM01</name>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-1.5'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='writeback'/>
      <source file='/home/oper/taas_vm/taas_vm_v3.qcow2' />
      <target bus='virtio' dev='vda'/>
    </disk>
    <serial type='pty'>
      <source path='/dev/pts/3'/>
      <target port='0'/>
    </serial>
    <!-- Management interface eth0 -->
    <interface type='network'>
	<model type='e1000' />
        <source network='br0'/>
        <address type='pci' domain='0x0000' bus='0x00' slot='0x00' function='0x0'/>
    </interface>
   <controller type='usb' index='0'/>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </memballoon>
  </devices>
</domain>

The below lines can be changed to customize the VM installation:

Line #2 the Name of the VM
Lines #3 & #4 the amount of System Memory for the VM
Line #5 the number of vCPU Core for the VM
Line #25 the Path to the qcow2 VM image file
Line #35 the name of the Linux bridge on the host machine

Create a Linux bridge configuration file (bridged-network.xml) for libvirt from the following template

vi bridged-network.xml

<network>
    <name>br0</name>
    <forward mode="bridge" />
    <bridge name="br0" />
</network>

Line #4 the name of the Linux bridge on the host machine

Define the Linux bridge for the VM

#Execute the below command to attach the VM to the Linux Bridge 
sonic@sonic-39:~$ virsh net-define bridged-network.xml
sonic@sonic-39:~$ virsh net-start br0
sonic@sonic-39:~$ virsh net-autostart br0
sonic@sonic-39:~$ virsh net-list
 Name                 State      Autostart     Persistent
----------------------------------------------------------
 br0                  active     yes           yes

sonic@sonic-39:~$

Start the VM

virsh create <VM XML configuration file>

#sonic@sonic-39:~$ virsh create ftas.xml 
#Domain FTAS_VM01 created from ftas.xml
#sonic@sonic-39:~$

If you see a permission error run the virsh command with sudo may fix the issue

Check the VM status

sonic@sonic-39:~$ virsh list
 Id    Name                           State
----------------------------------------------------
 8     FTAS_VM01                      running
sonic@sonic-39:~$

Configure the IP address on the VM

If there is a DHCP server on the management network the VM will obtain its IP configuration from the DHCP server
If there is no DHCP server or you want to configure the IP address statically, Follow the below steps

Enter VM console

sonic@sonic-39:~$ virsh console FTAS_VM01
Connected to domain ftas03
Escape character is ^]

ftasvm login:

The default username is 'oper' with the default password 'oper@123'

Check connections and devices

sudo nmcli con show

oper@ftasvm:~$ sudo nmcli con show
NAME                UUID                                  TYPE      DEVICE 
Wired connection 1  782de6d4-3867-3c5e-95fb-061ae39e5fae  ethernet  eth0   
oper@ftasvm:~$ 
# Capture the connection NAME of eth0 device

Release IP assigned by DHCP

sudo dhclient -v -r

oper@ftasvm:~$ sudo dhclient -v -r
Internet Systems Consortium DHCP Client 4.4.1
Copyright 2004-2018 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/

Listening on LPF/veth1dcacbe/b6:bc:e5:4a:7e:1f
Sending on   LPF/veth1dcacbe/b6:bc:e5:4a:7e:1f
<..>
Sending on   Socket/fallback
oper@ftasvm:~$

Configure static IP for the connection

sudo nmcli con mod "Wired connection 1" ipv4.addresses <ip address>/<prefix>

#Example - sudo nmcli con mod "Wired connection 1" ipv4.addresses 192.168.0.37/24

Set a default Gateway address

sudo nmcli connection modify "Wired connection 1" ipv4.gateway <GW Address>

#Example - sudo nmcli connection modify "Wired connection 1" ipv4.gateway 192.168.0.1

Set the IP configuration mode to manual

sudo nmcli con mod "Wired connection 1" ipv4.method manual

Reapply the configuration to the interface

sudo nmcli device reapply <dev_name>

#Example - sudo nmcli device reapply eth0

Verify the IP address

#verify the IP address
ip a

oper@ftasvm:~$ ip a
<..>
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:37:3c:5c brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.37/25 brd 192.168.0.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::70a4:9f2e:658c:4d29/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
<..>
oper@ftasvm:~$ 


#Verify IP method
oper@ftasvm:~$ sudo nmcli -f ipv4.method con show "Wired connection 1"
ipv4.method:                            manual
oper@ftasvm:~$

Test FTAS VM reachability from outside the VM, if the VM is not reachable please check the access rule at the below location,

Host Machine

sonic@sonic-39:~$ cat /proc/sys/net/bridge/bridge-nf-call-iptables 
1
sonic@sonic-39:~$

If the above value is 1 please change it to 0 and the reachability issue should be resolved

Host Machine

sudo echo 0 > /proc/sys/net/bridge/bridge-nf-call-iptables

There are some scaling scripts that require multiple network service servers (NTP, SYSLOG, TACACS+, etc.). In order to simulate this, we can add a secondary IP address to the VM NIC.

To add a secondary IP address, use the command

FTAS VM

sudo nmcli con mod "<con_name>" +ipv4.addresses <ip address>/<prefix>
#Example - sudo nmcli con mod "Wired connection 1" ipv4.addresses 192.168.0.42/24

# Reapply config
sudo nmcli device reapply <dev_name>
#Example - sudo nmcli device reapply eth0

# Show IP address to verify
ip a
# Restart docker containers so their services can listen on new IP addresses

Network services containers

The FTAS VM has docker containers running and the following docker images installed:

DHCP container image ztp_dhcp(DHCP sevice)

oper@ftasvm:~$ docker images
REPOSITORY    TAG       IMAGE ID       CREATED        SIZE
ztp_dhcp      v1        599313a03bfb   41 hours ago   83.3MB
netservices   v1        8a9c98506637   41 hours ago   259MB
oper@ftasvm:~$ d

ztp_dhcp(DHCP) services are not run by default as it might conflict with DHCP running in the DC infra.

Net Services container image netservices:v1 (NTP, SYSLOG, TACACS+ services). This container is run with the "--network=host" option. If you need to change the configurations of the services please find them in the following configuration files.

# NTP configuration file:
/etc/ntp.conf

# SYSLOG Configuration file
/etc/syslog-ng/conf.d/syslog_ng.conf

# Log files location:
/var/log/sonic_logs/<IP address of devices>.log

# TACACS+ configuraration
/etc/tacacs+/tac_plus.conf

Dockers running by default:

oper@ftasvm:~$ docker ps -a
CONTAINER ID   IMAGE            COMMAND                  CREATED        STATUS        PORTS     NAMES
8add12060a57   netservices:v1   "/usr/bin/supervisord"   41 hours ago   Up 41 hours             net_services
oper@ftasvm:~$

Test Bed Configuration

A Testbed parameter file is a Python script which defines the testbed parameters as variables. Testbed files are available in the ~/testbeds folder.

Full Mesh 4 DUTs Topology

All test scripts except Chaos can be run with full mesh 4 DUTs topology

Use the following sample script and steps to create your own testbed file.

//Sample TestBed file
LEAF01_Ports = {"s2_p1": "Ethernet32", "s2_p1_speed": 100000,
                "s2_p2": "Ethernet36", "s2_p2_speed": 100000,
                "s1_p1": "Ethernet0", "s1_p1_speed": 100000,
                "s1_p2": "Ethernet4", "s1_p2_speed": 100000,
                "l2_p1": "Ethernet16", "l2_p1_speed": 100000,
                "l2_p2": "Ethernet20", "l2_p2_speed": 100000,
                "ixia_p1": "Ethernet60", "ixia_p1_speed": 100000,
                "ixia_p2": "Ethernet48", "ixia_p2_speed": 100000,
                "port_mtu": 9100
                }
LEAF01 = {"IP": "10.4.4.66", 
          "PROTO": "http", "REST_PORT": 6002, "SSH_PORT": 22,
          "CLI_PROMPTS": params.CLI_PROMPTS,
          "cliErrors": params.CLI_ERROR_REGEXP, "cliWarnings": params.CLI_WARN_REGEXP,
          "ssh_user": "admin",
          "ssh_passwd": "YourPaSsWoRd",
          "Timeout": 30, "ports": LEAF01_Ports, "name": "MLNX-LEAF01",
          "backup_cfg_file": "clean_config.json"
          }

Update the above details in the testbed file with your DUTs - DUT connections, Management IP, Login credentials, Link Speed etc.
The "name" parameter is very important. Provide a string to identify the respective for this parameter. This name is displayed in logs for easy identification of devices.
Refer to the topology diagram to find the link variables and update their values
All testbed files can be found in the folder "~/testbeds/"

Testbed file variables

There are two variables to control the cleanup after each test run in the testbed file.

They are CLEANUP_BY_REBOOT and CLEANUP_BY_CFG_RELOAD:

CLEANUP_BY_REBOOT = True, The script will restore the switch's configuration from the /etc/sonic/clean_config.json file and then reboot the switch. This process consumes additional execution time, but it ensures that the DUTs (Devices Under Test) are consistently configured in a clean and proper manner for subsequent test scripts.
CLEANUP_BY_CFG_RELOAD = True, The script will restore the switch's configuration from the /etc/sonic/clean_config.json file and then issue the sudo config load command to load the clean config file. This method takes less time to have a clean configuration on switches but may not work correctly sometimes.
If both CLEANUP_BY_REBOOT and CLEANUP_BY_CFG_RELOAD are set to False, The scripts use the SONiC CLI procedure to un-configure whatever was configured on the switches by the scripts.
```
CFG_RELOAD_BY_REBOOT = False
```
Ideally, the below three parameters should be set to False
- CLEANUP_BY_REBOOT = False
- CLEANUP_BY_CFG_RELOAD = False
- CFG_RELOAD_BY_REBOOT = False
CLEANUP_BEFORE_TEST_RUN = False, It can be used to do the forceful clean-up of all the devices before the test run
- The following configuration will be cleaned up.
```
- ACL rules and table
- IPv4 routes 
- IPv4 interfaces 
- VLAN
- Port-channel
- All ports will be shut down
```

Apart from these, the following variables are specified in the testbed file:

CFG_RELOAD_BY_REBOOT = True, The scripts initiate device reboots in instances where the config reload command fails on the DUT due to certain reasons. This measure is taken as a workaround for such situations.
REBOOT_WAIT_TIME = 0, Maximum wait time for the device to reboot
NTP_SERVER = <FTAS VM IP>, FTAS VM serves as NTP server.
SYSLOG_SRVS = {"Servers": ["<FTAS VM IP>", "10.4.5.6"], "Log_Folder": "/var/log/sonic_logs"}, first list member should be set to the FTAS VM IP
MAX_V4_ACL , Maximum IPv4 ACL rules supported on the platform
MAX_V6_ACL , Maximum IPv6 ACL rules supported on the platform
MAX_SECONDARY_SUBNET, Maximum secondary subnets supported for SVI interface
MAX_IPV4_HOST_ROUTES, Maximum IPv4 host routes supported on the platform
MAX_IPV6_HOST_ROUTES, Maximum IPv6 host routes supported on the platform
MAX_IPV4_PREFIX_ROUTES, Maximum IPv4 prefix routes supported on the platform
MAX_IPV6_PREFIX_ROUTES, Maximum IPv6 prefix routes supported on the platform
TECHSUPPORT = True, Takes techsupport dump (If True) for the DUTs in case of failures
TECHSUPPORT_SINCE = "hour ago", Specifies the argument to the show techsupport command, in case TECHSUPPORT = True
TECHSUPPORT_TIMEOUT = 300, specifies the worst case timeout value for techsupport dump generation, in case TECHSUPPORT = True
ACCEPTABLE_DELTA = 0.5, Threshold value for acceptable packet/frame loss percentage.
CPU_MEM_THRESHOLD = 5 , Threshold value for acceptable change in CPU/memory utilization percentage
STRESS_AVAIL_CORES = 2 , Number of CPU cores reserved for system use; rest other CPU cores will undergo stress testing
STRESS_MEM_UTIL = 85 , Targeted percentage of total system memory to allocate for stress testing
SERVER_IP = "x.x.x.x" , IP address of the server hosting the stress-ng Docker image
SERVER_USER_ID = "oper" , User ID for SCP access to the server hosting the stress-ng Docker image
SERVER_PASSWORD = "oper@123" , Password for SCP access to securely transfer the stress-ng Docker image

Variable with Default values

Description

Configuring Ixia Variables

For UHD Chassis:

IXIA_Ports = {"s1_p1": "<port_no>", "s1_p2": "<port_no>", "s1_p3": "<port_no>", 
              "media": "fiber", "speed": "100G",
              "port_configs": {
              "<localuhd>/<8>": {"speed": "100G", "auto_negotiation": False, "rs_fec": True},

Specify the port connected to the DUT like this: "s1_p1": "8" , where the port number 8 is connected to Spine01_Port01 for Ixia traffic.
For example, Ixia port 8 can be configured like this: "localuhd/8", where localuhd refers to the Ixia chassis and 8 is the UHD port number.

For Novus Chassis:

IXIA_Ports = {"s1_p1": "<card_no>;<Port_no>", "s1_p2": "<card_no>;<Port_no>", "s1_p3": "<card_no>;<Port_no>", 
              "media": "fiber", "speed": "100G",
              "port_configs": {
              "<chassis_ip>;<card_no>;<Port_no>": {"speed": "100G", "auto_negotiation": False, "rs_fec": True},

Specify the port connected to the DUT like this: "s1_p1": "1;8" , where the port number 8 on Ixia card 1 is connected to Spine01_Port01 for Ixia traffic.
For example, Ixia port 1;8 can be configured like this: "<chassis_ip>;<card_no>;<Port_no>", where <chassis_ip> refers to the Ixia chassis IP and 1;8 is the port with the card number.

The FTAS VM has been extensively tested with UHD version 1.3.3003.118, It is recommended to use UHD with the same version for smooth operations.

Ixia port format varies and depends on the Ixia Chassis type

Control Ixia traffic rate globally

To avoid this situation a global parameter for the ixia traffic rate "global_traffic_rate" in percentage could be used to enforce traffic rate in all tests.

When global_traffic_rate is not defined all traffic streams in all test scripts would use their own static values for the traffic rate.

# Ixia Parameters #
IXIA_Ports = {"s1_p1": "8", "s1_p2": "9", "s1_p3": "10", 
              "s2_p1": "11", "s2_p2": "12",
              "l1_p1": "8", "l1_p2": "9", "l1_p3": "10",
              "l2_p1": "11", "l2_p2": "12",
              "media": "fiber", "speed": "100G", "global_traffic_rate": 25,
              ...

Here the "global_traffic_rate": 25 referrers to 25% speed for all Ixia ports

CLI_TIMEOUT, set the integer value for this variable when the command execution is slow or taking more than 30 sec to respond. Set the default value to 0 in the testbed file.

full_mesh_topo.py

"""
    Description: Testbed information
"""
from genlibs import params
from genlibs import const

gParams = params.GLOBAL_PARAMS
pParams = params.PLATFORM_PARAMS
# PLS. DON'T CHANGE ANYTHING FROM HERE TO TOP OF THE FILE
# =================================================
ALL_DUTS = ["LEAF01", "LEAF02", "SPINE01", "SPINE02"]
# =================================================
CLEANUP_BY_REBOOT = False
CLEANUP_BY_CFG_RELOAD = False
CFG_RELOAD_BY_REBOOT = False
CHECK_COMPATIBILITY = False
CLEANUP_BEFORE_TEST_RUN = False

NET_SERVICES_CONTAINER_NAME = "net_services"
NTP_SERVER = "10.4.5.245"
INTF_UP_WAIT_TIME = 30
REBOOT_WAIT_TIME = 300
CLI_TIMEOUT = 0
MAX_V4_ACL = 64
MAX_INGRESS_V4_ACL = 64
MAX_V6_ACL = 64
MAX_SECONDARY_SUBNET = 25
MAX_IPV4_HOST_ROUTES = 1000
MAX_IPV6_HOST_ROUTES = 1000
MAX_IPV4_PREFIX_ROUTES = 1000
MAX_IPV6_PREFIX_ROUTES = 32000
MAX_IPV4_ROUTES_PER_NEXTHOP = 256
MAX_IPV4_NEXTHOPS = 2048
TECHSUPPORT = True
TECHSUPPORT_SINCE = "hour ago"
TECHSUPPORT_TIMEOUT = 300
STRESS_AVAIL_CORES = 2  # Number of CPU cores reserved for system use; all other cores will undergo stress testing
STRESS_MEM_UTIL = 85    # Targeted percentage of total system memory to allocate for stress testing
SERVER_IP = "10.20.0.75" # IP address of the server hosting the stress-ng Docker image
SERVER_USER_ID = "oper"  # User ID for SCP access to the server hosting the stress-ng Docker image
SERVER_PASSWORD = "oper@123" # Password for SCP access to securely transfer the stress-ng Docker image

SYSLOG_SRVS = {"Servers": ["10.4.5.245", "10.4.5.6"], "Log_Folder": "/var/log/sonic_logs"}
TACACS_SRVS = [{"address": "10.4.5.177", "secret_key": "T@c@csSonic123"},
               {"address": "10.4.5.179", "secret_key": "T@c@csSonic123"}]
TACACS_USERS = {"admin_user": "tacadmin", "admin_passwd": "sadmin@123", "oper_user": "tacuser",
                "oper_passwd": "suser@123"}

IXIA_Ports = {"l1_p1": "21", "l1_p2": "24",
              "l2_p1": "22", "l2_p2": "23",
              "global_traffic_rate": 80,
              "media": "fiber", "speed": "100G",
              "port_configs": {
                  "localuhd/21": {"speed": "100G", "auto_negotiation": False, "rs_fec": True},
                  "localuhd/22": {"speed": "100G", "auto_negotiation": False, "rs_fec": True},
                  "localuhd/23": {"speed": "100G", "auto_negotiation": False, "rs_fec": True},
                  "localuhd/24": {"speed": "100G", "auto_negotiation": False, "rs_fec": True}
                  }
              }

IXIA = {"IP": "10.4.4.10", "username": "aviz", "password": "aviz@123", "ports": IXIA_Ports}

LOGSRV1 = {"IP": "10.1.1.11", "SSH_PORT": 22, "ssh_user": "aviz", "ssh_passwd": "IxiaAviz2020", "Timeout": 30,
           "name": "Syslog1", "CLI_PROMPTS": params.LINUX_PROMPTS}

# Linux Server to host services likes: NTP, Syslog, Tac_plus, etc. avtest user is in sudo group and no password
TESTSRV1 = {"IP": "10.109.9.112", "SSH_PORT": 22, "ssh_user": "avtest", "ssh_passwd": "avtest@123", "Timeout": 30,
            "name": "TestSrv1", "CLI_PROMPTS": params.LINUX_PROMPTS}

SPINE01_Ports = {"l1_p1": "Ethernet0", "l1_p1_speed": 100000,
                 "l1_p2": "Ethernet4", "l1_p2_speed": 100000,
                 "l2_p1": "Ethernet24", "l2_p1_speed": 100000,
                 "l2_p2": "Ethernet28", "l2_p2_speed": 100000,
                 "s2_p1": "Ethernet32", "s2_p1_speed": 100000,
                 "s2_p2": "Ethernet36", "s2_p2_speed": 100000,
                 "port_mtu": 9100
                 }
SPINE01 = {"IP": "10.4.4.65",
           "PROTO": "http", "REST_PORT": 9001, "SSH_PORT": 22,
           "CLI_PROMPTS": params.CLI_PROMPTS,
           "cliErrors": params.CLI_ERROR_REGEXP, "cliWarnings": params.CLI_WARN_REGEXP,
           "ssh_user": "admin",
           "ssh_passwd": "YourPaSsWoRd",
           "Timeout": 30, "ports": SPINE01_Ports, "name": "MLNX-SPINE01",
           "backup_cfg_file": "clean_config.json"
           }

SPINE02_Ports = {"l1_p1": "Ethernet16", "l1_p1_speed": 100000,
                 "l1_p2": "Ethernet20", "l1_p2_speed": 100000,
                 "l2_p1": "Ethernet0", "l2_p1_speed": 100000,
                 "l2_p2": "Ethernet4", "l2_p2_speed": 100000,
                 "s1_p1": "Ethernet32", "s1_p1_speed": 100000,
                 "s1_p2": "Ethernet36", "s1_p2_speed": 100000,
                 "port_mtu": 9100
                 }
SPINE02 = {"IP": "10.4.4.67",
           "PROTO": "http", "REST_PORT": 6018, "SSH_PORT": 22,
           "CLI_PROMPTS": params.CLI_PROMPTS,
           "cliErrors": params.CLI_ERROR_REGEXP, "cliWarnings": params.CLI_WARN_REGEXP,
           "ssh_user": "admin",
           "ssh_passwd": "YourPaSsWoRd",
           "Timeout": 30, "ports": SPINE02_Ports, "name": "MLNX-SPINE02",
           "backup_cfg_file": "clean_config.json"
           }

LEAF01_Ports = {"s2_p1": "Ethernet16", "s2_p1_speed": 100000,
                "s2_p2": "Ethernet20", "s2_p2_speed": 100000,
                "s1_p1": "Ethernet0", "s1_p1_speed": 100000,
                "s1_p2": "Ethernet4", "s1_p2_speed": 100000,
                "l2_p1": "Ethernet32", "l2_p1_speed": 100000,
                "l2_p2": "Ethernet36", "l2_p2_speed": 100000,
                # provide the breakout modes for the supp ports in the below format
                # "l2_p1": "Ethernet72", "l1_p3_speed": 100000,
                # "l2_p1_breakout": "1x100G[40G], 2x50G, 4x25G, 4x10G",
                "ixia_p1": "Ethernet60", "ixia_p1_speed": 100000,
                "ixia_p2": "Ethernet48", "ixia_p2_speed": 100000,
                "port_mtu": 9100
                }
LEAF01 = {"IP": "10.4.4.66",
          "PROTO": "http", "REST_PORT": 6002, "SSH_PORT": 22,
          "CLI_PROMPTS": params.CLI_PROMPTS,
          "cliErrors": params.CLI_ERROR_REGEXP, "cliWarnings": params.CLI_WARN_REGEXP,
          "ssh_user": "admin",
          "ssh_passwd": "YourPaSsWoRd",
          "Timeout": 30, "ports": LEAF01_Ports, "name": "MLNX-LEAF01",
          "backup_cfg_file": "clean_config.json"
          }

LEAF02_Ports = {"s2_p1": "Ethernet0", "s2_p1_speed": 100000,
                "s2_p2": "Ethernet4", "s2_p2_speed": 100000,
                "s1_p1": "Ethernet24", "s1_p1_speed": 100000,
                "s1_p2": "Ethernet28", "s1_p2_speed": 100000,
                "l1_p1": "Ethernet32", "l1_p1_speed": 100000,
                "l1_p2": "Ethernet36", "l1_p2_speed": 100000,
                # provide the breakout modes for the supp ports in the below format
                # "l1_p3": "Ethernet72", "l1_p3_speed": 100000,
                # "l1_p3_breakout": "1x100G[40G], 2x50G, 4x25G, 4x10G",
                "ixia_p1": "Ethernet60", "ixia_p1_speed": 100000,
                "ixia_p2": "Ethernet48", "ixia_p2_speed": 100000,
                "port_mtu": 9100
                }
LEAF02 = {"IP": "10.4.4.68",
          "PROTO": "http", "REST_PORT": 6002, "SSH_PORT": 22,
          "CLI_PROMPTS": params.CLI_PROMPTS,
          "cliErrors": params.CLI_ERROR_REGEXP, "cliWarnings": params.CLI_WARN_REGEXP,
          "ssh_user": "admin",
          "ssh_passwd": "YourPaSsWoRd",
          "Timeout": 30, "ports": LEAF02_Ports, "name": "MLNX-LEAF02",
          "backup_cfg_file": "clean_config.json"
          }

2 DUTs Topology

If you don't have 4 DUTs topology and want to run scripts that require only 2 DUTs, You can use 2dut_topo.py testbed parameter file

2dut_topo.py

"""
    Description: Testbed information
"""
from genlibs import params
from genlibs import const

gParams = params.GLOBAL_PARAMS
pParams = params.PLATFORM_PARAMS

# =================================================
ALL_DUTS = ['SPINE01', 'SPINE02']
# =================================================
CLEANUP_BY_REBOOT = False
CLEANUP_BY_CFG_RELOAD = False 
CFG_RELOAD_BY_REBOOT = False
CHECK_COMPATIBILITY = False
CLEANUP_BEFORE_TEST_RUN = False

NTP_SERVER = "10.4.5.4"

INTF_UP_WAIT_TIME = 30
REBOOT_WAIT_TIME = 120
CLI_TIMEOUT = 0

MAX_V4_ACL = 64
MAX_V6_ACL = 64
MAX_SECONDARY_SUBNET = 25
MAX_IPV4_HOST_ROUTES = 1000
MAX_IPV6_HOST_ROUTES = 1000
MAX_IPV4_PREFIX_ROUTES = 1000
MAX_IPV4_NEXTHOPS = 256

ZTP_PARAMS = {"ZTP_HTTP_SRV_ADDR": "10.4.5.177", "ZTP_HTTP_SRV_PORT": "8090", "ZTP_FOLDER": "/home/oper/reports/ztp",
              "DHCP_CONTAINER": "ztp_dhcp"}

NET_SERVICES_CONTAINER_NAME = "net_services"

SYSLOG_SRVS = {"Servers": ["<SYSLOG server IP1>", "<SYSLOG server IP2>"], 'Log_Folder': "/var/log/sonic_logs"}
TACACS_SRVS = [{"address": "<IP address1>", "secret_key": "T@c@csSonic123"},
               {"address": "<IP address2>", "secret_key": "T@c@csSonic123"}]
TACACS_USERS = {"admin_user": "tacadmin", "admin_passwd": "sadmin@123",
                "oper_user": "tacuser", "oper_passwd": "suser@123"}

# Ixia Parameters #
IXIA_Ports = {"s1_p1": "8", "s1_p2": "9", "s1_p3": "10",
              "s2_p1": "11", "s2_p2": "12",
              "l1_p1": "8", "l1_p2": "9", "l1_p3": "10",
              "l2_p1": "11", "l2_p2": "12",
              "media": "fiber", "speed": "100G",
              "port_configs": {
                  "localuhd/8": {"speed": "100G", "auto_negotiation": False, "rs_fec": True},
                  "localuhd/9": {"speed": "100G", "auto_negotiation": False, "rs_fec": True},
                  "localuhd/10": {"speed": "100G", "auto_negotiation": False, "rs_fec": True},
                  "localuhd/11": {"speed": "100G", "auto_negotiation": False, "rs_fec": True},
                  "localuhd/12": {"speed": "100G", "auto_negotiation": False, "rs_fec": True},
              }}

IXIA = {"IP": "<Ixia IP>", "username": "<ix username>", "password": "<Ixia passwd>", "ports": IXIA_Ports}

# DUTs Parameters #
SPINE01_Ports = {"s2_p1": "Ethernet0", "s2_p1_speed": 100000,
                 "s2_p2": "Ethernet8", "s2_p2_speed": 100000,
                 "ixia_p1": "Ethernet232",
                 "ixia_p2": "Ethernet240",
                 "ixia_p3": "Ethernet248"}

SPINE01 = {"IP": "<SPINE1 IP>", 'SSH_PORT': 22, "CLI_PROMPTS": params.CLI_PROMPTS,
           "cliErrors": params.CLI_ERROR_REGEXP,
           "cliWarnings": params.CLI_WARN_REGEXP,
           "climode": const.CliModes.SONiC_CLI, "ports": SPINE01_Ports,
           "ssh_user": "admin", "ssh_passwd": "Innovium123",
           "Timeout": 30, "name": "Spine1",
           "backup_cfg_file": "clean_config.json"}

SPINE02_Ports = {"s1_p1": "Ethernet0", "s1_p1_speed": 100000,
                 "s1_p2": "Ethernet8", "s1_p2_speed": 100000,
                 "ixia_p1": "Ethernet240",
                 "ixia_p2": "Ethernet248"}
SPINE02 = {"IP": "<SPINE2 IP>", 'SSH_PORT': 22, "CLI_PROMPTS": params.CLI_PROMPTS,
           "cliErrors": params.CLI_ERROR_REGEXP,
           "cliWarnings": params.CLI_WARN_REGEXP,
           "climode": const.CliModes.SONiC_CLI, "ports": SPINE02_Ports,
           "ssh_user": "admin", "ssh_passwd": "Innovium123",
           "Timeout": 30, "name": "Spine2",
           "backup_cfg_file": "clean_config.json"}

LEAF01_Ports = {"l2_p1": "Ethernet32", "l2_p1_speed": 100000,
                "l2_p2": "Ethernet40", "l2_p2_speed": 100000,
                "ixia_p1": "Ethernet240",
                "ixia_p2": "Ethernet248"}

LEAF01 = {"IP": "<SPINE1 IP>", 'SSH_PORT': 22, "CLI_PROMPTS": params.CLI_PROMPTS,
          "cliErrors": params.CLI_ERROR_REGEXP,
          "cliWarnings": params.CLI_WARN_REGEXP,
          "climode": const.CliModes.SONiC_CLI, "ports": LEAF01_Ports,
          "ssh_user": "admin", "ssh_passwd": "Innovium123",
          "Timeout": 30, "name": "Leaf1",
          "backup_cfg_file": "clean_config.json"}

LEAF02_Ports = {"l1_p1": "Ethernet32", "l1_p1_speed": 100000,
                "l1_p2": "Ethernet40", "l1_p2_speed": 100000,
                "ixia_p1": "Ethernet240",
                "ixia_p2": "Ethernet248"}

LEAF02 = {"IP": "<SPINE2 IP>", 'SSH_PORT': 22,
          "CLI_PROMPTS": params.CLI_PROMPTS,
          "cliErrors": params.CLI_ERROR_REGEXP,
          "cliWarnings": params.CLI_WARN_REGEXP,
          "climode": const.CliModes.SONiC_CLI, "ports": LEAF02_Ports,
          "ssh_user": "admin", "ssh_passwd": "Innovium123",
          "Timeout": 30, "name": "Leaf2",
          "backup_cfg_file": "clean_config.json"}

Full Mesh 4 DUTs Topology for Chaos

ftas_chaos_topo.py (sample topology file) the testbed parameter file is used for Chaos test scripts only.

ftas_chaos_topo.py

"""
    Description: Testbed information
"""
from genlibs import params
from genlibs import const

gParams = params.GLOBAL_PARAMS
pParams = params.PLATFORM_PARAMS

# =================================================
ALL_DUTS = ['LEAF01', 'LEAF02', 'SPINE01', 'SPINE02']
# =================================================
result_dir = "/home/oper/reports"

CLEANUP_BY_REBOOT = False
CLEANUP_BY_CFG_RELOAD = False

West_Ixia_Params = {"ports": [21], "ixmedia": "fiber", "ixspeed": "100G", "peer": "leaf1"}

East_Ixia_Params = {"ports": [22], "ixmedia": "fiber", "ixspeed": "100G", "peer": "leaf2"}
IXIA = {"IP": "10.4.4.10", "username": "aviz", "password": "aviz@123", "ixmedia": "fiber", "ixspeed": "100G"}
INTF_UP_WAIT_TIME = 30
REBOOT_WAIT_TIME = 300
CLI_TIMEOUT = 0
PCH_CONFIGURATION = False  # to disable the PCH configuration in chaos scripts, default is True
TECHSUPPORT = True
TECHSUPPORT_SINCE = "hour ago"
TECHSUPPORT_TIMEOUT = 120
ACCEPTABLE_DELTA = 0.5 #Threshold for acceptable packet/frame loss percentage
CPU_MEM_THRESHOLD = 5 ## Threshold for acceptable change in CPU/memory utilization percentage

# Use a network with prefixlen = 24
MASTER_NETWORK = "172.16.1.0/24"

S1_Ports = [
    # Links from Spine1 to Leaf1
    {"s1": "Ethernet0", "l1": "Ethernet0", "speed": 100000,
     "netinfo": {"spine_ip": "", "leaf_ip": ""}, "peer": "leaf1"},
#    {"s1": "Ethernet4", "l1": "Ethernet4", "speed": 100000,
#     "netinfo": {"spine_ip": "", "leaf_ip": ""}, "peer": "leaf1"},
    # Links from Spine1 to Leaf2
    {"s1": "Ethernet24", "l2": "Ethernet24", "speed": 100000,
     "netinfo": {"spine_ip": "", "leaf_ip": ""}, "peer": "leaf2"}]
#    {"s1": "Ethernet28", "l2": "Ethernet28", "speed": 100000,
#     "netinfo": {"spine_ip": "", "leaf_ip": ""}, "peer": "leaf2"}]

S2_Ports = [
    # Links from Spine2 to Leaf1
    {"s2": "Ethernet16", "l1": "Ethernet16", "speed": 100000,
     "netinfo": {"spine_ip": "", "leaf_ip": ""}, "peer": "leaf1"},
#    {"s2": "Ethernet20", "l1": "Ethernet20", "speed": 100000,
#     "netinfo": {"spine_ip": "", "leaf_ip": ""}, "peer": "leaf1"},
    # Links from Spine2 to Leaf2
    {"s2": "Ethernet0", "l2": "Ethernet0", "speed": 100000,
     "netinfo": {"spine_ip": "", "leaf_ip": ""}, "peer": "leaf2"}]
#    {"s2": "Ethernet4", "l2": "Ethernet4", "speed": 100000,
#     "netinfo": {"spine_ip": "", "leaf_ip": ""}, "peer": "leaf2"}]

L1_Ixia_Ports = [
    # Links from Leaf1 to Ixia
    {"ixia": "21", "l1_ixia": "Ethernet60", "speed": 100000,
     "netinfo": {"ixia_ip": "", "leaf_ip": ""},
     "port_configs": {"localuhd/21": {"speed": "100G", "auto_negotiation": False, "rs_fec": True}}}]

L2_Ixia_Ports = [
    # Links from Leaf1 to Ixia
    {"ixia": "22", "l2_ixia": "Ethernet60", "speed": 100000,
     "netinfo": {"ixia_ip": "", "leaf_ip": ""},
     "port_configs": {"localuhd/22": {"speed": "100G", "auto_negotiation": False, "rs_fec": True}}}]
#    {"ixia": "10", "l2_ixia": "Ethernet100", "speed": 1000000,
#     "netinfo": {"ixia_ip": "", "leaf_ip": ""}}]

SPINE01 = {"IP": "10.4.4.65", 'SSH_PORT': 22,
           "CLI_PROMPTS": params.CLI_PROMPTS,
           "cliErrors": params.CLI_ERROR_REGEXP,
           "cliWarnings": params.CLI_WARN_REGEXP,
           "climode": const.CliModes.SONiC_CLI,
           "ssh_user": "admin", "ssh_passwd": "YourPaSsWoRd",
           "Timeout": 30, "ports": S1_Ports, "name": "Spine1",
           "backup_cfg_file": "clean_config.json"}

SPINE02 = {"IP": "10.4.4.67", 'SSH_PORT': 22,
           "CLI_PROMPTS": params.CLI_PROMPTS,
           "cliErrors": params.CLI_ERROR_REGEXP,
           "cliWarnings": params.CLI_WARN_REGEXP,
           "climode": const.CliModes.SONiC_CLI,
           "ssh_user": "admin", "ssh_passwd": "YourPaSsWoRd",
           "Timeout": 30, "ports": S1_Ports, "name": "Spine2",
           "backup_cfg_file": "clean_config.json"}

LEAF01 = {"IP": "10.4.4.66", 'SSH_PORT': 22,
          "CLI_PROMPTS": params.CLI_PROMPTS,
          "cliErrors": params.CLI_ERROR_REGEXP,
          "cliWarnings": params.CLI_WARN_REGEXP,
          "climode": const.CliModes.SONiC_CLI,
          "ssh_user": "admin", "ssh_passwd": "YourPaSsWoRd",
          "Timeout": 30, "ports": S1_Ports + S2_Ports, "name": "Leaf1",
          "backup_cfg_file": "clean_config.json"}

LEAF02 = {"IP": "10.4.4.68", 'SSH_PORT': 22,
          "CLI_PROMPTS": params.CLI_PROMPTS,
          "cliErrors": params.CLI_ERROR_REGEXP,
          "cliWarnings": params.CLI_WARN_REGEXP,
          "climode": const.CliModes.SONiC_CLI,
          "ssh_user": "admin", "ssh_passwd": "YourPaSsWoRd",
          "Timeout": 30, "ports": S1_Ports + S2_Ports, "name": "Leaf2",
          "backup_cfg_file": "clean_config.json"}