Update the above details in the testbed file with your DUTs - DUT connections, Management IP, Login credentials, Link Speed etc.
The "name" parameter is very important. Provide a string to identify the respective for this parameter. This name is displayed in logs for easy identification of devices.
Refer to the topology diagram to find the link variables and update their values
All testbed files can be found in the folder "~/testbeds/"
Testbed file variables
There are two variables to control the cleanup after each test run in the testbed file.
They are CLEANUP_BY_REBOOT and CLEANUP_BY_CFG_RELOAD:
CLEANUP_BY_REBOOT = True, The script will restore the switch's configuration from the /etc/sonic/clean_config.json file and then reboot the switch. This process consumes additional execution time, but it ensures that the DUTs (Devices Under Test) are consistently configured in a clean and proper manner for subsequent test scripts.
CLEANUP_BY_CFG_RELOAD = True, The script will restore the switch's configuration from the /etc/sonic/clean_config.json file and then issue the sudo config load command to load the clean config file. This method takes less time to have a clean configuration on switches but may not work correctly sometimes.
If both CLEANUP_BY_REBOOT and CLEANUP_BY_CFG_RELOAD are set to False, The scripts use the SONiC CLI procedure to un-configure whatever was configured on the switches by the scripts.
CFG_RELOAD_BY_REBOOT = False
Ideally, the below three parameters should be set to False
CLEANUP_BY_REBOOT = False
CLEANUP_BY_CFG_RELOAD = False
CFG_RELOAD_BY_REBOOT = False
CLEANUP_BEFORE_TEST_RUN = False, It can be used to do the forceful clean-up of all the devices before the test run
The following configuration will be cleaned up.
- ACL rules and table
- IPv4 routes
- IPv4 interfaces
- VLAN
- Port-channel
- All ports will be shut down
Apart from these, the following variables are specified in the testbed file:
CFG_RELOAD_BY_REBOOT = True, The scripts initiate device reboots in instances where the config reload command fails on the DUT due to certain reasons. This measure is taken as a workaround for such situations.
REBOOT_WAIT_TIME = 0, Maximum wait time for the device to reboot
NTP_SERVER = <FTAS VM IP>, FTAS VM serves as NTP server.
SYSLOG_SRVS = {"Servers": ["<FTAS VM IP>", "10.4.5.6"], "Log_Folder": "/var/log/sonic_logs"}, first list member should be set to the FTAS VM IP
MAX_V4_ACL , Maximum IPv4 ACL rules supported on the platform
MAX_V6_ACL , Maximum IPv6 ACL rules supported on the platform
MAX_SECONDARY_SUBNET, Maximum secondary subnets supported for SVI interface
MAX_IPV4_HOST_ROUTES, Maximum IPv4 host routes supported on the platform
MAX_IPV6_HOST_ROUTES, Maximum IPv6 host routes supported on the platform
MAX_IPV4_PREFIX_ROUTES, Maximum IPv4 prefix routes supported on the platform
MAX_IPV6_PREFIX_ROUTES, Maximum IPv6 prefix routes supported on the platform
TECHSUPPORT = True, Takes techsupport dump (If True) for the DUTs in case of failures
TECHSUPPORT_SINCE = "hour ago", Specifies the argument to the show techsupport command, in case TECHSUPPORT = True
TECHSUPPORT_TIMEOUT = 300, specifies the worst case timeout value for techsupport dump generation, in case TECHSUPPORT = True
ACCEPTABLE_DELTA = 0.5, Threshold value for acceptable packet/frame loss percentage.
CPU_MEM_THRESHOLD = 5 , Threshold value for acceptable change in CPU/memory utilization percentage
STRESS_AVAIL_CORES = 2 , Number of CPU cores reserved for system use; rest other CPU cores will undergo stress testing
STRESS_MEM_UTIL = 85 , Targeted percentage of total system memory to allocate for stress testing
SERVER_IP = "x.x.x.x" , IP address of the server hosting the stress-ng Docker image
SERVER_USER_ID = "oper" , User ID for SCP access to the server hosting the stress-ng Docker image
SERVER_PASSWORD = "oper@123" , Password for SCP access to securely transfer the stress-ng Docker image
Variable with Default values
Description
INTF_UP_WAIT_TIME = 30
Timeout for the interface to be 'Operationally UP'
CLI_TIMEOUT = 0
Assign an integer value to this variable if the command execution on the device experiences slowness or takes longer than 30 seconds to respond.
If the value is not explicitly set, the default will be 0, implying a 30-second wait time for the command execution output.
MAX_V4_ACL = 64
The maximum number of supported IPV4 ACL rules.
The variable used in the testcase test_v4_acl_scale_max_supported in test script scalability/taas_qual_scale.py
MAX_V6_ACL = 64
The maximum number of supported IPV6 ACL rules.
The variable used in the testcase test_qual_v6_acl_scale_max_supported in test script scalability/taas_qual_scale.py
MAX_SECONDARY_SUBNET = 25
The maximum number of supported secondary subnets under a vlan.
The variable used in the testcase test_max_secondary_subnet_under_vlan in test script scalability/taas_qual_scale.py
MAX_IPV4_HOST_ROUTES = 1000
The maximum number of IPv4 host routes supported.
The variable used in the testcase test_v4_host_routes_scale_max_supported in test script scalability/taas_qual_scale.py
MAX_IPV6_HOST_ROUTES = 1000
The maximum number of IPV6 host routes supported.
The variable used in the testcase test_v6_host_routes_scale_max_supported in test script scalability/taas_qual_scale.py
MAX_IPV4_PREFIX_ROUTES = 1000
The maximum number of IPV4 prefix routes supported.
The variable used in the testcase test_v4_prefix_routes_scale_max_supported in test script scalability/taas_qual_scale.py
MAX_IPV4_NEXTHOPS = 256
The maximum number of IPV4 next-hop supported.
The variable used in the testcase test_v4_nexthops_scale_max_supported in test script scalability/taas_qual_scale.py
STRESS_AVAIL_CORES = 2
Number of CPU cores reserved for system use. Rest other CPU cores will undergo stress testing
STRESS_MEM_UTIL = 85
Targeted percentage of total system memory to allocate for stress testing
Specify the port connected to the DUT like this: "s1_p1": "1;8" , where the port number 8 on Ixia card 1 is connected to Spine01_Port01 for Ixia traffic.
For example, Ixia port 1;8 can be configured like this: "<chassis_ip>;<card_no>;<Port_no>", where <chassis_ip> refers to the Ixia chassis IP and 1;8 is the port with the card number.
The FTAS VM has been extensively tested with UHD version 1.3.3003.118, It is recommended to use UHD with the same version for smooth operations.
Ixia port format varies and depends on the Ixia Chassis type
Control Ixia traffic rate globally
For the testbeds where links between DUTs have less bandwidth than Ixia ports traffic tests may fail due to traffic drops on the least bandwidth links (i.e. DUTs may be connected with 1G ports and the Ixia link maybe 100G).
To avoid this situation a global parameter for the ixia traffic rate "global_traffic_rate" in percentage could be used to enforce traffic rate in all tests.
When the global_traffic_rate parameter is defined as a sub-parameter in the "Ixia_Ports" all traffic streams in all test scripts would override their own rate value with global_traffic_rate value configured.
When global_traffic_rateis not defined all traffic streams in all test scripts would use their own static values for the traffic rate.
Here the "global_traffic_rate": 25 referrers to 25% speed for all Ixia ports
CLI_TIMEOUT, set the integer value for this variable when the command execution is slow or taking more than 30 sec to respond. Set the default value to 0 in the testbed file.
full_mesh_topo.py
""" Description: Testbed information"""from genlibs import paramsfrom genlibs import constgParams = params.GLOBAL_PARAMSpParams = params.PLATFORM_PARAMS# PLS. DON'T CHANGE ANYTHING FROM HERE TO TOP OF THE FILE# =================================================ALL_DUTS = ["LEAF01","LEAF02","SPINE01","SPINE02"]# =================================================CLEANUP_BY_REBOOT =FalseCLEANUP_BY_CFG_RELOAD =FalseCFG_RELOAD_BY_REBOOT =FalseCHECK_COMPATIBILITY =FalseCLEANUP_BEFORE_TEST_RUN =FalseNET_SERVICES_CONTAINER_NAME ="net_services"NTP_SERVER ="10.4.5.245"INTF_UP_WAIT_TIME =30REBOOT_WAIT_TIME =300CLI_TIMEOUT =0MAX_V4_ACL =64MAX_INGRESS_V4_ACL =64MAX_V6_ACL =64MAX_SECONDARY_SUBNET =25MAX_IPV4_HOST_ROUTES =1000MAX_IPV6_HOST_ROUTES =1000MAX_IPV4_PREFIX_ROUTES =1000MAX_IPV6_PREFIX_ROUTES =32000MAX_IPV4_ROUTES_PER_NEXTHOP =256MAX_IPV4_NEXTHOPS =2048TECHSUPPORT =TrueTECHSUPPORT_SINCE ="hour ago"TECHSUPPORT_TIMEOUT =300STRESS_AVAIL_CORES =2# Number of CPU cores reserved for system use; all other cores will undergo stress testingSTRESS_MEM_UTIL =85# Targeted percentage of total system memory to allocate for stress testingSERVER_IP ="10.20.0.75"# IP address of the server hosting the stress-ng Docker imageSERVER_USER_ID ="oper"# User ID for SCP access to the server hosting the stress-ng Docker imageSERVER_PASSWORD ="oper@123"# Password for SCP access to securely transfer the stress-ng Docker imageSYSLOG_SRVS ={"Servers": ["10.4.5.245","10.4.5.6"],"Log_Folder":"/var/log/sonic_logs"}TACACS_SRVS = [{"address":"10.4.5.177","secret_key":"T@c@csSonic123"},{"address":"10.4.5.179","secret_key":"T@c@csSonic123"}]TACACS_USERS ={"admin_user":"tacadmin","admin_passwd":"sadmin@123","oper_user":"tacuser","oper_passwd":"suser@123"}IXIA_Ports ={"l1_p1":"21","l1_p2":"24","l2_p1":"22","l2_p2":"23","global_traffic_rate":80,"media":"fiber","speed":"100G","port_configs":{"localuhd/21":{"speed":"100G","auto_negotiation":False,"rs_fec":True},"localuhd/22":{"speed":"100G","auto_negotiation":False,"rs_fec":True},"localuhd/23":{"speed":"100G","auto_negotiation":False,"rs_fec":True},"localuhd/24":{"speed":"100G","auto_negotiation":False,"rs_fec":True}}}IXIA ={"IP":"10.4.4.10","username":"aviz","password":"aviz@123","ports": IXIA_Ports}LOGSRV1 ={"IP":"10.1.1.11","SSH_PORT":22,"ssh_user":"aviz","ssh_passwd":"IxiaAviz2020","Timeout":30,"name":"Syslog1","CLI_PROMPTS": params.LINUX_PROMPTS}# Linux Server to host services likes: NTP, Syslog, Tac_plus, etc. avtest user is in sudo group and no passwordTESTSRV1 ={"IP":"10.109.9.112","SSH_PORT":22,"ssh_user":"avtest","ssh_passwd":"avtest@123","Timeout":30,"name":"TestSrv1","CLI_PROMPTS": params.LINUX_PROMPTS}SPINE01_Ports ={"l1_p1":"Ethernet0","l1_p1_speed":100000,"l1_p2":"Ethernet4","l1_p2_speed":100000,"l2_p1":"Ethernet24","l2_p1_speed":100000,"l2_p2":"Ethernet28","l2_p2_speed":100000,"s2_p1":"Ethernet32","s2_p1_speed":100000,"s2_p2":"Ethernet36","s2_p2_speed":100000,"port_mtu":9100}SPINE01 ={"IP":"10.4.4.65","PROTO":"http","REST_PORT":9001,"SSH_PORT":22,"CLI_PROMPTS": params.CLI_PROMPTS,"cliErrors": params.CLI_ERROR_REGEXP,"cliWarnings": params.CLI_WARN_REGEXP,"ssh_user":"admin","ssh_passwd":"YourPaSsWoRd","Timeout":30,"ports": SPINE01_Ports,"name":"MLNX-SPINE01","backup_cfg_file":"clean_config.json"}SPINE02_Ports ={"l1_p1":"Ethernet16","l1_p1_speed":100000,"l1_p2":"Ethernet20","l1_p2_speed":100000,"l2_p1":"Ethernet0","l2_p1_speed":100000,"l2_p2":"Ethernet4","l2_p2_speed":100000,"s1_p1":"Ethernet32","s1_p1_speed":100000,"s1_p2":"Ethernet36","s1_p2_speed":100000,"port_mtu":9100}SPINE02 ={"IP":"10.4.4.67","PROTO":"http","REST_PORT":6018,"SSH_PORT":22,"CLI_PROMPTS": params.CLI_PROMPTS,"cliErrors": params.CLI_ERROR_REGEXP,"cliWarnings": params.CLI_WARN_REGEXP,"ssh_user":"admin","ssh_passwd":"YourPaSsWoRd","Timeout":30,"ports": SPINE02_Ports,"name":"MLNX-SPINE02","backup_cfg_file":"clean_config.json"}LEAF01_Ports ={"s2_p1":"Ethernet16","s2_p1_speed":100000,"s2_p2":"Ethernet20","s2_p2_speed":100000,"s1_p1":"Ethernet0","s1_p1_speed":100000,"s1_p2":"Ethernet4","s1_p2_speed":100000,"l2_p1":"Ethernet32","l2_p1_speed":100000,"l2_p2":"Ethernet36","l2_p2_speed":100000,# provide the breakout modes for the supp ports in the below format# "l2_p1": "Ethernet72", "l1_p3_speed": 100000,# "l2_p1_breakout": "1x100G[40G], 2x50G, 4x25G, 4x10G","ixia_p1":"Ethernet60","ixia_p1_speed":100000,"ixia_p2":"Ethernet48","ixia_p2_speed":100000,"port_mtu":9100}LEAF01 ={"IP":"10.4.4.66","PROTO":"http","REST_PORT":6002,"SSH_PORT":22,"CLI_PROMPTS": params.CLI_PROMPTS,"cliErrors": params.CLI_ERROR_REGEXP,"cliWarnings": params.CLI_WARN_REGEXP,"ssh_user":"admin","ssh_passwd":"YourPaSsWoRd","Timeout":30,"ports": LEAF01_Ports,"name":"MLNX-LEAF01","backup_cfg_file":"clean_config.json"}LEAF02_Ports ={"s2_p1":"Ethernet0","s2_p1_speed":100000,"s2_p2":"Ethernet4","s2_p2_speed":100000,"s1_p1":"Ethernet24","s1_p1_speed":100000,"s1_p2":"Ethernet28","s1_p2_speed":100000,"l1_p1":"Ethernet32","l1_p1_speed":100000,"l1_p2":"Ethernet36","l1_p2_speed":100000,# provide the breakout modes for the supp ports in the below format# "l1_p3": "Ethernet72", "l1_p3_speed": 100000,# "l1_p3_breakout": "1x100G[40G], 2x50G, 4x25G, 4x10G","ixia_p1":"Ethernet60","ixia_p1_speed":100000,"ixia_p2":"Ethernet48","ixia_p2_speed":100000,"port_mtu":9100}LEAF02 ={"IP":"10.4.4.68","PROTO":"http","REST_PORT":6002,"SSH_PORT":22,"CLI_PROMPTS": params.CLI_PROMPTS,"cliErrors": params.CLI_ERROR_REGEXP,"cliWarnings": params.CLI_WARN_REGEXP,"ssh_user":"admin","ssh_passwd":"YourPaSsWoRd","Timeout":30,"ports": LEAF02_Ports,"name":"MLNX-LEAF02","backup_cfg_file":"clean_config.json"}
2 DUTs Topology
If you don't have 4 DUTs topology and want to run scripts that require only 2 DUTs, You can use 2dut_topo.py testbed parameter file
There are two physical DUTs in the topology but the script might pick the name Spine and Leaf interchangeably. So in the testbed file, we should define parameters for both but they point to the same physical Spine and Leaf DUTs.
ftas_chaos_topo.py (sample topology file) the testbed parameter file is used for Chaos test scripts only.
A new parameter PCH_CONFIGURATION = False, has been added. This parameter determines if the tests are to be run on routed port (if False) or routed PortChannel (if True) configuration for the interfaces connecting DUTs.
ftas_chaos_topo.py
""" Description: Testbed information"""from genlibs import paramsfrom genlibs import constgParams = params.GLOBAL_PARAMSpParams = params.PLATFORM_PARAMS# =================================================ALL_DUTS = ['LEAF01','LEAF02','SPINE01','SPINE02']# =================================================result_dir ="/home/oper/reports"CLEANUP_BY_REBOOT =FalseCLEANUP_BY_CFG_RELOAD =FalseWest_Ixia_Params ={"ports": [21],"ixmedia":"fiber","ixspeed":"100G","peer":"leaf1"}East_Ixia_Params ={"ports": [22],"ixmedia":"fiber","ixspeed":"100G","peer":"leaf2"}IXIA ={"IP":"10.4.4.10","username":"aviz","password":"aviz@123","ixmedia":"fiber","ixspeed":"100G"}INTF_UP_WAIT_TIME =30REBOOT_WAIT_TIME =300CLI_TIMEOUT =0PCH_CONFIGURATION =False# to disable the PCH configuration in chaos scripts, default is TrueTECHSUPPORT =TrueTECHSUPPORT_SINCE ="hour ago"TECHSUPPORT_TIMEOUT =120ACCEPTABLE_DELTA =0.5#Threshold for acceptable packet/frame loss percentageCPU_MEM_THRESHOLD =5## Threshold for acceptable change in CPU/memory utilization percentage# Use a network with prefixlen = 24MASTER_NETWORK ="172.16.1.0/24"S1_Ports = [# Links from Spine1 to Leaf1{"s1":"Ethernet0","l1":"Ethernet0","speed":100000,"netinfo":{"spine_ip":"","leaf_ip":""},"peer":"leaf1"},# {"s1": "Ethernet4", "l1": "Ethernet4", "speed": 100000,# "netinfo": {"spine_ip": "", "leaf_ip": ""}, "peer": "leaf1"},# Links from Spine1 to Leaf2{"s1":"Ethernet24","l2":"Ethernet24","speed":100000,"netinfo":{"spine_ip":"","leaf_ip":""},"peer":"leaf2"}]# {"s1": "Ethernet28", "l2": "Ethernet28", "speed": 100000,# "netinfo": {"spine_ip": "", "leaf_ip": ""}, "peer": "leaf2"}]S2_Ports = [# Links from Spine2 to Leaf1{"s2":"Ethernet16","l1":"Ethernet16","speed":100000,"netinfo":{"spine_ip":"","leaf_ip":""},"peer":"leaf1"},# {"s2": "Ethernet20", "l1": "Ethernet20", "speed": 100000,# "netinfo": {"spine_ip": "", "leaf_ip": ""}, "peer": "leaf1"},# Links from Spine2 to Leaf2{"s2":"Ethernet0","l2":"Ethernet0","speed":100000,"netinfo":{"spine_ip":"","leaf_ip":""},"peer":"leaf2"}]# {"s2": "Ethernet4", "l2": "Ethernet4", "speed": 100000,# "netinfo": {"spine_ip": "", "leaf_ip": ""}, "peer": "leaf2"}]L1_Ixia_Ports = [# Links from Leaf1 to Ixia{"ixia":"21","l1_ixia":"Ethernet60","speed":100000,"netinfo":{"ixia_ip":"","leaf_ip":""},"port_configs":{"localuhd/21":{"speed":"100G","auto_negotiation":False,"rs_fec":True}}}]L2_Ixia_Ports = [# Links from Leaf1 to Ixia{"ixia":"22","l2_ixia":"Ethernet60","speed":100000,"netinfo":{"ixia_ip":"","leaf_ip":""},"port_configs":{"localuhd/22":{"speed":"100G","auto_negotiation":False,"rs_fec":True}}}]# {"ixia": "10", "l2_ixia": "Ethernet100", "speed": 1000000,# "netinfo": {"ixia_ip": "", "leaf_ip": ""}}]SPINE01 ={"IP":"10.4.4.65",'SSH_PORT':22,"CLI_PROMPTS": params.CLI_PROMPTS,"cliErrors": params.CLI_ERROR_REGEXP,"cliWarnings": params.CLI_WARN_REGEXP,"climode": const.CliModes.SONiC_CLI,"ssh_user":"admin","ssh_passwd":"YourPaSsWoRd","Timeout":30,"ports": S1_Ports,"name":"Spine1","backup_cfg_file":"clean_config.json"}SPINE02 ={"IP":"10.4.4.67",'SSH_PORT':22,"CLI_PROMPTS": params.CLI_PROMPTS,"cliErrors": params.CLI_ERROR_REGEXP,"cliWarnings": params.CLI_WARN_REGEXP,"climode": const.CliModes.SONiC_CLI,"ssh_user":"admin","ssh_passwd":"YourPaSsWoRd","Timeout":30,"ports": S1_Ports,"name":"Spine2","backup_cfg_file":"clean_config.json"}LEAF01 ={"IP":"10.4.4.66",'SSH_PORT':22,"CLI_PROMPTS": params.CLI_PROMPTS,"cliErrors": params.CLI_ERROR_REGEXP,"cliWarnings": params.CLI_WARN_REGEXP,"climode": const.CliModes.SONiC_CLI,"ssh_user":"admin","ssh_passwd":"YourPaSsWoRd","Timeout":30,"ports": S1_Ports + S2_Ports,"name":"Leaf1","backup_cfg_file":"clean_config.json"}LEAF02 ={"IP":"10.4.4.68",'SSH_PORT':22,"CLI_PROMPTS": params.CLI_PROMPTS,"cliErrors": params.CLI_ERROR_REGEXP,"cliWarnings": params.CLI_WARN_REGEXP,"climode": const.CliModes.SONiC_CLI,"ssh_user":"admin","ssh_passwd":"YourPaSsWoRd","Timeout":30,"ports": S1_Ports + S2_Ports,"name":"Leaf2","backup_cfg_file":"clean_config.json"}
