Test Setup

Before scheduling any jobs validate the physical testbed to make sure all links are connected and are operationally UP, also we need a clean configuration file with valid interface settings (breakout, speed, FEC, admin status, etc.), no IP interfaces, no BGP instances, and no QoS. The scripts will use this clean configuration file to restore the DUT to its default configuration as part of the clean-up process.

The cleanup configuration file should be created at /etc/sonic/clean_config.json. The clean configuration file follows the config_db.json format. It must include the port-related settings for lane mapping, speed, and admin status.

How to create a clean config file

• For every DUT in the testbed, backup the default config(if needed)

cd /etc/sonic/
cp config_db.json config_db.json.bak

• Start with the default configuration after a fresh installation of SONiC

#reset to factory default
sudo rm /etc/sonic/config_db.json
sudo config-setup factory
sudo reboot

• Alternatively, you can also create a clean config by editing the config_db.json as below

  • Edit config_db.json and remove the following configuration blocks and save the file

    • VLAN

    • VLAN_MEMBER

    • PORTCHANNEL

    • PORTCHANNEL_MEMBER

    • BGP configuration

    • Loopback interfaces

• Edit "DEVICE_METADATA" in /etc/sonic/config_db.json as below

  • Configure "hostname" for each device (Example: Leaf01, Leaf02, Spine01, Spine02)

  • Add "docker_routing_config_mode": "split" configuration

// Sample DEVICE_METADATA
    "DEVICE_METADATA": {
        "localhost": {
            "buffer_model": "traditional",
            "default_bgp_status": "up",
            "default_pfcwd_status": "disable",
            "docker_routing_config_mode": "split",
            "hostname": "INVM32K-02",
            "hwsku": "Wistron_sw_to3200k_32x100",
            "mac": "00:30:64:6f:61:ad",
            "platform": "x86_64-wistron_sw_to3200k-r0",
            "type": "not-provisioned"
        }
    },

• Configure the below list through config_db.json:

  • The management IP address for eth0 and gateway

    "MGMT_INTERFACE": {
        "eth0|10.4.4.65/23": {
            "gwaddr": "10.4.4.1"
        }
    },

• Port breakout (if any)

• Port speed, FEC, Auto-negotiation on links connected to other devices and Ixia

• Configure "admin_status": "down"

Sample port configuration block

        "Ethernet0": {
            "admin_status": "down",
            "alias": "etp1",
            "index": "1",
            "lanes": "0,1,2,3",
            "mtu": "9100",
            "speed": "100000"
        },

• Save config_db.json

• Copy config_db.json to /etc/sonic/clean_config.json

sudo cp /etc/sonic/config_db.json /etc/sonic/clean_config.json

clean_config.json should be built using the text editor or SONiC CLI, but not both.

• Load the configuration on the device

sudo config reload -y -f

• Add the following line to /etc/sonic/frr/vtysh.conf

service integrated-vtysh-config

• Cleanup the BGP configuration from FRR

vtysh
show run
config t

<remove all BGP configurations>

write memory

Chaos Test Setup

TestBed File

Build the Chaos testbed file using "~/testbeds/ftas_chaos_topo.py"

Disable all cleanup options

The chaos test suite loads a base configuration to all DUTs and Ixia for its test scripts. So ensure the following variables are set to False in the testbed file.

CLEANUP_BY_REBOOT = False
CLEANUP_BY_CFG_RELOAD = False

Clean up DUTs Configuration Manually

Please make sure DUTs have a clean or default configuration before the Chaos test run.

Remove the test statistics report file

Chaos suite generates a statistics report file at ~/reports/report.txt to track its execution status and metric data for all test scripts. So please make sure to remove this file before running the Chaos test to avoid Ixia library errors

cd ~/reports/
rm report.txt