Test Suite Configuration

Test suite configuration is a text file where you can list all test functions (each test function is a test case) for a batch run. Test suite files are stored in the "~/testsuites" folder

oper@ftasvm:~$ cd testsuites/
oper@ftasvm:~/testsuites$ ls -lrth
total 384
-rw-rw-r-- 1 oper oper 18247 Mar  2 17:12 wistron_6512_ecs2.2.2.suite
-rw-rw-r-- 1 oper oper 16342 Mar  2 17:12 wistron_3200_ecs2.2.2.suite
-rw-rw-r-- 1 oper oper   405 Mar  2 17:12 verify_yaml.py
-rw-rw-r-- 1 oper oper   872 Mar  2 17:12 verify_testbed.suite
-rw-rw-r-- 1 oper oper  6877 Mar  2 17:12 sanity.suite
-rw-rw-r-- 1 oper oper 14429 Mar  2 17:12 nvidia_msn2100.suite
-rw-rw-r-- 1 oper oper 14741 Mar  2 17:12 nvidia_202205.suite
-rw-rw-r-- 1 oper oper  7697 Mar  2 17:12 mgmt_complete.suite
-rw-rw-r-- 1 oper oper   808 Mar  2 17:12 mgmt_3dut.suite
-rw-rw-r-- 1 oper oper  5338 Mar  2 17:12 mgmt_2dut.suite
-rw-rw-r-- 1 oper oper  2075 Mar  2 17:12 mgmt_1dut.suite
-rw-rw-r-- 1 oper oper 21071 Mar  2 17:12 edgecore_9736.suite
-rw-rw-r-- 1 oper oper 22136 Mar  2 17:12 edgecore_9716_202111.suite
-rw-rw-r-- 1 oper oper 24399 Mar  2 17:12 edgecore_7726_202111.suite
-rw-rw-r-- 1 oper oper 17023 Mar  2 17:12 edgecore_4630_202111.suite
-rw-rw-r-- 1 oper oper   838 Mar  2 17:12 edge_sonic.suite
-rw-rw-r-- 1 oper oper 22218 Mar  2 17:12 ds3000_data_complete.suite
-rw-rw-r-- 1 oper oper  8960 Mar  2 17:12 cisco_data_2dut.suite
-rw-rw-r-- 1 oper oper 10884 Mar  2 17:12 ccs_regular.suite
-rw-r--r-- 1 oper oper 12166 Jun 22 15:34 regular.suite
-rw-r--r-- 1 oper oper 17003 Jul  1 13:59 ones_agent_validation.suite
-rw-r--r-- 1 oper oper  9304 Jul  1 13:59 data_4dut.suite
-rw-r--r-- 1 oper oper  2215 Jul  1 13:59 data_3dut.suite
-rw-r--r-- 1 oper oper 12461 Jul  1 13:59 data_2dut.suite
-rw-r--r-- 1 oper oper  6038 Jul  1 13:59 data_1dut.suite
-rw-r--r-- 1 oper oper 16411 Jul  1 13:59 PD.suite
-rw-r--r-- 1 oper oper 28312 Jul  1 17:49 data_complete.suite
-rw-r--r-- 1 oper oper  8924 Jul  2 07:40 PI.suite
oper@ftasvm:~/testsuites$ 

Below is an example of how a test suite file is structured:

PI.suite

---
 "TESTBED_ROOT_FOLDER": "/home/oper/testbeds"
 "TESTSCRIPT_ROOT_FOLDER": "/home/oper"
 "TESTSUITE_ROOT_FOLDER": ""
 "TEST_REPORT_ROOT_FOLDER": "/home/oper/reports"
 "CHECK_COMPATIBILITY": true
 "TEST_CONTACT": "[e-mail]"
 "TESTSUITES": {
   # 2 DUT test suites/scripts
    "./feature/taas_acl_test.py": {
     "SKIP": false,
     "COMMON_TESTBED": "full_mesh_topo.py",
     "TESTCASES": [{"test_acl_001": ""}, {"test_acl_002": ""}, {"test_acl_003": ""},{"test_acl_004": ""},{"test_acl_005": ""},{"test_acl_006": ""},
                   {"test_acl_007": ""},{"test_acl_008": ""},{"test_acl_009": ""},{"test_acl_010": ""},{"test_acl_011": ""},{"test_acl_012": ""}]
   },
    "./feature/taas_mtu_test.py": {
     "SKIP": false,
     "COMMON_TESTBED": "full_mesh_topo.py",
     "TESTCASES": [{"test_mtu_001": ""}]
   },
    "./feature/taas_mgmt_Syslog_test.py": {
     "SKIP": false,
     "COMMON_TESTBED": "full_mesh_topo.py",
     "TESTCASES": [{"test_syslog_002": ""},{"test_syslog_004": ""}]
   },
    "./feature/taas_autoneg_test.py": {
     "SKIP": false,
     "COMMON_TESTBED": "full_mesh_topo.py",
     "TESTCASES": [{"test_autoneg_001": ""}]
   },
    "./feature/taas_layer2_Vlan_ixia_test.py": {
     "SKIP": false,
     "COMMON_TESTBED": "full_mesh_topo.py",
     "TESTCASES": [{"test_vlan_014": ""},{"test_vlan_016": ""}]
   },
    "./feature/taas_layer2_LACP_test.py": {
     "SKIP": false,
     "COMMON_TESTBED": "full_mesh_topo.py",
     "TESTCASES": [{"test_lacp_003": ""}, {"test_lacp_005": ""}, {"test_lacp_011": ""},{"test_lacp_012": ""}]
   },
    "./feature/taas_layer2_LLDP_test.py": {
     "SKIP": false,
     "COMMON_TESTBED": "full_mesh_topo.py",
     "TESTCASES": [{"test_lldp_001": ""}, {"test_lldp_002": ""}, {"test_lldp_013": ""}]
   },
     "./feature/taas_layer3_ARP_test.py": {
     "SKIP": false,
     "COMMON_TESTBED": "full_mesh_topo.py",
     "TESTCASES": [{"test_arp_003": ""}, {"test_arp_007": ""}, {"test_arp_011": ""},{"test_arp_012": ""}]
   },
     "./feature/taas_layer3_IP_test.py": {
     "SKIP": false,
     "COMMON_TESTBED": "full_mesh_topo.py",
     "TESTCASES": [{"test_IP_001": ""}, {"test_IP_002": ""}, {"test_IP_005": ""},{"test_IP_006": ""},{"test_IP_011": ""},{"test_IP_014": ""}]
   },
     "./feature/taas_mgmt_Ping_test.py": {
     "SKIP": false,
     "COMMON_TESTBED": "full_mesh_topo.py",
     "TESTCASES": [{"test_ping_001": ""}, {"test_ping_009": ""}, {"test_ping_011": ""}, {"test_ping_013": ""}]
   },
      "./feature/taas_platform_Interface_test.py": {
     "SKIP": false,
     "COMMON_TESTBED": "full_mesh_topo.py",
     "TESTCASES": [{"test_ports_002": ""}, {"test_ports_005": ""}, {"test_ports_006": ""},{"test_ports_008": ""},{"test_ports_009_14": ""}]
   },
    "./scalability/taas_qual_Scale.py": {
     "SKIP": false,
     "COMMON_TESTBED": "full_mesh_topo.py",
     "TESTCASES": [{"test_10_secondary_subnet_under_vlan": ""}, {"test_20_secondary_subnet_under_vlan": ""}, {"test_max_secondary_subnet_under_vlan": ""}, {"test_v4_host_routes_scale_2k": ""},
                   {"test_v4_host_routes_scale_4k": ""}, {"test_v4_host_routes_scale_max_supported": ""}, {"test_v6_host_routes_scale_2k": ""}, {"test_v6_host_routes_scale_4k": ""},
                   {"test_v6_host_routes_scale_max_supported": ""}, {"test_v4_prefix_routes_scale_2k": ""}, {"test_v4_prefix_routes_scale_4k": ""}, {"test_v4_prefix_routes_scale_max_supported": ""},
                   {"test_v4_nexthops_scale_512": ""}, {"test_v4_nexthops_scale_1024": ""}, {"test_v4_nexthops_scale_max_supported": ""}, {"test_qual_scale_007": ""}, {"test_qual_scale_011": ""},
                   {"test_v4_acl_scale_128": ""}, {"test_v4_acl_scale_256": ""}, {"test_v4_acl_scale_512": ""}, {"test_v4_acl_scale_max_supported": ""},
                   {"test_qual_v6_acl_scale_128": ""}, {"test_qual_v6_acl_scale_256": ""}, {"test_qual_v6_acl_scale_512": ""},
                   {"test_qual_v6_acl_scale_max_supported": ""}]
   },
    "./feature/taas_mgmt_SSH_test.py": {
     "SKIP": false,
     "COMMON_TESTBED": "full_mesh_topo.py",
     "TESTCASES": [{"test_ssh_001": ""},{"test_ssh_002": ""},{"test_ssh_003": ""}]
   },
   "./feature/taas_layer3_bgp_netops_test.py": {
     "SKIP": false,
     "COMMON_TESTBED": "ftas_2duts_topo.py",
     "TESTCASES": [{"test_bgp_netops_002": ""}, {"test_bgp_netops_005_006": ""}, {"test_bgp_netops_008": ""},
                   {"test_bgp_netops_010": ""}]
   },
    "./feature/taas_layer3_BGP_ixia_test.py": {
     "SKIP": false,
     "COMMON_TESTBED": "full_mesh_topo.py",
     "TESTCASES": [{"test_bgp_004": ""},{"test_bgp_005": ""}]
   },
   "./feature/taas_qual_PortCfg_test.py": {
     "SKIP":false,
     "COMMON_TESTBED": "full_mesh_topo.py",
     "TESTCASES": [{"test_ports_fec_001": ""},{"test_ports_mtu_002": ""},{"test_ports_counters": ""}]
   },
   "./feature/taas_qual_L3_test.py": {
     "SKIP": false,
     "COMMON_TESTBED": "full_mesh_topo.py",
     "TESTCASES": [{"test_qual_bgp_004": ""},{"test_qual_bgp_007": ""},{"test_qual_bgp_008": ""},
                   {"test_qual_bgp_009": ""},{"test_qual_bgp_010": ""},{"test_qual_bgp_013": ""},{"test_qual_bgp_014": ""},
                   {"test_qual_bgp_015": ""},{"test_qual_bgp_016": ""},{"test_qual_bgp_017": ""},
                   {"test_qual_bgp_019": ""},{"test_qual_vlan1": ""},{"test_qual_vrf": ""}]
   },
    "./feature/taas_qual_Mgmt_test.py": {
     "SKIP": false,
     "COMMON_TESTBED": "full_mesh_topo.py",
     "TESTCASES": [{"test_port_span_001": ""},{"test_port_span_002": ""},{"test_port_span_003": ""},
                   {"test_front_panel_ports_ipv6": ""}]
   },

     "./feature/taas_layer2_Vlan_test.py": {
     "SKIP": false,
     "COMMON_TESTBED": "full_mesh_topo.py",
     "TESTCASES": [{ "test_vlan_004": "" },{ "test_vlan_005": "" },{ "test_vlan_007": "" }, {"test_vlan_008": ""},{"test_vlan_011": ""}]
   }, 

    "./feature/taas_qual_Security_test.py": {
     "SKIP": false,
     "COMMON_TESTBED": "full_mesh_topo.py",
     "TESTCASES": [{"test_qual_ip6_acl_001": ""},{"test_qual_ip6_acl_002": ""},{"test_qual_ip6_acl_003": ""},
                   {"test_qual_ip6_acl_004": ""},{"test_qual_ip6_acl_005": ""},{"test_qual_ip6_acl_006": ""}]
   },
    "./feature/taas_qual_SNMP_test.py": {
     "SKIP": false,
     "COMMON_TESTBED": "ftas_mesh_topo.py",
     "TESTCASES": [{"test_snmp_walk_inf_admin_oper": ""}, {"test_snmp_walk_ip_inf_index": ""}, {"test_snmp_walk_ip_to_mac": ""}]
   }
}

• TEST***_FOLDER - Path to the test artefacts including testbeds, testsuites and reports
• TEST_CONTACT - Email address of the test owner. This information is included in the test report
• TESTSUITES - Define the sub-test suites in a key-value pair for execution

  • "./essential/taas_platform_Interface_test.py" - Location of the test script files.
  • SKIP - Defines whether a particular test script will be included or excluded from the execution. If set to "SKIP": true, then the tests will be excluded from execution.
  • COMMON_TESTBED - The testbed file all test cases use.
  • TESTCASES - A list of test functions (test cases) with the structure of "TESTCASES": [{"test_syslog_004": ""},{"test_syslog_002": ""}].

Customer suites

FTAS comes with the following predefined suites which can be run directly on the applicable platform and release combinations. These suite files are present in ~/testsuites directory

Suitefile	Description	Number of testcases
PI.suite	All platform independent testcases	148
PD.suite	All platform dependent testcases	232
data_1dut.suite	All one DUT testcases applicable on data switches	86
data_2dut.suite	All two DUT testcases applicable on data switches	199
data_3dut.suite	All three DUT testcases applicable on data switches	25
data_4dut.suite	All four DUT testcases applicable on data switches	130
data_complete.suite	All testcases applicable on data switches	448
mgmt_1dut.suite	All one DUT testcases applicable on management switches	23
mgmt_2dut.suite	All two DUT testcases applicable on management switches	87
mgmt_3dut.suite	All three DUT testcases applicable on management switches	8
mgmt_complete.suite	All testcases applicable on management switches	126
edgecore_4630_202111.suite	All testcases applicable on Edgecore 4630 platform for 202111 release	282
edgecore_9716_202111.suite	All testcases applicable on Edgecore 9716 platform for 202111 release	359
nvidia_202205.suite	All testcases applicable on NVIDIA platforms for 202205 release	240
wistron_3200_ecs2.2.2.suite	All testcases applicable on Wistron 3200 platforms for ECS 2.2.2 release	265
wistron_6512_ecs2.0.0.suite	All testcases applicable on Wistron 6512 platforms for ECS 2.2.2 release	292