Test Suite Configuration

Test suite configuration is a text file where you can list all test functions (each test function is a test case) for a batch run. Test suite files are stored in the "~/testsuites" folder

oper@ftasvm:~$ cd testsuites/
oper@ftasvm:~/testsuites$ ls -lrth
total 156K
-rw-rw-r-- 1 oper oper  405 Jul  4 07:57 verify_yaml.py
-rw-rw-r-- 1 oper oper  539 Jul  4 07:57 verify_testbed.suite
-rw-rw-r-- 1 oper oper  808 Jul  4 07:57 mgmt_3dut.suite
-rw-rw-r-- 1 oper oper 4.7K Jul  4 07:57 mgmt_2dut.suite
-rw-rw-r-- 1 oper oper 1.8K Jul  4 07:57 mgmt_1dut.suite
-rw-rw-r-- 1 oper oper  11K Jul  4 07:57 edgecore_9716_202111.suite
-rw-rw-r-- 1 oper oper  839 Jul  4 07:57 data_3dut.suite
-rw-rw-r-- 1 oper oper 1.8K Jul  4 07:57 data_1dut.suite
-rw-rw-r-- 1 oper oper 6.9K Jul  4 07:57 PD.suite
-rw-r--r-- 1 oper oper  12K Jul  6 16:13 wistron_6512_ecs2.0.0.suite
-rw-r--r-- 1 oper oper  12K Jul  6 16:13 wistron_3200_ecs2.0.0.suite
-rw-r--r-- 1 oper oper  12K Jul  6 16:13 nvidia_202205.suite
-rw-r--r-- 1 oper oper 6.5K Jul  6 16:13 mgmt_complete.suite
-rw-r--r-- 1 oper oper 7.7K Jul  6 16:13 edgecore_4630_202111.suite
-rw-r--r-- 1 oper oper 6.4K Jul  6 16:13 data_2dut.suite
drwxrwxr-x 2 oper oper 4.0K Jul  7 07:27 ebay
-rw-r--r-- 1 oper oper 5.9K Jul  7 07:27 PI.suite
-rw-r--r-- 1 oper oper  12K Jul 14 11:02 data_complete.suite
-rw-r--r-- 1 oper oper 4.5K Jul 16 10:37 data_4dut.suite
-rw-r--r-- 1 oper oper 8.2K Jul 17 05:58 copy_data_complete.suite
oper@ftasvm:~/testsuites$ 

Below is an example of how a test suite file is structured:

PI.suite

---
 "TESTBED_ROOT_FOLDER": "/home/oper/testbeds"
 "TESTSCRIPT_ROOT_FOLDER": "/home/oper"
 "TESTSUITE_ROOT_FOLDER": ""
 "TEST_REPORT_ROOT_FOLDER": "/home/oper/reports"
 "CHECK_COMPATIBILITY": true
 "TEST_CONTACT": "[e-mail]"
 "TESTSUITES": {
   # 2 DUT test suites/scripts
    "./feature/taas_acl_test.py": {
     "SKIP": false,
     "COMMON_TESTBED": "full_mesh_topo.py",
     "TESTCASES": [{"test_acl_001": ""}, {"test_acl_002": ""}, {"test_acl_003": ""},{"test_acl_004": ""},{"test_acl_005": ""},{"test_acl_006": ""},
                   {"test_acl_007": ""},{"test_acl_008": ""},{"test_acl_009": ""},{"test_acl_010": ""},{"test_acl_011": ""},{"test_acl_012": ""}]
   },
    "./feature/taas_mtu_test.py": {
     "SKIP": false,
     "COMMON_TESTBED": "full_mesh_topo.py",
     "TESTCASES": [{"test_mtu_001": ""}]
   },
    "./feature/taas_mgmt_Syslog_test.py": {
     "SKIP": false,
     "COMMON_TESTBED": "full_mesh_topo.py",
     "TESTCASES": [{"test_syslog_002": ""},{"test_syslog_004": ""}]
   },
    "./feature/taas_autoneg_test.py": {
     "SKIP": false,
     "COMMON_TESTBED": "full_mesh_topo.py",
     "TESTCASES": [{"test_autoneg_001": ""}]
   },
    "./feature/taas_layer2_Vlan_ixia_test.py": {
     "SKIP": false,
     "COMMON_TESTBED": "full_mesh_topo.py",
     "TESTCASES": [{"test_vlan_014": ""},{"test_vlan_016": ""}]
   },
    "./feature/taas_layer2_LACP_test.py": {
     "SKIP": false,
     "COMMON_TESTBED": "full_mesh_topo.py",
     "TESTCASES": [{"test_lacp_003": ""}, {"test_lacp_005": ""}, {"test_lacp_011": ""},{"test_lacp_012": ""}]
   },
    "./feature/taas_layer2_LLDP_test.py": {
     "SKIP": false,
     "COMMON_TESTBED": "full_mesh_topo.py",
     "TESTCASES": [{"test_lldp_001": ""}, {"test_lldp_002": ""}, {"test_lldp_013": ""}]
   },
     "./feature/taas_layer3_ARP_test.py": {
     "SKIP": false,
     "COMMON_TESTBED": "full_mesh_topo.py",
     "TESTCASES": [{"test_arp_003": ""}, {"test_arp_007": ""}, {"test_arp_011": ""},{"test_arp_012": ""}]
   },
     "./feature/taas_layer3_IP_test.py": {
     "SKIP": false,
     "COMMON_TESTBED": "full_mesh_topo.py",
     "TESTCASES": [{"test_IP_001": ""}, {"test_IP_002": ""}, {"test_IP_005": ""},{"test_IP_006": ""},{"test_IP_011": ""},{"test_IP_014": ""}]
   },
     "./feature/taas_mgmt_Ping_test.py": {
     "SKIP": false,
     "COMMON_TESTBED": "full_mesh_topo.py",
     "TESTCASES": [{"test_ping_001": ""}, {"test_ping_009": ""}, {"test_ping_011": ""}, {"test_ping_013": ""}]
   },
      "./feature/taas_platform_Interface_test.py": {
     "SKIP": false,
     "COMMON_TESTBED": "full_mesh_topo.py",
     "TESTCASES": [{"test_ports_002": ""}, {"test_ports_005": ""}, {"test_ports_006": ""},{"test_ports_008": ""},{"test_ports_009_14": ""}]
   },
    "./scalability/taas_qual_Scale.py": {
     "SKIP": false,
     "COMMON_TESTBED": "full_mesh_topo.py",
     "TESTCASES": [{"test_10_secondary_subnet_under_vlan": ""}, {"test_20_secondary_subnet_under_vlan": ""}, {"test_max_secondary_subnet_under_vlan": ""}, {"test_v4_host_routes_scale_2k": ""},
                   {"test_v4_host_routes_scale_4k": ""}, {"test_v4_host_routes_scale_max_supported": ""}, {"test_v6_host_routes_scale_2k": ""}, {"test_v6_host_routes_scale_4k": ""},
                   {"test_v6_host_routes_scale_max_supported": ""}, {"test_v4_prefix_routes_scale_2k": ""}, {"test_v4_prefix_routes_scale_4k": ""}, {"test_v4_prefix_routes_scale_max_supported": ""},
                   {"test_v4_nexthops_scale_512": ""}, {"test_v4_nexthops_scale_1024": ""}, {"test_v4_nexthops_scale_max_supported": ""}, {"test_qual_scale_007": ""}, {"test_qual_scale_011": ""},
                   {"test_v4_acl_scale_128": ""}, {"test_v4_acl_scale_256": ""}, {"test_v4_acl_scale_512": ""}, {"test_v4_acl_scale_max_supported": ""},
                   {"test_qual_v6_acl_scale_128": ""}, {"test_qual_v6_acl_scale_256": ""}, {"test_qual_v6_acl_scale_512": ""},
                   {"test_qual_v6_acl_scale_max_supported": ""}]
   },
    "./feature/taas_mgmt_SSH_test.py": {
     "SKIP": false,
     "COMMON_TESTBED": "full_mesh_topo.py",
     "TESTCASES": [{"test_ssh_001": ""},{"test_ssh_002": ""},{"test_ssh_003": ""}]
   },
   "./feature/taas_layer3_bgp_netops_test.py": {
     "SKIP": false,
     "COMMON_TESTBED": "ftas_2duts_topo.py",
     "TESTCASES": [{"test_bgp_netops_002": ""}, {"test_bgp_netops_005_006": ""}, {"test_bgp_netops_008": ""},
                   {"test_bgp_netops_010": ""}]
   },
    "./feature/taas_layer3_BGP_ixia_test.py": {
     "SKIP": false,
     "COMMON_TESTBED": "full_mesh_topo.py",
     "TESTCASES": [{"test_bgp_004": ""},{"test_bgp_005": ""}]
   },
   "./feature/taas_qual_PortCfg_test.py": {
     "SKIP":false,
     "COMMON_TESTBED": "full_mesh_topo.py",
     "TESTCASES": [{"test_ports_fec_001": ""},{"test_ports_mtu_002": ""},{"test_ports_counters": ""}]
   },
   "./feature/taas_qual_L3_test.py": {
     "SKIP": false,
     "COMMON_TESTBED": "full_mesh_topo.py",
     "TESTCASES": [{"test_qual_bgp_004": ""},{"test_qual_bgp_007": ""},{"test_qual_bgp_008": ""},
                   {"test_qual_bgp_009": ""},{"test_qual_bgp_010": ""},{"test_qual_bgp_013": ""},{"test_qual_bgp_014": ""},
                   {"test_qual_bgp_015": ""},{"test_qual_bgp_016": ""},{"test_qual_bgp_017": ""},
                   {"test_qual_bgp_019": ""},{"test_qual_vlan1": ""},{"test_qual_vrf": ""}]
   },
    "./feature/taas_qual_Mgmt_test.py": {
     "SKIP": false,
     "COMMON_TESTBED": "full_mesh_topo.py",
     "TESTCASES": [{"test_port_span_001": ""},{"test_port_span_002": ""},{"test_port_span_003": ""},
                   {"test_front_panel_ports_ipv6": ""}]
   },

     "./feature/taas_layer2_Vlan_test.py": {
     "SKIP": false,
     "COMMON_TESTBED": "full_mesh_topo.py",
     "TESTCASES": [{ "test_vlan_004": "" },{ "test_vlan_005": "" },{ "test_vlan_007": "" }, {"test_vlan_008": ""},{"test_vlan_011": ""}]
   }, 

    "./feature/taas_qual_Security_test.py": {
     "SKIP": false,
     "COMMON_TESTBED": "full_mesh_topo.py",
     "TESTCASES": [{"test_qual_ip6_acl_001": ""},{"test_qual_ip6_acl_002": ""},{"test_qual_ip6_acl_003": ""},
                   {"test_qual_ip6_acl_004": ""},{"test_qual_ip6_acl_005": ""},{"test_qual_ip6_acl_006": ""}]
   },
    "./feature/taas_qual_SNMP_test.py": {
     "SKIP": false,
     "COMMON_TESTBED": "ftas_mesh_topo.py",
     "TESTCASES": [{"test_snmp_walk_inf_admin_oper": ""}, {"test_snmp_walk_ip_inf_index": ""}, {"test_snmp_walk_ip_to_mac": ""}]
   }
}

• TEST***_FOLDER - Path to the test artefacts including testbeds, testsuites and reports

• TEST_CONTACT - Email address of the test owner. This information is included in the test report

• TESTSUITES - Define the sub-test suites in a key-value pair for execution

  • "./essential/taas_platform_Interface_test.py" - Location of the test script files.

  • SKIP - Defines whether a particular test script will be included or excluded from the execution. If set to "SKIP": true, then the tests will be excluded from execution.

  • COMMON_TESTBED - The testbed file all test cases use.

  • TESTCASES - A list of test functions (test cases) with the structure of "TESTCASES": [{"test_syslog_004": ""},{"test_syslog_002": ""}].

Customer suites

FTAS comes with the following predefined suites which can be run directly on the applicable platform and release combinations. These suite files are present in ~/testsuites directory

Suitefile	Description	Number of testcases
PI.suite	All platform independent testcases	105
PD.suite	All platform dependent testcases	96
data_1dut.suite	All one DUT testcases applicable on data switches	19
data_2dut.suite	All two DUT testcases applicable on data switches	112
data_3dut.suite	All three DUT testcases applicable on data switches	8
data_4dut.suite	All four DUT testcases applicable on data switches	61
data_complete.suite	All testcases applicable on data switches	201
mgmt_1dut.suite	All one DUT testcases applicable on management switches	18
mgmt_2dut.suite	All two DUT testcases applicable on management switches	77
mgmt_3dut.suite	All three DUT testcases applicable on management switches	8
mgmt_complete.suite	All testcases applicable on management switches	111
edgecore_4630_202111.suite	All testcases applicable on Edgecore 4630 platform for 202111 release	142
edgecore_9716_202111.suite	All testcases applicable on Edgecore 9716 platform for 202111 release	181
nvidia_202205.suite	All testcases applicable on NVIDIA platforms for 202205 release	200
wistron_3200_ecs2.0.0.suite	All testcases applicable on Wistron 3200 platforms for ECS 2.0.0 release	184
wistron_6512_ecs2.0.0.suite	All testcases applicable on Wistron 6512 platforms for ECS 2.0.0 release	184