OpenStack & GoBGP

OpenStack setup bringup

This section covers how FTAS works with OpenStack and GoBGP in a lab environment. Use it to understand where these components fit in the setup, what role they play in route and service validation, and what to prepare before running BGP-based test cases.

Two-node OpenStack cluster

OS02 — Controller + Compute OS03 — Compute

Ubuntu 22.04.5 LTS OpenStack Bobcat (2024.1) KVM/QEMU Hypervisor

1. Purpose and scope

This document is the definitive reference guide to run the automated OpenStack VM live-migration script.

The script creates a virtual machine using a lightweight CirrOS image and then continuously moves it between OS02 and OS03 using OpenStack Nova live migration. The VM stays powered on during the move.

2. Environment overview

The OpenStack cloud consists of two physical servers connected to a leaf-spine fabric. One server hosts all OpenStack control-plane services inside Linux containers, and both servers act as compute nodes where VMs run.

2.1 Physical server summary

Server	Hostname	Management IP	Role	Uplink
Server 1	os02	10.20.x.x	Controller + Compute	10G to Leaf1
Server 2	os03	10.20.x.x	Dedicated Compute	10G to Leaf2

2.2 Key IP addresses example

IP address	Hostname or service	Purpose
10.20.x.x	os02	OS02 physical management interface (br-mgmt)
10.20.x.x	os03	OS03 physical management interface (br-mgmt)
10.20.x.x	VIP (Load Balancer)	All OpenStack API endpoints are reachable at this virtual IP
192.168.20.11	os02 mig-int	OS02 live-migration interface on VLAN 20
192.168.20.12	os03 mig-int	OS03 live-migration interface on VLAN 20
10.20.x.x/24	Management subnet	LXC containers and hypervisor management traffic
10.10.x.x/24	Tunnel subnet	VXLAN overlay tunnel traffic between nodes
192.168.20.0/24	VM and migration subnet	Tenant VMs and libvirt live-migration TCP stream

Keep all 192.168.x.x IPs the same as documented here. If you change one value, update every related command, port setting, route, and interface reference to match.

2.3 OpenStack LXC containers on OS02

All OpenStack control-plane services run inside Linux containers on OS02. These containers must all be in RUNNING state before executing the script.

Container name	Management IP	Function
os02-galera-container-0bb9da1e	10.20.x.x	MySQL/Galera database
os02-rabbit-mq-container-e57ae9a1	10.20.x.x	RabbitMQ message queue
os02-keystone-container-1d601d44	10.20.x.x	Keystone identity and authentication
os02-glance-container-142bd293	10.20.x.x	Glance image service
os02-nova-api-container-820a150c	10.20.x.x	Nova API, scheduler, and conductor
os02-neutron-server-container-7731d725	10.20.x.x	Neutron networking service
os02-cinder-api-container-752dfe2b	10.20.x.x	Cinder block storage service
os02-heat-api-container-ff75cc2a	10.20.x.x	Heat orchestration service
os02-horizon-container-7703010a	10.20.x.x	Horizon web dashboard
os02-memcached-container-6f74c143	10.20.x.x	Memcached token and session cache
os02-placement-container-f88213a0	10.20.x.x	Placement API
os02-repo-container-3f6647ec	10.20.x.x	Internal package repository mirror
os02-utility-container-9616c3ff	10.20.x.x	Utility container where the script runs

The utility container os02-utility-container-9616c3ff is the execution point for the migration script. It must be RUNNING, and the OpenRC credentials file must exist at /root/openrc.

3. Network architecture

Understanding the network layout is critical for diagnosing migration failures. Three independent networks serve different purposes in this environment.

3.1 Network planes summary

Network plane	Subnet	Bridge	Purpose
Management	10.20.x.x/24	br-mgmt	LXC communication and OpenStack API traffic
Tunnel (VXLAN)	10.10.x.x/24	br-vxlan	Overlay traffic between hosts
Provider (VLAN)	N/A (trunk)	br-vlan	Tenant VM traffic on physical VLANs
Migration (VLAN 20)	192.168.20.0/24	mig-int	Dedicated libvirt TCP channel for live memory transfer

3.2 VM network (mobility-test-net)

The script creates VMs on a pre-existing OpenStack network. This network must exist before the script runs.

• Network name: mobility-test-net
• Network type: VLAN (provider: physnet1, segmentation ID 20)
• Subnet name: mobility-test-subnet
• Subnet CIDR: 192.168.20.0/24
• DHCP: enabled with allocation pool 192.168.20.2 to 192.168.20.254
• Gateway: 192.168.20.1

The VM network on VLAN 20 is also the live-migration network. The mig-int interface on OS02 (192.168.20.11) and OS03 (192.168.20.12) is tagged inside this VLAN on the OVS br-vlan bridge.

3.3 Live migration network

Live migration transfers a running VM's memory state from one hypervisor to the other over the network.

• Transport: plain TCP on port 16509
• Interface: mig-int on each host
• OS02 migration IP: 192.168.20.11
• OS03 migration IP: 192.168.20.12
• Nova configuration key: live_migration_scheme = tcp
• libvirt authentication: auth_tcp = none

If TCP port 16509 is not reachable between OS02 and OS03 on the 192.168.20.0/24 subnet, all live migrations will fail.

4. Required OpenStack resources

The following resources must already exist before the script is executed.

4.1 Glance image

Attribute	Value
Image name	cirros
Image ID	26b0e1a3-e211-44a9-a9d1-84d52f04f797
Format	qcow2
Status	active
Size	~21 MB
Virtual size	~117 MB
Visibility	public
CPU mode	host-model (hw_cpu_mode=host-model)
Storage	File-backed at /var/lib/glance/images/ on OS02

CirrOS is a lightweight Linux image for cloud testing. It boots quickly and uses minimal memory, which makes it ideal for rapid VM creation and migration validation.

4.2 Nova flavor

Attribute	Value
Flavor name	m1.tiny
RAM	512 MB
vCPUs	1
Root disk	1 GB
Ephemeral disk	0 GB
Swap	0 MB
Public	Yes

4.3 Nova compute services

Two compute nodes must be registered and in up state.

Binary	Host	Status	State	Role
nova-conductor	os02-nova-api-container-*	enabled	up	Manages VM state in the database
nova-scheduler	os02-nova-api-container-*	enabled	up	Selects the target host
nova-compute	os02	enabled	up	Runs VMs on OS02
nova-compute	os03	enabled	up	Runs VMs on OS03

4.4 Nova hypervisors

Hypervisor	Host IP	Type and version	State	Status
os02	10.20.x.x	QEMU v6.2.0	up	enabled
os03	10.20.x.x	QEMU v6.2.0	up	enabled

5. Prerequisites checklist

Before running the migration script, verify every item in this checklist.

5.1 Hardware and OS prerequisites

ID	Requirement	Verification command
H1	Both OS02 and OS03 are powered on and reachable via SSH	ping 10.20.x.x && ping 10.20.x.x
H2	Ubuntu 22.04.5 LTS is installed on both servers	lsb_release -a
H3	Kernel version 5.15.0+ is installed and KVM modules are loaded	uname -r && lsmod | grep kvm
H4	CPU virtualization is enabled	egrep -c '(vmx|svm)' /proc/cpuinfo
H5	Both hosts have at least 512 MB free RAM	free -h
H6	Both hosts have at least 2 GB free in /var/lib/nova	df -h /var/lib/nova

5.2 Network prerequisites

ID	Requirement	Verification command
N1	br-mgmt is up on OS02	ip -br addr show br-mgmt
N2	br-mgmt is up on OS03	ip -br addr show br-mgmt
N3	VIP 10.20.x.x is reachable	ping -c 3 10.20.x.x
N4	mig-int is up on OS02	ip -br addr show mig-int
N5	mig-int is up on OS03	ip -br addr show mig-int
N6	OS02 can reach OS03 on the migration port	nc -zv 192.168.20.12 16509
N7	OS03 can reach OS02 on the migration port	nc -zv 192.168.20.11 16509
N8	br-vlan is up on both hosts	ip -br link show br-vlan
N9	OVS bridges br-int and br-vlan exist	ovs-vsctl show
N10	Leaf uplinks are active	ethtool enp24s0f0 | grep 'Link detected'

5.3 OpenStack service prerequisites

Run these commands in the utility container unless stated otherwise.

ID	Requirement	Verification command
S1	All 13 LXC containers on OS02 are in RUNNING state	lxc-ls -f
S2	Keystone authentication works	openstack token issue
S3	nova-compute on os02 is up	openstack compute service list
S4	nova-compute on os03 is up	openstack compute service list
S5	nova-conductor and nova-scheduler are up	openstack compute service list
S6	Both hypervisors are visible and up	openstack hypervisor list
S7	The cirros image exists and is active	openstack image list
S8	The m1.tiny flavor exists	openstack flavor list
S9	The mobility-test-net network exists	openstack network list
S10	The mobility-test-subnet subnet exists	openstack subnet list
S11	Glance is healthy	openstack image list
S12	Neutron agents are healthy on both nodes	openstack network agent list

5.4 libvirt prerequisites

ID	Requirement	Verification command
L1	libvirtd is running on OS02	systemctl status libvirtd
L2	libvirtd is running on OS03	systemctl status libvirtd
L3	libvirt listens on TCP port 16509 on OS02	ss -tulnp | grep 16509
L4	libvirt listens on TCP port 16509 on OS03	ss -tulnp | grep 16509
L5	nova-compute was restarted after any kernel upgrade	systemctl status nova-compute
L6	No stale virsh domains block migration slots	virsh list --all

5.5 Credential prerequisites

ID	Requirement	Verification
C1	OpenRC exists at /root/openrc in the utility container	cat /root/openrc
C2	OS_AUTH_URL points to http://10.20.x.x:5000/v3	grep OS_AUTH_URL /root/openrc
C3	Admin credentials are valid	source /root/openrc && openstack token issue
C4	Root SSH between OS02 and OS03 works	ssh 10.20.x.x hostname

6. Accessing the utility container

The script must be launched from inside the OpenStack utility container.

Step 1: SSH into OS02

ssh root@10.20.x.x

Step 2: Attach to the utility container

sudo lxc-attach -n os02-utility-container-9616c3ff

After this command, the shell prompt changes to:

root@os02-utility-container-9616c3ff:/#

Step 3: Load OpenStack credentials

source /root/openrc

Load the credentials each time you open a new shell inside the container.

To verify that credentials are loaded, run openstack token issue. If it returns a token ID, authentication is working.

Step 4: Verify the environment is ready

openstack compute service list
openstack hypervisor list
openstack image list
openstack network list

7. What the script does

The script automates a continuous demonstration of OpenStack live migration.

Phase 1 — VM creation

The script creates a new VM using:

• Image: cirros
• Flavor: m1.tiny
• Network: mobility-test-net

The VM receives an IP address from 192.168.20.0/24 through DHCP and starts in ACTIVE state.

Phase 2 — Continuous live migration loop

The script:

1. identifies the current host of the VM
2. triggers openstack server migrate --live toward the other host
3. waits for the VM to return to ACTIVE
4. repeats the move in the opposite direction

During migration, Nova transfers the VM memory state between the hosts over mig-int on TCP port 16509.

Migration types demonstrated

Type	What moves	VM downtime
Live migration	VM memory is copied while the guest keeps running	Near-zero
Block migration	VM disk data is also copied because shared storage is absent	Near-zero, but longer

This environment does not use shared storage such as Ceph or NFS. Nova therefore uses block migration automatically.

8. Post-run verification commands

Run these commands inside the utility container after the script starts.

8.1 Check VM location and status

openstack server show <vm-name> -c OS-EXT-SRV-ATTR:host -c status -c addresses

8.2 List all VMs

openstack server list

Expected result: one or more VMs in ACTIVE state on mobility-test-net.

8.3 List running VMs on each hypervisor

openstack server list --host os02
openstack server list --host os03

8.4 Check compute services

openstack compute service list

Every service should show State = up and Status = enabled.

8.5 Verify migration traffic flow

From OS02:

nc -zv 192.168.20.12 16509

From OS03:

nc -zv 192.168.20.11 16509

Both commands must return succeeded or open.

GoBGP setup bringup

Configure allowed address pairs for a VM port

For mobility and multi-IP test cases, the VM port must accept additional IP addresses. In OpenStack, configure this with allowed address pairs on the Neutron port attached to the VM.

Find the VM and its port

Log in to the utility container and load the OpenStack environment:

lxc-attach -n os02-utility-container-9616c3ff
source /root/openrc

List the VM and capture its server ID:

openstack server list

Example output:

+--------------------------------------+------------+--------+---------------------------------+--------------+-----------+
| ID                                   | Name       | Status | Networks                        | Image        | Flavor    |
+--------------------------------------+------------+--------+---------------------------------+--------------+-----------+
| 73a5b2f8-d974-4415-ae5d-c37e78043693 | ubuntu-vm1 | ACTIVE | mobility-test-net=192.168.20.36 | ubuntu-22.04 | frr.large |
+--------------------------------------+------------+--------+---------------------------------+--------------+-----------+

Use the server ID to find the attached port:

openstack port list --server 73a5b2f8-d974-4415-ae5d-c37e78043693

Example output:

+--------------------------------------+------+-------------------+------------------------------------------------------------------------------+--------+
| ID                                   | Name | MAC Address       | Fixed IP Addresses                                                           | Status |
+--------------------------------------+------+-------------------+------------------------------------------------------------------------------+--------+
| 3bf433bb-81ef-45c5-aff9-5cf70a890435 |      | fa:16:3e:42:49:ab | ip_address='192.168.20.36', subnet_id='4bbf0452-4240-40e2-89ca-181946a3a54f' | ACTIVE |
+--------------------------------------+------+-------------------+------------------------------------------------------------------------------+--------+

Verify the port details

Inspect the port before making changes:

openstack port show 3bf433bb-81ef-45c5-aff9-5cf70a890435

Confirm these values:

• The port status is ACTIVE.
• The port is bound to the expected compute host.
• The VM MAC address matches the interface you plan to use.

Add the extra IP addresses

Set the allowed address pairs on the port:

openstack port set 3bf433bb-81ef-45c5-aff9-5cf70a890435 \
  --allowed-address ip-address=10.20.30.33 \
  --allowed-address ip-address=192.168.100.1 \
  --allowed-address ip-address=192.168.100.101

This keeps the port MAC address unchanged and allows the VM to use the additional IP addresses on the same interface.

Verify the update

Check that the allowed address pairs were applied:

openstack port show 3bf433bb-81ef-45c5-aff9-5cf70a890435 -f value -c allowed_address_pairs

Example output:

[{'mac_address': 'fa:16:3e:42:49:ab', 'ip_address': '10.20.30.33'}, {'mac_address': 'fa:16:3e:42:49:ab', 'ip_address': '192.168.100.1'}, {'mac_address': 'fa:16:3e:42:49:ab', 'ip_address': '192.168.100.101'}]

Use allowed address pairs when the VM must source or receive traffic for IP addresses beyond its primary fixed IP. This is common in mobility and service validation workflows.

Configure a GoBGP remote VTEP on Ubuntu

This example uses an Ubuntu 22.04 VM as a remote VXLAN VTEP for a SONiC leaf. The VM connects on Vlan20, advertises its loopback through IPv4 unicast, and sends an EVPN Type-3 IMET route through GoBGP.

Prerequisites

• Install the GoBGP package before setup.
• Use GoBGP version 3.14.0.
• Ensure the gobgpd and gobgp binaries are available in /usr/local/bin.

root@ubuntu-gobgp:/home/ubuntu# wget https://github.com/osrg/gobgp/releases/download/v3.14.0/gobgp_3.14.0_linux_amd64.tar.gz

Key requirements:

• The VM interface must use a static IP on the VLAN-facing network.
• The VTEP loopback IP must be reachable from the switch.
• The OpenStack port must allow the extra loopback IP.
• The EVPN Type-3 route must include a PMSI ingress-replication attribute.

If the VM runs in OpenStack, add the VTEP loopback IP to the VM port allowed address pairs before testing. Without this, OVN port security drops traffic destined to the loopback IP.

Example setup script

gobgp-vxlan-server-setup.sh

#!/bin/bash
#============================================================================
# GoBGP + VXLAN Server Setup — Remote VTEP for SONiC Leaf
#============================================================================
#
# Topology:
#   Switch (SONiC DS3000-6035):
#     - Vlan20 SVI:    192.168.20.1/24
#     - VTEP Loopback: 2.2.2.2 (Loopback99)
#     - VNI 10020 -> Vlan20
#     - AS 65000
#
#   Server (Ubuntu 22.04 VM — OpenStack):
#     - ens3:          192.168.20.36/24 (directly connected to switch Vlan20)
#     - VTEP Loopback: 10.20.30.33/32 (on lo)
#     - VNI 10020
#     - AS 65001
#     - GoBGP 3.14.0 (/usr/local/bin/gobgpd)
#
# Switch port: Ethernet56, VLAN 20 tagged
# Connection: Switch Ethernet56 <-> Server ens3 (via OpenStack OVS/OVN)
#
# Prerequisites:
#   - GoBGP 3.14.0 installed at /usr/local/bin/gobgpd and /usr/local/bin/gobgp
#   - Ubuntu 22.04 VM with ens3 interface
#   - If OpenStack VM: allowed-address-pairs must include the VTEP loopback IP
#     openstack port set <port-id> --allowed-address ip-address=10.20.30.33
#     (Port ID for this VM: 3bf433bb-81ef-45c5-aff9-5cf70a890435)
#
# Key lessons learned:
#   1. PMSI ingress-repl is MANDATORY on Type-3 IMET — without it, SONiC's
#      zebra will not push the remote VTEP into the kernel FDB, and
#      "show vxlan remotevtep" stays empty
#   2. accept_local=1 required so kernel accepts packets destined to
#      loopback IPs arriving on ens3
#   3. OpenStack OVN port security drops traffic to IPs not in
#      allowed-address-pairs — this blocks ping to VTEP loopback
#   4. VTEP loopback must be advertised via IPv4 unicast so the switch
#      has a route to it
#   5. RD must be <vtep-ip>:2 to match SONiC's auto-derived format
#
#============================================================================

set -e

#============================================================================
# STEP 1: Netplan — Static IP on ens3
#============================================================================
# File: /etc/netplan/00-cloud-init.yaml
# Actual running config from the server.
#============================================================================

echo "=== Step 1: Writing netplan config ==="
sudo tee /etc/netplan/00-cloud-init.yaml > /dev/null << 'NETPLAN'
network:
  version: 2
  ethernets:
    ens3:
      dhcp4: no
      addresses: [192.168.20.36/24]
      routes:
        - to: default
          via: 192.168.20.1
      nameservers:
        addresses: [8.8.8.8, 1.1.1.1]
NETPLAN
sudo netplan apply
echo "Netplan applied: ens3 = 192.168.20.36/24, default gw = 192.168.20.1"

#============================================================================
# STEP 2: Kernel sysctl tuning
#============================================================================
# accept_local=1 : Allow kernel to accept packets destined to loopback IPs
#                   (10.20.30.33) arriving on ens3. Without this, the switch
#                   can never ping the VTEP loopback.
# rp_filter=0    : Disable reverse path filtering so replies from 10.20.30.33
#                   going out ens3 aren't dropped.
#============================================================================

echo "=== Step 2: Kernel sysctl tuning ==="
sudo sysctl -w net.ipv4.conf.lo.accept_local=1
sudo sysctl -w net.ipv4.conf.ens3.accept_local=1
sudo sysctl -w net.ipv4.conf.all.rp_filter=0
sudo sysctl -w net.ipv4.conf.ens3.rp_filter=0

# Make persistent across reboots
sudo tee /etc/sysctl.d/99-vxlan-vtep.conf > /dev/null << 'SYSCTL'
# Required for VXLAN VTEP on loopback
net.ipv4.conf.lo.accept_local = 1
net.ipv4.conf.ens3.accept_local = 1
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.ens3.rp_filter = 0
SYSCTL
echo "Sysctl applied and persisted to /etc/sysctl.d/99-vxlan-vtep.conf"

#============================================================================
# STEP 3: Loopback IP — VTEP source address
#============================================================================
# This is the IP that will be used as:
#   - VXLAN tunnel source (local parameter on vxlan interface)
#   - Next-hop in EVPN Type-3 IMET route
#   - Appears in "show vxlan remotevtep" DIP column on the switch
#============================================================================

echo "=== Step 3: VTEP loopback IP ==="
if ! ip addr show lo | grep -q "10.20.30.33/32"; then
    sudo ip addr add 10.20.30.33/32 dev lo
    echo "Added 10.20.30.33/32 to lo"
else
    echo "10.20.30.33/32 already on lo, skipping"
fi

#============================================================================
# STEP 4: VXLAN interface
#============================================================================
# VNI 10020 matches the switch's VNI-to-VLAN mapping (10020 -> Vlan20)
# local = VTEP loopback IP (NOT ens3 IP)
# dstport 4789 = standard VXLAN UDP port
# nolearning = we use EVPN for MAC learning, not data-plane learning
#============================================================================

echo "=== Step 4: VXLAN interface ==="
sudo ip link del vxlan10020 2>/dev/null || true
sudo ip link add vxlan10020 type vxlan id 10020 local 10.20.30.33 dstport 4789 nolearning
sudo ip link set vxlan10020 up
echo "vxlan10020 created: VNI 10020, local 10.20.30.33, dstport 4789, nolearning"

# Verify
echo "--- Verification ---"
ip -d link show vxlan10020

#============================================================================
# STEP 5: GoBGP configuration
#============================================================================
# - router-id = VTEP loopback IP
# - neighbor = switch Vlan20 SVI IP
# - local-address = server physical IP (ens3) since BGP TCP session
#   runs over the directly connected link, not the loopback
# - l2vpn-evpn: for EVPN Type-2/Type-3 routes
# - ipv4-unicast: to advertise VTEP loopback reachability to the switch
#============================================================================

echo "=== Step 5: GoBGP config ==="
sudo tee /etc/gobgp.conf > /dev/null << 'GOBGP_CONF'
[global.config]
  as = 65001
  router-id = "10.20.30.33"

[[neighbors]]
  [neighbors.config]
    neighbor-address = "192.168.20.1"
    peer-as = 65000

  [neighbors.transport.config]
    local-address = "192.168.20.36"

  [[neighbors.afi-safis]]
    [neighbors.afi-safis.config]
      afi-safi-name = "l2vpn-evpn"

  [[neighbors.afi-safis]]
    [neighbors.afi-safis.config]
      afi-safi-name = "ipv4-unicast"
GOBGP_CONF
echo "GoBGP config written to /etc/gobgp.conf"

#============================================================================
# STEP 6: Start GoBGP daemon
#============================================================================

echo "=== Step 6: Start GoBGP ==="
sudo pkill gobgpd 2>/dev/null || true
sleep 2
sudo /usr/local/bin/gobgpd -f /etc/gobgp.conf &>/var/log/gobgpd.log &
echo "GoBGP daemon started (log: /var/log/gobgpd.log)"

echo "Waiting for BGP session to establish..."
sleep 8

# Verify neighbor
echo "--- BGP Neighbor Status ---"
/usr/local/bin/gobgp neighbor

#============================================================================
# STEP 7: Advertise VTEP loopback via IPv4 unicast
#============================================================================
# The switch needs a route to 10.20.30.33 to:
#   a) Respond to ping (verify reachability)
#   b) Resolve the next-hop in EVPN Type-3 routes
#   c) Forward VXLAN-encapsulated traffic
#
# We advertise it with nexthop = 192.168.20.36 (ens3 IP) because that's
# the directly connected next-hop the switch can ARP for on Vlan20.
#
# On the switch this shows up as:
#   B>* 10.20.30.33/32 [20/0] via 192.168.20.36, Vlan20
#============================================================================

echo "=== Step 7: Advertise VTEP loopback via IPv4 ==="
/usr/local/bin/gobgp global rib add 10.20.30.33/32 -a ipv4 nexthop 192.168.20.36
echo "Advertised 10.20.30.33/32 nexthop 192.168.20.36"

#============================================================================
# STEP 8: Add EVPN Type-3 IMET route
#============================================================================
# This is THE route that makes "show vxlan remotevtep" work on the switch.
#
# Parameters explained:
#   multicast 10.20.30.33  = Originating router IP (VTEP IP)
#   etag 0                 = Ethernet Tag, must be 0 for VLAN-based service
#   rd 10.20.30.33:2       = Route Distinguisher — MUST match SONiC's format
#                             (SONiC auto-derives as <router-id>:<vni-index>)
#                             The :2 is the VNI index, visible in
#                             "show bgp l2vpn evpn vni" on the switch
#   rt 65000:10020         = Route Target — MUST match the switch's import RT
#                             (visible in "show bgp l2vpn evpn vni")
#   nexthop 10.20.30.33    = VTEP IP (NOT the physical ens3 IP)
#   encap vxlan            = Encapsulation type extended community
#   pmsi ingress-repl 10020 10.20.30.33
#                           = PMSI Tunnel Attribute — THIS IS CRITICAL
#                             Without it, FRR/zebra will NOT install the
#                             remote VTEP into the kernel FDB, and SONiC's
#                             vxlanmgrd/fpmsyncd will never see it.
#                             Format: pmsi <type> <label/vni> <tunnel-id/vtep-ip>
#============================================================================

echo "=== Step 8: Add EVPN Type-3 IMET route ==="
/usr/local/bin/gobgp global rib -a evpn add multicast \
  10.20.30.33 \
  etag 0 \
  rd 10.20.30.33:2 \
  rt 65000:10020 \
  nexthop 10.20.30.33 \
  encap vxlan \
  pmsi ingress-repl 10020 10.20.30.33
echo "Added Type-3 IMET with PMSI ingress-replication"

#============================================================================
# STEP 9: Verify everything
#============================================================================

echo ""
echo "=== Final Verification ==="
echo ""
echo "--- IPv4 RIB ---"
/usr/local/bin/gobgp global rib -a ipv4
echo ""
echo "--- EVPN RIB ---"
/usr/local/bin/gobgp global rib -a evpn
echo ""
echo "--- Interface State ---"
ip addr show lo | grep "10.20.30.33"
ip -d link show vxlan10020 | grep "vxlan id"
echo ""
echo "--- Route to switch VTEP ---"
ip route get 2.2.2.2

echo ""
echo "============================================"
echo "  SETUP COMPLETE"
echo "============================================"
echo ""
echo "Now verify on the SONiC switch (192.168.20.1):"
echo ""
echo "  ping 10.20.30.33                              # Must succeed"
echo "  show vxlan remotevtep                         # Should show 10.20.30.33"
echo "  vtysh -c 'show bgp l2vpn evpn route type 3'  # Type-3 from server"
echo "  vtysh -c 'show evpn vni 10020'               # Remote VTEPs list"
echo ""
echo "============================================"
echo "  OPENSTACK REQUIREMENT"
echo "============================================"
echo ""
echo "If this server is an OpenStack VM, you MUST add allowed-address-pairs"
echo "on the Neutron port so OVN doesn't drop traffic to the VTEP loopback."
echo ""
echo "From the OpenStack controller:"
echo "  openstack port set 3bf433bb-81ef-45c5-aff9-5cf70a890435 \\"
echo "    --allowed-address ip-address=10.20.30.33"
echo ""
echo "Without this, the switch can ping 192.168.20.36 but NOT 10.20.30.33,"
echo "and the VXLAN tunnel will never come up."
echo ""
echo "============================================"
echo "  TROUBLESHOOTING CHECKLIST"
echo "============================================"
echo ""
echo "1. show vxlan remotevtep is empty?"
echo "   -> Check Type-3 has PMSI attribute: gobgp global rib -a evpn"
echo "   -> Look for: {Pmsi: type: ingress-repl, label: 10020, tunnel-id: 10.20.30.33}"
echo ""
echo "2. Switch can't ping VTEP loopback?"
echo "   -> Check sysctl accept_local=1 on server"
echo "   -> Check OpenStack allowed-address-pairs"
echo "   -> Check: vtysh -c 'show ip route 10.20.30.33' on switch"
echo ""
echo "3. BGP session not establishing?"
echo "   -> Check: gobgp neighbor"
echo "   -> Verify switch has BGP neighbor 192.168.20.36 configured"
echo "   -> Verify l2vpn-evpn address family enabled on both sides"
echo ""
echo "4. Type-3 received on switch but remotevtep empty?"
echo "   -> RT mismatch: check vtysh -c 'show bgp l2vpn evpn vni' on switch"
echo "   -> Next-hop unreachable: check vtysh -c 'show ip route 10.20.30.33'"
echo "   -> Missing PMSI: re-add Type-3 with pmsi ingress-repl"

What this script configures

• Static addressing on ens3
• Kernel handling for loopback-based VTEP traffic
• A local VXLAN interface for VNI 10020
• A GoBGP session toward the switch
• IPv4 advertisement for loopback reachability
• An EVPN Type-3 IMET route with PMSI ingress replication

Validated runtime state

The following state was verified on the Ubuntu VM after setup:

• ens3 uses 192.168.20.36/24
• the default route points to 192.168.20.1
• lo includes 10.20.30.33/32
• vxlan10020 is up with local 10.20.30.33
• net.ipv4.conf.lo.accept_local = 1
• net.ipv4.conf.ens3.accept_local = 1
• net.ipv4.conf.all.rp_filter = 0
• net.ipv4.conf.ens3.rp_filter = 0
• GoBGP 3.14.0 runs from /usr/local/bin/gobgpd
• the BGP session to 192.168.20.1 in AS 65000 is established

In this lab, the loopback also carries these additional /32 IP addresses:

• 192.168.100.1/32
• 192.168.100.101/32

If you want to inspect the netplan file directly, use sudo:

sudo cat /etc/netplan/*.yaml

Re-add the IMET route with PMSI

If the switch receives the Type-3 route but show vxlan remotevtep stays empty, remove the old IMET route and add it again with the PMSI attribute:

gobgp global rib -a evpn del multicast \
  10.20.30.33 etag 0 rd 10.20.30.33:2 \
  rt 65000:10020 nexthop 10.20.30.33 encap vxlan

gobgp global rib -a evpn add multicast \
  10.20.30.33 etag 0 rd 10.20.30.33:2 \
  rt 65000:10020 nexthop 10.20.30.33 encap vxlan \
  pmsi ingress-repl 10020 10.20.30.33

After the re-add, verify that the EVPN RIB includes:

• route type multicast
• rd 10.20.30.33:2
• rt 65000:10020
• nexthop 10.20.30.33
• Pmsi: type: ingress-repl, label: 10020, tunnel-id: 10.20.30.33

Verify on the SONiC side

After the script runs, validate the control plane and VTEP programming on the switch:

ping 10.20.30.33
show vxlan remotevtep
vtysh -c 'show bgp l2vpn evpn route type 3'
vtysh -c 'show evpn vni 10020'