VPP on Debian with BIRD and ConnectX-4: How We Built a Faster Router Edge

We previously ran VyOS with VPP in our routing stack. It gave us the attraction of an appliance-style network operating system and a fast packet path, but bugs in the combination made changes and fault isolation harder than we wanted. The issue was not that VyOS or VPP is universally unsuitable; it was that the joined stack was no longer the right operational boundary for our network.

Our replacement is deliberately less magical. We now run two separate physical router hosts on Debian. Each host uses BIRD as the BGP control plane, FD.io VPP as the packet-forwarding plane and ConnectX-4 Ethernet adapters for the high-speed interfaces. Debian owns the system lifecycle, BIRD owns route policy, and VPP owns moving packets.

Why We Moved Away from the Previous Stack

Bundling the network operating system, routing daemon and VPP integration into one product made the system convenient until a fault crossed those boundaries. When forwarding, interface state or route synchronisation behaved unexpectedly, it took longer to establish which layer was responsible. Upgrades also moved several parts at once.

Debian gives us a small, familiar operating-system base with explicit package versions and standard observability. BIRD can be upgraded, validated and rolled back as the routing daemon. VPP can be pinned and tested as the forwarding engine. That separation gives each component a narrower job and makes a failed change easier to contain.

How BIRD, Linux and VPP Fit Together

FD.io describes the Linux Control Plane model as Linux providing the familiar network stack while VPP behaves like a software forwarding ASIC. VPP owns the physical interface. An LCP interface pair creates a corresponding Linux TAP or TUN interface so Linux applications can use it, while the Linux netlink plugin synchronises interface state, addresses, neighbours and routes.

1. BIRD establishes BGP sessions through Linux-facing LCP interfaces.
2. Import and export filters decide which prefixes are accepted and announced.
3. BIRD exports selected routes into a dedicated Linux kernel routing table.
4. The VPP Linux netlink integration mirrors the approved routes and neighbours into VPP.
5. Packets enter a ConnectX-4 queue and are forwarded by a VPP worker rather than the regular Linux data path.

This is an important control boundary: VPP should never receive an unfiltered internet table merely because BIRD learned it. Policy lives in BIRD, and only the routes intended for forwarding are exported to the kernel and synchronised onward.

Hardware, Firmware and Debian Baseline

Use a current supported Debian release, a CPU with enough physical cores for dedicated VPP workers, and RAM local to the NIC's NUMA node. ConnectX-4 and ConnectX-4 Lx cards vary by model, port speed and PCIe layout, so confirm the exact part number rather than assuming every card has the same capabilities.

• Two independent hosts, power feeds and management paths where possible
• ConnectX-4 Ethernet adapters with current, tested firmware
• CPU cores reserved for VPP workers rather than shared with busy system services
• Enough huge pages and buffers for the expected route and traffic profile
• Out-of-band access for a failed network configuration
• A staged BGP policy and an explicit rollback plan

sudo apt update
sudo apt full-upgrade -y
sudo apt install -y curl ca-certificates gnupg pciutils ethtool rdma-core bird2

lspci -nn | grep -i -E 'Mellanox|NVIDIA'
ethtool -i enp65s0f0np0
rdma link
numactl --hardware

The interface names in this guide are examples. Record the PCI address, Linux name, MAC address, NUMA node, switch port and intended role for every physical port before VPP takes ownership of it.

Install VPP on Debian

Use the package repository for a currently supported FD.io release and follow the matching installation page rather than copying a repository URL from an old guide. VPP plugins and configuration syntax can change between release trains, so pin the version that passed your lab and failover tests.

# After adding the current official FD.io Debian repository:
sudo apt update
sudo apt install -y vpp vpp-plugin-core vpp-plugin-dpdk
sudo systemctl enable vpp

vppctl show version
vppctl show plugins
vppctl show threads

Start conservatively. Reserve one main core and a small number of workers, place workers on the NIC's NUMA node, and increase buffers only after observing pressure. Do not blindly paste another operator's core map.

unix {
  nodaemon
  cli-listen /run/vpp/cli.sock
}

cpu {
  main-core 2
  corelist-workers 4-7
}

buffers {
  buffers-per-numa 131072
}

plugins {
  plugin linux_cp_plugin.so { enable }
  plugin linux_nl_plugin.so { enable }
  plugin rdma_plugin.so { enable }
}

Treat these values as a starting shape, not production truth. Validate the plugin filenames against vppctl show plugins for your installed release.

Create the ConnectX-4 RDMA Interfaces

VPP's RDMA driver is an Ethernet driver built on Linux rdma-core and documents ConnectX-4/5 support. It is not RoCE or InfiniBand configuration. The Linux mlx5 driver still discovers the device, while VPP creates an Ethernet interface from the host interface.

sudo modprobe ib_uverbs
rdma link
ethtool -i enp65s0f0np0

sudo vppctl create interface rdma host-if enp65s0f0np0 name transit0
sudo vppctl create interface rdma host-if enp65s0f1np1 name fabric0
sudo vppctl set interface state transit0 up
sudo vppctl set interface state fabric0 up
sudo vppctl show hardware-interfaces
sudo vppctl show interface

Restrict access to /dev/infiniband/uverbs*. A process allowed to control a raw Ethernet queue can divert traffic for the associated MAC address. Keep VPP privileged, keep its API and CLI sockets local, and isolate the management plane from forwarding interfaces.

Create Linux Control Plane Interface Pairs

Create a host-facing Linux interface for every VPP interface that BIRD or the Linux control plane needs. The exact command is release-dependent; current LCP builds expose an lcp create command. Confirm it with vppctl help lcp before applying the configuration.

sudo vppctl lcp create transit0 host-if transit0-host
sudo vppctl lcp create fabric0 host-if fabric0-host
sudo vppctl show lcp

sudo ip link set transit0-host up
sudo ip link set fabric0-host up
sudo ip addr add 192.0.2.2/31 dev transit0-host
sudo ip -6 addr add 2001:db8:100::2/127 dev transit0-host

Use addresses from your own point-to-point plan. For a routed host, disable redirects and source routing, turn off router advertisements where they are not expected, and choose a reverse-path-filtering policy that does not break legitimate asymmetric routing.

net.ipv4.ip_forward=1
net.ipv6.conf.all.forwarding=1
net.ipv4.conf.all.send_redirects=0
net.ipv4.conf.all.accept_source_route=0
net.ipv4.conf.all.rp_filter=0
net.ipv6.conf.all.accept_ra=0

Configure BIRD as the BGP Control Plane

BIRD should import only what the peer is permitted to announce and export only HYEHOST-owned or customer-authorised prefixes. The example below shows the shape of a BIRD 2 configuration; replace the documentation addresses, ASNs, filters and table names with reviewed production policy.

router id 192.0.2.2;

ipv4 table edge4;
ipv6 table edge6;

protocol device { scan time 10; }

filter peer4_in {
  if net ~ [ 0.0.0.0/0{0,24} ] then accept;
  reject;
}

filter peer4_out {
  if net ~ [ 203.0.113.0/24 ] then accept;
  reject;
}

protocol kernel kernel4 {
  ipv4 { table edge4; import none; export all; };
  kernel table 100;
  scan time 10;
}

protocol bgp upstream4 {
  local 192.0.2.2 as 64500;
  neighbor 192.0.2.3 as 64501;
  ipv4 {
    table edge4;
    import filter peer4_in;
    export filter peer4_out;
  };
}

Production filters should also reject martians, too-specific prefixes, invalid origin states where RPKI validation is available, unexpected first ASNs and routes beyond the agreed maximum-prefix count. Keep IPv4 and IPv6 policy explicit rather than relying on a permissive default.

Use Our BIRD Peer Summary for Daily Checks

Raw birdc output is invaluable when diagnosing one protocol in depth, but it is noisy when an engineer only needs a quick view of every BGP session. We use our open-source BIRD Peer Summary script, exposed as bps, on all of our BIRD routers.

The tool is a read-only wrapper around birdc. It does not edit configuration, reload BIRD, reset sessions or write to the daemon. One command summarises the peer name, neighbour address, neighbour ASN, session state, imported and exported route counts, state start time and a short detail string reported by BIRD. That gives the on-call engineer a useful first screen without combining several commands or opening a full monitoring dashboard.

sudo apt install -y git
git clone https://github.com/Lostepic/Bird-Peer-Summary.git
cd Bird-Peer-Summary
chmod +x install.sh
sudo ./install.sh

# Summary of every BGP peer
bps

# Established sessions only
bps --up

# Find a peer by name or ASN
bps --match linx
bps --match 47272

# Refresh a compact view every second
bps --watch 1 --compact

# Plain output for logs or tickets
bps --compact --no-color

The installer places bird-peer-summary and the shorter bps command in /usr/local/bin by default. If birdc is installed elsewhere, set the BIRDC environment variable to its full path. We still use native birdc show protocols all and route-export commands for deeper investigation; bps is the clean operational overview that tells us where to look next.

Roll Out Two Routers Without Creating One Shared Failure

Our design uses two separate Debian hosts. That means a kernel update, VPP restart, failed NIC or host-level fault does not automatically remove both forwarding paths. The benefit only exists if dependencies are also separated: management, switch ports, power, peer sessions and monitoring should not quietly converge on one shared point.

1. Build and soak the first router with no production announcements.
2. Load the expected routes and confirm memory, buffers and next hops.
3. Bring up one controlled BGP session with conservative local preference.
4. Move a small traffic slice and compare loss, latency and error counters.
5. Drain and restore the router to prove the rollback path.
6. Repeat on the second host, then test complete withdrawal of each router.

Tune for Throughput and Predictable Latency

VPP's vector packet processing handles packets in batches and can keep hot forwarding data in CPU caches. That can increase packets per second and reduce forwarding jitter when the router is busy. It does not promise a dramatic improvement to an idle single ping; the meaningful result is usually steadier latency and lower loss at load.

• Keep NIC queues and VPP workers on the same NUMA node.
• Reserve worker cores and avoid SMT siblings used by noisy services.
• Match RX queues to workers, then measure queue imbalance.
• Set a consistent MTU across switch, NIC, VPP and Linux LCP interfaces.
• Watch PCIe width and speed; a fast NIC in a constrained slot still bottlenecks.
• Change descriptors and buffers only when counters prove starvation.
• Benchmark IPv4, IPv6, small packets, large packets and mixed traffic separately.

sudo vppctl show threads
sudo vppctl show runtime
sudo vppctl show interface
sudo vppctl show errors
sudo vppctl show buffers
sudo ethtool -S enp65s0f0np0

Validate Routes, Traffic and Failure Behaviour

Measure before and after with the same traffic profile. Record packets per second, throughput, drop rate, CPU per worker and p50, p95 and p99 round-trip time. Then repeat during a full BGP table refresh and during withdrawal of one router.

sudo birdc show protocols all
sudo birdc show route count
sudo birdc show route export upstream4
ip route show table 100
sudo vppctl show ip fib summary
sudo vppctl show ip6 fib summary
sudo vppctl show ip neighbors
sudo vppctl show errors

Use the public HYEHOST Looking Glass to observe reachability from our network, but use controlled packet generation and telemetry for capacity testing. A looking glass is a diagnostic view, not a load generator.

Common VPP, LCP and BIRD Problems

VPP on Debian FAQ

Can VPP run on Debian?

Yes. FD.io publishes Debian packages. Pin a tested VPP release and verify its Linux Control Plane, netlink and driver plugins before production rollout.

How do BIRD and VPP work together?

BIRD speaks BGP and applies route policy through Linux. Approved routes enter a Linux kernel table, and the VPP Linux Control Plane integration synchronises routes, addresses and neighbours into the forwarding plane.

Does VPP support ConnectX-4 network cards?

FD.io lists ConnectX-4 and ConnectX-5 as production-supported by the RDMA Ethernet driver. Exact queue, speed and offload capabilities depend on the card, firmware, PCIe placement and VPP release.

Does VPP lower ping?

It can reduce forwarding jitter and tail latency under load when the system is tuned correctly. Idle-path latency may change very little, so we measure distributions under realistic traffic rather than advertising one best-case ping.

Why use two physical router hosts?

Two hosts let us drain one router for maintenance and preserve a forwarding path during many host, NIC or software failures. BGP convergence and upstream diversity still need to be designed and tested.

Should VPP use DPDK or the RDMA driver?

Both can be valid. We use ConnectX-4 hardware and evaluate the VPP RDMA path because it integrates through rdma-core and supports the card family. Choose from measured compatibility and operational needs, not labels alone.

A Faster Edge Is an Operational Result

The migration gives us a forwarding plane designed for sustained packet rates, but the larger improvement is control. Debian, BIRD and VPP now have clear responsibilities; each router can be drained and changed independently; and failures are easier to place in one layer.

That is how HYEHOST can carry more traffic with more predictable latency and fewer surprises as the network grows. Customers can inspect live paths through our Looking Glass, learn about AS47272, or deploy automated BGP through our IP transit platform.

Official Resources

• FD.io VPP installation guide
• VPP Linux Control Plane integration
• VPP RDMA Ethernet driver documentation
• VPP supported features
• BIRD Internet Routing Daemon
• HYEHOST BIRD Peer Summary
• NVIDIA ConnectX driver documentation