Linux bridge开启hairpin模拟测试macvlan vepa模式

看到网上介绍可以通过Linux bridge 开启hairpin方式测试macvlan vepa模式，但是没有找到详细资料。我尝试测试总提示错误信息，无法实现，经过几天的研究，我总算实现模拟测试，记录如下：

参考

1.Linux Macvlan
2.图解几个与Linux网络虚拟化相关的虚拟网卡-VETH/MACVLAN/MACVTAP/IPVLAN
3.kube-proxy IPVS 模式的工作原理
4.Linux brctl 命令，虚拟网络设备 LinuxBridge 管理工具
5.Linux 虚拟网络设备 bridge
6.Linux虚拟网络设备—之使用Veth pair连接linux网桥bridge
7.brctl快速入门与基础

环境

1. 操作系统

Centos7.9

2. 安装包

安装测试环境需要的包

[root@centos7-10 ~]# yum install -y net-tools iputils telnet traceroute iproute bridge-utils NetworkManager 

• net-tools：netstat命令
• iputils：ping命令
• telnet：telnet命令
• traceroute：traceroute命令
• iproute：ip命令
• bridge-utils：brctl命令
• NetworkManager 网络管理命令

如果是ubuntu 命令如下：

apt install -y net-tools inetutils-ping telnet traceroute iproute2 bridge-utils network-manager

Linux bridge 介绍

Bridge概念详见：Linux brctl 命令，虚拟网络设备 LinuxBridge 管理工具

Macvlan 介绍

Macvlan概念详见：Linux brctl 命令，虚拟网络设备 LinuxBridge 管理工具

模拟测试

1. 测试流程

因交换不支持802.1q，故采用Linux bridge开启hairpin方式，模拟测试macvlan vepa模式，步骤如下：

• 创建Linux bridge br0
• 创建veth pair：veth0和veth0_1
• veth0加入br0
• veth0_1作为父网卡，创建两个macvlan子网卡veth0_1.101****和veth0_1.102，模式vepa
• 创建两个namespace：ns101和ns102
• veth0_1.101加入ns101，配置IP 10.211.55.101，启用
• veth0_1.102加入ns102，配置IP 10.211.55.102，启用
• 测试br0下关闭和开启接口veth0 hairpin时，macvlan vepa网络通讯情况

详见下图：

2. 创建Linux bridge br0

• 查看当前bridge

[root@centos7-18 ~]# brctl show
bridge name     bridge id               STP enabled     interfaces
virbr0          8000.5254009f1377       yes             virbr0-nic

• 创建bridge br0

// 创建br0
[root@centos7-18 ~]# brctl addbr br0
// 启用br0
[root@centos7-18 ~]# ip link set br0 up
// 查看bridge
[root@centos7-18 ~]# brctl show
bridge name     bridge id               STP enabled     interfaces
br0             8000.000000000000       no
virbr0          8000.5254009f1377       yes             virbr0-nic

3. 创建veth pair：veth0和veth0_1

• 查看当前网卡

[root@centos7-18 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp0s5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:1c:42:60:87:b2 brd ff:ff:ff:ff:ff:ff
    inet 10.211.55.18/24 brd 10.211.55.255 scope global enp0s5
       valid_lft forever preferred_lft forever
    inet6 fdb2:2c26:f4e4:0:21c:42ff:fe60:87b2/64 scope global mngtmpaddr dynamic 
       valid_lft 2591486sec preferred_lft 604286sec
    inet6 fe80::21c:42ff:fe60:87b2/64 scope link 
       valid_lft forever preferred_lft forever
3: enp0s6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 00:1c:42:d1:70:62 brd ff:ff:ff:ff:ff:ff
    inet 10.211.55.21/24 brd 10.211.55.255 scope global noprefixroute dynamic enp0s6
       valid_lft 1322sec preferred_lft 1322sec
    inet6 fdb2:2c26:f4e4:0:2a52:f262:86d:6cd5/64 scope global noprefixroute dynamic 
       valid_lft 2591486sec preferred_lft 604286sec
    inet6 fe80::bfab:127:7500:dd3c/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:9f:13:77 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
5: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast master virbr0 state DOWN group default qlen 1000
    link/ether 52:54:00:9f:13:77 brd ff:ff:ff:ff:ff:ff
6: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 86:13:97:70:a2:e2 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::8413:97ff:fe70:a2e2/64 scope link 
       valid_lft forever preferred_lft forever
[root@centos7-18 ~]# 

• 创建虚拟网卡veth0和veth0_1

// 创建veth0和veth0_1
[root@centos7-18 ~]# ip link add veth0 type veth peer name veth0_1
// 启用veth0和veth0_1
[root@centos7-18 ~]# ip link set veth0 up
[root@centos7-18 ~]# ip link set veth0_1 up
// 查看veth
[root@centos7-18 ~]# ip a | grep -A4 veth0
7: veth0_1@veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 7a:87:ef:c6:77:9b brd ff:ff:ff:ff:ff:ff
    inet6 fe80::7887:efff:fec6:779b/64 scope link 
       valid_lft forever preferred_lft forever
8: veth0@veth0_1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 86:08:8e:91:09:fe brd ff:ff:ff:ff:ff:ff
    inet6 fe80::8408:8eff:fe91:9fe/64 scope link 
       valid_lft forever preferred_lft forever
[root@centos7-18 ~]# 

4. veth0加入br0

// veth0加入br0
[root@centos7-18 ~]# brctl addif br0 veth0
[root@centos7-18 ~]# brctl show
bridge name     bridge id               STP enabled     interfaces
br0             8000.86088e9109fe       no              veth0
virbr0          8000.5254009f1377       yes             virbr0-nic

5. veth0_1作为父网卡，创建两个macvlan子网卡veth0_1.101和veth0_1.102，模式vepa

• 父网卡 veth0_1
• 子网卡 veth0_1.101
• 子网卡 veth0_1.102

// 创建两个macvlan子网卡veth0_1.101和veth0_1.102，模式vepa
[root@centos7-18 ~]# ip link add link veth0_1 name veth0_1.101 type macvlan mode vepa
[root@centos7-18 ~]# ip link add link veth0_1 name veth0_1.102 type macvlan mode vepa
// 查看创建结果
[root@centos7-18 ~]# ip a | grep -A5 veth0
7: veth0_1@veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 7a:87:ef:c6:77:9b brd ff:ff:ff:ff:ff:ff
    inet6 fe80::7887:efff:fec6:779b/64 scope link 
       valid_lft forever preferred_lft forever
8: veth0@veth0_1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default qlen 1000
    link/ether 86:08:8e:91:09:fe brd ff:ff:ff:ff:ff:ff
    inet6 fe80::8408:8eff:fe91:9fe/64 scope link 
       valid_lft forever preferred_lft forever
9: veth0_1.101@veth0_1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether b2:3e:6e:ae:74:57 brd ff:ff:ff:ff:ff:ff
10: veth0_1.102@veth0_1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 22:f8:d5:8b:c1:63 brd ff:ff:ff:ff:ff:ff

6. 创建两个namespace：ns101和ns102

// 创建ns101和ns102
[root@centos7-18 ~]# ip netns add ns101
[root@centos7-18 ~]# ip netns add ns102
// 查看结果
[root@centos7-18 ~]# ip netns list
ns102
ns101

7. veth子网卡加入namespace，配置网卡并启用

ns101和ns102网络隔离，将两个macvlan子网卡（veth0_1.101和veth0_1.102）分别加入其中

• veth0_1.101加入ns101，配置IP 10.211.55.101，启用

// veth0_1.101加入ns101
[root@centos7-18 ~]# ip link set veth0_1.101 netns ns101
// 查看ns101的网卡
[root@centos7-18 ~]# ip netns exec ns101 ip a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
9: veth0_1.101@if7: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether b2:3e:6e:ae:74:57 brd ff:ff:ff:ff:ff:ff link-netnsid 0

// ns101启用lo。不启用ping自己的IP，会不通
[root@centos7-18 ~]# ip netns exec ns101 ip link set lo up
// ns101配置IP 10.211.55.101
[root@centos7-18 ~]# ip netns exec ns101 ip addr add 10.211.55.101/24 dev veth0_1.101
// ns101启用veth0_1.101。
[root@centos7-18 ~]# ip netns exec ns101 ip link set veth0_1.101 up

// 查看ns101 网卡
[root@centos7-18 ~]# ip netns exec ns101 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
9: veth0_1.101@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether b2:3e:6e:ae:74:57 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.211.55.101/24 scope global veth0_1.101
       valid_lft forever preferred_lft forever
    inet6 fe80::b03e:6eff:feae:7457/64 scope link 
       valid_lft forever preferred_lft forever
[root@centos7-18 ~]# 
[root@centos7-18 ~]# ip netns exec ns101 ping -c2 10.211.55.101
PING 10.211.55.101 (10.211.55.101) 56(84) bytes of data.
64 bytes from 10.211.55.101: icmp_seq=1 ttl=64 time=0.037 ms
64 bytes from 10.211.55.101: icmp_seq=2 ttl=64 time=0.058 ms

--- 10.211.55.101 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.037/0.047/0.058/0.012 ms

• veth0_1.102加入ns102，配置IP 10.211.55.102，启用

// veth0_1.102加入ns102
[root@centos7-18 ~]# ip link set veth0_1.102 netns ns102
[root@centos7-18 ~]# ip netns exec ns102 ip link set lo up
[root@centos7-18 ~]# ip netns exec ns102 ip addr add 10.211.55.102/24 dev veth0_1.102
[root@centos7-18 ~]# ip netns exec ns102 ip link set veth0_1.102 up
[root@centos7-18 ~]# 

// 查看ns102 网卡
[root@centos7-18 ~]# ip netns exec ns102 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
10: veth0_1.102@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 22:f8:d5:8b:c1:63 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.211.55.102/24 scope global veth0_1.102
       valid_lft forever preferred_lft forever
    inet6 fe80::20f8:d5ff:fe8b:c163/64 scope link 
       valid_lft forever preferred_lft forever

[root@centos7-18 ~]# ip netns exec ns102 ping -c2 10.211.55.102
PING 10.211.55.102 (10.211.55.102) 56(84) bytes of data.
64 bytes from 10.211.55.102: icmp_seq=1 ttl=64 time=0.035 ms
64 bytes from 10.211.55.102: icmp_seq=2 ttl=64 time=0.055 ms

--- 10.211.55.102 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.035/0.045/0.055/0.010 ms
[root@centos7-18 ~]# 

8. 模拟测试macvlan vepa网络通讯情况

测试br0在关闭和开启接口veth0 hairpin时，macvlan vepa网络通讯情况

• br0关闭veth0 hairpin时（默认状态是off），macvlan子网卡无法互相访问

// ns101 无法ping通 ns102的10.211.55.102
[root@centos7-18 ~]# ip netns exec ns101 ip a | grep veth
9: veth0_1.101@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 10.211.55.101/24 scope global veth0_1.101
[root@centos7-18 ~]# 
[root@centos7-18 ~]# ip netns exec ns101 ping -c2 10.211.55.102
PING 10.211.55.102 (10.211.55.102) 56(84) bytes of data.
From 10.211.55.18 icmp_seq=1 Destination Host Unreachable
From 10.211.55.18 icmp_seq=2 Destination Host Unreachable

--- 10.211.55.102 ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1000ms
// ns102 无法ping通 ns101的10.211.55.101
[root@centos7-18 ~]# ip netns exec ns102 ip a | grep veth
10: veth0_1.102@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 10.211.55.102/24 scope global veth0_1.102
[root@centos7-18 ~]# 
[root@centos7-18 ~]# ip netns exec ns102 ping -c2 10.211.55.101
PING 10.211.55.101 (10.211.55.101) 56(84) bytes of data.
From 10.211.55.18 icmp_seq=1 Destination Host Unreachable
From 10.211.55.18 icmp_seq=2 Destination Host Unreachable

--- 10.211.55.101 ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 999ms

• br0开启veth0 hairpin时，macvlan子网卡经过br0转发，macvlan子网卡可以互相访问
  • 开启veth0 hairpin
  • 测试macvaln网络联通性

// 开启hairpin
[root@centos7-18 ~]# brctl hairpin br0 veth0 on
// 查看br0的veth0开启hairpin结果
[root@centos7-18 ~]# bridge -d link  | grep -A5 veth0
8: veth0 state UP @veth0_1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 2 
    hairpin on guard off root_block off fastleave off learning on flood on mcast_flood on 
[root@centos7-18 ~]# 

// ns101 可以ping通 ns102的10.211.55.102
[root@centos7-18 ~]# ip netns exec ns101 ip a | grep -A5 veth0
9: veth0_1.101@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether b2:3e:6e:ae:74:57 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.211.55.101/24 scope global veth0_1.101
       valid_lft forever preferred_lft forever
    inet6 fe80::b03e:6eff:feae:7457/64 scope link 
       valid_lft forever preferred_lft forever
[root@centos7-18 ~]# 
[root@centos7-18 ~]# ip netns exec ns101 ping -c2 10.211.55.102
PING 10.211.55.102 (10.211.55.102) 56(84) bytes of data.
64 bytes from 10.211.55.102: icmp_seq=1 ttl=64 time=0.048 ms
64 bytes from 10.211.55.102: icmp_seq=2 ttl=64 time=0.095 ms

--- 10.211.55.102 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 0.048/0.071/0.095/0.024 ms
[root@centos7-18 ~]# 
// ns102 可以ping通 ns101的10.211.55.101
[root@centos7-18 ~]# ip netns exec ns102 ip a | grep -A5 veth0
10: veth0_1.102@if7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 22:f8:d5:8b:c1:63 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.211.55.102/24 scope global veth0_1.102
       valid_lft forever preferred_lft forever
    inet6 fe80::20f8:d5ff:fe8b:c163/64 scope link 
       valid_lft forever preferred_lft forever
[root@centos7-18 ~]# 
[root@centos7-18 ~]# ip netns exec ns102 ping -c2 10.211.55.101
PING 10.211.55.101 (10.211.55.101) 56(84) bytes of data.
64 bytes from 10.211.55.101: icmp_seq=1 ttl=64 time=0.047 ms
64 bytes from 10.211.55.101: icmp_seq=2 ttl=64 time=0.077 ms

--- 10.211.55.101 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.047/0.062/0.077/0.015 ms
[root@centos7-18 ~]# 

总结

通过Linux bridge 开启接口hairpin的方式，模拟macvlan vepa在外部交换支持802.1q的情况下，同一父网卡下的多个子网卡之间是可以通讯的。

之所以使用bridge、veth pair和macvlan组合进行模拟测试，是因为macvlan的父网卡不能属于其它bridge，如果尝试加入会报以下错误信息：

[root@centos7-18 ~]# brctl addif br0 veth0_1
device veth0_1 is already a member of a bridge; can't enslave it to bridge br0.
[root@centos7-18 ~]#