tcpdump使用pcap-filter抓Vxlan包内数据

1. 背景

看vxlan协议时，发现可以使用tcpdump高级用法（pcap-filter）抓包内数据，记录如下

2. 参考

1.tcpdump在生产中的常见用法与复杂过滤规则
2.在vxlan网络中使用tcpdump精确过滤抓包
3.IP头部详解
4.IP到十六进制转换器
5.tcpdump与BPF expression

3. 概念

man tcpdump
man pcap-filter

4. 环境

在这里插入图片描述

Docker overlay网络
- Docker自定义overlay网络（ol_network），vxlan id =4098，网段 172.16.0.0/16
- 节点1:宿主机centos7-10，网卡enp0s5，IP 10.211.55.10；运行容器IP 172.16.0.28；
- 节点2:宿主机centos7-18，网卡enp0s5，IP 10.211.55.18；运行容器IP 172.16.0.42；
- 节点3:宿主机centos7-22，网卡enp0s5，IP 10.211.55.22；运行容器IP 172.16.0.43.
查看节点1：宿主机centos7-10上自定义overlay网络（ol_network）
- 网段IP 172.16.0.0/16
- 容器IP 172.16.0.28/16
- vxlanid 4098

[root@centos7-10 ~]# docker network inspect ol_network
[{"Name": "ol_network","Id": "9u6c1vlfy4kvli7ihka8f5mjf","Created": "2024-02-26T17:59:24.77121473+08:00","Scope": "swarm","Driver": "overlay","EnableIPv6": false,"IPAM": {"Driver": "default","Options": null,"Config": [{"Subnet": "172.16.0.0/16","Gateway": "172.16.0.1"}]},"Internal": false,"Attachable": false,"Ingress": false,"ConfigFrom": {"Network": ""},"ConfigOnly": false,"Containers": {"3f75b0e3f12428bac0099ccd9c5a40e7e69ac50e6a6368c13f677e52a1c8557a": {"Name": "my-nginx.1.elw6yuwr0yid85477cqysbx5c","EndpointID": "9f2afa4733fa92595372ec4a97345186d22b9569aaf3c392b9d14e550e33dd6e","MacAddress": "02:42:ac:10:00:1c","IPv4Address": "172.16.0.28/16","IPv6Address": ""},"lb-ol_network": {"Name": "ol_network-endpoint","EndpointID": "b87c0028a6b83ed61ba3bfcbf2bf17370ed4af39631d23b6e972f8ee63ef452a","MacAddress": "02:42:ac:10:00:1f","IPv4Address": "172.16.0.31/16","IPv6Address": ""}},"Options": {"com.docker.network.driver.overlay.vxlanid_list": "4098"},"Labels": {},"Peers": [{"Name": "36c12810b62d","IP": "10.211.55.10"},{"Name": "d1c0ea4bd8fd","IP": "10.211.55.18"},{"Name": "8c9ea1baadab","IP": "10.211.55.22"}]}
]

查看节点3：宿主机centos7-22上自定义overlay网络（ol_network）
- 网段IP 172.16.0.0/16
- 容器IP 172.16.0.43/16
- vxlanid 4098

[root@centos7-22 ~]# docker network inspect ol_network 
[{"Name": "ol_network","Id": "9u6c1vlfy4kvli7ihka8f5mjf","Created": "2024-02-27T23:30:10.421963035+08:00","Scope": "swarm","Driver": "overlay","EnableIPv6": false,"IPAM": {"Driver": "default","Options": null,"Config": [{"Subnet": "172.16.0.0/16","Gateway": "172.16.0.1"}]},"Internal": false,"Attachable": false,"Ingress": false,"ConfigFrom": {"Network": ""},"ConfigOnly": false,"Containers": {"4e9f5b2912973facbf3d05d3e097bbc75fe6bb69aa90dcb68af9655a67bcc7d1": {"Name": "my-nginx.3.nmza4vtpz6rr2gg2f2spdddvx","EndpointID": "f129a0ab99d0c4c2438e6b3a488ef61c9c111e4994a7bc0a4a1e51508cf2914b","MacAddress": "02:42:ac:10:00:2b","IPv4Address": "172.16.0.43/16","IPv6Address": ""},"lb-ol_network": {"Name": "ol_network-endpoint","EndpointID": "4b45d74c5f676042a72ecf9f6f74010caa22dc9aedf6627c2e633ba349416059","MacAddress": "02:42:ac:10:00:2d","IPv4Address": "172.16.0.45/16","IPv6Address": ""}},"Options": {"com.docker.network.driver.overlay.vxlanid_list": "4098"},"Labels": {},"Peers": [{"Name": "8c9ea1baadab","IP": "10.211.55.22"},{"Name": "d1c0ea4bd8fd","IP": "10.211.55.18"},{"Name": "36c12810b62d","IP": "10.211.55.10"}]}
]

5. 用法

5.1 抓vxlan通讯中的icmp包

查看centos7-10网卡enp0s5信息

[root@centos7-10 ~]# ip a show enp0s5
2: enp0s5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000link/ether 00:1c:42:ae:b6:41 brd ff:ff:ff:ff:ff:ffinet 10.211.55.10/24 brd 10.211.55.255 scope global noprefixroute dynamic enp0s5valid_lft 1436sec preferred_lft 1436secinet6 fdb2:2c26:f4e4:0:233e:38df:2cbd:cec1/64 scope global noprefixroute dynamic valid_lft 2591789sec preferred_lft 604589secinet6 fe80::7e0c:1902:e1ca:4324/64 scope link tentative noprefixroute dadfailed valid_lft forever preferred_lft foreverinet6 fe80::567a:248b:5e94:5d19/64 scope link noprefixroute valid_lft forever preferred_lft forever

查看centos7-10上overlay网络（ol_network）中运行容器3f75b0e3f124的信息
- 容器网卡668: eth1@if669，IP 172.16.0.28

// 查看容器
[root@centos7-10 ~]# docker ps -a
CONTAINER ID   IMAGE                 COMMAND                   CREATED      STATUS      PORTS     NAMES
3f75b0e3f124   nginx-sj:2024022601   "nginx -g 'daemon of…"   2 days ago   Up 2 days             my-nginx.1.elw6yuwr0yid85477cqysbx5c
// 查看容器网卡和IP
[root@centos7-10 ~]# docker exec -it 3f ip a show eth1
668: eth1@if669: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default link/ether 02:42:ac:10:00:1c brd ff:ff:ff:ff:ff:ff link-netnsid 1inet 172.16.0.28/16 brd 172.16.255.255 scope global eth1valid_lft forever preferred_lft forever

节点1（10.211.55.10）中容器（172.16.0.28）ping 节点3（10.211.55.22）中容器的IP 172.16.0.43
- 经过Vxlan封装后走物理网络通讯，网络接口是宿主机的enp0s5

// 节点1（10.211.55.10）中容器（172.16.0.28）ping 节点3（10.211.55.22）中容器的IP 172.16.0.43
[root@centos7-10 ~]#  docker exec -it 3f ping 172.16.0.43 -c3
PING 172.16.0.43 (172.16.0.43): 56 data bytes
64 bytes from 172.16.0.43: icmp_seq=0 ttl=64 time=0.770 ms
64 bytes from 172.16.0.43: icmp_seq=1 ttl=64 time=0.924 ms
64 bytes from 172.16.0.43: icmp_seq=2 ttl=64 time=0.599 ms
--- 172.16.0.43 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.599/0.764/0.924/0.133 ms

节点1宿主机（centos7-10）网卡enp0s5（10.211.55.10），抓vxlan数据包
- tcpdump ‘udp[39]=1’ -nv -i enp0s5 ，下文详细解释命令；
- 抓到外层是10.211.55.10.35720 > 10.211.55.22.4789:VXLAN协议包，VNI=4098；内层是172.16.0.28 > 172.16.0.43: ICMP echo request；
- 也抓到10.211.55.22.42854 > 10.211.55.10.4789: VXLAN协议包，VNI=4098；内层是172.16.0.43 > 172.16.0.28: ICMP echo reply。

//节点1宿主机网卡enp0s5（10.211.55.10），tcpdump抓vxlan数据包
[root@centos7-10 ~]# tcpdump 'udp[39]=1' -nv -i enp0s5 
tcpdump: listening on enp0s5, link-type EN10MB (Ethernet), capture size 262144 bytes
17:12:37.448751 IP (tos 0x0, ttl 64, id 28517, offset 0, flags [none], proto UDP (17), length 134)10.211.55.10.35720 > 10.211.55.22.4789: VXLAN, flags [I] (0x08), vni 4098
IP (tos 0x0, ttl 64, id 57957, offset 0, flags [DF], proto ICMP (1), length 84)172.16.0.28 > 172.16.0.43: ICMP echo request, id 236, seq 0, length 64
17:12:37.449082 IP (tos 0x0, ttl 64, id 27909, offset 0, flags [none], proto UDP (17), length 134)10.211.55.22.42854 > 10.211.55.10.4789: VXLAN, flags [I] (0x08), vni 4098
IP (tos 0x0, ttl 64, id 22136, offset 0, flags [none], proto ICMP (1), length 84)172.16.0.43 > 172.16.0.28: ICMP echo reply, id 236, seq 0, length 64
17:12:38.456550 IP (tos 0x0, ttl 64, id 29233, offset 0, flags [none], proto UDP (17), length 134)10.211.55.10.35720 > 10.211.55.22.4789: VXLAN, flags [I] (0x08), vni 4098
IP (tos 0x0, ttl 64, id 58863, offset 0, flags [DF], proto ICMP (1), length 84)172.16.0.28 > 172.16.0.43: ICMP echo request, id 236, seq 1, length 64
17:12:38.457139 IP (tos 0x0, ttl 64, id 28823, offset 0, flags [none], proto UDP (17), length 134)10.211.55.22.42854 > 10.211.55.10.4789: VXLAN, flags [I] (0x08), vni 4098
IP (tos 0x0, ttl 64, id 22638, offset 0, flags [none], proto ICMP (1), length 84)172.16.0.43 > 172.16.0.28: ICMP echo reply, id 236, seq 1, length 64
17:12:39.457441 IP (tos 0x0, ttl 64, id 30184, offset 0, flags [none], proto UDP (17), length 134)10.211.55.10.35720 > 10.211.55.22.4789: VXLAN, flags [I] (0x08), vni 4098
IP (tos 0x0, ttl 64, id 59514, offset 0, flags [DF], proto ICMP (1), length 84)172.16.0.28 > 172.16.0.43: ICMP echo request, id 236, seq 2, length 64
17:12:39.457827 IP (tos 0x0, ttl 64, id 29810, offset 0, flags [none], proto UDP (17), length 134)10.211.55.22.42854 > 10.211.55.10.4789: VXLAN, flags [I] (0x08), vni 4098
IP (tos 0x0, ttl 64, id 22700, offset 0, flags [none], proto ICMP (1), length 84)172.16.0.43 > 172.16.0.28: ICMP echo reply, id 236, seq 2, length 64
^C
6 packets captured
6 packets received by filter
0 packets dropped by kernel

5.2 tcpdump抓包命令解析

5.2.1 tcpdump命令说明

tcpdump可以使用pcap-filter进行筛选过滤。

pcap-filter过滤器proto [ expr : size ]用法
- proto 报文中的协议，可以是ether, fddi, tr, wlan, ppp, slip, link, ip, arp, rarp, tcp, udp, icmp, ip6 or radio中任一一种。
- expr 报文中相对protol首部的字节偏移量（字节）
- size 根据expr确定的位置，向后的数据长度

[root@centos7-10 ~]# man pcap-filter
PCAP-FILTER(7)                                                   Miscellaneous Information Manual                                                  PCAP-FILTER(7)NAMEpcap-filter - packet filter syntax
......expr relop exprTrue if the relation holds, where relop is one of >, <, >=, <=, =, !=,  and  expr  is  an  arithmetic  expression  composed  of  integer  constants(expressed  in  standard  C  syntax), the normal binary operators [+, -, *, /, &, |, <<, >>], a length operator, and special packet data accessors.Note that all comparisons are unsigned, so that, for example, 0x80000000 and 0xffffffff are > 0.  To access data inside the packet, use the follow‐ing syntax:proto [ expr : size ]Proto is one of ether, fddi, tr, wlan, ppp, slip, link, ip, arp, rarp, tcp, udp, icmp, ip6 or radio, and indicates the protocol layer for the indexoperation.  (ether, fddi, wlan, tr, ppp, slip and link all refer to the link layer. radio refers to the "radio header" added to  some  802.11  cap‐tures.)  Note that tcp, udp and other upper-layer protocol types only apply to IPv4, not IPv6 (this will be fixed in the future).  The byte offset,relative to the indicated protocol layer, is given by expr.  Size is optional and indicates the number of bytes in the field of interest; it can beeither one, two, or four, and defaults to one.  The length operator, indicated by the keyword len, gives the length of the packet.For  example,  `ether[0]  &  1  != 0' catches all multicast traffic.  The expression `ip[0] & 0xf != 5' catches all IPv4 packets with options.  Theexpression `ip[6:2] & 0x1fff = 0' catches only unfragmented IPv4 datagrams and frag zero of fragmented IPv4 datagrams.  This  check  is  implicitlyapplied to the tcp and udp index operations.  For instance, tcp[0] always means the first byte of the TCP header, and never means the first byte ofan intervening fragment.
......

tcpdump ‘udp[39]=1’ -nv -i enp0s5 解析
- tcpdump 抓包命令
- -n 不把网络地址转换为名字
- -v 输出稍微详细的信息
- -i enp0s5 抓接口enp0s5的包
- ‘udp[39]=1’ 从协议UDP header开始计算，偏移40个字节（启始位置从0开始计算，0-39=40字节），获取对应位置的值=1（对应位置是报文中的协议位置，icmp协议号=1）

5.2.2 Vxlan协议报文解析

解释vxlan中’udp[39]=1’的含义
Vxlan报文格式是在原始报文前封装了Vxlan报文，命令中的“39”是指从OUT UDP header协议报文启始位置0偏移至39字节（偏移量40字节），其中包括：OUT UDP header（8字节）+VXLAN header(8字节）+Inner Ethernet header（14字节）+Inner IP header中Protocol位置（10字节，详见下文“解释IP header报文格式的含义”）=40字节，详见下图红色部分。

在这里插入图片描述

解释IP header报文格式的含义
在Inner IP header报文中，从启始位置0“Version” 偏移10个字节位置是“Protocol”位置，详见下图红色部分。
常见的协议号包括：
1 ICMP
2 IGMP
6 TCP
17 UDP
88 IGRP
89 OSPF

下图中每一行4字节，例如：第一行是0-3字节

5.3 其他抓包例子

5.3.1 抓包示例1

仍然使用上面的环境，继续抓Vxlan网络通讯，要求内部源IP是172.16.0.28或172.16.0.43，具体命令如下：

tcpdump ‘udp[39]=1 and (udp[42:4]=0xac10001c or udp[42:4]=0xac10002b)’ -nv -i enp0s5
- 条件1：udp[39]=1，从UDP Header 0启始偏移39字节，Protocol协议号=1（ICMP）
- 条件2：udp[42:4]=0xac10001c，从UDP Header 启始位置0偏移42字节，数据长度4字节，Source Address=0xac10001c（172.16.0.28，节点1容器的IP）
- 条件3：udp[42:4]=0xac10002b，从UDP Header 启始位置0偏移42字节，Source Address=0xac10002b（172.16.0.43，节点3容器的IP）
- 表达式：条件1 &&（条件2 || 条件3）

IP转16进制可以查询：IP到十六进制转换器
172.16.0.28 --> ac.10.00.1c (0xac10001c)，0x表明16进制整数。
172.16.0.43 --> ac.10.00.2b (0xac10002b)，0x表明16进制整数。

[root@centos7-10 ~]# tcpdump 'udp[39]=1 and (udp[42:4]=0xac10001c or udp[42:4]=0xac10002b)' -nv -i enp0s5
tcpdump: listening on enp0s5, link-type EN10MB (Ethernet), capture size 262144 bytes
09:46:27.895978 IP (tos 0x0, ttl 64, id 48103, offset 0, flags [none], proto UDP (17), length 134)10.211.55.10.35720 > 10.211.55.22.4789: VXLAN, flags [I] (0x08), vni 4098
IP (tos 0x0, ttl 64, id 11611, offset 0, flags [DF], proto ICMP (1), length 84)172.16.0.28 > 172.16.0.43: ICMP echo request, id 266, seq 0, length 64
09:46:27.896365 IP (tos 0x0, ttl 64, id 37189, offset 0, flags [none], proto UDP (17), length 134)10.211.55.22.42854 > 10.211.55.10.4789: VXLAN, flags [I] (0x08), vni 4098
IP (tos 0x0, ttl 64, id 63007, offset 0, flags [none], proto ICMP (1), length 84)172.16.0.43 > 172.16.0.28: ICMP echo reply, id 266, seq 0, length 64
^C
2 packets captured
2 packets received by filter
0 packets dropped by kernel
[root@centos7-10 ~]#

5.3.2 抓包示例2

抓访问宿主机centos7-10（10.211.55.10）ssh服务（22端口）的包（用作示例演示，实际使用中不需要这么麻烦，直接抓TCP 22端口）

tcpdump ‘ip[9]=6 && ip[16:4]=0x0ad3370a && port 22’ -nv -i enp0s5 -c3
- 条件1：ip[9]=6，从IP Header 0启始偏移9字节，Protocol协议号=6（TCP）
- 条件2：ip[16:4]=0x0ad3370a，从IP Header 启始位置0偏移16字节，数据长度4字节，Destination Address=0x0ad3370a（10.211.55.10）
- 条件3：port 22，端口是22（ssh）
- 表达式：条件1 && 条件2 && 条件3

10.211.55.10 --> 0a.d3.37.0a (0x0ad3370a)，0x表明16进制整数。

[root@centos7-10 ~]# tcpdump 'ip[9]=6 && ip[16:4]=0x0ad3370a && port 22' -nv -i enp0s5 -c3
tcpdump: listening on enp0s5, link-type EN10MB (Ethernet), capture size 262144 bytes
10:40:52.437795 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52)10.211.55.2.52199 > 10.211.55.10.ssh: Flags [.], cksum 0x9d4c (correct), ack 1654340393, win 2045, options [nop,nop,TS val 1112225161 ecr 179414603], length 0
10:40:52.438241 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52)10.211.55.2.52199 > 10.211.55.10.ssh: Flags [.], cksum 0x9bee (correct), ack 353, win 2042, options [nop,nop,TS val 1112225161 ecr 179414604], length 0
10:40:52.438506 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 52)10.211.55.2.52199 > 10.211.55.10.ssh: Flags [.], cksum 0x9a9e (correct), ack 689, win 2042, options [nop,nop,TS val 1112225161 ecr 179414604], length 0
3 packets captured
3 packets received by filter
0 packets dropped by kernel
[root@centos7-10 ~]#