January 2023 – 一江春水向东流

KUBERNETES

为什么 kubernetes 环境要求开启 bridge-nf-call-iptables

为什么 kubernetes 环境要求开启 bridge-nf-call-iptables ? 背景 Kubernetes 环境中，很多时候都要求节点内核参数开启 bridge-nf-call-iptables:

sysctl -w net.bridge.bridge-nf-call-iptables=1

1 2	sysctl -w net.bridge.bridge-nf-call-iptables=1

参考官方文档 Network Plugin Requirements 如果不开启或中途因某些操作导致参数被关闭了，就可能造成一些奇奇怪怪的网络问题，排查起来非常麻烦。为什么要开启呢？本文就来跟你详细掰扯下。基于网桥的容器网络 Kubernetes 集群网络有很多种实现，有很大一部分都用到了 Linux 网桥: 每个 Pod 的网卡都是 veth 设备，veth pair 的另一端连上宿主机上的网桥。由于网桥是虚拟的二层设备，同节点的 Pod 之间通信直接走二层转发，跨节点通信才会经过宿主机 eth0。 Service 同节点通信问题不管是 iptables 还是 ipvs 转发模式，Kubernetes 中访问 Service 都会进行 DNAT，将原本访问 ClusterIP:Port 的数据包 DNAT 成 Service 的某个 Endpoint Read more…

By sean, 2 years01/12/2023 ago

NETWORK

calico网络问题排查

问题：今天同事安装了一套k8s环境, 结果发现calico网络互相访问不通，同台机器上的pod之间的calico ip都ping不通分析解决： calico支持三种网络模式，可通过修过calico.yaml进行配置: overlay之ipip overlay之vxlan underlay之BGP overlay — ipip traffic flow overlay — vxlan traffic flow underlay — BGP traffic flow 本环境用的是CrossSubnet且都在同网段，用的是BGP 下面是node信息：

[root@localhost ~]# kubectl get no -owide
NAME                    STATUS   ROLES                                      AGE   VERSION    INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                     KERNEL-VERSION            CONTAINER-RUNTIME
yks1                    Ready    control-plane,ingress,master,node,system   22h   v1.20.15   10.16.245.46   <none>        NFSChina Server 4.0 (G196)   4.19.113-14.nfs4.x86_64   docker://20.10.8
yks2                    Ready    control-plane,ingress,master,node,system   22h   v1.20.15   10.16.245.47   <none>        NFSChina Server 4.0 (G196)   4.19.113-14.nfs4.x86_64   docker://20.10.8
yks3                    Ready    ingress,node,system                        22h   v1.20.15   10.16.245.65   <none>        NFSChina Server 4.0 (G196)   4.19.113-14.nfs4.x86_64   docker://20.10.8

[root@localhost ~]# kubectl get no -owide

NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME

yks1 Ready control-plane,ingress,master,node,system 22h v1.20.15 10.16.245.46 <none> NFSChina Server 4.0 (G196) 4.19.113-14.nfs4.x86_64 docker://20.10.8

yks2 Ready control-plane,ingress,master,node,system 22h v1.20.15 10.16.245.47 <none> NFSChina Server 4.0 (G196) 4.19.113-14.nfs4.x86_64 docker://20.10.8

yks3 Ready ingress,node,system 22h v1.20.15 10.16.245.65 <none> NFSChina Server 4.0 (G196) 4.19.113-14.nfs4.x86_64 docker://20.10.8

查看yks2上的路由：

[root@yks3 ~]# ip r
default via 10.16.245.1 dev ens3 proto dhcp metric 100 
10.16.245.0/24 dev ens3 proto kernel scope link src 10.16.245.65 metric 100 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
172.23.27.0/24 via 10.16.245.47 dev ens3 proto bird 
172.23.39.0/24 via 10.16.245.34 dev ens3 proto bird 
blackhole 172.23.119.0/24 proto bird 
172.23.119.19 dev cali112980e2900 scope link 
172.23.119.20 dev cali7ad105f6052 scope link 
172.23.119.21 dev cali850bd215908 scope link 
172.23.119.22 dev cali014e534f978 scope link 
172.23.119.23 dev cali90505cacde3 scope link 
172.23.119.24 dev cali1e38a050268 scope link 
172.23.119.25 dev cali55015378d77 scope link 
172.23.119.26 dev caliad0c2e2531c scope link 
172.23.255.0/24 via 10.16.245.46 dev ens3 proto bird

[root@yks3 ~]# ip r

default via 10.16.245.1 dev ens3 proto dhcp metric 100

10.16.245.0/24 dev ens3 proto kernel scope link src 10.16.245.65 metric 100

172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown

172.23.27.0/24 via 10.16.245.47 dev ens3 proto bird

172.23.39.0/24 via 10.16.245.34 dev ens3 proto bird

blackhole 172.23.119.0/24 proto bird

172.23.119.19 dev cali112980e2900 scope link

172.23.119.20 dev cali7ad105f6052 scope link

172.23.119.21 dev cali850bd215908 scope link

172.23.119.22 dev cali014e534f978 scope link

172.23.119.23 dev cali90505cacde3 scope link

172.23.119.24 dev cali1e38a050268 scope link

172.23.119.25 dev cali55015378d77 scope link

172.23.119.26 dev caliad0c2e2531c scope link

172.23.255.0/24 via 10.16.245.46 dev ens3 proto bird

可以看到， 172.23.27.0/24 via 10.16.245.47 dev ens3 proto bird 这条路由，172.23.27.0/24这个地址是直接到yks2主机上的

yks             nfs-server-fc7d676c9-8dkq5                 1/1     Running   0          21h     172.23.119.4    yks3 
kube-system     coredns-6d7c6648db-4stmp                   1/1 Running       0          23h     172.23.27.2     yks2

1 2	yks nfs-server-fc7d676c9-8dkq5 1/1 Running 0 21h 172.23.119.4 yks3 kube-system coredns-6d7c6648db-4stmp 1/1 Running 0 23h 172.23.27.2 yks2

在yks2上ping coredns:

[root@yks1 ~]# ping 172.23.27.2
PING 172.23.27.2 (172.23.27.2) 56(84) bytes of data.

1 2	[root@yks1 ~]# ping 172.23.27.2 PING 172.23.27.2 (172.23.27.2) 56(84) bytes of data.

上主机yks2上抓包：首先抓宿主机网卡：

[root@yks2 ~]# tcpdump -i ens3 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes

15:42:16.286242 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 20, length 64
15:42:17.310198 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 21, length 64
15:42:18.334192 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 22, length 64
15:42:19.358197 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 23, length 64
15:42:20.382236 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 24, length 64
15:42:21.410206 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 25, length 64
15:42:22.430279 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 26, length 64
15:42:23.454333 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 27, length 64
15:42:24.478138 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 28, length 64
15:42:25.502182 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 29, length 64
15:42:26.526196 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 30, length 64
15:42:27.550195 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 31, length 64
^C
12 packets captured
13 packets received by filter
0 packets dropped by kernels

[root@yks2 ~]# tcpdump -i ens3 icmp

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes

15:42:16.286242 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 20, length 64

15:42:17.310198 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 21, length 64

15:42:18.334192 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 22, length 64

15:42:19.358197 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 23, length 64

15:42:20.382236 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 24, length 64

15:42:21.410206 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 25, length 64

15:42:22.430279 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 26, length 64

15:42:23.454333 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 27, length 64

15:42:24.478138 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 28, length 64

15:42:25.502182 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 29, length 64

15:42:26.526196 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 30, length 64

15:42:27.550195 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 31, length 64

12 packets captured

13 packets received by filter

0 packets dropped by kernels

可以接收到包看下yks2的路由：

[root@yks2 ~]# ip r
default via 10.16.245.1 dev ens3 proto dhcp metric 100 
10.16.245.0/24 dev ens3 proto kernel scope link src 10.16.245.47 metric 100 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 
172.18.0.0/16 dev br-34de7dda2079 proto kernel scope link src 172.18.0.1 
blackhole 172.23.27.0/24 proto bird 
172.23.27.1 dev calid894dc87e68 scope link 
172.23.27.2 dev calif945d263abc scope link 
172.23.27.3 dev califc5d5492c2d scope link 
172.23.27.5 dev cali420874a7614 scope link 
172.23.27.7 dev calif2ef5efbec8 scope link 
172.23.27.8 dev calif508601809c scope link 
172.23.27.9 dev calida55188c614 scope link 
172.23.27.10 dev cali3e6ad545e92 scope link 
172.23.27.11 dev cali2c5cdc71fac scope link 
172.23.27.12 dev cali1be0e00985c scope link 
172.23.27.13 dev calif00b7a22fc0 scope link 
172.23.27.14 dev cali56eb38c7099 scope link 
172.23.39.0/24 via 10.16.245.34 dev ens3 proto bird 
172.23.119.0/24 via 10.16.245.65 dev ens3 proto bird 
172.23.255.0/24 via 10.16.245.46 dev ens3 proto bird

[root@yks2 ~]# ip r

default via 10.16.245.1 dev ens3 proto dhcp metric 100

10.16.245.0/24 dev ens3 proto kernel scope link src 10.16.245.47 metric 100

172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1

172.18.0.0/16 dev br-34de7dda2079 proto kernel scope link src 172.18.0.1

blackhole 172.23.27.0/24 proto bird

172.23.27.1 dev calid894dc87e68 scope link

172.23.27.2 dev calif945d263abc scope link

172.23.27.3 dev califc5d5492c2d scope link

172.23.27.5 dev cali420874a7614 scope link

172.23.27.7 dev calif2ef5efbec8 scope link

172.23.27.8 dev calif508601809c scope link

172.23.27.9 dev calida55188c614 scope link

172.23.27.10 dev cali3e6ad545e92 scope link

172.23.27.11 dev cali2c5cdc71fac scope link

172.23.27.12 dev cali1be0e00985c scope link

172.23.27.13 dev calif00b7a22fc0 scope link

172.23.27.14 dev cali56eb38c7099 scope link

172.23.39.0/24 via 10.16.245.34 dev ens3 proto bird

172.23.119.0/24 via 10.16.245.65 dev ens3 proto bird

172.23.255.0/24 via 10.16.245.46 dev ens3 proto bird