NETWORK
calico网络问题排查
问题：今天同事安装了一套k8s环境, 结果发现calico网络互相访问不通，同台机器上的pod之间的calico ip都ping不通 分析解决： calico支持三种网络模式，可通过修过calico.yaml进行配置: overlay之ipip overlay之vxlan underlay之BGP overlay — ipip traffic flow overlay — vxlan traffic flow underlay — BGP traffic flow 本环境用的是CrossSubnet且都在同网段，用的是BGP 下面是node信息：
|
1 2 3 4 5 |
[root@localhost ~]# kubectl get no -owide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME yks1 Ready control-plane,ingress,master,node,system 22h v1.20.15 10.16.245.46 <none> NFSChina Server 4.0 (G196) 4.19.113-14.nfs4.x86_64 docker://20.10.8 yks2 Ready control-plane,ingress,master,node,system 22h v1.20.15 10.16.245.47 <none> NFSChina Server 4.0 (G196) 4.19.113-14.nfs4.x86_64 docker://20.10.8 yks3 Ready ingress,node,system 22h v1.20.15 10.16.245.65 <none> NFSChina Server 4.0 (G196) 4.19.113-14.nfs4.x86_64 docker://20.10.8 |
查看yks2上的路由：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
[root@yks3 ~]# ip r default via 10.16.245.1 dev ens3 proto dhcp metric 100 10.16.245.0/24 dev ens3 proto kernel scope link src 10.16.245.65 metric 100 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 172.23.27.0/24 via 10.16.245.47 dev ens3 proto bird 172.23.39.0/24 via 10.16.245.34 dev ens3 proto bird blackhole 172.23.119.0/24 proto bird 172.23.119.19 dev cali112980e2900 scope link 172.23.119.20 dev cali7ad105f6052 scope link 172.23.119.21 dev cali850bd215908 scope link 172.23.119.22 dev cali014e534f978 scope link 172.23.119.23 dev cali90505cacde3 scope link 172.23.119.24 dev cali1e38a050268 scope link 172.23.119.25 dev cali55015378d77 scope link 172.23.119.26 dev caliad0c2e2531c scope link 172.23.255.0/24 via 10.16.245.46 dev ens3 proto bird |
可以看到， 172.23.27.0/24 via 10.16.245.47 dev ens3 proto bird 这条路由，172.23.27.0/24这个地址是直接到yks2主机上的
|
1 2 |
yks nfs-server-fc7d676c9-8dkq5 1/1 Running 0 21h 172.23.119.4 yks3 kube-system coredns-6d7c6648db-4stmp 1/1 Running 0 23h 172.23.27.2 yks2 |
在yks2上ping coredns:
|
1 2 |
[root@yks1 ~]# ping 172.23.27.2 PING 172.23.27.2 (172.23.27.2) 56(84) bytes of data. |
上主机yks2上抓包： 首先抓宿主机网卡：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
[root@yks2 ~]# tcpdump -i ens3 icmp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes 15:42:16.286242 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 20, length 64 15:42:17.310198 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 21, length 64 15:42:18.334192 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 22, length 64 15:42:19.358197 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 23, length 64 15:42:20.382236 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 24, length 64 15:42:21.410206 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 25, length 64 15:42:22.430279 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 26, length 64 15:42:23.454333 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 27, length 64 15:42:24.478138 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 28, length 64 15:42:25.502182 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 29, length 64 15:42:26.526196 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 30, length 64 15:42:27.550195 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 31, length 64 ^C 12 packets captured 13 packets received by filter 0 packets dropped by kernels |
可以接收到包 看下yks2的路由：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
[root@yks2 ~]# ip r default via 10.16.245.1 dev ens3 proto dhcp metric 100 10.16.245.0/24 dev ens3 proto kernel scope link src 10.16.245.47 metric 100 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 172.18.0.0/16 dev br-34de7dda2079 proto kernel scope link src 172.18.0.1 blackhole 172.23.27.0/24 proto bird 172.23.27.1 dev calid894dc87e68 scope link 172.23.27.2 dev calif945d263abc scope link 172.23.27.3 dev califc5d5492c2d scope link 172.23.27.5 dev cali420874a7614 scope link 172.23.27.7 dev calif2ef5efbec8 scope link 172.23.27.8 dev calif508601809c scope link 172.23.27.9 dev calida55188c614 scope link 172.23.27.10 dev cali3e6ad545e92 scope link 172.23.27.11 dev cali2c5cdc71fac scope link 172.23.27.12 dev cali1be0e00985c scope link 172.23.27.13 dev calif00b7a22fc0 scope link 172.23.27.14 dev cali56eb38c7099 scope link 172.23.39.0/24 via 10.16.245.34 dev ens3 proto bird 172.23.119.0/24 via 10.16.245.65 dev ens3 proto bird 172.23.255.0/24 via 10.16.245.46 dev ens3 proto bird |