March 2020 – 一江春水向东流

KUBERNETES

使用 Sealos 在 3 分钟内快速部署一个生产级别的 Kubernetes 高可用集群

前段时间，我们在「使用 Kind 在 5 分钟内快速部署一个 Kubernetes 高可用集群」一文中介绍了如何使用 Kind 这个开箱即可快速部署 Kubernetes 高可用集群的神器，相信不少同学用上这个神器后大大的降低了 Kubernetes 集群的部署难度和提高了 Kubernetes 集群的部署速度。不过有一点比较遗憾的是 Kind 当前仅仅支持在本地快速构建一个开发或者测试环境，目前暂时还是不支持在生产环境中部署 Kubernetes 高可用集群的。今天，我们就要给大家介绍另一款可以支持在生产环境中部署 Kubernetes 高可用集群的利器 Sealos。什么是 Sealos ？ Sealos 是一个 Go 语言开发的简单干净且轻量的 Kubernetes 集群部署工具，Sealos 能很好的支持在生产环境中部署高可用的 Kubernetes 集群。 Sealos 架构图 Sealos 特性与优势支持离线安装，工具与部署资源包分离，方便不同版本间快速升级。证书有效期默认延期至 99 年。工具使用非常简单。支持使用自定义配置文件，可灵活完成集群环境定制。使用内核进行本地负载，稳定性极高，故障排查也极其简单。 Sealos 设计原则和工作原理 1. Read more…

By sean, 5 years03/23/2020 ago

KUBERNETES

kubernetes生产环境下的iptables安全配置

控制节点（主节点）所需要的端口如下：协议端口服务 TCP 6443 k8s API server TCP 2379-2380 etcd server client API TCP 10250 Kubelet API TCP 10251 kube-scheduler TCP 10252 kube-controller-manager TCP 10255 Read-only Kubelet API UDP 8472 flannel TCP 9099,9100 calico TCP 179 bgp TCP 111 rpcbind 使用 NFS 作为持久化存储 IPIP Calico 需要允许 IPIP 协议 Read more…

By sean, 5 years03/21/2020 ago

KUBERNETES

NetworkPolicy网络策略

资源隔离设计本案例使用NetworkPolicy来进行资源隔离要实现的目的: 用户isv应用不能访问k8s集群应用 k8s集群应用可以访问isv应用 isv应用可以访问外网本案例的架构图: 创建isv-demon namespace

ns=isv-demo
kubectl create ns $ns
kubectl config set-context \
  --current \
  --namespace $ns
  
 kubectl label ns $ns name=$ns
 # 如果kube-system没有label, 则打上
 kubectl lable ns kube-system name=kube-system

ns=isv-demo

kubectl create ns $ns

kubectl config set-context \

--current \

--namespace $ns

kubectl label ns $ns name=$ns

# 如果kube-system没有label, 则打上

kubectl lable ns kube-system name=kube-system

定义资源为了不跟宿主机端口冲突, web的service端口定义为8800

apiVersion: apps/v1
kind: Deployment
metadata:
  name: db
  labels:
    app: db
spec:
  replicas: 1
  selector:
    matchLabels:
      app: db
  template:
    metadata:
      labels:
        app: db
    spec:
      containers:
      - name: couchdb
        image: "couchdb"
        ports:
        - containerPort: 5984
        env:
        - name: COUCHDB_USER
          value: admin
        - name: COUCHDB_PASSWORD
          value: password

---
apiVersion: v1
kind: Service
metadata:
  name: db
spec:
  selector:
    app: db
  ports:
    - name: db
      port: 15984
      targetPort: 5984
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  labels:
    app: api
spec:
  replicas: 1
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
      - name: nodebrady
        image: mabenoit/nodebrady
        ports:
        - containerPort: 3000
---
apiVersion: v1
kind: Service
metadata:
  name: api
spec:
  selector:
    app: api
  ports:
    - name: api
      port: 8080
      targetPort: 3000
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  labels:
    app: web
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector:
    app: web
  ports:
    - name: web
      port: 8800
      targetPort: 80
  type: ClusterIP

100

101

102

103

104

105

106

107

108

109

110

apiVersion: apps/v1

kind: Deployment

metadata:

labels:

app: db

spec:

replicas: 1

selector:

matchLabels:

app: db

template:

metadata:

labels:

app: db

spec:

containers:

- name: couchdb

image: "couchdb"

ports:

- containerPort: 5984

env:

- name: COUCHDB_USER

value: admin

- name: COUCHDB_PASSWORD

value: password

---

apiVersion: v1

kind: Service

metadata:

spec:

selector:

app: db

ports:

- name: db

port: 15984

targetPort: 5984

type: ClusterIP

---

apiVersion: apps/v1

kind: Deployment

metadata:

labels:

app: api

spec:

replicas: 1

selector:

matchLabels:

app: api

template:

metadata:

labels:

app: api

spec:

containers:

- name: nodebrady

image: mabenoit/nodebrady

ports:

- containerPort: 3000

---

apiVersion: v1

kind: Service

metadata:

spec:

selector:

app: api

ports:

- name: api

port: 8080

targetPort: 3000

type: ClusterIP

---

apiVersion: apps/v1

kind: Deployment

metadata:

labels:

app: web

spec:

replicas: 1

selector:

matchLabels:

app: web

template:

metadata:

labels:

app: web

spec:

containers:

- name: nginx

image: nginx

ports:

- containerPort: 80

---

apiVersion: v1

kind: Service

metadata:

spec:

selector:

app: web

ports:

- name: web

port: 8800

targetPort: 80

type: ClusterIP

创建WEB, API and DB Pods/Services

kubectl apply \
 -f db-api-web-deployments.yaml

1 2	kubectl apply \ -f db-api-web-deployments.yaml

测试:

web=$(kubectl get svc web -o jsonpath='{.spec.clusterIP}'):8800)
echo $web
# curl web.isv-demo:8800
curl $(kubectl get svc web -o jsonpath='{.spec.clusterIP}'):8800
kubectl run curl-$RANDOM --image=radial/busyboxplus:curl --rm -it --generator=run-pod/v1 -n isv-demo
[ root@curl-3899:/ ]$ curl www.baidu.com
<!DOCTYPE html>
<!--STATUS OK--><html> <head><meta http-equiv=con
[ root@curl-3899:/ ]$ curl http://db:15984
{"couchdb":"Welcome","version":"3.0.0","git_sha":"03a77db6c","uuid":"05616a1b6f1eccbed4f24d3e6d5526d2","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
[ root@curl-3899:/ ]$ # exit

web=$(kubectl get svc web -o jsonpath='{.spec.clusterIP}'):8800)

echo $web

# curl web.isv-demo:8800

curl $(kubectl get svc web -o jsonpath='{.spec.clusterIP}'):8800

kubectl run curl-$RANDOM --image=radial/busyboxplus:curl --rm -it --generator=run-pod/v1 -n isv-demo

[ root@curl-3899:/ ]$ curl www.baidu.com

<!DOCTYPE html>

<html> <head><meta http-equiv=con

[ root@curl-3899:/ ]$ curl http://db:15984

{"couchdb":"Welcome","version":"3.0.0","git_sha":"03a77db6c","uuid":"05616a1b6f1eccbed4f24d3e6d5526d2","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}

[ root@curl-3899:/ ]$ # exit

定义networkpolicy: deny-all

# deny-all-netpol.yaml

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all
  namespace: isv-demo
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

# deny-all-netpol.yaml

apiVersion: networking.k8s.io/v1

kind: NetworkPolicy

metadata:

namespace: isv-demo

spec:

podSelector: {}

policyTypes:

- Ingress

- Egress

定义进栈和出栈访问:

# isv-demo-in-out.yaml
# 实现: 
# 1. namespace: isv-demo里的所有pod可以互相访问
# 2. namespace: isv-demo里的所有pod可以访问namespace: kube-system里的coredns的53端口
# 3. namespace: kube-system里的所有pod可以访问namespace: isv-demo里的所有pod
# 4. 访问外网限制: namespace: isv-demo里的所有pod可以访问除172.20.0.0/16(物理机)和172.23.0.0/16(calico)网段外的所有网段的80和443端口

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: isv-demo-in-out
  namespace: isv-demo
spec:
  podSelector:
    matchLabels:
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: isv-demo
  - from:
    - namespaceSelector:
        matchLabels:
          name: kube-system
  # 这块定义了这三个端口全开放, 可根据需要进行限制
  - from: []
    ports:
      - protocol: TCP
        port: 8800
      - protocol: TCP
        port: 80
      - protocol: TCP
        port: 443
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          name: isv-demo
  # to kube-dns
  - to:
    - namespaceSelector:
        matchLabels:
          name: kube-system
      podSelector:
        matchLabels:
          k8s-app: kube-dns
    ports:
      - port: 53
        protocol: UDP
      - port: 53
        protocol: TCP
  # to outside website
  - to:
    - ipBlock:
        cidr: 0.0.0.0/0
        except:
          - 172.20.0.0/16
          - 172.23.0.0/16
    ports:
      - protocol: TCP
        port: 80
      - protocol: TCP
        port: 443


# 注: developer-center为开发者中心namespace, 线上为c87e2267-1001-4c70-bb2a-ab41f3b81aa3

# isv-demo-in-out.yaml

# 实现:

# 1. namespace: isv-demo里的所有pod可以互相访问

# 2. namespace: isv-demo里的所有pod可以访问namespace: kube-system里的coredns的53端口

# 3. namespace: kube-system里的所有pod可以访问namespace: isv-demo里的所有pod

# 4. 访问外网限制: namespace: isv-demo里的所有pod可以访问除172.20.0.0/16(物理机)和172.23.0.0/16(calico)网段外的所有网段的80和443端口

kind: NetworkPolicy

apiVersion: networking.k8s.io/v1

metadata:

namespace: isv-demo

spec:

podSelector:

matchLabels:

ingress:

- from:

- namespaceSelector:

matchLabels:

- from:

- namespaceSelector:

matchLabels:

# 这块定义了这三个端口全开放, 可根据需要进行限制

- from: []

ports:

- protocol: TCP

port: 8800

- protocol: TCP

port: 80

- protocol: TCP

port: 443

egress:

- to:

- namespaceSelector:

matchLabels:

# to kube-dns

- to:

- namespaceSelector:

matchLabels:

podSelector:

matchLabels:

k8s-app: kube-dns

ports:

- port: 53

protocol: UDP

- port: 53

protocol: TCP

# to outside website

- to:

- ipBlock:

cidr: 0.0.0.0/0

except:

- 172.20.0.0/16

- 172.23.0.0/16

ports:

- protocol: TCP

port: 80

- protocol: TCP

port: 443

# 注: developer-center为开发者中心namespace, 线上为c87e2267-1001-4c70-bb2a-ab41f3b81aa3

创建NetworkPolicy

# Apply the first NetworkPolicy: deny all Ingress/Egress
kubectl apply \
  -f deny-all-netpol.yaml
 
# Apply the NetworkPolicy definition related to isv-demo-in-out
kubectl apply \
  -f isv-demo-in-out.yaml

# Apply the first NetworkPolicy: deny all Ingress/Egress

kubectl apply \

-f deny-all-netpol.yaml

# Apply the NetworkPolicy definition related to isv-demo-in-out

kubectl apply \

-f isv-demo-in-out.yaml

测试:

1. namespace: isv-demo里的所有pod可以互相访问
# 2. namespace: isv-demo里的所有pod可以访问namespace: kube-system里的coredns的53端口
# 3. namespace: kube-system里的所有pod可以访问namespace: isv-demo里的所有pod
# 4. 访问外网限制: namespace: isv-demo里的所有pod可以访问除172.20.0.0/16(物理机)和172.23.0.0/16(calico)网段外的所有网段的80和443端口

web=$(kubectl get svc web -o jsonpath='{.spec.clusterIP}'):8800
#======在namespace:isv-demo里测试=======
kubectl run curl-$RANDOM --image=radial/busyboxplus:curl --rm -it --generator=run-pod/v1 -n isv-demo
# --> 访问isv-demo的web服务
[ root@curl-24497:/ ]$ curl web:8800
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
# --> 访问isv-demo的db服务
[ root@curl-24497:/ ]$ curl http://db:15984
{"couchdb":"Welcome","version":"3.0.0","git_sha":"03a77db6c","uuid":"05616a1b6f1eccbed4f24d3e6d5526d2","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
# --> 访问外网baidu
[ root@curl-24497:/ ]$ curl www.baidu.com
<!DOCTYPE html>
<!--STATUS OK--><html>
# --> ping外网baidu
[ root@curl-24497:/ ]$ ping www.baidu.com -c3 -W2
PING www.baidu.com (61.135.169.125): 56 data bytes

--- www.baidu.com ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss
# --> 访问开发者中心的web地址
[ root@curl-24497:/ ]$ curl 172.20.58.132 --connect-timeout 5
curl: (28) Connection timed out after 5001 milliseconds
# --> 访问kube-system的某Pod的calico的IP
[ root@curl-24497:/ ]$ ping 172.23.166.156 -c3 -W2
PING 172.23.166.156 (172.23.166.156): 56 data bytes

--- 172.23.166.156 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss
# --> 访问isv-demo里的pod的calico ip
[ root@curl-24497:/ ]$ ping 172.23.166.151
PING 172.23.166.151 (172.23.166.151): 56 data bytes
64 bytes from 172.23.166.151: seq=0 ttl=63 time=0.243 ms
[ root@curl-24497:/ ]$ exit


#======在namespace:kube-system里测试=======
kubectl run curl-$RANDOM --image=radial/busyboxplus:curl --rm -it --generator=run-pod/v1 -n kube-system
[ root@curl-23248:/ ]$ curl web.isv-demo:8800
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
[ root@curl-23248:/ ]$ curl web.isv-demo:8800
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
[ root@curl-23248:/ ]$ curl http://db.isv-demo:15984
{"couchdb":"Welcome","version":"3.0.0","git_sha":"03a77db6c","uuid":"05616a1b6f1eccbed4f24d3e6d5526d2","features":["access-ready","partitioned","pluggable-storage-engines","reshard","scheduler"],"vendor":{"name":"The Apache Software Foundation"}}
[ root@curl-23248:/ ]$ # exit

1. namespace: isv-demo里的所有pod可以互相访问

# 2. namespace: isv-demo里的所有pod可以访问namespace: kube-system里的coredns的53端口

# 3. namespace: kube-system里的所有pod可以访问namespace: isv-demo里的所有pod

# 4. 访问外网限制: namespace: isv-demo里的所有pod可以访问除172.20.0.0/16(物理机)和172.23.0.0/16(calico)网段外的所有网段的80和443端口

web=$(kubectl get svc web -o jsonpath='{.spec.clusterIP}'):8800

#======在namespace:isv-demo里测试=======

kubectl run curl-$RANDOM --image=radial/busyboxplus:curl --rm -it --generator=run-pod/v1 -n isv-demo

# --> 访问isv-demo的web服务

[ root@curl-24497:/ ]$ curl web:8800

<!DOCTYPE html>

<html>

<head>

<title>Welcome to nginx!</title>

# --> 访问isv-demo的db服务

[ root@curl-24497:/ ]$ curl http://db:15984

# --> 访问外网baidu

[ root@curl-24497:/ ]$ curl www.baidu.com

<!DOCTYPE html>

<html>

# --> ping外网baidu

[ root@curl-24497:/ ]$ ping www.baidu.com -c3 -W2

PING www.baidu.com (61.135.169.125): 56 data bytes

--- www.baidu.com ping statistics ---

3 packets transmitted, 0 packets received, 100% packet loss

# --> 访问开发者中心的web地址

[ root@curl-24497:/ ]$ curl 172.20.58.132 --connect-timeout 5

curl: (28) Connection timed out after 5001 milliseconds

# --> 访问kube-system的某Pod的calico的IP

[ root@curl-24497:/ ]$ ping 172.23.166.156 -c3 -W2

PING 172.23.166.156 (172.23.166.156): 56 data bytes

--- 172.23.166.156 ping statistics ---

3 packets transmitted, 0 packets received, 100% packet loss

# --> 访问isv-demo里的pod的calico ip

[ root@curl-24497:/ ]$ ping 172.23.166.151

PING 172.23.166.151 (172.23.166.151): 56 data bytes

64 bytes from 172.23.166.151: seq=0 ttl=63 time=0.243 ms

[ root@curl-24497:/ ]$ exit

#======在namespace:kube-system里测试=======

kubectl run curl-$RANDOM --image=radial/busyboxplus:curl --rm -it --generator=run-pod/v1 -n kube-system

[ root@curl-23248:/ ]$ curl web.isv-demo:8800

<!DOCTYPE html>

<html>

<head>

<title>Welcome to nginx!</title>

[ root@curl-23248:/ ]$ curl web.isv-demo:8800

<!DOCTYPE html>

<html>

<head>

<title>Welcome to nginx!</title>

[ root@curl-23248:/ ]$ curl http://db.isv-demo:15984

[ root@curl-23248:/ ]$ # exit

总结: 首先一定要给namespace打label, 因为NetworkPolicy是根据lable来match的参考: https://alwaysupalwayson.blogspot.com/2019/09/kubernetes-network-policies-how-to.html https://github.com/mathieu-benoit/k8s-netpol 和 https://ahmet.im/blog/kubernetes-network-policy/ https://github.com/ahmetb/kubernetes-network-policy-recipes Read more…

By sean, 5 years03/20/2020 ago