January 2019 – 一江春水向东流

未分类

Kubernetes & Docker 容器网络终极之战

与 Docker 默认的网络模型不同，Kubernetes 形成了一套自己的网络模型，该网络模型更加适应传统的网络模式，应用能够平滑的从非容器环境迁移到 Kubernetes 环境中。自从 Docker 容器出现，容器的网络通信一直是众人关注的焦点，而容器的网络方案又可以分为两大部分：单主机的容器间通信；跨主机的容器间通信。一、单主机 Docker 网络通信利用 Net Namespace 可以为 Docker 容器创建隔离的网络环境，容器具有完全独立的网络栈，与宿主机隔离。也可以使 Docker 容器共享主机或者其他容器的网络命名空间。我们在使用docker run创建 Docker 容器时，可以使用–network=选项指定容器的网络模式，Docker 有以下 4 种网络模式： host 模式，使用–network=host指定，不支持多主机； bridge 模式，使用–network=bridge指定，默认设置，不支持多主机； container 模式，使用–network=container:NAME_or_ID指定，即joiner 容器，不支持多主机； none 模式，使用–network=none指定，不支持多主机。 1.1、host 模式连接到 host 网络的容器共享 Docker host 的网络栈，容器的网络配置与 host 完全一样。我们先查看一下主机的网络。

[root@datanode03 ~]# ifconfig
docker0: flags=4099<span class="hljs-tag">&lt;<span class="hljs-name">UP,BROADCAST,MULTICAST</span>&gt;</span>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        ether 02:42:44:8d:48:70  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp1s0: flags=4163<span class="hljs-tag">&lt;<span class="hljs-name">UP,BROADCAST,RUNNING,MULTICAST</span>&gt;</span>  mtu 1500
        inet 192.168.1.203  netmask 255.255.255.0  broadcast 192.168.1.255
        inet6 fe80::2e0:70ff:fe92:4779  prefixlen 64  scopeid 0x20<span class="hljs-tag">&lt;<span class="hljs-name">link</span>&gt;</span>
        ether 00:e0:70:92:47:79  txqueuelen 1000  (Ethernet)
        RX packets 46093  bytes 66816291 (63.7 MiB)
        RX errors 0  dropped 1  overruns 0  frame 0
        TX packets 24071  bytes 1814769 (1.7 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<span class="hljs-tag">&lt;<span class="hljs-name">UP,LOOPBACK,RUNNING</span>&gt;</span>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<span class="hljs-tag">&lt;<span class="hljs-name">host</span>&gt;</span>
        loop  txqueuelen 0  (Local Loopback)
        RX packets 170  bytes 107720 (105.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 170  bytes 107720 (105.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@datanode03 ~]# ifconfig

docker0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500

inet 172.17.0.1 netmask 255.255.0.0 broadcast 172.17.255.255

ether 02:42:44:8d:48:70 txqueuelen 0 (Ethernet)

RX packets 0 bytes 0 (0.0 B)

RX errors 0 dropped 0 overruns 0 frame 0

TX packets 0 bytes 0 (0.0 B)

TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

enp1s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500

inet 192.168.1.203 netmask 255.255.255.0 broadcast 192.168.1.255

inet6 fe80::2e0:70ff:fe92:4779 prefixlen 64 scopeid 0x20<link>

ether 00:e0:70:92:47:79 txqueuelen 1000 (Ethernet)

RX packets 46093 bytes 66816291 (63.7 MiB)

RX errors 0 dropped 1 overruns 0 frame 0

TX packets 24071 bytes 1814769 (1.7 MiB)

TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536

inet 127.0.0.1 netmask 255.0.0.0

inet6 ::1 prefixlen 128 scopeid 0x10<host>

loop txqueuelen 0 (Local Loopback)

RX packets 170 bytes 107720 (105.1 KiB)

RX errors 0 dropped 0 overruns 0 frame 0

TX packets 170 bytes 107720 (105.1 KiB)

TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

未分类

FreeIPA删除记录时报 record not found的错误

DNS服务器之前加过一条DNS Forward Zones, 后来删除时出现了报错, 报DNS record not found的错误. 下面是报错时的截图: 原因: dns里删除了此记录, 但是ldap里还存在. 解决: 手动删除ldap里的记录首先, 查询所有记录:

ldapsearch -x -D 'cn=Directory Manager' -W > ldap_records.txt

1	ldapsearch -x -D 'cn=Directory Manager' -W > ldap_records.txt

经过查看发现, 有问题的记录是:

# yonyou.com. + 4da8c981-ca9311e7-bb5cd39b-89b4247b, dns, yonyouiuap.com
dn: idnsname=yonyou.com.+nsuniqueid=4da8c981-ca9311e7-bb5cd39b-89b4247b,cn=dns
 ,dc=yonyouiuap,dc=com
idnsName: yonyou.com.
idnsForwardPolicy: first
idnsForwarders: 192.168.8.57
idnsForwarders: 10.10.6.11
idnsZoneActive: TRUE
objectClass: top
objectClass: idnsforwardzone

# yonyou.com. + 4da8c981-ca9311e7-bb5cd39b-89b4247b, dns, yonyouiuap.com

dn: idnsname=yonyou.com.+nsuniqueid=4da8c981-ca9311e7-bb5cd39b-89b4247b,cn=dns

,dc=yonyouiuap,dc=com

idnsName: yonyou.com.

idnsForwardPolicy: first

idnsForwarders: 192.168.8.57

idnsForwarders: 10.10.6.11

idnsZoneActive: TRUE

objectClass: top

objectClass: idnsforwardzone

手动删除记录:

ldapmodify -x -D 'cn=Directory Manager' -W
Enter LDAP Password: 
dn: idnsname=yonyou.com.+nsuniqueid=4da8c981-ca9311e7-bb5cd39b-89b4247b,cn=dns,dc=yonyouiuap,dc=com
changetype: delete

deleting entry "idnsname=yonyou.com.+nsuniqueid=4da8c981-ca9311e7-bb5cd39b-89b4247b,cn=dns,dc=yonyouiuap,dc=com"

ldapmodify -x -D 'cn=Directory Manager' -W

Enter LDAP Password:

dn: idnsname=yonyou.com.+nsuniqueid=4da8c981-ca9311e7-bb5cd39b-89b4247b,cn=dns,dc=yonyouiuap,dc=com

changetype: delete

deleting entry "idnsname=yonyou.com.+nsuniqueid=4da8c981-ca9311e7-bb5cd39b-89b4247b,cn=dns,dc=yonyouiuap,dc=com"

再次刷新页面, 发现记录已被删除

By sean, 6 years ago

DIARY

统一运维

继续推动运维标准化和流程化标准化和流程化的落地，往往还会伴随着对已存在的系统部署方式进行迁移至标准化的操作。这其实要求我们在执行标准化和流程化以后的所有运维操作完全按照运维的标准和流程进行，对于执行以前的运维工作要进行迁移，目的是要实现所有系统和运维的标准、流程化。对于迁移，一般比较好的方案是：首先保留已存在非标准化系统，同时搭建标准化环境，进行已存在系统的部署并进行测试，测试无误后，与已存在非标准化系统并行运行，同时提供对外服务一段时间后，在评估，最后将已存在非标准化系统进行下线，以达到系统平滑得由非标准化迁移至标准化的目的。建设CMDB 目前线上环境都跑在阿里云，其它环境基本都是用友云cloudstack, 虽然开发者中心有资源池各主机的信息，但是目前服务器的管理还是依靠手工去登录处理，信息维护也是依靠用户接入或删除主机到资源池，具体信息维护起来麻烦且没有一个直观、统一去监控和配置管理的地方要做的：建立用友CMDB系统，结合阿里云SDK、用友cloudstack SDK，开发者中心资源池信息等，建立一个CMDB平台，第一步，能实现所有主机信息的实时显示和统计，如主机配置、资源使用情况、上面跑的应用。通过一个业务，如www.diwork.com, 能查出访问链路图，如首先经过哪个SLB, 然后到哪台nginx, 再到哪个容器和哪个RDS等。第二步，能根据业务的使用情况，对重要业务能实现根据资源池的负载来自动创建和销毁主机，如果负载高，则自动创建按量付费的主机，如果负载低，则可自动销毁主机以节省开支。第三步，能根据不同的业务制定不同的执行策略，如遇到CPU,硬盘报警、业务报错不可用，访问超时等，可被自动调用处理。监控监控是整个运维环节，乃至整个产品生命周期中最重要的一环，事前及时预警发现故障，事后提供翔实的数据用于追查定位问题，分析业务指标等。目前的不足之处：之前用Nagios, ganglia, open-falcon, zabbix等作监控报警，但是只能监控到主机基本指标，在开发者中心大量使用容器自动分发部署后，如何监控容器及k8s及各业务指标成为了重要需求，目前主要监控报警是用prometheus系列来做的，另外还有一些自写的自动化监控采集脚本去周期性执行并触发报警，目前监控报警能做到主机基本指标、域名和api调用可用性和访问效率、容器及业务应用的可用性监控，发送到微信，友空间及邮件。目前的问题是：报警泛滥，没有达到很好的关联性分析，合并，降噪没有很好的去做自动处理，更多依赖于手工处理现实情况出现的很多故障都并没有那么简单，大部分情况下，这种单一唯独的监控都没有办法很好的覆盖业务的监控需求，发现问题后不能快速定位，只能去一个个手动去查日志分析下一步要做的：制定统一的监控标准，推行统一的最佳实践，提供统一强大便捷的metrics lib库支撑。这样才能更容易的推进自动化进程，以及更高的监控指标覆盖度 1.坚持业务指标采集是代码的一部分原则不动摇，提高指标覆盖率 2.监控方式和指标要标准化，工具支撑系统化原则1，没有人比模块的研发人员，更清楚其工作机制，更关心其运行状态，模块自身的可运维性就是代码功能的重要组成部分。每次业务逻辑部分的代码变更，都应该伴随着监控指标采集的相应更改。有了原则1，还远远不够，没有好的工具和体系支撑，提高指标覆盖率就是一句空话。在运维层面，应该制定统一的监控标准，推行统一的最佳实践，提供统一强大便捷的metrics lib库支撑。这样才能更容易的推进自动化进程，以及更高的监控指标覆盖度。谈谈监控标准化，标准如何定义？ 1.每个业务的每个接口，都要可被监控 2.每个接口的监控指标，必须至少包含： cps latency-50th/75th/95th/99th… error_rate error_count Read more…

By sean, 6 years01/17/2019 ago

KUBERNETES

二进制部署Kubernetes v1.11.x(1.12.x) HA可选

#!/usr/bin/env bash
set -e
# 声明集群成员信息
declare -A MasterArray otherMaster NodeArray
MasterArray=(['k8s-m1']=172.20.23.167 ['k8s-m2']=172.20.23.168 ['k8s-m3']=172.20.23.169)
otherMaster=(['k8s-m2']=172.20.23.168 ['k8s-m3']=172.20.23.169)
NodeArray=(['k8s-n1']=172.20.23.173)

export         VIP=172.20.23.4
export INGRESS_VIP=172.20.23.11
[ "${#MasterArray[@]}" -eq 1 ]  && export VIP=${MasterArray[@]} || export API_PORT=8443
export KUBE_APISERVER=https://${VIP}:${API_PORT:-6443}

#声明需要安装的的k8s版本
export KUBE_VERSION=v1.12.3

# 网卡名
export interface=ens160

export K8S_DIR=/etc/kubernetes
export PKI_DIR=${K8S_DIR}/pki
export ETCD_SSL=/etc/etcd/ssl
export MANIFESTS_DIR=/etc/kubernetes/manifests/
# cni
export CNI_URL="https://github.com/containernetworking/plugins/releases/download"
export CNI_VERSION=v0.7.4
# cfssl
export CFSSL_URL="https://pkg.cfssl.org/R1.2"
# etcd
export ETCD_version=v3.3.10

#首先在k8s-m1通过git获取部署要用到的二进制配置文件和yml
# curl  https://storage.googleapis.com/kubernetes-release/release/${KUBE_VERSION}/kubernetes-server-linux-amd64.tar.gz > kubernetes-server-linux-amd64.tar.gz
# tar -zxvf kubernetes-server-linux-amd64.tar.gz  --strip-components=3 -C /usr/local/bin kubernetes/server/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy}

#分发master相关组件到其他master上
#(这边不想master跑pod的话就不复制kubelet和kube-proxy过去,以及后面master节点上的kubelet的相关操作)
# for NODE in "${!otherMaster[@]}"; do
#     echo "--- $NODE ${otherMaster[$NODE]} ---"
#     scp /usr/local/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy} ${otherMaster[$NODE]}:/usr/local/bin/ 
# done
#分发node的kubernetes二进制文件
# for NODE in "${!NodeArray[@]}"; do
#     echo "--- $NODE ${NodeArray[$NODE]} ---"
#     scp /usr/local/bin/kube{let,-proxy} ${NodeArray[$NODE]}:/usr/local/bin/ 
# done

#在k81-m1下载Kubernetes CNI 二进制文件并分发
#分发cni文件到otherMaster
# mkdir -p /opt/cni/bin
# wget  "${CNI_URL}/${CNI_VERSION}/cni-plugins-amd64-${CNI_VERSION}.tgz" 
# tar -zxf cni-plugins-amd64-${CNI_VERSION}.tgz -C /opt/cni/bin

# 分发cni文件
# for NODE in "${!otherMaster[@]}"; do
#     echo "--- $NODE ${otherMaster[$NODE]} ---"
#     ssh ${otherMaster[$NODE]} 'mkdir -p /opt/cni/bin'
#     scp /opt/cni/bin/* ${otherMaster[$NODE]}:/opt/cni/bin/
# done

# #分发cni文件到node
# for NODE in "${!NodeArray[@]}"; do
#     echo "--- $NODE ${NodeArray[$NODE]} ---"
#     ssh ${NodeArray[$NODE]} 'mkdir -p /opt/cni/bin'
#     scp /opt/cni/bin/* ${NodeArray[$NODE]}:/opt/cni/bin/
# done

#在k8s-m1需要安裝CFSSL工具,这将会用來建立 TLS Certificates。
# wget "${CFSSL_URL}/cfssl_linux-amd64" -O /usr/local/bin/cfssl
# wget "${CFSSL_URL}/cfssljson_linux-amd64" -O /usr/local/bin/cfssljson
# chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson


#建立集群CA keys 与Certificates

#Etcd CA
#首先在k8s-m1建立/etc/etcd/ssl文件夹
cd ~/k8s-manual-files/pki
mkdir -p ${ETCD_SSL}

#从CSR json文件ca-config.json与etcd-ca-csr.json生成etcd的CA keys与Certificate：
cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare ${ETCD_SSL}/etcd-ca

#生成Etcd证书
cfssl gencert \
  -ca=${ETCD_SSL}/etcd-ca.pem \
  -ca-key=${ETCD_SSL}/etcd-ca-key.pem \
  -config=ca-config.json \
  -hostname=127.0.0.1,$(xargs -n1<<<${MasterArray[@]} | sort  | paste -d, -s -) \
  -profile=kubernetes \
  etcd-csr.json | cfssljson -bare ${ETCD_SSL}/etcd
#-hostname值为所有masters节点的IP,如果后续master节点扩容此处可以多预留ip到证书里

#完成后删除不必要文件,确认/etc/etcd/ssl有以下文件
rm -rf ${ETCD_SSL}/*.csr
ls $ETCD_SSL
#etcd-ca-key.pem  etcd-ca.pem  etcd-key.pem  etcd.pem

#在k8s-m1上复制相关文件至其他Etcd节点,这边etcd跑在所有master节点上,所以etcd的证书复制到其他mster节点：
for NODE in "${!otherMaster[@]}"; do
    echo "--- $NODE ${otherMaster[$NODE]} ---"
    ssh ${otherMaster[$NODE]} "mkdir -p ${ETCD_SSL}"
    for FILE in etcd-ca-key.pem  etcd-ca.pem  etcd-key.pem  etcd.pem; do
      scp ${ETCD_SSL}/${FILE} ${otherMaster[$NODE]}:${ETCD_SSL}/${FILE}
    done
done

#ETCD二进制文件 
#etcd所有标准版本可以在下面url查看
#https://github.com/etcd-io/etcd/releases
#在k8s-m1上下载etcd的二进制文件,单台的话建议使用v3.1.9因为有bug
#[ "${#MasterArray[@]}" -eq 1 ] && ETCD_version=v3.3.10 || :
cd ~/k8s-manual-files

#在k8s-m1上分发etcd的二进制文件到其他master上
for NODE in "${!otherMaster[@]}"; do
    echo "--- $NODE ${otherMaster[$NODE]} ---"
    scp /usr/local/bin/etcd* ${otherMaster[$NODE]}:/usr/local/bin/
done

#在k8s-m1上配置etcd配置文件并分发相关文件
#配置文件存放在/etc/etcd/etcd.config.yml里
#注入基础变量
cd ~/k8s-manual-files/master/
etcd_servers=$( xargs -n1<<<${MasterArray[@]} | sort | sed 's#^#https://#;s#$#:2379#;$s#\n##' | paste -d, -s - )
etcd_initial_cluster=$( for i in ${!MasterArray[@]};do  echo $i=https://${MasterArray[$i]}:2380; done | sort | paste -d, -s - )
sed -ri "/initial-cluster:/s#'.+'#'${etcd_initial_cluster}'#" etc/etcd/config.yml

#分发systemd和配置文件
for NODE in "${!MasterArray[@]}"; do
    echo "--- $NODE ${MasterArray[$NODE]} ---"
    ssh ${MasterArray[$NODE]} "mkdir -p $MANIFESTS_DIR /etc/etcd /var/lib/etcd"
    scp systemd/etcd.service ${MasterArray[$NODE]}:/usr/lib/systemd/system/etcd.service
    scp etc/etcd/config.yml ${MasterArray[$NODE]}:/etc/etcd/etcd.config.yml
    ssh ${MasterArray[$NODE]} "sed -i "s/{HOSTNAME}/$NODE/g" /etc/etcd/etcd.config.yml"
    ssh ${MasterArray[$NODE]} "sed -i "s/{PUBLIC_IP}/${MasterArray[$NODE]}/g" /etc/etcd/etcd.config.yml"
    ssh ${MasterArray[$NODE]} 'systemctl daemon-reload'
done

#在k8s-m1上启动所有etcd
#etcd 进程首次启动时会等待其它节点的 etcd 加入集群，命令 systemctl start etcd 会卡住一段时间，为正常现象
#可以全部启动后后面的etcdctl命令查看状态确认正常否
for NODE in "${!MasterArray[@]}"; do
    echo "--- $NODE ${MasterArray[$NODE]} ---"
    ssh ${MasterArray[$NODE]} 'systemctl enable --now etcd' &
done
wait

#然后输出到终端了的时候多按几下回车直到等光标回到终端状态
#k8s-m1上执行下面命令验证 ETCD 集群状态,下面第二个是使用3的api去查询集群的键值
etcdctl \
   --cert-file /etc/etcd/ssl/etcd.pem \
   --key-file /etc/etcd/ssl/etcd-key.pem  \
   --ca-file /etc/etcd/ssl/etcd-ca.pem \
   --endpoints $etcd_servers cluster-health
#使用3的api去查询集群的键值
ETCDCTL_API=3 \
    etcdctl   \
    --cert=/etc/etcd/ssl/etcd.pem    \
    --key=/etc/etcd/ssl/etcd-key.pem     \
    --cacert /etc/etcd/ssl/etcd-ca.pem    \
    --endpoints $etcd_servers get / --prefix --keys-only

#Kubernetes CA
#为确保安全，kubernetes 系统各组件需要使用 x509 证书对通信进行加密和认证。
#在k8s-m1建立pki文件夹,并生成根CA凭证用于签署其它的k8s证书
mkdir -p ${PKI_DIR}
cd ~/k8s-manual-files/pki
cfssl gencert -initca ca-csr.json | cfssljson -bare ${PKI_DIR}/ca
ls ${PKI_DIR}/ca*.pem
#/etc/kubernetes/pki/ca-key.pem  /etc/kubernetes/pki/ca.pem

#API Server Certificate
#此凭证将被用于API Server和Kubelet Client通信使用,使用下面命令生成kube-apiserver凭证
cfssl gencert \
  -ca=${PKI_DIR}/ca.pem \
  -ca-key=${PKI_DIR}/ca-key.pem \
  -config=ca-config.json \
  -hostname=10.96.0.1,${VIP},127.0.0.1,kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.default.svc.cluster.local,$(xargs -n1<<<${MasterArray[@]} | sort  | paste -d, -s -) \
  -profile=kubernetes \
  apiserver-csr.json | cfssljson -bare ${PKI_DIR}/apiserver

ls ${PKI_DIR}/apiserver*.pem
#/etc/kubernetes/pki/apiserver-key.pem  /etc/kubernetes/pki/apiserver.pem
#这边-hostname的10.96.0.1是Cluster IP的Kubernetes端点(默认占用第一个ip,用于给集群里的pod要调用Kubernetes的API server);
#kubernetes.default为Kubernets DN。
#如果使用域名可以加上域名
#如果后续master节点扩容此处可以多预留ip到证书里

#Front Proxy Certificate
#此凭证将被用于Authenticating Proxy的功能上,而该功能主要是提供API Aggregation的认证。使用下面命令生成CA:
cfssl gencert \
  -initca front-proxy-ca-csr.json | cfssljson -bare ${PKI_DIR}/front-proxy-ca

ls ${PKI_DIR}/front-proxy-ca*.pem
#/etc/kubernetes/pki/front-proxy-ca-key.pem  /etc/kubernetes/pki/front-proxy-ca.pem

#接着生成front-proxy-client凭证(hosts的warning忽略即可)：
cfssl gencert \
  -ca=${PKI_DIR}/front-proxy-ca.pem \
  -ca-key=${PKI_DIR}/front-proxy-ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  front-proxy-client-csr.json | cfssljson -bare ${PKI_DIR}/front-proxy-client

ls ${PKI_DIR}/front-proxy-client*.pem
#front-proxy-client-key.pem  front-proxy-client.pem

#Controller Manager Certificate
#凭证会建立system:kube-controller-manager的使用者(凭证 CN),并被绑定在RBAC 
#Cluster Role中的system:kube-controller-manager来让Controller Manager 
#元件能够存取需要的API object。
#这边通过以下命令生成 Controller Manager 凭证(hosts的warning忽略即可)
cfssl gencert \
  -ca=${PKI_DIR}/ca.pem \
  -ca-key=${PKI_DIR}/ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  manager-csr.json | cfssljson -bare ${PKI_DIR}/controller-manager

ls ${PKI_DIR}/controller-manager*.pem
#controller-manager-key.pem  controller-manager.pem

#接着利用kubectl生成Controller Manager的kubeconfig文件

# controller-manager set cluster

kubectl config set-cluster kubernetes \
    --certificate-authority=${PKI_DIR}/ca.pem \
    --embed-certs=true \
    --server=${KUBE_APISERVER} \
    --kubeconfig=${K8S_DIR}/controller-manager.kubeconfig

# controller-manager set credentials

kubectl config set-credentials system:kube-controller-manager \
    --client-certificate=${PKI_DIR}/controller-manager.pem \
    --client-key=${PKI_DIR}/controller-manager-key.pem \
    --embed-certs=true \
    --kubeconfig=${K8S_DIR}/controller-manager.kubeconfig

# controller-manager set context

kubectl config set-context system:kube-controller-manager@kubernetes \
    --cluster=kubernetes \
    --user=system:kube-controller-manager \
    --kubeconfig=${K8S_DIR}/controller-manager.kubeconfig

# controller-manager set default context

kubectl config use-context system:kube-controller-manager@kubernetes \
    --kubeconfig=${K8S_DIR}/controller-manager.kubeconfig

#Scheduler Certificate
#凭证会建立system:kube-scheduler的使用者(凭证 CN),并被绑定在 RBAC Cluster Role 
#中的system:kube-scheduler来让 Scheduler 元件能够存取需要的 API object。
#这边通过以下命令生成 Scheduler 凭证(hosts的warning忽略即可)：
cfssl gencert \
  -ca=${PKI_DIR}/ca.pem \
  -ca-key=${PKI_DIR}/ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  scheduler-csr.json | cfssljson -bare ${PKI_DIR}/scheduler

ls ${PKI_DIR}/scheduler*.pem
#/etc/kubernetes/pki/scheduler-key.pem  /etc/kubernetes/pki/scheduler.pem

#接着利用kubectl生成Scheduler的kubeconfig文件
# scheduler set cluster

kubectl config set-cluster kubernetes \
    --certificate-authority=${PKI_DIR}/ca.pem \
    --embed-certs=true \
    --server=${KUBE_APISERVER} \
    --kubeconfig=${K8S_DIR}/scheduler.kubeconfig

# scheduler set credentials

kubectl config set-credentials system:kube-scheduler \
    --client-certificate=${PKI_DIR}/scheduler.pem \
    --client-key=${PKI_DIR}/scheduler-key.pem \
    --embed-certs=true \
    --kubeconfig=${K8S_DIR}/scheduler.kubeconfig

# scheduler set context

kubectl config set-context system:kube-scheduler@kubernetes \
    --cluster=kubernetes \
    --user=system:kube-scheduler \
    --kubeconfig=${K8S_DIR}/scheduler.kubeconfig

# scheduler use default context

kubectl config use-context system:kube-scheduler@kubernetes \
    --kubeconfig=${K8S_DIR}/scheduler.kubeconfig

#Admin Certificate
#Admin 被用来绑定 RBAC Cluster Role 中 cluster-admin,当想要(最常见的就是使用kubectl)操作
#所有 Kubernetes 集群功能时,就必须利用这边生成的 kubeconfig 文件。
cfssl gencert \
  -ca=${PKI_DIR}/ca.pem \
  -ca-key=${PKI_DIR}/ca-key.pem \
  -config=ca-config.json \
  -profile=kubernetes \
  admin-csr.json | cfssljson -bare ${PKI_DIR}/admin

ls ${PKI_DIR}/admin*.pem
#/etc/kubernetes/pki/admin-key.pem  /etc/kubernetes/pki/admin.pem

#kubectl 默认从 ~/.kube/config 文件读取 kube-apiserver 地址、证书、用户名等信息，
#如果没有配置，执行 kubectl 命令时可能会出错(因为默认连接8080匿名端口)
#接着利用kubectl生成 Admin 的kubeconfig文件
# admin set cluster
kubectl config set-cluster kubernetes \
    --certificate-authority=${PKI_DIR}/ca.pem \
    --embed-certs=true \
    --server=${KUBE_APISERVER} \
    --kubeconfig=${K8S_DIR}/admin.kubeconfig

# admin set credentials
kubectl config set-credentials kubernetes-admin \
    --client-certificate=${PKI_DIR}/admin.pem \
    --client-key=${PKI_DIR}/admin-key.pem \
    --embed-certs=true \
    --kubeconfig=${K8S_DIR}/admin.kubeconfig

# admin set context
kubectl config set-context kubernetes-admin@kubernetes \
    --cluster=kubernetes \
    --user=kubernetes-admin \
    --kubeconfig=${K8S_DIR}/admin.kubeconfig

# admin set default context
kubectl config use-context kubernetes-admin@kubernetes \
    --kubeconfig=${K8S_DIR}/admin.kubeconfig


# Master Kubelet Certificate
# 这边使用 Node authorizer 来让节点的 kubelet 能够存取如 services、endpoints 等 API,
# 而使用 Node authorizer 需定义system:nodesCLusterRole(凭证的 Organization),
# 并且包含system:node:<nodeName>的使用者名称(凭证的 Common Name)。
# 首先在k8s-m1节点生成所有 master 节点的 kubelet 凭证,这边通过下面命令來生成：
for NODE in "${!MasterArray[@]}"; do
    echo "--- $NODE ---"
    \cp kubelet-csr.json kubelet-$NODE-csr.json;
    sed -i "s/\$NODE/$NODE/g" kubelet-$NODE-csr.json;
    cfssl gencert \
      -ca=${PKI_DIR}/ca.pem \
      -ca-key=${PKI_DIR}/ca-key.pem \
      -config=ca-config.json \
      -hostname=$NODE \
      -profile=kubernetes \
      kubelet-$NODE-csr.json | cfssljson -bare ${PKI_DIR}/kubelet-$NODE;
    rm -f kubelet-$NODE-csr.json
  done

ls ${PKI_DIR}/kubelet*.pem
#/etc/kubernetes/pki/kubelet-k8s-m1-key.pem  /etc/kubernetes/pki/kubelet-k8s-m2.pem
#/etc/kubernetes/pki/kubelet-k8s-m1.pem      /etc/kubernetes/pki/kubelet-k8s-m3-key.pem
#/etc/kubernetes/pki/kubelet-k8s-m2-key.pem  /etc/kubernetes/pki/kubelet-k8s-m3.pem
#这边需要依据节点修改-hostname与$NODE。

#完成后复制kubelet凭证至所有master节点：
for NODE in "${!MasterArray[@]}"; do
    echo "--- $NODE ${MasterArray[$NODE]} ---"
    ssh ${MasterArray[$NODE]} "mkdir -p ${PKI_DIR}"
    scp ${PKI_DIR}/ca.pem ${MasterArray[$NODE]}:${PKI_DIR}/ca.pem
    scp ${PKI_DIR}/kubelet-$NODE-key.pem ${MasterArray[$NODE]}:${PKI_DIR}/kubelet-key.pem
    scp ${PKI_DIR}/kubelet-$NODE.pem ${MasterArray[$NODE]}:${PKI_DIR}/kubelet.pem
    rm -f ${PKI_DIR}/kubelet-$NODE-key.pem ${PKI_DIR}/kubelet-$NODE.pem
done

#接着在k8s-m1执行以下命令给所有master产生kubelet的kubeconfig文件
for NODE in "${!MasterArray[@]}"; do
    echo "--- $NODE ---"
    ssh ${MasterArray[$NODE]} "cd ${PKI_DIR} && \
      kubectl config set-cluster kubernetes \
        --certificate-authority=${PKI_DIR}/ca.pem \
        --embed-certs=true \
        --server=${KUBE_APISERVER} \
        --kubeconfig=${K8S_DIR}/kubelet.kubeconfig && \
      kubectl config set-credentials system:node:${NODE} \
        --client-certificate=${PKI_DIR}/kubelet.pem \
        --client-key=${PKI_DIR}/kubelet-key.pem \
        --embed-certs=true \
        --kubeconfig=${K8S_DIR}/kubelet.kubeconfig && \
      kubectl config set-context system:node:${NODE}@kubernetes \
        --cluster=kubernetes \
        --user=system:node:${NODE} \
        --kubeconfig=${K8S_DIR}/kubelet.kubeconfig && \
      kubectl config use-context system:node:${NODE}@kubernetes \
        --kubeconfig=${K8S_DIR}/kubelet.kubeconfig"
done


# Service Account Key
# Kubernetes Controller Manager 利用 Key pair 生成与签署 Service Account 的 tokens,
# 而这边不能通过 CA 做认证,而是建立一组公私钥来让 API Server 与 Controller Manager 使用：
# 在k8s-m1执行以下指令
openssl genrsa -out ${PKI_DIR}/sa.key 2048
openssl rsa -in ${PKI_DIR}/sa.key -pubout -out ${PKI_DIR}/sa.pub
ls ${PKI_DIR}/sa.*
#/etc/kubernetes/pki/sa.key  /etc/kubernetes/pki/sa.pub

# 删除不必要文件
# 所有资讯准备完成后,就可以将一些不必要文件删除
rm -f ${PKI_DIR}/*.csr \
    ${PKI_DIR}/scheduler*.pem \
    ${PKI_DIR}/controller-manager*.pem \
    ${PKI_DIR}/admin*.pem \
    ${PKI_DIR}/kubelet*.pem

# 复制文件至其他节点
# 复制凭证文件至其他master节点
for NODE in "${!otherMaster[@]}"; do
    echo "--- $NODE ${otherMaster[$NODE]}---"
    for FILE in $(ls ${PKI_DIR}); do
      scp ${PKI_DIR}/${FILE} ${otherMaster[$NODE]}:${PKI_DIR}/${FILE}
    done
  done

# 复制Kubernetes config文件至其他master节点：
for NODE in "${!otherMaster[@]}"; do
    echo "--- $NODE ${otherMaster[$NODE]}---"
    for FILE in admin.kubeconfig controller-manager.kubeconfig scheduler.kubeconfig; do
      scp ${K8S_DIR}/${FILE} ${otherMaster[$NODE]}:${K8S_DIR}/${FILE}
    done
  done

# Kubernetes Masters

# HA(haproxy+keepalived) 单台master就不要用HA了
for NODE in "${!MasterArray[@]}"; do
    echo "--- $NODE ${MasterArray[$NODE]} ---"
    ssh ${MasterArray[$NODE]} 'yum install haproxy keepalived -y' &
done
wait


# 在k8s-m1节点下把相关配置文件配置后再分发
cd ~/k8s-manual-files/master/etc

# 修改haproxy.cfg配置文件
sed -i '$r '<(paste <( seq -f'  server k8s-api-%g'  ${#MasterArray[@]} ) <( xargs -n1<<<${MasterArray[@]} | sort | sed 's#$#:6443  check#')) haproxy/haproxy.cfg

# 修改keepalived(网卡和VIP写进去,使用下面命令)

sed -ri "s#\{\{ VIP \}\}#${VIP}#" keepalived/*
sed -ri "s#\{\{ interface \}\}#${interface}#" keepalived/keepalived.conf 
sed -i '/unicast_peer/r '<(xargs -n1<<<${MasterArray[@]} | sort | sed 's#^#\t#') keepalived/keepalived.conf

# 分发文件
for NODE in "${!MasterArray[@]}"; do
    echo "--- $NODE ${MasterArray[$NODE]} ---"
    scp -r haproxy/ ${MasterArray[$NODE]}:/etc
    scp -r keepalived/ ${MasterArray[$NODE]}:/etc
    ssh ${MasterArray[$NODE]} 'systemctl enable --now haproxy keepalived'
done


# ping下vip看看能通否,先等待大概四五秒等keepalived和haproxy起来
ping -c1 $VIP

# 如果vip没起来就是keepalived没起来就每个节点上去restart下keepalived或者确认下配置文件
# /etc/keepalived/keepalived.conf里网卡名和ip是否注入成功
for NODE in "${!MasterArray[@]}"; do
    echo "--- $NODE ${MasterArray[$NODE]} ---"
    ssh ${MasterArray[$NODE]}  'systemctl restart haproxy keepalived'
done


# Master组件
# 在k8s-m1节点下把相关配置文件配置后再分发
cd ~/k8s-manual-files/master/
etcd_servers=$( xargs -n1<<<${MasterArray[@]} | sort | sed 's#^#https://#;s#$#:2379#;$s#\n##' | paste -d, -s - )

# 注入VIP和etcd_servers
sed -ri '/--advertise-address/s#=.+#='"$VIP"' \\#' systemd/kube-apiserver.service
sed -ri '/--etcd-servers/s#=.+#='"$etcd_servers"' \\#' systemd/kube-apiserver.service

# 修改encryption.yml
ENCRYPT_SECRET=$( head -c 32 /dev/urandom | base64 )
sed -ri "/secret:/s#(: ).+#\1${ENCRYPT_SECRET}#" encryption/config.yml

# 分发文件(不想master跑pod的话就不复制kubelet的配置文件)
for NODE in "${!MasterArray[@]}"; do
    echo "--- $NODE ${MasterArray[$NODE]} ---"
    ssh ${MasterArray[$NODE]} "mkdir -p $MANIFESTS_DIR /etc/systemd/system/kubelet.service.d /var/lib/kubelet /var/log/kubernetes"
    scp systemd/kube-*.service ${MasterArray[$NODE]}:/usr/lib/systemd/system/

    scp encryption/config.yml ${MasterArray[$NODE]}:/etc/kubernetes/encryption.yml
    scp audit/policy.yml ${MasterArray[$NODE]}:/etc/kubernetes/audit-policy.yml

    scp systemd/kubelet.service ${MasterArray[$NODE]}:/lib/systemd/system/kubelet.service
    scp systemd/10-kubelet.conf ${MasterArray[$NODE]}:/etc/systemd/system/kubelet.service.d/10-kubelet.conf
    scp etc/kubelet/kubelet-conf.yml ${MasterArray[$NODE]}:/etc/kubernetes/kubelet-conf.yml
done


#在k8s-m1上给所有master机器启动kubelet 服务并设置kubectl补全脚本:
for NODE in "${!MasterArray[@]}"; do
    echo "--- $NODE ${MasterArray[$NODE]} ---"
    ssh ${MasterArray[$NODE]} 'systemctl enable --now kubelet kube-apiserver kube-controller-manager kube-scheduler;
    \cp /etc/kubernetes/admin.kubeconfig ~/.kube/config;
    kubectl completion bash > /etc/bash_completion.d/kubectl'
done


# 建立TLS Bootstrapping RBAC 与Secret
# 后面kubectl命令只需要在任何一台master执行就行了
# 首先在k8s-m1建立一个变数来产生BOOTSTRAP_TOKEN,并建立bootstrap-kubelet.conf的Kubernetes config文件
export TOKEN_ID=$(openssl rand 3 -hex)
export TOKEN_SECRET=$(openssl rand 8 -hex)
export BOOTSTRAP_TOKEN=${TOKEN_ID}.${TOKEN_SECRET}

# bootstrap set cluster
kubectl config set-cluster kubernetes \
    --certificate-authority=${PKI_DIR}/ca.pem \
    --embed-certs=true \
    --server=${KUBE_APISERVER} \
    --kubeconfig=${K8S_DIR}/bootstrap-kubelet.kubeconfig

# bootstrap set credentials
kubectl config set-credentials tls-bootstrap-token-user \
    --token=${BOOTSTRAP_TOKEN} \
    --kubeconfig=${K8S_DIR}/bootstrap-kubelet.kubeconfig

# bootstrap set context
kubectl config set-context tls-bootstrap-token-user@kubernetes \
    --cluster=kubernetes \
    --user=tls-bootstrap-token-user \
    --kubeconfig=${K8S_DIR}/bootstrap-kubelet.kubeconfig

# bootstrap use default context
kubectl config use-context tls-bootstrap-token-user@kubernetes \
    --kubeconfig=${K8S_DIR}/bootstrap-kubelet.kubeconfig

#接着在k8s-m1建立TLS bootstrap secret来提供自动签证使用：
cd ~/k8s-manual-files/master

# 注入变量

sed -ri "s#\{TOKEN_ID\}#${TOKEN_ID}#g" resources/bootstrap-token-Secret.yml
sed -ri "/token-id/s#\S+\$#'&'#" resources/bootstrap-token-Secret.yml
sed -ri "s#\{TOKEN_SECRET\}#${TOKEN_SECRET}#g" resources/bootstrap-token-Secret.yml
kubectl create -f resources/bootstrap-token-Secret.yml
# 下面是输出
#secret "bootstrap-token-65a3a9" created

#在k8s-m1建立 TLS Bootstrap Autoapprove RBAC来自动处理 CSR
kubectl apply -f resources/kubelet-bootstrap-rbac.yml
# 下面是输出
#clusterrolebinding.rbac.authorization.k8s.io "kubelet-bootstrap" created
#clusterrolebinding.rbac.authorization.k8s.io "node-autoapprove-bootstrap" created
#clusterrolebinding.rbac.authorization.k8s.io "node-autoapprove-certificate-rotation" created

#为了方便管理集群,因此需要通过 kubectl logs 来查看,但由于 API 权限,故需要建立一个 RBAC Role 来获取存取权限,这边在k8s-m1节点执行下面命令创建
kubectl apply -f resources/apiserver-to-kubelet-rbac.yml
# 下面是输出
#clusterrole.rbac.authorization.k8s.io "system:kube-apiserver-to-kubelet" configured
#clusterrolebinding.rbac.authorization.k8s.io "system:kube-apiserver" configured

#设定master节点加上污点Taint不让(没有声明容忍该污点的)pod跑在master节点上：
kubectl taint nodes node-role.kubernetes.io/master="":NoSchedule --all
# 下面是输出
# node "k8s-m1" tainted
# node "k8s-m2" tainted
# node "k8s-m3" tainted


# Kubernetes Nodes
# 本部分将说明如何建立与设定Kubernetes Node 角色,Node 是主要执行容器实例(Pod)的工作节点。
# 在开始部署前,先在k8-m1将需要用到的文件复制到所有node节点上
cd ${PKI_DIR}
for NODE in "${!NodeArray[@]}"; do
    echo "--- $NODE ${NodeArray[$NODE]} ---"
    ssh ${NodeArray[$NODE]} "mkdir -p ${PKI_DIR} ${ETCD_SSL}"
    # Etcd
    for FILE in etcd-ca.pem etcd.pem etcd-key.pem; do
      scp ${ETCD_SSL}/${FILE} ${NodeArray[$NODE]}:${ETCD_SSL}/${FILE}
    done
    # Kubernetes
    for FILE in pki/ca.pem pki/ca-key.pem pki/front-proxy-ca.pem bootstrap-kubelet.kubeconfig ; do
      scp ${K8S_DIR}/${FILE} ${NodeArray[$NODE]}:${K8S_DIR}/${FILE}
    done
done

# 部署与设定
# 在k8s-m1节点分发kubelet.service相关文件到每台node上去管理kubelet：
cd ~/k8s-manual-files/
for NODE in "${!NodeArray[@]}"; do
    echo "--- $NODE ${NodeArray[$NODE]} ---"
    ssh ${NodeArray[$NODE]} "mkdir -p /var/lib/kubelet /var/log/kubernetes /etc/systemd/system/kubelet.service.d $MANIFESTS_DIR"
    scp node/systemd/kubelet.service ${NodeArray[$NODE]}:/lib/systemd/system/kubelet.service
    scp node/systemd/10-kubelet.conf ${NodeArray[$NODE]}:/etc/systemd/system/kubelet.service.d/10-kubelet.conf
    scp node/etc/kubelet/kubelet-conf.yml ${NodeArray[$NODE]}:/etc/kubernetes/kubelet-conf.yml
done

# 最后在k8s-m1上去启动每个node节点的kubelet 服务:
for NODE in "${!NodeArray[@]}"; do
    echo "--- $NODE ${NodeArray[$NODE]} ---"
    ssh ${NodeArray[$NODE]} 'systemctl enable --now kubelet.service'
done

# Kubernetes Core Addons部署
# daemonSet方式部署
cd ~/k8s-manual-files
# 注入变量
sed -ri "/server:/s#(: ).+#\1${KUBE_APISERVER}#" addons/kube-proxy/kube-proxy.yml
sed -ri "/image:.+kube-proxy/s#:[^:]+\$#:$KUBE_VERSION#" addons/kube-proxy/kube-proxy.yml
kubectl apply -f addons/kube-proxy/kube-proxy.yml
# 下面是输出
# serviceaccount "kube-proxy" created
# clusterrolebinding.rbac.authorization.k8s.io "system:kube-proxy" created
# configmap "kube-proxy" created
# daemonset.apps "kube-proxy" created

# Calico
# 拉取镜像
curl -s https://zhangguanzhang.github.io/bash/pull.sh | bash -s -- quay.io/calico/node:v3.1.3
curl -s https://zhangguanzhang.github.io/bash/pull.sh | bash -s -- quay.io/calico/cni:v3.1.3

sed -ri "s#\{\{ interface \}\}#${interface}#" addons/calico/v3.1/calico.yml
kubectl apply -f addons/calico/v3.1
kubectl -n kube-system get pod --all-namespaces

# CoreDNS
# 这里节点使用的是hostname,所以建议把hosts关系写到Coredns的解析里
sed  -i '57r '<(echo '        hosts {';for NODE in "${!MasterArray[@]}";do  echo "          ${MasterArray[$NODE]} $NODE"; done;for NODE in "${!NodeArray[@]}";do echo "          ${NodeArray[$NODE]} $NODE";done;echo '        }';) addons/coredns/coredns.yml

# 如果后期增加类似解析记录的话可以改cm后(注意cm是yaml格式写的,所以不要使用tab必须用空格)用kill信号让coredns去reload,
# 因为主进程是前台也就是PID为1,找到对应的pod执行即可,也可以利用deploy的更新机制去伪更新实现重启
# kubectl exec coredns-xxxxxx -- kill -SIGUSR1 1

# 在k8s-m1通过 kubeclt 执行下面命令來创建,并检查是否部署成功:
# 注: 目前coredns官方有bug, 在此不使用
# kubectl apply -f addons/coredns/coredns.yml
# kubectl -n kube-system get po -l k8s-app=kube-dns

# 测试coredns正常否看脸,可以下面创建pod来测试
# 先创建一个dnstool的pod
cat<<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: busybox
  namespace: default
spec:
  containers:
  - name: busybox
    image: busybox:1.28
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
  restartPolicy: Always
EOF
# nslookup下看看能返回地址不
kubectl exec -ti busybox -- nslookup kubernetes

# 如果已安装的话, 如果CoreDNS工作不正常,先删掉它,删掉后确保coredns的pod和svc不存在
# kubectl delete -f addons/coredns/coredns.yml
# kubectl -n kube-system get pod,svc -l k8s-app=kube-dns

# KubeDNS(如果遇到上面的CoreDNS的bug的话使用它)
kubectl apply -f addons/Kubedns/kubedns.yml
#查看pod状态
kubectl -n kube-system get pod,svc -l k8s-app=kube-dns
#检查集群dns正常否
kubectl exec -ti busybox -- nslookup kubernetes
# 等待官方修复的话可以后期先创建出coredns的deploy,svc会负载到coredns之后再删掉kubedns的副本控制器和pod

# Metrics Server(1.8+的k8s)
kubectl create -f addons/metric-server/metrics-server-1.12+.yml

# Kubernets Extra Addons部署
cd ~/k8s-manual-files

# Dashboard
# 在k8s-m1通过kubectl来建立kubernetes dashboard即可：
kubectl apply -f ExtraAddons/dashboard
kubectl -n kube-system get po,svc -l k8s-app=kubernetes-dashboard
# 完成后,就可以通过浏览器存取Dashboard https://{YOUR_VIP}:6443/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/
# 用如下命令得到token, 复制token,然后贴到Kubernetes dashboard。注意这边一般来说要针对不同User开启特定存取权限。
kubectl -n kube-system describe secrets | sed -rn '/\sdashboard-token-/,/^token/{/^token/s#\S+\s+##p}'


# 这边会额外建立一个名称为anonymous-dashboard-proxy的 Cluster Role(Binding) 
# 来让system:anonymous这个匿名使用者能够通过 API Server 来 proxy 到 Kubernetes Dashboard,
# 而这个 RBAC 规则仅能够存取services/proxy资源,以及https:kubernetes-dashboard:资源名称同时
# 在 1.7 版本以后的 Dashboard 将不再提供所有权限,因此需要建立一个 service account 来绑定 
# cluster-admin role(这系列已经写在dashboard/anonymous-proxy-rbac.yml里)
# 完成后,就可以通过浏览器存取Dashboard https://{YOUR_VIP}:6443/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/。
kubectl -n kube-system describe secrets | sed -rn '/\sdashboard-token-/,/^token/{/^token/s#\S+\s+##p}'
# 复制token,然后贴到Kubernetes dashboard。注意这边一般来说要针对不同User开启特定存取权限。

# Ingress Controller
# 首先在k8s-m1执行下列命令来建立 Ingress Controller,并检查是否部署正常：
sed -ri 's#\{\{ INGRESS_VIP \}\}#'"${INGRESS_VIP}"'#' ExtraAddons/ingress-controller/ingress-controller-svc.yml
kubectl create ns ingress-nginx
kubectl apply -f ExtraAddons/ingress-controller/
kubectl -n ingress-nginx get po,svc

# External DNS(非公有云下且内网没DNS下很实用)
# 首先在k8s-m1执行下面命令来建立 CoreDNS Server,并检查是否部署正常
sed -ri 's#\{\{ INGRESS_VIP \}\}#'"${INGRESS_VIP}"'#' ExtraAddons/external-dns/coredns/coredns-svc-tcp.yml
sed -ri 's#\{\{ INGRESS_VIP \}\}#'"${INGRESS_VIP}"'#' ExtraAddons/external-dns/coredns/coredns-svc-udp.yml
kubectl create ns external-dns
kubectl create -f ExtraAddons/external-dns/coredns/
kubectl -n external-dns get po,svc

# 接着部署 ExternalDNS 来与 CoreDNS 同步资源记录：
kubectl apply -f ExtraAddons/external-dns/external-dns/
kubectl -n external-dns get po -l k8s-app=external-dns

# Prometheus Operator
# 在k8s-m1通过kubectl来建立Prometheus 需要的元件
kubectl apply -f ExtraAddons/prometheus/
kubectl apply -f ExtraAddons/prometheus/operator/

# 这边要等 operator 起來并建立好 CRDs 才能进行
kubectl apply -f ExtraAddons/prometheus/alertmanater/
kubectl apply -f ExtraAddons/prometheus/node-exporter/
kubectl apply -f ExtraAddons/prometheus/kube-state-metrics/
kubectl apply -f ExtraAddons/prometheus/grafana/
kubectl apply -f ExtraAddons/prometheus/kube-service-discovery/

# 这里建议先执行后面的补全ep信息部分的操作后再回来
kubectl apply -f ExtraAddons/prometheus/prometheus/
kubectl apply -f ExtraAddons/prometheus/servicemonitor/

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

580

581

582

583

584

585

586

587

588

589

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

615

616

617

618

619

620

621

622

623

624

625

626

627

628

629

630

631

632

633

634

635

636

637

638

639

640

641

642

643

644

645

646

647

648

649

650

651

652

653

654

655

656

657

658

659

660

661

662

663

664

665

666

667

668

669

670

671

672

673

674

675

676

677

678

679

680

681

682

683

684

685

686

687

688

689

690

691

692

693

694

695

696

697

698

699

700

701

702

703

704

705

706

707

708

709

710

711

712

713

714

715

716

717

718

719

720

721

722

723

724

725

726

727

728

729

730

#!/usr/bin/env bash

set -e

# 声明集群成员信息

declare -A MasterArray otherMaster NodeArray

MasterArray=(['k8s-m1']=172.20.23.167 ['k8s-m2']=172.20.23.168 ['k8s-m3']=172.20.23.169)

otherMaster=(['k8s-m2']=172.20.23.168 ['k8s-m3']=172.20.23.169)

NodeArray=(['k8s-n1']=172.20.23.173)

export VIP=172.20.23.4

export INGRESS_VIP=172.20.23.11

[ "${#MasterArray[@]}" -eq 1 ] && export VIP=${MasterArray[@]} || export API_PORT=8443

export KUBE_APISERVER=https://${VIP}:${API_PORT:-6443}

#声明需要安装的的k8s版本

export KUBE_VERSION=v1.12.3

# 网卡名

export interface=ens160

export K8S_DIR=/etc/kubernetes

export PKI_DIR=${K8S_DIR}/pki

export ETCD_SSL=/etc/etcd/ssl

export MANIFESTS_DIR=/etc/kubernetes/manifests/

# cni

export CNI_URL="https://github.com/containernetworking/plugins/releases/download"

export CNI_VERSION=v0.7.4

# cfssl

export CFSSL_URL="https://pkg.cfssl.org/R1.2"

# etcd

export ETCD_version=v3.3.10

#首先在k8s-m1通过git获取部署要用到的二进制配置文件和yml

# curl https://storage.googleapis.com/kubernetes-release/release/${KUBE_VERSION}/kubernetes-server-linux-amd64.tar.gz > kubernetes-server-linux-amd64.tar.gz

# tar -zxvf kubernetes-server-linux-amd64.tar.gz --strip-components=3 -C /usr/local/bin kubernetes/server/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy}

#分发master相关组件到其他master上

#(这边不想master跑pod的话就不复制kubelet和kube-proxy过去,以及后面master节点上的kubelet的相关操作)

# for NODE in "${!otherMaster[@]}"; do

# echo "--- $NODE ${otherMaster[$NODE]} ---"

# scp /usr/local/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy} ${otherMaster[$NODE]}:/usr/local/bin/

# done

#分发node的kubernetes二进制文件

# for NODE in "${!NodeArray[@]}"; do

# echo "--- $NODE ${NodeArray[$NODE]} ---"

# scp /usr/local/bin/kube{let,-proxy} ${NodeArray[$NODE]}:/usr/local/bin/

# done

#在k81-m1下载Kubernetes CNI 二进制文件并分发

#分发cni文件到otherMaster

# mkdir -p /opt/cni/bin

# wget "${CNI_URL}/${CNI_VERSION}/cni-plugins-amd64-${CNI_VERSION}.tgz"

# tar -zxf cni-plugins-amd64-${CNI_VERSION}.tgz -C /opt/cni/bin

# 分发cni文件

# for NODE in "${!otherMaster[@]}"; do

# echo "--- $NODE ${otherMaster[$NODE]} ---"

# ssh ${otherMaster[$NODE]} 'mkdir -p /opt/cni/bin'

# scp /opt/cni/bin/* ${otherMaster[$NODE]}:/opt/cni/bin/

# done

# #分发cni文件到node

# for NODE in "${!NodeArray[@]}"; do

# echo "--- $NODE ${NodeArray[$NODE]} ---"

# ssh ${NodeArray[$NODE]} 'mkdir -p /opt/cni/bin'

# scp /opt/cni/bin/* ${NodeArray[$NODE]}:/opt/cni/bin/

# done

#在k8s-m1需要安裝CFSSL工具,这将会用來建立 TLS Certificates。

# wget "${CFSSL_URL}/cfssl_linux-amd64" -O /usr/local/bin/cfssl

# wget "${CFSSL_URL}/cfssljson_linux-amd64" -O /usr/local/bin/cfssljson

# chmod +x /usr/local/bin/cfssl /usr/local/bin/cfssljson

#建立集群CA keys 与Certificates

#Etcd CA

#首先在k8s-m1建立/etc/etcd/ssl文件夹

cd ~/k8s-manual-files/pki

mkdir -p ${ETCD_SSL}

#从CSR json文件ca-config.json与etcd-ca-csr.json生成etcd的CA keys与Certificate：

cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare ${ETCD_SSL}/etcd-ca

#生成Etcd证书

cfssl gencert \

-ca=${ETCD_SSL}/etcd-ca.pem \

-ca-key=${ETCD_SSL}/etcd-ca-key.pem \

-config=ca-config.json \

-hostname=127.0.0.1,$(xargs -n1<<<${MasterArray[@]} | sort | paste -d, -s -) \

-profile=kubernetes \

etcd-csr.json | cfssljson -bare ${ETCD_SSL}/etcd

#-hostname值为所有masters节点的IP,如果后续master节点扩容此处可以多预留ip到证书里

#完成后删除不必要文件,确认/etc/etcd/ssl有以下文件

rm -rf ${ETCD_SSL}/*.csr

ls $ETCD_SSL

#etcd-ca-key.pem etcd-ca.pem etcd-key.pem etcd.pem

#在k8s-m1上复制相关文件至其他Etcd节点,这边etcd跑在所有master节点上,所以etcd的证书复制到其他mster节点：

for NODE in "${!otherMaster[@]}"; do

echo "--- $NODE ${otherMaster[$NODE]} ---"

ssh ${otherMaster[$NODE]} "mkdir -p ${ETCD_SSL}"

for FILE in etcd-ca-key.pem etcd-ca.pem etcd-key.pem etcd.pem; do

scp ${ETCD_SSL}/${FILE} ${otherMaster[$NODE]}:${ETCD_SSL}/${FILE}

done

#ETCD二进制文件

#etcd所有标准版本可以在下面url查看

#https://github.com/etcd-io/etcd/releases

#在k8s-m1上下载etcd的二进制文件,单台的话建议使用v3.1.9因为有bug

#[ "${#MasterArray[@]}" -eq 1 ] && ETCD_version=v3.3.10 || :

cd ~/k8s-manual-files

#在k8s-m1上分发etcd的二进制文件到其他master上

for NODE in "${!otherMaster[@]}"; do

echo "--- $NODE ${otherMaster[$NODE]} ---"

scp /usr/local/bin/etcd* ${otherMaster[$NODE]}:/usr/local/bin/

done

#在k8s-m1上配置etcd配置文件并分发相关文件

#配置文件存放在/etc/etcd/etcd.config.yml里

#注入基础变量

cd ~/k8s-manual-files/master/

etcd_servers=$( xargs -n1<<<${MasterArray[@]} | sort | sed 's#^#https://#;s#$#:2379#;$s#\n##' | paste -d, -s - )

etcd_initial_cluster=$( for i in ${!MasterArray[@]};do echo $i=https://${MasterArray[$i]}:2380; done | sort | paste -d, -s - )

sed -ri "/initial-cluster:/s#'.+'#'${etcd_initial_cluster}'#" etc/etcd/config.yml

#分发systemd和配置文件

for NODE in "${!MasterArray[@]}"; do

echo "--- $NODE ${MasterArray[$NODE]} ---"

ssh ${MasterArray[$NODE]} "mkdir -p $MANIFESTS_DIR /etc/etcd /var/lib/etcd"

scp systemd/etcd.service ${MasterArray[$NODE]}:/usr/lib/systemd/system/etcd.service

scp etc/etcd/config.yml ${MasterArray[$NODE]}:/etc/etcd/etcd.config.yml

ssh ${MasterArray[$NODE]} "sed -i "s/{HOSTNAME}/$NODE/g" /etc/etcd/etcd.config.yml"

ssh ${MasterArray[$NODE]} "sed -i "s/{PUBLIC_IP}/${MasterArray[$NODE]}/g" /etc/etcd/etcd.config.yml"

ssh ${MasterArray[$NODE]} 'systemctl daemon-reload'

done

#在k8s-m1上启动所有etcd

#etcd 进程首次启动时会等待其它节点的 etcd 加入集群，命令 systemctl start etcd 会卡住一段时间，为正常现象

#可以全部启动后后面的etcdctl命令查看状态确认正常否

for NODE in "${!MasterArray[@]}"; do

echo "--- $NODE ${MasterArray[$NODE]} ---"

ssh ${MasterArray[$NODE]} 'systemctl enable --now etcd' &

done

wait

#然后输出到终端了的时候多按几下回车直到等光标回到终端状态

#k8s-m1上执行下面命令验证 ETCD 集群状态,下面第二个是使用3的api去查询集群的键值

etcdctl \

--cert-file /etc/etcd/ssl/etcd.pem \

--key-file /etc/etcd/ssl/etcd-key.pem \

--ca-file /etc/etcd/ssl/etcd-ca.pem \

--endpoints $etcd_servers cluster-health

#使用3的api去查询集群的键值

ETCDCTL_API=3 \

etcdctl \

--cert=/etc/etcd/ssl/etcd.pem \

--key=/etc/etcd/ssl/etcd-key.pem \

--cacert /etc/etcd/ssl/etcd-ca.pem \

--endpoints $etcd_servers get / --prefix --keys-only

#Kubernetes CA

#为确保安全，kubernetes 系统各组件需要使用 x509 证书对通信进行加密和认证。

#在k8s-m1建立pki文件夹,并生成根CA凭证用于签署其它的k8s证书

mkdir -p ${PKI_DIR}

cd ~/k8s-manual-files/pki

cfssl gencert -initca ca-csr.json | cfssljson -bare ${PKI_DIR}/ca

ls ${PKI_DIR}/ca*.pem

#/etc/kubernetes/pki/ca-key.pem /etc/kubernetes/pki/ca.pem

#API Server Certificate

#此凭证将被用于API Server和Kubelet Client通信使用,使用下面命令生成kube-apiserver凭证

cfssl gencert \

-ca=${PKI_DIR}/ca.pem \

-ca-key=${PKI_DIR}/ca-key.pem \

-config=ca-config.json \

-hostname=10.96.0.1,${VIP},127.0.0.1,kubernetes,kubernetes.default,kubernetes.default.svc,kubernetes.default.svc.cluster,kubernetes.default.svc.cluster.local,$(xargs -n1<<<${MasterArray[@]} | sort | paste -d, -s -) \

-profile=kubernetes \

apiserver-csr.json | cfssljson -bare ${PKI_DIR}/apiserver

ls ${PKI_DIR}/apiserver*.pem

#/etc/kubernetes/pki/apiserver-key.pem /etc/kubernetes/pki/apiserver.pem

#这边-hostname的10.96.0.1是Cluster IP的Kubernetes端点(默认占用第一个ip,用于给集群里的pod要调用Kubernetes的API server);

#kubernetes.default为Kubernets DN。

#如果使用域名可以加上域名

#如果后续master节点扩容此处可以多预留ip到证书里

#Front Proxy Certificate

#此凭证将被用于Authenticating Proxy的功能上,而该功能主要是提供API Aggregation的认证。使用下面命令生成CA:

cfssl gencert \

-initca front-proxy-ca-csr.json | cfssljson -bare ${PKI_DIR}/front-proxy-ca

ls ${PKI_DIR}/front-proxy-ca*.pem

#/etc/kubernetes/pki/front-proxy-ca-key.pem /etc/kubernetes/pki/front-proxy-ca.pem

#接着生成front-proxy-client凭证(hosts的warning忽略即可)：

cfssl gencert \

-ca=${PKI_DIR}/front-proxy-ca.pem \

-ca-key=${PKI_DIR}/front-proxy-ca-key.pem \

-config=ca-config.json \

-profile=kubernetes \

front-proxy-client-csr.json | cfssljson -bare ${PKI_DIR}/front-proxy-client

ls ${PKI_DIR}/front-proxy-client*.pem

#front-proxy-client-key.pem front-proxy-client.pem

#Controller Manager Certificate

#凭证会建立system:kube-controller-manager的使用者(凭证 CN),并被绑定在RBAC

#Cluster Role中的system:kube-controller-manager来让Controller Manager

#元件能够存取需要的API object。

#这边通过以下命令生成 Controller Manager 凭证(hosts的warning忽略即可)

cfssl gencert \

-ca=${PKI_DIR}/ca.pem \

-ca-key=${PKI_DIR}/ca-key.pem \

-config=ca-config.json \

-profile=kubernetes \

manager-csr.json | cfssljson -bare ${PKI_DIR}/controller-manager

ls ${PKI_DIR}/controller-manager*.pem

#controller-manager-key.pem controller-manager.pem

#接着利用kubectl生成Controller Manager的kubeconfig文件

# controller-manager set cluster

kubectl config set-cluster kubernetes \

--certificate-authority=${PKI_DIR}/ca.pem \

--embed-certs=true \

--server=${KUBE_APISERVER} \

--kubeconfig=${K8S_DIR}/controller-manager.kubeconfig

# controller-manager set credentials

kubectl config set-credentials system:kube-controller-manager \

--client-certificate=${PKI_DIR}/controller-manager.pem \

--client-key=${PKI_DIR}/controller-manager-key.pem \

--embed-certs=true \

--kubeconfig=${K8S_DIR}/controller-manager.kubeconfig

# controller-manager set context

kubectl config set-context system:kube-controller-manager@kubernetes \

--cluster=kubernetes \

--user=system:kube-controller-manager \

--kubeconfig=${K8S_DIR}/controller-manager.kubeconfig

# controller-manager set default context

kubectl config use-context system:kube-controller-manager@kubernetes \

--kubeconfig=${K8S_DIR}/controller-manager.kubeconfig

#Scheduler Certificate

#凭证会建立system:kube-scheduler的使用者(凭证 CN),并被绑定在 RBAC Cluster Role

#中的system:kube-scheduler来让 Scheduler 元件能够存取需要的 API object。

#这边通过以下命令生成 Scheduler 凭证(hosts的warning忽略即可)：

cfssl gencert \

-ca=${PKI_DIR}/ca.pem \

-ca-key=${PKI_DIR}/ca-key.pem \

-config=ca-config.json \

-profile=kubernetes \

scheduler-csr.json | cfssljson -bare ${PKI_DIR}/scheduler

ls ${PKI_DIR}/scheduler*.pem

#/etc/kubernetes/pki/scheduler-key.pem /etc/kubernetes/pki/scheduler.pem

#接着利用kubectl生成Scheduler的kubeconfig文件

# scheduler set cluster

kubectl config set-cluster kubernetes \

--certificate-authority=${PKI_DIR}/ca.pem \

--embed-certs=true \

--server=${KUBE_APISERVER} \

--kubeconfig=${K8S_DIR}/scheduler.kubeconfig

# scheduler set credentials

kubectl config set-credentials system:kube-scheduler \

--client-certificate=${PKI_DIR}/scheduler.pem \

--client-key=${PKI_DIR}/scheduler-key.pem \

--embed-certs=true \

--kubeconfig=${K8S_DIR}/scheduler.kubeconfig

# scheduler set context

kubectl config set-context system:kube-scheduler@kubernetes \

--cluster=kubernetes \

--user=system:kube-scheduler \

--kubeconfig=${K8S_DIR}/scheduler.kubeconfig

# scheduler use default context

kubectl config use-context system:kube-scheduler@kubernetes \

--kubeconfig=${K8S_DIR}/scheduler.kubeconfig

#Admin Certificate

#Admin 被用来绑定 RBAC Cluster Role 中 cluster-admin,当想要(最常见的就是使用kubectl)操作

#所有 Kubernetes 集群功能时,就必须利用这边生成的 kubeconfig 文件。

cfssl gencert \

-ca=${PKI_DIR}/ca.pem \

-ca-key=${PKI_DIR}/ca-key.pem \

-config=ca-config.json \

-profile=kubernetes \

admin-csr.json | cfssljson -bare ${PKI_DIR}/admin

ls ${PKI_DIR}/admin*.pem

#/etc/kubernetes/pki/admin-key.pem /etc/kubernetes/pki/admin.pem

#kubectl 默认从 ~/.kube/config 文件读取 kube-apiserver 地址、证书、用户名等信息，

#如果没有配置，执行 kubectl 命令时可能会出错(因为默认连接8080匿名端口)

#接着利用kubectl生成 Admin 的kubeconfig文件

# admin set cluster

kubectl config set-cluster kubernetes \

--certificate-authority=${PKI_DIR}/ca.pem \

--embed-certs=true \

--server=${KUBE_APISERVER} \

--kubeconfig=${K8S_DIR}/admin.kubeconfig

# admin set credentials

kubectl config set-credentials kubernetes-admin \

--client-certificate=${PKI_DIR}/admin.pem \

--client-key=${PKI_DIR}/admin-key.pem \

--embed-certs=true \

--kubeconfig=${K8S_DIR}/admin.kubeconfig

# admin set context

kubectl config set-context kubernetes-admin@kubernetes \

--cluster=kubernetes \

--user=kubernetes-admin \

--kubeconfig=${K8S_DIR}/admin.kubeconfig

# admin set default context

kubectl config use-context kubernetes-admin@kubernetes \

--kubeconfig=${K8S_DIR}/admin.kubeconfig

# Master Kubelet Certificate

# 这边使用 Node authorizer 来让节点的 kubelet 能够存取如 services、endpoints 等 API,

# 而使用 Node authorizer 需定义system:nodesCLusterRole(凭证的 Organization),

# 并且包含system:node:<nodeName>的使用者名称(凭证的 Common Name)。

# 首先在k8s-m1节点生成所有 master 节点的 kubelet 凭证,这边通过下面命令來生成：

for NODE in "${!MasterArray[@]}"; do

echo "--- $NODE ---"

\cp kubelet-csr.json kubelet-$NODE-csr.json;

sed -i "s/\$NODE/$NODE/g" kubelet-$NODE-csr.json;

cfssl gencert \

-ca=${PKI_DIR}/ca.pem \

-ca-key=${PKI_DIR}/ca-key.pem \

-config=ca-config.json \

-hostname=$NODE \

-profile=kubernetes \

kubelet-$NODE-csr.json | cfssljson -bare ${PKI_DIR}/kubelet-$NODE;

rm -f kubelet-$NODE-csr.json

done

ls ${PKI_DIR}/kubelet*.pem

#/etc/kubernetes/pki/kubelet-k8s-m1-key.pem /etc/kubernetes/pki/kubelet-k8s-m2.pem

#/etc/kubernetes/pki/kubelet-k8s-m1.pem /etc/kubernetes/pki/kubelet-k8s-m3-key.pem

#/etc/kubernetes/pki/kubelet-k8s-m2-key.pem /etc/kubernetes/pki/kubelet-k8s-m3.pem

#这边需要依据节点修改-hostname与$NODE。

#完成后复制kubelet凭证至所有master节点：

for NODE in "${!MasterArray[@]}"; do

echo "--- $NODE ${MasterArray[$NODE]} ---"

ssh ${MasterArray[$NODE]} "mkdir -p ${PKI_DIR}"

scp ${PKI_DIR}/ca.pem ${MasterArray[$NODE]}:${PKI_DIR}/ca.pem

scp ${PKI_DIR}/kubelet-$NODE-key.pem ${MasterArray[$NODE]}:${PKI_DIR}/kubelet-key.pem

scp ${PKI_DIR}/kubelet-$NODE.pem ${MasterArray[$NODE]}:${PKI_DIR}/kubelet.pem

rm -f ${PKI_DIR}/kubelet-$NODE-key.pem ${PKI_DIR}/kubelet-$NODE.pem

done

#接着在k8s-m1执行以下命令给所有master产生kubelet的kubeconfig文件

for NODE in "${!MasterArray[@]}"; do

echo "--- $NODE ---"

ssh ${MasterArray[$NODE]} "cd ${PKI_DIR} && \

kubectl config set-cluster kubernetes \

--certificate-authority=${PKI_DIR}/ca.pem \

--embed-certs=true \

--server=${KUBE_APISERVER} \

--kubeconfig=${K8S_DIR}/kubelet.kubeconfig && \

kubectl config set-credentials system:node:${NODE} \

--client-certificate=${PKI_DIR}/kubelet.pem \

--client-key=${PKI_DIR}/kubelet-key.pem \

--embed-certs=true \

--kubeconfig=${K8S_DIR}/kubelet.kubeconfig && \

kubectl config set-context system:node:${NODE}@kubernetes \

--cluster=kubernetes \

--user=system:node:${NODE} \

--kubeconfig=${K8S_DIR}/kubelet.kubeconfig && \

kubectl config use-context system:node:${NODE}@kubernetes \

--kubeconfig=${K8S_DIR}/kubelet.kubeconfig"

done

# Service Account Key

# Kubernetes Controller Manager 利用 Key pair 生成与签署 Service Account 的 tokens,

# 而这边不能通过 CA 做认证,而是建立一组公私钥来让 API Server 与 Controller Manager 使用：

# 在k8s-m1执行以下指令

openssl genrsa -out ${PKI_DIR}/sa.key 2048

openssl rsa -in ${PKI_DIR}/sa.key -pubout -out ${PKI_DIR}/sa.pub

ls ${PKI_DIR}/sa.*

#/etc/kubernetes/pki/sa.key /etc/kubernetes/pki/sa.pub

# 删除不必要文件

# 所有资讯准备完成后,就可以将一些不必要文件删除

rm -f ${PKI_DIR}/*.csr \

${PKI_DIR}/scheduler*.pem \

${PKI_DIR}/controller-manager*.pem \

${PKI_DIR}/admin*.pem \

${PKI_DIR}/kubelet*.pem

# 复制文件至其他节点

# 复制凭证文件至其他master节点

for NODE in "${!otherMaster[@]}"; do

echo "--- $NODE ${otherMaster[$NODE]}---"

for FILE in $(ls ${PKI_DIR}); do

scp ${PKI_DIR}/${FILE} ${otherMaster[$NODE]}:${PKI_DIR}/${FILE}

done

# 复制Kubernetes config文件至其他master节点：

for NODE in "${!otherMaster[@]}"; do

echo "--- $NODE ${otherMaster[$NODE]}---"

for FILE in admin.kubeconfig controller-manager.kubeconfig scheduler.kubeconfig; do

scp ${K8S_DIR}/${FILE} ${otherMaster[$NODE]}:${K8S_DIR}/${FILE}

done

# Kubernetes Masters

# HA(haproxy+keepalived) 单台master就不要用HA了

for NODE in "${!MasterArray[@]}"; do

echo "--- $NODE ${MasterArray[$NODE]} ---"

ssh ${MasterArray[$NODE]} 'yum install haproxy keepalived -y' &

done

wait

# 在k8s-m1节点下把相关配置文件配置后再分发

cd ~/k8s-manual-files/master/etc

# 修改haproxy.cfg配置文件

sed -i '$r '<(paste <( seq -f' server k8s-api-%g' ${#MasterArray[@]} ) <( xargs -n1<<<${MasterArray[@]} | sort | sed 's#$#:6443 check#')) haproxy/haproxy.cfg

# 修改keepalived(网卡和VIP写进去,使用下面命令)

sed -ri "s#\{\{ VIP \}\}#${VIP}#" keepalived/*

sed -ri "s#\{\{ interface \}\}#${interface}#" keepalived/keepalived.conf

sed -i '/unicast_peer/r '<(xargs -n1<<<${MasterArray[@]} | sort | sed 's#^#\t#') keepalived/keepalived.conf

# 分发文件

for NODE in "${!MasterArray[@]}"; do

echo "--- $NODE ${MasterArray[$NODE]} ---"

scp -r haproxy/ ${MasterArray[$NODE]}:/etc

scp -r keepalived/ ${MasterArray[$NODE]}:/etc

ssh ${MasterArray[$NODE]} 'systemctl enable --now haproxy keepalived'

done

# ping下vip看看能通否,先等待大概四五秒等keepalived和haproxy起来

ping -c1 $VIP

# 如果vip没起来就是keepalived没起来就每个节点上去restart下keepalived或者确认下配置文件

# /etc/keepalived/keepalived.conf里网卡名和ip是否注入成功

for NODE in "${!MasterArray[@]}"; do

echo "--- $NODE ${MasterArray[$NODE]} ---"

ssh ${MasterArray[$NODE]} 'systemctl restart haproxy keepalived'

done

# Master组件

# 在k8s-m1节点下把相关配置文件配置后再分发

cd ~/k8s-manual-files/master/

etcd_servers=$( xargs -n1<<<${MasterArray[@]} | sort | sed 's#^#https://#;s#$#:2379#;$s#\n##' | paste -d, -s - )

# 注入VIP和etcd_servers

sed -ri '/--advertise-address/s#=.+#='"$VIP"' \\#' systemd/kube-apiserver.service

sed -ri '/--etcd-servers/s#=.+#='"$etcd_servers"' \\#' systemd/kube-apiserver.service

# 修改encryption.yml

ENCRYPT_SECRET=$( head -c 32 /dev/urandom | base64 )

sed -ri "/secret:/s#(: ).+#\1${ENCRYPT_SECRET}#" encryption/config.yml

# 分发文件(不想master跑pod的话就不复制kubelet的配置文件)

for NODE in "${!MasterArray[@]}"; do

echo "--- $NODE ${MasterArray[$NODE]} ---"

ssh ${MasterArray[$NODE]} "mkdir -p $MANIFESTS_DIR /etc/systemd/system/kubelet.service.d /var/lib/kubelet /var/log/kubernetes"

scp systemd/kube-*.service ${MasterArray[$NODE]}:/usr/lib/systemd/system/

scp encryption/config.yml ${MasterArray[$NODE]}:/etc/kubernetes/encryption.yml

scp audit/policy.yml ${MasterArray[$NODE]}:/etc/kubernetes/audit-policy.yml

scp systemd/kubelet.service ${MasterArray[$NODE]}:/lib/systemd/system/kubelet.service

scp systemd/10-kubelet.conf ${MasterArray[$NODE]}:/etc/systemd/system/kubelet.service.d/10-kubelet.conf

scp etc/kubelet/kubelet-conf.yml ${MasterArray[$NODE]}:/etc/kubernetes/kubelet-conf.yml

done

#在k8s-m1上给所有master机器启动kubelet 服务并设置kubectl补全脚本:

for NODE in "${!MasterArray[@]}"; do

echo "--- $NODE ${MasterArray[$NODE]} ---"

ssh ${MasterArray[$NODE]} 'systemctl enable --now kubelet kube-apiserver kube-controller-manager kube-scheduler;

\cp /etc/kubernetes/admin.kubeconfig ~/.kube/config;

kubectl completion bash > /etc/bash_completion.d/kubectl'

done

# 建立TLS Bootstrapping RBAC 与Secret

# 后面kubectl命令只需要在任何一台master执行就行了

# 首先在k8s-m1建立一个变数来产生BOOTSTRAP_TOKEN,并建立bootstrap-kubelet.conf的Kubernetes config文件

export TOKEN_ID=$(openssl rand 3 -hex)

export TOKEN_SECRET=$(openssl rand 8 -hex)

export BOOTSTRAP_TOKEN=${TOKEN_ID}.${TOKEN_SECRET}

# bootstrap set cluster

kubectl config set-cluster kubernetes \

--certificate-authority=${PKI_DIR}/ca.pem \

--embed-certs=true \

--server=${KUBE_APISERVER} \

--kubeconfig=${K8S_DIR}/bootstrap-kubelet.kubeconfig

# bootstrap set credentials

kubectl config set-credentials tls-bootstrap-token-user \

--token=${BOOTSTRAP_TOKEN} \

--kubeconfig=${K8S_DIR}/bootstrap-kubelet.kubeconfig

# bootstrap set context

kubectl config set-context tls-bootstrap-token-user@kubernetes \

--cluster=kubernetes \

--user=tls-bootstrap-token-user \

--kubeconfig=${K8S_DIR}/bootstrap-kubelet.kubeconfig

# bootstrap use default context

kubectl config use-context tls-bootstrap-token-user@kubernetes \

--kubeconfig=${K8S_DIR}/bootstrap-kubelet.kubeconfig

#接着在k8s-m1建立TLS bootstrap secret来提供自动签证使用：

cd ~/k8s-manual-files/master

# 注入变量

sed -ri "s#\{TOKEN_ID\}#${TOKEN_ID}#g" resources/bootstrap-token-Secret.yml

sed -ri "/token-id/s#\S+\$#'&'#" resources/bootstrap-token-Secret.yml

sed -ri "s#\{TOKEN_SECRET\}#${TOKEN_SECRET}#g" resources/bootstrap-token-Secret.yml

kubectl create -f resources/bootstrap-token-Secret.yml

# 下面是输出

#secret "bootstrap-token-65a3a9" created

#在k8s-m1建立 TLS Bootstrap Autoapprove RBAC来自动处理 CSR

kubectl apply -f resources/kubelet-bootstrap-rbac.yml

# 下面是输出

#clusterrolebinding.rbac.authorization.k8s.io "kubelet-bootstrap" created

#clusterrolebinding.rbac.authorization.k8s.io "node-autoapprove-bootstrap" created

#clusterrolebinding.rbac.authorization.k8s.io "node-autoapprove-certificate-rotation" created

#为了方便管理集群,因此需要通过 kubectl logs 来查看,但由于 API 权限,故需要建立一个 RBAC Role 来获取存取权限,这边在k8s-m1节点执行下面命令创建

kubectl apply -f resources/apiserver-to-kubelet-rbac.yml

# 下面是输出

#clusterrole.rbac.authorization.k8s.io "system:kube-apiserver-to-kubelet" configured

#clusterrolebinding.rbac.authorization.k8s.io "system:kube-apiserver" configured

#设定master节点加上污点Taint不让(没有声明容忍该污点的)pod跑在master节点上：

kubectl taint nodes node-role.kubernetes.io/master="":NoSchedule --all

# 下面是输出

# node "k8s-m1" tainted

# node "k8s-m2" tainted

# node "k8s-m3" tainted

# Kubernetes Nodes

# 本部分将说明如何建立与设定Kubernetes Node 角色,Node 是主要执行容器实例(Pod)的工作节点。

# 在开始部署前,先在k8-m1将需要用到的文件复制到所有node节点上

cd ${PKI_DIR}

for NODE in "${!NodeArray[@]}"; do

echo "--- $NODE ${NodeArray[$NODE]} ---"

ssh ${NodeArray[$NODE]} "mkdir -p ${PKI_DIR} ${ETCD_SSL}"

# Etcd

for FILE in etcd-ca.pem etcd.pem etcd-key.pem; do

scp ${ETCD_SSL}/${FILE} ${NodeArray[$NODE]}:${ETCD_SSL}/${FILE}

done

# Kubernetes

for FILE in pki/ca.pem pki/ca-key.pem pki/front-proxy-ca.pem bootstrap-kubelet.kubeconfig ; do

scp ${K8S_DIR}/${FILE} ${NodeArray[$NODE]}:${K8S_DIR}/${FILE}

done

# 部署与设定

# 在k8s-m1节点分发kubelet.service相关文件到每台node上去管理kubelet：

cd ~/k8s-manual-files/

for NODE in "${!NodeArray[@]}"; do

echo "--- $NODE ${NodeArray[$NODE]} ---"

ssh ${NodeArray[$NODE]} "mkdir -p /var/lib/kubelet /var/log/kubernetes /etc/systemd/system/kubelet.service.d $MANIFESTS_DIR"

scp node/systemd/kubelet.service ${NodeArray[$NODE]}:/lib/systemd/system/kubelet.service

scp node/systemd/10-kubelet.conf ${NodeArray[$NODE]}:/etc/systemd/system/kubelet.service.d/10-kubelet.conf

scp node/etc/kubelet/kubelet-conf.yml ${NodeArray[$NODE]}:/etc/kubernetes/kubelet-conf.yml

done

# 最后在k8s-m1上去启动每个node节点的kubelet 服务:

for NODE in "${!NodeArray[@]}"; do

echo "--- $NODE ${NodeArray[$NODE]} ---"

ssh ${NodeArray[$NODE]} 'systemctl enable --now kubelet.service'

done

# Kubernetes Core Addons部署

# daemonSet方式部署

cd ~/k8s-manual-files

# 注入变量

sed -ri "/server:/s#(: ).+#\1${KUBE_APISERVER}#" addons/kube-proxy/kube-proxy.yml

sed -ri "/image:.+kube-proxy/s#:[^:]+\$#:$KUBE_VERSION#" addons/kube-proxy/kube-proxy.yml

kubectl apply -f addons/kube-proxy/kube-proxy.yml

# 下面是输出

# serviceaccount "kube-proxy" created

# clusterrolebinding.rbac.authorization.k8s.io "system:kube-proxy" created

# configmap "kube-proxy" created

# daemonset.apps "kube-proxy" created

# Calico

# 拉取镜像

curl -s https://zhangguanzhang.github.io/bash/pull.sh | bash -s -- quay.io/calico/node:v3.1.3

curl -s https://zhangguanzhang.github.io/bash/pull.sh | bash -s -- quay.io/calico/cni:v3.1.3

sed -ri "s#\{\{ interface \}\}#${interface}#" addons/calico/v3.1/calico.yml

kubectl apply -f addons/calico/v3.1

kubectl -n kube-system get pod --all-namespaces

# CoreDNS

# 这里节点使用的是hostname,所以建议把hosts关系写到Coredns的解析里

sed -i '57r '<(echo ' hosts {';for NODE in "${!MasterArray[@]}";do echo " ${MasterArray[$NODE]} $NODE"; done;for NODE in "${!NodeArray[@]}";do echo " ${NodeArray[$NODE]} $NODE";done;echo ' }';) addons/coredns/coredns.yml

# 如果后期增加类似解析记录的话可以改cm后(注意cm是yaml格式写的,所以不要使用tab必须用空格)用kill信号让coredns去reload,

# 因为主进程是前台也就是PID为1,找到对应的pod执行即可,也可以利用deploy的更新机制去伪更新实现重启

# kubectl exec coredns-xxxxxx -- kill -SIGUSR1 1

# 在k8s-m1通过 kubeclt 执行下面命令來创建,并检查是否部署成功:

# 注: 目前coredns官方有bug, 在此不使用

# kubectl apply -f addons/coredns/coredns.yml

# kubectl -n kube-system get po -l k8s-app=kube-dns

# 测试coredns正常否看脸,可以下面创建pod来测试

# 先创建一个dnstool的pod

cat<<EOF | kubectl apply -f -

apiVersion: v1

kind: Pod

metadata:

name: busybox

namespace: default

spec:

containers:

- name: busybox

image: busybox:1.28

command:

- sleep

- "3600"

imagePullPolicy: IfNotPresent

restartPolicy: Always

EOF

# nslookup下看看能返回地址不

kubectl exec -ti busybox -- nslookup kubernetes

# 如果已安装的话, 如果CoreDNS工作不正常,先删掉它,删掉后确保coredns的pod和svc不存在

# kubectl delete -f addons/coredns/coredns.yml

# kubectl -n kube-system get pod,svc -l k8s-app=kube-dns

# KubeDNS(如果遇到上面的CoreDNS的bug的话使用它)

kubectl apply -f addons/Kubedns/kubedns.yml

#查看pod状态

kubectl -n kube-system get pod,svc -l k8s-app=kube-dns

#检查集群dns正常否

kubectl exec -ti busybox -- nslookup kubernetes

# 等待官方修复的话可以后期先创建出coredns的deploy,svc会负载到coredns之后再删掉kubedns的副本控制器和pod

# Metrics Server(1.8+的k8s)

kubectl create -f addons/metric-server/metrics-server-1.12+.yml

# Kubernets Extra Addons部署

cd ~/k8s-manual-files

# Dashboard

# 在k8s-m1通过kubectl来建立kubernetes dashboard即可：

kubectl apply -f ExtraAddons/dashboard

kubectl -n kube-system get po,svc -l k8s-app=kubernetes-dashboard

# 完成后,就可以通过浏览器存取Dashboard https://{YOUR_VIP}:6443/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/

# 用如下命令得到token, 复制token,然后贴到Kubernetes dashboard。注意这边一般来说要针对不同User开启特定存取权限。

kubectl -n kube-system describe secrets | sed -rn '/\sdashboard-token-/,/^token/{/^token/s#\S+\s+##p}'

# 这边会额外建立一个名称为anonymous-dashboard-proxy的 Cluster Role(Binding)

# 来让system:anonymous这个匿名使用者能够通过 API Server 来 proxy 到 Kubernetes Dashboard,

# 而这个 RBAC 规则仅能够存取services/proxy资源,以及https:kubernetes-dashboard:资源名称同时

# 在 1.7 版本以后的 Dashboard 将不再提供所有权限,因此需要建立一个 service account 来绑定

# cluster-admin role(这系列已经写在dashboard/anonymous-proxy-rbac.yml里)

# 完成后,就可以通过浏览器存取Dashboard https://{YOUR_VIP}:6443/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/。

kubectl -n kube-system describe secrets | sed -rn '/\sdashboard-token-/,/^token/{/^token/s#\S+\s+##p}'

# 复制token,然后贴到Kubernetes dashboard。注意这边一般来说要针对不同User开启特定存取权限。

# Ingress Controller

# 首先在k8s-m1执行下列命令来建立 Ingress Controller,并检查是否部署正常：

sed -ri 's#\{\{ INGRESS_VIP \}\}#'"${INGRESS_VIP}"'#' ExtraAddons/ingress-controller/ingress-controller-svc.yml

kubectl create ns ingress-nginx

kubectl apply -f ExtraAddons/ingress-controller/

kubectl -n ingress-nginx get po,svc

# External DNS(非公有云下且内网没DNS下很实用)

# 首先在k8s-m1执行下面命令来建立 CoreDNS Server,并检查是否部署正常

sed -ri 's#\{\{ INGRESS_VIP \}\}#'"${INGRESS_VIP}"'#' ExtraAddons/external-dns/coredns/coredns-svc-tcp.yml

sed -ri 's#\{\{ INGRESS_VIP \}\}#'"${INGRESS_VIP}"'#' ExtraAddons/external-dns/coredns/coredns-svc-udp.yml

kubectl create ns external-dns

kubectl create -f ExtraAddons/external-dns/coredns/

kubectl -n external-dns get po,svc

# 接着部署 ExternalDNS 来与 CoreDNS 同步资源记录：

kubectl apply -f ExtraAddons/external-dns/external-dns/

kubectl -n external-dns get po -l k8s-app=external-dns

# Prometheus Operator

# 在k8s-m1通过kubectl来建立Prometheus 需要的元件

kubectl apply -f ExtraAddons/prometheus/

kubectl apply -f ExtraAddons/prometheus/operator/

# 这边要等 operator 起來并建立好 CRDs 才能进行

kubectl apply -f ExtraAddons/prometheus/alertmanater/

kubectl apply -f ExtraAddons/prometheus/node-exporter/

kubectl apply -f ExtraAddons/prometheus/kube-state-metrics/

kubectl apply -f ExtraAddons/prometheus/grafana/

kubectl apply -f ExtraAddons/prometheus/kube-service-discovery/

# 这里建议先执行后面的补全ep信息部分的操作后再回来

kubectl apply -f ExtraAddons/prometheus/prometheus/

kubectl apply -f ExtraAddons/prometheus/servicemonitor/

转自: https://zhangguanzhang.github.io/2018/09/18/kubernetes-1-11-x-bin/

By sean, 6 years01/07/2019 ago