一， ansible安装k3s集群
下载ansible安装k3s脚本：
ansible-galaxy install xanmanning.k3s
ansible role目录结构：
├── hosts.ini
├── roles
│ ├── k3s
│ │ ├── defaults
│ │ │ └── main.yml
│ │ ├── files
│ │ │ ├── docker
│ │ │ │ ├── bin
│ │ │ │ │ ├── containerd
│ │ │ │ │ ├── containerd-shim
│ │ │ │ │ ├── ctr
│ │ │ │ │ ├── docker
│ │ │ │ │ ├── docker-compose
│ │ │ │ │ ├── dockerd
│ │ │ │ │ ├── docker-init
│ │ │ │ │ ├── docker-proxy
│ │ │ │ │ └── runc
│ │ │ │ ├── docker
│ │ │ │ ├── docker-tag
│ │ │ │ └── libseccomp-2.3.1-4.el7.x86_64.rpm
│ │ │ ├── images
│ │ │ │ └── images.tar.gz
│ │ │ ├── k3s-v1.18.2+k3s1.tar.gz
│ │ │ ├── kube-nginx.tar.gz
│ │ │ └── push_images.sh
│ │ ├── handlers
│ │ │ └── main.yml
│ │ ├── tasks
│ │ │ ├── build
│ │ │ │ ├── configure-k3s-cluster.yml
│ │ │ │ ├── download-k3s.yml
│ │ │ │ ├── get-systemd-context.yml
│ │ │ │ ├── get-version.yml
│ │ │ │ ├── install-docker-amazon.yml
│ │ │ │ ├── install-docker-opensuse-leap.yml
│ │ │ │ ├── install-docker-prerequisites-debian.yml
│ │ │ │ ├── install-docker-prerequisites-redhat.yml
│ │ │ │ ├── install-docker-prerequisites-suse.yml
│ │ │ │ ├── install-docker-suse.yml
│ │ │ │ ├── install-docker.yml
│ │ │ │ ├── install-k3s.yml
│ │ │ │ ├── install-kube-nginx.yml
│ │ │ │ ├── load_images.yml
│ │ │ │ ├── optimization-host.yml
│ │ │ │ ├── preconfigure-k3s-auto-deploying-manifests.yml
│ │ │ │ ├── preconfigure-k3s.yml
│ │ │ │ └── prepare-host-optimization.yml
│ │ │ ├── main.yml
│ │ │ ├── operate
│ │ │ │ ├── start-k3s.yml
│ │ │ │ └── stop-k3s.yml
│ │ │ ├── state-downloaded.yml
│ │ │ ├── state-installed.yml
│ │ │ ├── state-restarted.yml
│ │ │ ├── state-started.yml
│ │ │ ├── state-stopped.yml
│ │ │ ├── state-uninstalled.yml
│ │ │ ├── teardown
│ │ │ │ ├── drain-and-remove-nodes.yml
│ │ │ │ ├── uninstall-docker-amazon.yml
│ │ │ │ ├── uninstall-docker-opensuse-leap.yml
│ │ │ │ ├── uninstall-docker-prerequisites-debian.yml
│ │ │ │ ├── uninstall-docker-prerequisites-redhat.yml
│ │ │ │ ├── uninstall-docker-prerequisites-suse.yml
│ │ │ │ ├── uninstall-docker-suse.yml
│ │ │ │ ├── uninstall-docker.yml
│ │ │ │ ├── uninstall-k3s.yml
│ │ │ │ └── uninstall-kube-nginx.yml
│ │ │ └── validate
│ │ │ ├── check-environment.yml
│ │ │ ├── check-experimental-variables.yml
│ │ │ ├── check-master-count.yml
│ │ │ ├── check-uninstalled.yml
│ │ │ ├── check-unsupported-rootless.yml
│ │ │ ├── check-variables.yml
│ │ │ └── main.yml
│ │ ├── templates
│ │ │ ├── docker
│ │ │ │ ├── containerd.service.j2
│ │ │ │ ├── daemon.json.j2
│ │ │ │ ├── docker.service.j2
│ │ │ │ └── docker.socket.j2
│ │ │ ├── k3s-killall.sh.j2
│ │ │ ├── k3s.service.j2
│ │ │ ├── k3s-uninstall.sh.j2
│ │ │ ├── kube-nginx
│ │ │ │ ├── kube-nginx.conf.j2
│ │ │ │ └── kube-nginx.service.j2
│ │ │ └── manifests
│ │ │ ├── calico.yaml.j2
│ │ │ ├── cert-manager-no-webhook.yaml.j2
│ │ │ ├── coredns.yaml.j2
│ │ │ ├── kuboard.yaml.j2
│ │ │ ├── local-storage.yaml.j2
│ │ │ ├── metrics-server-deployment.yaml.j2
│ │ │ ├── metrics-server.yaml
│ │ │ ├── nginx-ingress-controller.yaml.j2
│ │ │ └── nginx-ingress.yaml.j2
│ │ └── vars
│ │ └── main.yml│ └── registry
│ ├── defaults
│ │ └── main.yml
│ ├── files
│ │ ├── k3s-images
│ │ │ └── k3s-images.tar.gz
│ │ └── registry-images
│ │ └── registry_2.7.1.tar
│ ├── tasks
│ │ ├── main.yml
│ │ ├── state-installed.yml
│ │ ├── state-uninstalled.yml
│ │ └── uninstall.yml
│ └── templates
│ ├── config.yml.j2
│ └── docker-compose.yml.j2
├── run.yml
其中的坑有:
- k3s的apiserver高可用默认用的是ipvs, 可通过k3s_kubelet_args定义, 可以通过k3s_control_node_address变量来定义一个VIP或SLB(IP地址或域名)来代理apiserver集群; 在没有VIP或SLB的情况下, 可通过在每个节点上安装一个nginx来做apiserver的代理, 我命名为了kube-nginx, 并且定义了一个本地端口:8443, 即: k3s_https_port=6443 (apiserver), k3s_control_node_https_port=8443(nginx), 这样就可将k3s_control_node_address定义为’127.0.0.1′, 来使k3s连接本机8443端口
- k3s_node_name要是主机名, 不能是IP, 如果定义成IP的话, 服务器上运行kubectl logs/exec 会报错:x509: certificate is valid for 127.0.0.1, not 172.20.48.132
- 安装完成后, 一个master节点的calico/node始终启不来, 报错:
Warning Unhealthy 2m30s (x48 over 10m) kubelet, ncc-gpaas-group07-node05 (combined from similar events): Readiness probe failed: 2020-06-14 09:53:11.875 [INFO][2190] confd/health.go 177: Number of node(s) with BGP peering established = 0
calico/node is not ready: BIRD is not ready: BGP not established with 172.20.48.161,172.20.47.65原因是:
参考了如下：
calico/node is not ready: BIRD is not ready: BGP not established (Calico 3.6 / k8s 1.14.1)
https://stackoverflow.com/questions/54465963/calico-node-is-not-ready-bird-is-not-ready-bgp-not-established
https://q.cnblogs.com/q/125235/
其中提问者改变网段后问题解决了https://github.com/projectcalico/calico/issues/2561#issuecomment-485648104
有其他人通过改变绑定的网络接口解决问题
https://github.com/projectcalico/calico/issues/2561#issuecomment-531537534总结一下,应该是calico在多网络接口时自动检测到错误的网络接口,导致网络无法连通
在calico.yml.j2文件里加上如下:
1234- name: CALICO_IPV4POOL_CIDRvalue: "{{ k3s_cluster_cidr }}"- name: IP_AUTODETECTION_METHODvalue: "interface=eth.*" - 当定义多个master节点时, 需要使用etcd/mysql/dqlite/pg等存储, 而使用dqlite时还要将k3s_use_experimental设为true, 而且要保证奇数个master节点, 不然k3s会报错
ansible playbook其中做了如下改动：
hosts.ini:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 |
[k3s_nodes] 172.20.48.132 nodeRole=system,ingress k3s_control_node=true registry=true 172.20.48.161 nodeRole=system,ingress k3s_control_node=true 172.20.47.65 nodeRole=node k3s_control_node=true [all:vars] k3s_cluster_state=installed registry_state=installed ansible_ssh_user=root ansible_ssh_port=22 k3s_release_version='v1.18.2+k3s1' bin_dir="/usr/local/bin" k3s_cluster_cidr='10.42.0.0/16' k3s_service_cidr='10.43.0.0/16' # 集群 DNS 服务 IP (从 SERVICE_CIDR 中预分配) k3s_dns_svc_ip="10.43.0.10" # k3s_control_node_address可以指定一个高可用VIP或域名, 当127.0.0.1时会安装kube-nginx代理apiserver k3s_control_node_address='127.0.0.1' # nginx ingress ingress_http_port=8000 ingress_https_port=4443 k3s_https_port=6443 k3s_control_node_https_port=8443 # docker registry registry_url="{% for h in groups['k3s_nodes'] %}{% if 'registry' in hostvars[h] and hostvars[h].registry %}{{ h }}{% endif %}{% endfor %}" registry_port=5000 registry_addr="{{ registry_url }}:{{ registry_port }}" registry_version=2.7.1 registry_dir='/data/registry/{{ registry_url }}' docker_storage_dir='/data/docker' k3s_control_workers=true # 当定义多个master时， 需要定义如下变量（mysql, dqlite等选一, 如用dqlite的话， 请用奇数个节点）： ; k3s_datastore_endpoint="mysql://root:mypasswd@tcp(172.20.58.132:3306)/k3sdb" k3s_dqlite_datastore=true k3s_use_experimental=true |
defaults/main.yml 增加如下：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 |
--- # k3s cluster state, options: installed, started, stopped, restarted # (default: installed) k3s_cluster_state: installed install_online: false #------- start for docker ---------- # # 清华大学开源软件镜像站 docker_repo_cn: "https://mirrors4.tuna.tsinghua.edu.cn/docker-ce" # docker_repo_offical: "https://download.docker.com" # docker_version: "19.03.9-3" # containerd_io_version: "1.2.13-3.2" IMAGE_REPOSITORY: "reg.yyuap.io:81" docker_daemon_conf_dir: "/etc/docker" # 国内镜像加速 docker_reg_mirror_1: "https://registry.docker-cn.com" docker_reg_mirror_2: "https://docker.mirrors.ustc.edu.cn" # docker日志相关 docker_log_driver: "json-file" docker_log_level: "warn" docker_log_max_size: "10m" docker_log_max_file: 3 # docker容器存储目录 docker_storage_dir: "/data/docker" # 开启Restful API enable_docker_remote_api: false #------- end for docker ---------- # for nginx-ingress ingress_http_port: 80 ingress_https_port: 443 # for kube-nginx nginx_prefix: /opt/kube-nginx # Use a specific k3s version, if set to "false" we will get the latest # k3s_release_version: v0.1.0 k3s_release_version: false # When multiple play_hosts are present, attempt to cluster the nodes. # Using false will create multiple standalone nodes. # (default: true) k3s_build_cluster: true # URL for GitHub project k3s_github_url: https://github.com/rancher/k3s # Installation directory for k3s k3s_install_dir: /usr/local/bin # Path for additional Kubernetes Manifests # https://rancher.com/docs/k3s/latest/en/advanced/#auto-deploying-manifests k3s_server_manifests_dir: /var/lib/rancher/k3s/server/manifests # A list of templates used for preconfigure the cluster. k3s_server_manifests_templates: [] # Use experimental features in k3s? k3s_use_experimental: false # Use a database or etcd cluster to enable HA. Examples below: # MySQL: # k3s_datastore_endpoint "mysql://username:password@tcp(hostname:3306)/database-name" # PostgreSQL: # k3s_datastore_endpoint: "postgres://username:password@hostname:port/database-name" # Etcd: # k3s_datastore_endpoint: "https://etcd-host-1:2379,https://etcd-host-2:2379,https://etcd-host-3:2379" k3s_datastore_endpoint: false # If using a database endpoint for HA, you can optionally set the CA file, # Cert file and Key file for connecting to the database using TLS. # # These need to already be present on the play hosts. # # k3s_datastore_cafile: /path/to/ca.crt # k3s_datastore_certfile: /path/to/cert.crt # k3s_datastore_keyfile: /path/to/key.pem # Use DQLite for HA Datastore? (EXPERIMENTAL) k3s_dqlite_datastore: false # Are control hosts also worker nodes? k3s_control_workers: true # HTTPS Listening port k3s_https_port: 6443 # Ensure Docker is installed on nodes k3s_use_docker: false # Disable flannel, you will need to install your own CNI driver. k3s_no_flannel: false # Flannel backend ('none', 'vxlan', 'ipsec', or 'wireguard') # k3s_flannel_backend: vxlan # Disable CoreDNS, you will need to install your own DNS provider. k3s_no_coredns: false # Cluster IP for CoreDNS service. Should be in your service-cidr range. # Use `false` to use default k3s_cluster_dns: false # Cluster Domain (default: "cluster.local") # k3s_cluster_domain: cluster.local # Disable Traefik k3s_no_traefik: false # Disable Service Load Balancer, you will need to install your own # load balancer, such as MetalLB. Must be disabled if using your own # load balancer service. k3s_no_servicelb: false # Do not use local storage k3s_no_local_storage: false # Do not deploy metrics server k3s_no_metrics_server: false # Disable default k3s scheduler k3s_disable_scheduler: false # Disable k3s cloud controller k3s_disable_cloud_controller: false # Disable k3s network policy controller k3s_disable_network_policy: false # Default local storage path for local provisioner storage class, if set to "false" we will use the default k3s_default_local_storage_path: false # Use secret encryption at rest (EXPERIMENTAL) k3s_secrets_encryption: false # with become privileges for k3s_become_for_all: false k3s_become_for_systemd: null k3s_become_for_install_dir: null k3s_become_for_usr_local_bin: null k3s_become_for_package_install: null k3s_become_for_kubectl: null k3s_become_for_uninstall: null k3s_tls_san_default: - "{{ k3s_control_node_address }}" - "k3s.apiserver.io" - "kubernetes" - "kubernetes.default" - "kubernetes.default.svc" - "kubernetes.default.svc.cluster.local" |
tasks/state-installed.yml: 将preconfigure-k3s-auto-deploying-manifests.yml和apply manifests放到最后执行， 因为k3s启动时会在/var/lib/rancher/k3s/server/manifests/目录生成一些要启动的manifests, 需要等k3s启动完成后, 再通过模块生成将此目录下所需文件进行覆盖，然后再apply
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 |
--- - import_tasks: validate/check-environment.yml - import_tasks: build/preconfigure-k3s.yml - import_tasks: teardown/drain-and-remove-nodes.yml - import_tasks: build/get-version.yml when: k3s_release_version is not defined or not k3s_release_version or k3s_release_version is not regex('\\+k3s[1-9]$') - import_tasks: validate/main.yml - import_tasks: build/get-systemd-context.yml - import_tasks: build/optimization-host.yml - include_tasks: build/install-docker-prerequisites-{{ ansible_os_family | lower }}.yml when: k3s_use_docker and ((k3s_control_workers) or (not k3s_control_workers and not k3s_control_node)) and (k3s_non_root is not defined or not k3s_non_root) and install_online - import_tasks: build/install-docker.yml when: k3s_use_docker and ((k3s_control_workers) or (not k3s_control_workers and not k3s_control_node)) and ansible_distribution | replace(" ", "-") | lower not in ['amazon', 'suse', 'opensuse-leap'] and (k3s_non_root is not defined or not k3s_non_root) - include_tasks: build/install-docker-{{ ansible_distribution | replace(" ", "-") | lower }}.yml when: k3s_use_docker and ((k3s_control_workers) or (not k3s_control_workers and not k3s_control_node)) and ansible_distribution | replace(" ", "-") | lower in ['amazon', 'suse', 'opensuse-leap'] and (k3s_non_root is not defined or not k3s_non_root) - include_role: name: registry when: registry is defined and registry - import_tasks: build/download-k3s.yml # - import_tasks: build/preconfigure-k3s-auto-deploying-manifests.yml # when: k3s_control_node # and k3s_server_manifests_templates | length > 0 - import_tasks: build/install-k3s.yml - import_tasks: build/install-kube-nginx.yml when: k3s_control_node_address == '127.0.0.1' - import_tasks: build/configure-k3s-cluster.yml when: play_hosts | length > 1 and k3s_build_cluster is defined and k3s_build_cluster # - name: delete default metrics # file: # path: "{{ k3s_server_manifests_dir }}/{{ item }}" # state: absent # with_items: # - "coredns.yaml" # - "metrics-server/metrics-server-deployment.yaml" # when: k3s_server_manifests_templates | length > 0 - import_tasks: build/preconfigure-k3s-auto-deploying-manifests.yml when: k3s_control_node and k3s_server_manifests_templates | length > 0 - name: apply manifests shell: >- find {{ k3s_server_manifests_dir }} -name "*.yaml" -exec kubectl apply -f {} \; delegate_to: "{{ k3s_control_delegate }}" run_once: true when: k3s_control_node and k3s_server_manifests_templates | length > 0 |
roles/k3s/tasks/build/optimization-host.yml, 安装前进行系统优化:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 |
- name: "set system.conf" lineinfile: dest: "/etc/systemd/system.conf" line: "{{ item }}" with_items: - "DefaultLimitMEMLOCK=infinity" - "DefaultLimitCORE=infinity" - "DefaultCPUAccounting=yes" - "DefaultMemoryAccounting=yes" - "DefaultLimitNOFILE=1024000" - "DefaultLimitNPROC=1024000" - name: "set limits.conf" lineinfile: dest: "/etc/security/limits.conf" line: "{{ item }}" with_items: - "* soft nproc 1024000" - "* hard nproc 1024000" - "* soft nofile 1024000" - "* hard nofile 1024000" - "* soft core 1024000" - "* hard core 1024000" - name: set sysctl.conf shell: cmd: | # 优化OS cp /etc/sysctl.conf /etc/sysctl.conf.bak cat > /etc/sysctl.conf << EOF net.bridge.bridge-nf-call-ip6tables=1 net.bridge.bridge-nf-call-iptables=1 net.ipv4.ip_forward=1 net.ipv4.conf.all.forwarding=1 net.ipv4.neigh.default.gc_thresh1=4096 net.ipv4.neigh.default.gc_thresh2=6144 net.ipv4.neigh.default.gc_thresh3=8192 net.ipv4.neigh.default.gc_interval=60 net.ipv4.neigh.default.gc_stale_time=120 # 参考 https://github.com/prometheus/node_exporter#disabled-by-default kernel.perf_event_paranoid=-1 #sysctls for k8s node config net.ipv4.tcp_slow_start_after_idle=0 net.core.rmem_max=16777216 fs.inotify.max_user_watches=524288 kernel.softlockup_all_cpu_backtrace=1 kernel.softlockup_panic=0 kernel.watchdog_thresh=30 fs.file-max=2097152 fs.inotify.max_user_instances=8192 fs.inotify.max_queued_events=16384 vm.max_map_count=262144 fs.may_detach_mounts=1 net.core.netdev_max_backlog=16384 net.ipv4.tcp_wmem=4096 12582912 16777216 net.core.wmem_max=16777216 net.core.somaxconn=32768 net.ipv4.ip_forward=1 net.ipv4.tcp_max_syn_backlog=8096 net.ipv4.tcp_rmem=4096 12582912 16777216 net.ipv6.conf.all.disable_ipv6=1 net.ipv6.conf.default.disable_ipv6=1 net.ipv6.conf.lo.disable_ipv6=1 kernel.yama.ptrace_scope=0 vm.swappiness=0 # 可以控制core文件的文件名中是否添加pid作为扩展。 kernel.core_uses_pid=1 # Do not accept source routing net.ipv4.conf.default.accept_source_route=0 net.ipv4.conf.all.accept_source_route=0 # Promote secondary addresses when the primary address is removed net.ipv4.conf.default.promote_secondaries=1 net.ipv4.conf.all.promote_secondaries=1 # Enable hard and soft link protection fs.protected_hardlinks=1 fs.protected_symlinks=1 # 源路由验证 # see details in https://help.aliyun.com/knowledge_detail/39428.html net.ipv4.conf.all.rp_filter=0 net.ipv4.conf.default.rp_filter=0 net.ipv4.conf.default.arp_announce = 2 net.ipv4.conf.lo.arp_announce=2 net.ipv4.conf.all.arp_announce=2 # see details in https://help.aliyun.com/knowledge_detail/41334.html net.ipv4.tcp_max_tw_buckets=5000 net.ipv4.tcp_syncookies=1 net.ipv4.tcp_fin_timeout=30 net.ipv4.tcp_synack_retries=2 kernel.sysrq=1 EOF sysctl -p # 关闭selinux sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config setenforce 0 # 关闭防火墙 systemctl stop firewalld.service && systemctl disable firewalld.service # 修改时区 ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime # 关闭swap swapoff -a && sysctl -w vm.swappiness=0 && sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab |
build/preconfigure-k3s-auto-deploying-manifests.yml: 创建metrics-server时进行了修改， 将此放入metrics-server子目录， 与rancher自动生成的保持一致
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
--- - name: Ensure that the manifests directory exists file: state: directory path: "{{ k3s_server_manifests_dir }}" when: k3s_server_manifests_templates | length > 0 # https://rancher.com/docs/k3s/latest/en/advanced/#auto-deploying-manifests - name: Create metrics-server sub-directory file: path: "{{ k3s_server_manifests_dir }}/metrics-server" state: directory when: k3s_server_manifests_templates | length > 0 - name: Ensure Auto-Deploying Manifests are copied to controllers template: src: "{{ item }}" dest: "{{ k3s_server_manifests_dir }}/{% if 'metrics' in item %}metrics-server/{%endif%}{{ item | basename | replace('.j2','') }}" loop: "{{ k3s_server_manifests_templates }}" |
tasks/build/download-k3s.yml： 加入离线安装步骤
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 |
--- - block: - name: Ensure target host architecture information is set as a fact set_fact: k3s_arch: "{{ k3s_arch_lookup[ansible_architecture].arch }}" k3s_arch_suffix: "{{ k3s_arch_lookup[ansible_architecture].suffix }}" - name: Ensure URLs are set as facts for downloading binaries set_fact: k3s_binary_url: "{{ k3s_github_download_url }}/{{ k3s_release_version }}/k3s{{ k3s_arch_suffix }}" k3s_hash_url: "{{ k3s_github_download_url }}/{{ k3s_release_version }}/sha256sum-{{ k3s_arch }}.txt" - name: Ensure the k3s hashsum is downloaded uri: url: "{{ k3s_hash_url }}" return_content: true register: k3s_hash_sum_raw - name: Ensure sha256sum is set from hashsum variable set_fact: k3s_hash_sum: "{{ (k3s_hash_sum_raw.content.split('\n') | select('search', 'k3s' + k3s_arch_suffix) | reject('search', 'images') | first).split() | first }}" changed_when: false - name: Ensure installation directory exists file: path: "{{ k3s_install_dir }}" state: directory - name: Ensure k3s binary is downloaded get_url: url: "{{ k3s_binary_url }}" dest: "{{ k3s_install_dir }}/k3s-{{ k3s_release_version }}" checksum: "sha256:{{ k3s_hash_sum }}" mode: 0755 use_proxy: yes become: "{{ k3s_become_for_install_dir | ternary(true, false, k3s_become_for_all) }}" when: install_online # 离线安装 - block: - name: Ensure k3s binary is downloaded and Unarchived unarchive: src: "k3s-{{ k3s_release_version }}.tar.gz" dest: "{{ k3s_install_dir }}/" mode: 0755 become: "{{ k3s_become_for_install_dir | ternary(true, false, k3s_become_for_all) }}" - name: remove installation zip files file: path: "{{ k3s_install_dir }}/k3s-{{ k3s_release_version }}.tar.gz" state: absent when: not install_online |
tasks/build/install-docker-prerequisites-redhat.yml: 在线安装时使用, 如果要通过外网在线安装docker，由于官方源下载太慢，所以修改为国内源， 其中下面注释的地方打开后可以使用国内源
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 |
--- - name: Ensure python-dnf is installed package: name: "{{ 'python-dnf' if ansible_python_version is version_compare('3.0.0', '<') else 'python3-dnf' }}" state: present register: ensure_python_dnf_installed become: "{{ k3s_become_for_package_install | ternary(true, false, k3s_become_for_all) }}" until: ensure_python_dnf_installed is succeeded retries: 3 delay: 10 when: ansible_pkg_mgr == 'dnf' - name: Ensure Docker prerequisites are installed yum: name: - yum-utils - device-mapper-persistent-data - lvm2 state: present register: ensure_docker_prerequisites_installed until: ensure_docker_prerequisites_installed is succeeded retries: 3 delay: 10 become: "{{ k3s_become_for_package_install | ternary(true, false, k3s_become_for_all) }}" - name: Check to see if Docker repository is available for this distribution uri: url: "{{ docker_repo_cn }}/linux/{{ ansible_distribution | lower }}/{{ ansible_distribution_major_version }}" register: k3s_redhat_repo_check failed_when: false changed_when: false - name: Ensure Docker repository is installed and configured yum_repository: name: docker-ce description: Docker CE Repository baseurl: "{{ docker_repo_cn }}/linux/{{ ansible_distribution | lower }}/{{ ansible_distribution_major_version }}/$basearch/stable" gpgkey: "{{ docker_repo_cn }}/linux/{{ ansible_distribution | lower }}/gpg" enabled: true gpgcheck: true state: present when: ansible_distribution | lower not in ['amazon'] and k3s_redhat_repo_check.status == 200 become: "{{ k3s_become_for_package_install | ternary(true, false, k3s_become_for_all) }}" - name: Ensure Docker repository is installed and configured from file #command: yum-config-manager --add-repo=https://download.docker.com/linux/centos/docker-ce.repo command: yum-config-manager --add-repo={{ docker_repo_cn }}/linux/centos/docker-ce.repo args: creates: /etc/yum.repos.d/docker-ce.repo when: ansible_distribution | lower not in ['amazon'] and k3s_redhat_repo_check.status != 200 become: "{{ k3s_become_for_package_install | ternary(true, false, k3s_become_for_all) }}" |
tasks/build/install-docker.yml 增加配置docker
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 |
--- - name: Get docker service state shell: "systemctl status docker.service|grep Active || echo 'false'" register: dockerd_status ignore_errors: true # install_online - block: - name: Ensure docker is installed package: name: - docker-ce - docker-ce-cli - containerd.io state: present register: ensure_docker_installed until: ensure_docker_installed is succeeded retries: 3 delay: 10 # notify: # - restart docker become: "{{ k3s_become_for_package_install | ternary(true, false, k3s_become_for_all) }}" # config docker daemon - name: Ensure docker storage and configure directory exists file: path: "{{ item }}" state: directory recurse: true with_items: - "{{ docker_daemon_conf_dir }}" - "{{ docker_storage_dir }}" become: "{{ k3s_become_for_package_install | ternary(true, false, k3s_become_for_all) }}" - name: config docker daemon template: src: docker/daemon.json.j2 dest: "{{ docker_daemon_conf_dir }}/daemon.json" notify: - restart docker become: "{{ k3s_become_for_package_install | ternary(true, false, k3s_become_for_all) }}" when: install_online and "running" not in dockerd_status.stdout # install offline - block: - name: "copy libseccomp packages to remote servers" copy: src: "docker/libseccomp-2.3.1-4.el7.x86_64.rpm" dest: /tmp/libseccomp-2.3.1-4.el7.x86_64.rpm mode: 0755 - name: install libseccomp shell: yum localinstall -y /tmp/libseccomp-2.3.1-4.el7.x86_64.rpm - name: Create directors for docker file: name={{ item }} state=directory with_items: - "{{ bin_dir }}" - "{{ docker_storage_dir }}" - "{{ docker_daemon_conf_dir }}" - name: Install docker binaries copy: src=docker/bin/{{ item }} dest={{ bin_dir }}/{{ item }} mode=0755 with_items: - containerd - containerd-shim - docker-init - runc - docker - dockerd - docker-proxy - docker-compose #- ctr tags: upgrade_docker, download_docker - name: Configure docker auto completion copy: src=docker/docker dest=/etc/bash_completion.d/docker mode=0644 - name: config docker daemon template: src: docker/daemon.json.j2 dest: "{{ docker_daemon_conf_dir }}/daemon.json" - name: Flush iptables shell: "iptables -P INPUT ACCEPT \ && iptables -F && iptables -X \ && iptables -F -t nat && iptables -X -t nat \ && iptables -F -t raw && iptables -X -t raw \ && iptables -F -t mangle && iptables -X -t mangle" - name: Write docker.service systemd file template: src=docker/{{ item }}.j2 dest=/usr/lib/systemd/system/{{ item }} tags: upgrade_docker, download_docker with_items: - docker.service - docker.socket - containerd.service - name: Ensure group docker exists group: name: docker state: present - name: Start docker service shell: systemctl enable docker && systemctl daemon-reload && systemctl restart docker - name: Copy docker-tag copy: src=docker/docker-tag dest={{ bin_dir }}/docker-tag mode=0755 - name: Wait for docker service to be started shell: "systemctl status docker.service|grep Active" register: docker_status until: '"running" in docker_status.stdout' retries: 8 delay: 2 tags: upgrade_docker - name: Create docker command link file: src={{ bin_dir }}/docker dest=/usr/bin/docker state=link force=yes when: bin_dir != '/usr/bin' when: not install_online and "running" not in dockerd_status.stdout - meta: flush_handlers |
templates/docker/daemon.json.j2:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
{ "registry-mirrors": ["http://{{ IMAGE_REPOSITORY }}", "{{ docker_reg_mirror_1 }}", "{{ docker_reg_mirror_2 }}"], "max-concurrent-downloads": 10, "log-driver": "{{ docker_log_driver }}", "log-level": "{{ docker_log_level }}", "storage-driver": "overlay2", "log-opts": { "max-size": "{{ docker_log_max_size }}", "max-file": "{{ docker_log_max_file }}" }, "data-root": "{{ docker_storage_dir }}", "insecure-registries": ["0.0.0.0/0"] {% if enable_docker_remote_api %} , "hosts": ["tcp://0.0.0.0:2376", "unix:///var/run/docker.sock"] {% endif %} } |
templates/k3s.service.j2, 其中增加了k3s_node_labels, k3s_tls_san 定义,
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 |
[Unit] Description=Lightweight Kubernetes Documentation=https://k3s.io After=network.target [Service] Type={{ 'notify' if k3s_control_node else 'exec' }} ExecStartPre=-/sbin/modprobe br_netfilter ExecStartPre=-/sbin/modprobe overlay {% filter regex_replace('\s+', ' ') %} {% filter replace('\n', ' ') %} ExecStart={{ k3s_install_dir }}/k3s {% if k3s_control_node %} server{{ ' --disable-agent' if not k3s_control_workers else '' }} {% if k3s_non_root is defined and k3s_non_root %} --rootless {% endif %} {% if k3s_https_port != 6443 %} --https-listen-port {{ k3s_https_port }} {% endif %} {% if k3s_disable_scheduler %} --disable-scheduler {% endif %} {% if k3s_disable_cloud_controller %} --disable-cloud-controller {% endif %} {% if k3s_disable_network_policy %} --disable-network-policy {% endif %} {% if k3s_no_flannel %} {% if (k3s_release_version | replace('v', '')) is version_compare('1.0.0', '>=') %} --flannel-backend none {% else %} --no-flannel {% endif %} {% endif %} {% if k3s_cluster_cidr is defined %} --cluster-cidr {{ k3s_cluster_cidr }} {% endif %} {% if k3s_service_cidr is defined %} --service-cidr {{ k3s_service_cidr }} {% endif %} {% if k3s_flannel_backend is defined and not k3s_no_flannel %} --flannel-backend {{ k3s_flannel_backend }} {% endif %} {% if k3s_no_coredns is defined or k3s_no_traefik is defined or k3s_no_servicelb is defined %} {% if k3s_no_coredns or k3s_no_traefik or k3s_no_servicelb %} {{ ' --no-deploy coredns' if k3s_no_coredns else '' }}{{ ' --no-deploy servicelb' if k3s_no_servicelb else '' }}{{ ' --no-deploy traefik' if k3s_no_traefik else '' }}{{ ' --no-deploy local-storage' if k3s_no_local_storage else '' }}{{ ' --no-deploy metrics-server' if k3s_no_metrics_server else '' }} {% endif %} {% endif %} {% if not k3s_no_local_storage and k3s_default_local_storage_path is defined and k3s_default_local_storage_path %} --default-local-storage-path {{ k3s_default_local_storage_path }} {% endif %} {% if k3s_cluster_dns is defined and k3s_cluster_dns %} --cluster-dns {{ k3s_cluster_dns }} {% endif %} {% if k3s_cluster_domain is defined and k3s_cluster_domain != "cluster.local" %} --cluster-domain {{ k3s_cluster_domain }} {% endif %} {% if k3s_datastore_endpoint is defined and k3s_datastore_endpoint %} --datastore-endpoint "{{ k3s_datastore_endpoint }}" {% if k3s_datastore_cafile is defined and k3s_datastore_cafile %} --datastore-cafile {{ k3s_datastore_cafile }} {% endif %} {% if k3s_datastore_certfile is defined and k3s_datastore_certfile %} --datastore-certfile {{ k3s_datastore_certfile }} {% endif %} {% if k3s_datastore_keyfile is defined and k3s_datastore_keyfile %} --datastore-keyfile {{ k3s_datastore_keyfile }} {% endif %} {% endif %} {% if k3s_dqlite_datastore is defined and k3s_dqlite_datastore %} {% if k3s_primary_control_node is defined and k3s_primary_control_node %} --cluster-init {% else %} --server https://{{ k3s_control_node_address }}:{{ k3s_control_node_https_port }} --token {{ k3s_control_token }} {% endif %} {% endif %} {% if k3s_secrets_encryption is defined and k3s_secrets_encryption %} --secrets-encryption {% endif %} {% else %} agent --server https://{{ k3s_control_node_address }}:{{ k3s_control_node_https_port }} --token {{ k3s_control_token }} {% endif %} {% if k3s_resolv_conf is defined and k3s_resolv_conf %} --resolv-conf {{ k3s_resolv_conf }} {% endif %} {% if k3s_tls_san is defined and k3s_tls_san %} {% for tls in k3s_tls_san %} --tls-san {{ tls }} {% endfor %} {% endif %} {% if k3s_node_data_dir is defined %} --data-dir {{ k3s_node_data_dir }} {% endif %} {% if k3s_use_docker %} --docker {% endif %} {% if k3s_flannel_interface is defined and not k3s_no_flannel %} --flannel-iface {{ k3s_flannel_interface }} {% endif %} {% if k3s_bind_address is defined %} --bind-address {{ k3s_bind_address }} {% endif %} {% if k3s_node_name is defined %} --node-name {{ k3s_node_name }} {% endif %} {% if k3s_node_id is defined %} --with-node-id {{ k3s_node_id }} {% endif %} {% if k3s_node_ip_address is defined %} --node-ip {{ k3s_node_ip_address }} {% endif %} {% if k3s_node_external_address is defined %} --node-external-ip {{ k3s_node_external_address }} {% endif %} {% if k3s_write_kubeconfig_mode is defined %} --write-kubeconfig-mode {{ k3s_write_kubeconfig_mode }} {% endif %} {% if k3s_node_labels is defined and k3s_node_labels is iterable %} {% for label in k3s_node_labels %} {% for key, value in label.items() %} --node-label {{ key }}={{ value }} {% endfor %} {% endfor %} {% endif %} {% if k3s_node_taints is defined and k3s_node_taints is iterable %} {% for taint in k3s_node_taints %} {% for key, value in taint.items() %} --node-taint {{ key }}={{ value }} {% endfor %} {% endfor %} {% endif %} {% if k3s_kubelet_args is defined and k3s_kubelet_args is iterable %} {% for arg in k3s_kubelet_args %} {% for key, value in arg.items() %} --kubelet-arg {{ key }}={{ value }} {% endfor %} {% endfor %} {% endif %} {% endfilter %} {% endfilter %} KillMode=process Delegate=yes LimitNOFILE=infinity LimitNPROC=infinity LimitCORE=infinity TasksMax=infinity TimeoutStartSec=0 Restart=always RestartSec=5s [Install] WantedBy=multi-user.target |
templates/kube-nginx/kube-nginx.conf.j2:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
worker_processes 1; events { worker_connections 1024; } stream { upstream backend { hash $remote_addr consistent; {% for h in play_hosts %} {% if 'k3s_control_node' in hostvars[h] and hostvars[h].k3s_control_node %} server {{ h }}:6443 max_fails=3 fail_timeout=30s; {% endif %} {% endfor %} } server { listen 127.0.0.1:8443; proxy_connect_timeout 1s; proxy_pass backend; } } |
templates/kube-nginx/kube-nginx.service.j2
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
[Unit] Description=kube-apiserver nginx proxy After=network.target After=network-online.target Wants=network-online.target [Service] Type=forking ExecStartPre={{ nginx_prefix }}/sbin/kube-nginx -c {{ nginx_prefix }}/conf/kube-nginx.conf -p {{ nginx_prefix }} -t ExecStart={{ nginx_prefix }}/sbin/kube-nginx -c {{ nginx_prefix }}/conf/kube-nginx.conf -p {{ nginx_prefix }} ExecReload={{ nginx_prefix }}/sbin/kube-nginx -c {{ nginx_prefix }}/conf/kube-nginx.conf -p {{ nginx_prefix }} -s reload PrivateTmp=true Restart=always RestartSec=5 StartLimitInterval=0 LimitNOFILE=65536 [Install] WantedBy=multi-user.target |
tasks/build/install-kube-nginx.yml
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
--- - name: Create kube-nginx directories file: name={{ item }} state=directory with_items: - "{{ nginx_prefix }}/conf" - "{{ nginx_prefix }}/logs" - "{{ nginx_prefix }}/sbin" - name: copy and unarchive kube-nginx binary file unarchive: src: "kube-nginx.tar.gz" dest: "{{ nginx_prefix }}/sbin/" mode: 0755 - name: Create kube-nginx config file template: src=kube-nginx/kube-nginx.conf.j2 dest={{ nginx_prefix }}/conf/kube-nginx.conf - name: Create kube-nginx service file template: src=kube-nginx/kube-nginx.service.j2 dest=/etc/systemd/system/kube-nginx.service - name: restart kube-nginx systemd: name: kube-nginx enabled: true state: restarted daemon_reload: yes |
tasks/build/load_images.yml:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
# load images - block: - name: Ensure images is downloaded and Unarchived unarchive: src: "images/images.tar.gz" dest: "/tmp/" mode: 0755 become: "{{ k3s_become_for_install_dir | ternary(true, false, k3s_become_for_all) }}" # delegate_to: "{{ k3s_control_node }}" # run_once: true - name: load docker image shell: >- for image in $(find /tmp/images -type f); do docker load -i $image done when: not install_online - block: - block: - name: load docker image shell: >- for image in $(find /tmp/images -type f); do docker load -i $image done |
总的playbook: k3s.yml
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 |
- hosts: k3s_nodes vars: #k3s_control_node_address: 172.20.47.250 # k3s_datastore_endpoint: "mysql://root:mypassword@tcp(172.20.40.20:3306)/k3sdb" k3s_cluster_cidr: 10.42.0.0/16 k3s_service_cidr: 10.43.0.0/16 k3s_use_docker: true k3s_no_flannel: true k3s_no_traefik: true # k3s_control_node_address: "k8s.apiserver.io" # k3s_build_cluster: true ingress_http_port: 8000 ingress_https_port: 4443 # if set k3s_disable_network_policy=false, # it will show error: User "system:k3s-controller" cannot get resource "nodes" in API group "" at the cluster scope # refer to: https://github.com/rancher/k3s/issues/1792 k3s_disable_network_policy: true k3s_server_manifests_templates: - "manifests/calico.yaml.j2" - "manifests/coredns.yaml.j2" - "manifests/local-storage.yaml.j2" - "manifests/metrics-server-deployment.yaml.j2" - "manifests/nginx-ingress.yaml.j2" - "manifests/kuboard.yaml.j2" - "manifests/cert-manager-no-webhook.yaml.j2" pre_tasks: - name: Set k3s_node_name set_fact: k3s_node_name: "{{ inventory_hostname }}" - name: Set node labels set_fact: # k3s_node_labels: "{{ k3s_node_labels|default([]) | combine({'node.kubernetes.io/'+item: ''}) }}" k3s_node_labels: "{{ k3s_node_labels|default([]) + [{'node.kubernetes.io/'+item: ''}] }}" with_items: - "{{ nodeRole.split(',') }}" when: nodeRole is defined roles: - { role: k3s, k3s_release_version: v1.18.2+k3s1 } |
其中hosts.ini文件可以这样定义：
|
1 2 3 4 |
[k3s_nodes] 172.20.48.132 nodeRole=system,ingress k3s_control_node=true registry=true 172.20.48.161 nodeRole=system,ingress k3s_control_node=true 172.20.47.65 nodeRole=node k3s_control_node=true |
说明：
会选择有k3s_control_node=true的来安装master172.20.47.2: k3s master, 并将node打上label: node.kubernetes.io/system: “” 和node.kubernetes.io/ingress: “”, 会安装registry镜像仓库172.20.47.3 k3s master, 并将node打上label: node.kubernetes.io/node: “” 和node.kubernetes.io/ingress: “”172.20.47.4 k3s node, 并将node打上label: node.kubernetes.io/node: “”其中主机有system标签是用来跑k3s核心应用的，ingress标签用来跑nginx-ingress-controllor, node标签可以跑业务应用
安装镜像仓库registry的playbook:
roles/registry/tasks/state-installed.yml:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 |
- block: - name: Create directors for registry file: name={{ item }} state=directory with_items: - "{{ registry_dir }}/config" - "{{ registry_dir }}/data" - name: Copy registry files copy: src: "{{ item }}" dest: "{{ registry_dir }}" with_items: - "registry-images" - "k3s-images" - name: load registry image shell: cmd: | cd {{ registry_dir }}/registry-images docker load -i *.tar - name: render registry config file template: src: "config.yml.j2" dest: "{{ registry_dir }}/config/config.yml" - name: render registry docker-compose file template: src: "docker-compose.yml.j2" dest: "{{ registry_dir }}/docker-compose.yml" - name: Make sure compose service is up shell: cmd: | cd {{ registry_dir }} docker-compose -f docker-compose.yml up -d - name: Wait for registry to be ready command: "curl -sL {{ inventory_hostname }}:{{ registry_port }}/v2/_catalog" changed_when: false failed_when: false register: get_registry_status until: get_registry_status.rc == 0 and get_registry_status.stdout.find("repositories") != -1 retries: 30 delay: 20 - name: load k3s images shell: cmd: | cd {{ registry_dir }}/k3s-images tar -zxvf k3s-images.tar.gz find ./ -name *.tar -exec docker load -i {} \; rm -rf k3s-images.tar.gz - name: push images to registry shell: cmd: | images=$(docker images|egrep "^calico|^quay.io.*.cert-manager|^rancher|^kubernetes" | awk '{print $1":"$2}') regs="{{ registry_url }}:{{ registry_port }}" for reg in $regs; do for image in $images; do target_image="$reg/$image" if [[ $image =~ 'cert-manager' ]]; then target_image="$reg/tools/$(basename $image)" fi docker tag $image $target_image && docker push $target_image done done # echo "$password" | docker login --username admin $reg --password-stdin when: registry is defined and registry |
roles/registry/templates/config.yml.j2
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
version: 0.1 log: level: warn fields: service: registry storage: cache: blobdescriptor: inmemory filesystem: rootdirectory: /var/lib/registry delete: enabled: true http: addr: :{{ registry_port }} headers: X-Content-Type-Options: [nosniff] health: storagedriver: enabled: true interval: 10s threshold: 3 # proxy: # remoteurl: https://registry-1.docker.io # username: [username] # password: [password] #auth: # false |
roles/registry/templates/docker-compose.yml.j2:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
version: '3.3' services: registry: container_name: registry restart: always image: registry:{{ registry_version }} ports: - "{{ registry_port }}:5000" volumes: - ./config/config.yml:/etc/docker/registry/config.yml:ro - ./data:/var/lib/registry:rw #environment: #- "STANDALONE=true" #- "MIRROR_SOURCE=https://registry-1.docker.io" #- "MIRROR_SOURCE_INDEX=https://index.docker.io" |
安装完成后， 可能会造成x509报错，证书报错kubectl unable to connect to server: x509: certificate signed by unknown authority， 可以加 kubectl get no –insecure-skip-tls-verify=true
如果pull 镜像的时候失败,利用以下操作可以解决：
|
1 2 3 4 5 6 |
echo -n | openssl s_client -showcerts -connect registry-1.docker.io:443 2>/dev/null | \ sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > \ /usr/local/share/ca-certificates/docker.crt 使用以上命令，获取网站证书。 然后拷贝至/etc/ssl/certs/目录之下. |
二， 运行rancher server并加入k3s集群
参考： https://www.yuque.com/dimension/shu93f/td5aq3
使用 CAShell 生成 CA 证书：https://github.com/yhyddr/CAShell
create_self-signed-cert.sh：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 |
#!/bin/bash -e help () { echo ' ================================================================ ' echo ' --ssl-domain: 生成ssl证书需要的主域名，如不指定则默认为www.rancher.local，如果是ip访问服务，则可忽略；' echo ' --ssl-trusted-ip: 一般ssl证书只信任域名的访问请求，有时候需要使用ip去访问server，那么需要给ssl证书添加扩展IP，多个IP用逗号隔开；' echo ' --ssl-trusted-domain: 如果想多个域名访问，则添加扩展域名（SSL_TRUSTED_DOMAIN）,多个扩展域名用逗号隔开；' echo ' --ssl-size: ssl加密位数，默认2048；' echo ' --ssl-date: ssl有效期，默认10年；' echo ' --ca-date: ca有效期，默认10年；' echo ' --ssl-cn: 国家代码(2个字母的代号),默认CN;' echo ' 使用示例:' echo ' ./create_self-signed-cert.sh --ssl-domain=www.test.com --ssl-trusted-domain=www.test2.com \ ' echo ' --ssl-trusted-ip=1.1.1.1,2.2.2.2,3.3.3.3 --ssl-size=2048 --ssl-date=3650' echo ' ================================================================' } case "$1" in -h|--help) help; exit;; esac if [[ $1 == '' ]];then help; exit; fi CMDOPTS="$*" for OPTS in $CMDOPTS; do key=$(echo ${OPTS} | awk -F"=" '{print $1}' ) value=$(echo ${OPTS} | awk -F"=" '{print $2}' ) case "$key" in --ssl-domain) SSL_DOMAIN=$value ;; --ssl-trusted-ip) SSL_TRUSTED_IP=$value ;; --ssl-trusted-domain) SSL_TRUSTED_DOMAIN=$value ;; --ssl-size) SSL_SIZE=$value ;; --ssl-date) SSL_DATE=$value ;; --ca-date) CA_DATE=$value ;; --ssl-cn) CN=$value ;; esac done # CA相关配置 CA_DATE=${CA_DATE:-3650} CA_KEY=${CA_KEY:-cakey.pem} CA_CERT=${CA_CERT:-cacerts.pem} CA_DOMAIN=cattle-ca # ssl相关配置 SSL_CONFIG=${SSL_CONFIG:-$PWD/openssl.cnf} SSL_DOMAIN=${SSL_DOMAIN:-'www.rancher.local'} SSL_DATE=${SSL_DATE:-3650} SSL_SIZE=${SSL_SIZE:-2048} ## 国家代码(2个字母的代号),默认CN; CN=${CN:-CN} SSL_KEY=$SSL_DOMAIN.key SSL_CSR=$SSL_DOMAIN.csr SSL_CERT=$SSL_DOMAIN.crt echo -e "\033[32m ---------------------------- \033[0m" echo -e "\033[32m | 生成 SSL Cert | \033[0m" echo -e "\033[32m ---------------------------- \033[0m" if [[ -e ./${CA_KEY} ]]; then echo -e "\033[32m ====> 1. 发现已存在CA私钥，备份"${CA_KEY}"为"${CA_KEY}"-bak，然后重新创建 \033[0m" mv ${CA_KEY} "${CA_KEY}"-bak openssl genrsa -out ${CA_KEY} ${SSL_SIZE} else echo -e "\033[32m ====> 1. 生成新的CA私钥 ${CA_KEY} \033[0m" openssl genrsa -out ${CA_KEY} ${SSL_SIZE} fi if [[ -e ./${CA_CERT} ]]; then echo -e "\033[32m ====> 2. 发现已存在CA证书，先备份"${CA_CERT}"为"${CA_CERT}"-bak，然后重新创建 \033[0m" mv ${CA_CERT} "${CA_CERT}"-bak openssl req -x509 -sha256 -new -nodes -key ${CA_KEY} -days ${CA_DATE} -out ${CA_CERT} -subj "/C=${CN}/CN=${CA_DOMAIN}" else echo -e "\033[32m ====> 2. 生成新的CA证书 ${CA_CERT} \033[0m" openssl req -x509 -sha256 -new -nodes -key ${CA_KEY} -days ${CA_DATE} -out ${CA_CERT} -subj "/C=${CN}/CN=${CA_DOMAIN}" fi echo -e "\033[32m ====> 3. 生成Openssl配置文件 ${SSL_CONFIG} \033[0m" cat > ${SSL_CONFIG} <<EOM [req] req_extensions = v3_req distinguished_name = req_distinguished_name [req_distinguished_name] [ v3_req ] basicConstraints = CA:FALSE keyUsage = nonRepudiation, digitalSignature, keyEncipherment extendedKeyUsage = clientAuth, serverAuth EOM if [[ -n ${SSL_TRUSTED_IP} || -n ${SSL_TRUSTED_DOMAIN} ]]; then cat >> ${SSL_CONFIG} <<EOM subjectAltName = @alt_names [alt_names] EOM IFS="," dns=(${SSL_TRUSTED_DOMAIN}) dns+=(${SSL_DOMAIN}) for i in "${!dns[@]}"; do echo DNS.$((i+1)) = ${dns[$i]} >> ${SSL_CONFIG} done if [[ -n ${SSL_TRUSTED_IP} ]]; then ip=(${SSL_TRUSTED_IP}) for i in "${!ip[@]}"; do echo IP.$((i+1)) = ${ip[$i]} >> ${SSL_CONFIG} done fi fi echo -e "\033[32m ====> 4. 生成服务SSL KEY ${SSL_KEY} \033[0m" openssl genrsa -out ${SSL_KEY} ${SSL_SIZE} echo -e "\033[32m ====> 5. 生成服务SSL CSR ${SSL_CSR} \033[0m" openssl req -sha256 -new -key ${SSL_KEY} -out ${SSL_CSR} -subj "/C=${CN}/CN=${SSL_DOMAIN}" -config ${SSL_CONFIG} echo -e "\033[32m ====> 6. 生成服务SSL CERT ${SSL_CERT} \033[0m" openssl x509 -sha256 -req -in ${SSL_CSR} -CA ${CA_CERT} \ -CAkey ${CA_KEY} -CAcreateserial -out ${SSL_CERT} \ -days ${SSL_DATE} -extensions v3_req \ -extfile ${SSL_CONFIG} echo -e "\033[32m ====> 7. 证书制作完成 \033[0m" echo echo -e "\033[32m ====> 8. 以YAML格式输出结果 \033[0m" echo "----------------------------------------------------------" echo "ca_key: |" cat $CA_KEY | sed 's/^/ /' echo echo "ca_cert: |" cat $CA_CERT | sed 's/^/ /' echo echo "ssl_key: |" cat $SSL_KEY | sed 's/^/ /' echo echo "ssl_csr: |" cat $SSL_CSR | sed 's/^/ /' echo echo "ssl_cert: |" cat $SSL_CERT | sed 's/^/ /' echo echo -e "\033[32m ====> 9. 附加CA证书到Cert文件 \033[0m" cat ${CA_CERT} >> ${SSL_CERT} echo "ssl_cert: |" cat $SSL_CERT | sed 's/^/ /' echo echo -e "\033[32m ====> 10. 重命名服务证书 \033[0m" echo "cp ${SSL_DOMAIN}.key tls.key" cp ${SSL_DOMAIN}.key tls.key echo "cp ${SSL_DOMAIN}.crt tls.crt" cp ${SSL_DOMAIN}.crt tls.crt |
通过create.sh脚本来调用上述脚本来生成证书：
|
1 2 3 4 5 6 7 8 9 10 11 12 |
#脚本参数 # --ssl-domain: 生成ssl证书需要的主域名，如不指定则默认为www.rancher.local，如果是ip访问服务，则可忽略； # --ssl-trusted-ip: 一般ssl证书只信任域名的访问请求，有时候需要使用ip去访问server，那么需要给ssl证书添加扩展IP，多个IP用逗号隔开； # --ssl-trusted-domain: 如果想多个域名访问，则添加扩展域名（TRUSTED_DOMAIN）,多个TRUSTED_DOMAIN用逗号隔开； # --ssl-size: ssl加密位数，默认2048； # --ssl-date: ssl有效期，默认10年； # --ca-date: ca有效期，默认10年； # --ssl-cn: 国家代码(2个字母的代号),默认CN； mkdir ssl cd ssl sh ../create_self-signed-cert.sh --ssl-domain=rancher.yyuap.io --ssl-trusted-domain=rancher.yyuap.io --ssl-trusted-ip=172.20.47.254 --ssl-size=2048 --ssl-date=3650 |
docker run rancher
|
1 2 3 4 5 6 7 8 9 |
docker run -d --restart=unless-stopped \ -p 8001:80 -p 4433:443 \ -v /root/var/lib/rancher:/var/lib/rancher/ \ -v /var/log/rancher/auditlog:/var/log/auditlog \ -e AUDIT_LEVEL=3 \ -v /root/ca/ssl/tls.crt:/etc/rancher/ssl/cert.pem \ -v /root/ca/ssl/tls.key:/etc/rancher/ssl/key.pem \ -v /root/ca/ssl/cacerts.pem:/etc/rancher/ssl/cacerts.pem \ rancher/rancher:latest |
如果你只是想试试（ rancher 自动签名一个证书供使用）
|
1 2 3 4 5 6 7 |
docker run -d --restart=unless-stopped \ -p 80:80 -p 443:443 \ -v /root/var/lib/rancher:/var/lib/rancher/ \ -v /root/var/log/auditlog:/var/log/auditlog \ -e CATTLE_SYSTEM_CATALOG=bundled \ -e AUDIT_LEVEL=3 \ rancher/rancher:stable (或者rancher/rancher:latest) |
然后选择添加现有集群即可
注：
其中可以通过使用cert-manager实现Ingress https
可参考：
https://xuchao918.github.io/2019/03/14/%E4%BD%BF%E7%94%A8cert-manager%E5%AE%9E%E7%8E%B0Ingress-https/
https://blog.indexyz.me/archives/starting-using-k3s/
https://itocm.com/a/879888B9399245969D0AF4ADB403D3F2
0 Comments