November 2021

通过ansible获取文件的几种方式

通过ansible获取文件的几种方式: 通过lookup读取ansible管理节点上的的内容，并将其内容赋值给变量。如: 获取管理节点上的ssh key:

- name: Autoops | set ssh key
  set_fact:
    autoops_public_key: "{{ autoops_public_key | default(lookup('file', '/root/.ssh/id_rsa.pub')) }}"
    autoops_private_key: "{{ autoops_private_key | default(lookup('file', '/root/.ssh/id_rsa')) }}"
  delegate_to: localhost
  connection: local
  run_once: yes

- name: Autoops | set ssh key

set_fact:

autoops_public_key: "{{ autoops_public_key | default(lookup('file', '/root/.ssh/id_rsa.pub')) }}"

autoops_private_key: "{{ autoops_private_key | default(lookup('file', '/root/.ssh/id_rsa')) }}"

delegate_to: localhost

connection: local

run_once: yes

fetch 模块的作用就是”Fetches a file from remote nodes，fetch为”拿来”之意，当我们需要将受管主机中的文件拉取到ansible主机时，则可以使用此模块, 可以从远端将文件拉取到本地, 如获取k8s master节点上的ca证书到本地:

- hosts: kube_control_plane:localhost
  gather_facts: False
  pre_tasks:
  - setup:
      gather_subset: 'min'
    tags: 
      - kubeconfig_localhost
      - user_init
  tasks:
  - import_role: 
      name: kubespray-defaults
  - name: "set_fact for current_user"
    set_fact:
      current_user: "{{ hostvars[inventory_hostname]['ansible_env'].USER }}"
      current_user_home: "{{ hostvars[inventory_hostname]['ansible_env'].HOME }}"
    tags: 
      - kubeconfig_localhost
      - user_init

  - block:
    # edit admin.conf
    - name: modify admin.conf to api server domain
      lineinfile:
        dest: "{{ current_user_home }}/.kube/config"
        state: present
        regexp: '(^\s*.server: .*://).*(:.*$)'
        line: '\1{{ apiserver_loadbalancer_domain_name }}\2'
        backrefs: yes
      when: loadbalancer_apiserver is not defined
    delegate_to: localhost
    connection: local
    delegate_facts: yes
    become: no
    run_once: yes
    when:
      - kubeconfig_localhost
    tags: kubeconfig_localhost

  - name: Fetch ca.crt from master to ansible host
    fetch:
      src: "{{ kube_cert_compat_dir }}/ca.crt"
      dest: "{{ kube_cert_compat_dir }}/ca.crt"
      flat: yes
      validate_checksum: no
    register: copy_ca_result
    until: copy_ca_result is not failed
    retries: 20
    become: no
    run_once: yes
    when:
      - kubeconfig_localhost
      - "'HOSTNAME' in hostvars['localhost']['ansible_env']"
      - "hostvars['localhost']['ansible_env'].get('HOSTNAME') not in groups['kube_control_plane']"
    tags: kubeconfig_localhost

- hosts: kube_control_plane:localhost

gather_facts: False

pre_tasks:

- setup:

gather_subset: 'min'

tags:

- kubeconfig_localhost

- user_init

tasks:

- import_role:

name: kubespray-defaults

- name: "set_fact for current_user"

set_fact:

current_user: "{{ hostvars[inventory_hostname]['ansible_env'].USER }}"

current_user_home: "{{ hostvars[inventory_hostname]['ansible_env'].HOME }}"

tags:

- kubeconfig_localhost

- user_init

- block:

# edit admin.conf

- name: modify admin.conf to api server domain

lineinfile:

dest: "{{ current_user_home }}/.kube/config"

state: present

regexp: '(^\s*.server: .*://).*(:.*$)'

line: '\1{{ apiserver_loadbalancer_domain_name }}\2'

backrefs: yes

when: loadbalancer_apiserver is not defined

delegate_to: localhost

connection: local

delegate_facts: yes

become: no

run_once: yes

when:

- kubeconfig_localhost

tags: kubeconfig_localhost

- name: Fetch ca.crt from master to ansible host

fetch:

src: "{{ kube_cert_compat_dir }}/ca.crt"

dest: "{{ kube_cert_compat_dir }}/ca.crt"

flat: yes

validate_checksum: no

until: copy_ca_result is not failed

retries: 20

become: no

run_once: yes

when:

- kubeconfig_localhost

- "'HOSTNAME' in hostvars['localhost']['ansible_env']"

- "hostvars['localhost']['ansible_env'].get('HOSTNAME') not in groups['kube_control_plane']"

tags: kubeconfig_localhost

使用cat测试到变量 slurp 模块用于拉取远端文件的 base64 码, 可通过b64decode进行解码任务: 获取etcd主机的ssl文件信息注册到变量, 用来生成etcdkeeper配置文件, 首先用cat来获取: tasks/main.yml:

- block:
  - name: get remote file contents
    command: "cat {{ etcdkeeper_cacert_file }}"
    register: etcdkeeper_cacert_file

  - name: get remote file contents
    command: "cat {{ etcdkeeper_cert_file }}"
    register: etcdkeeper_cert_file

  - name: get remote file contents
    command: "cat {{ etcdkeeper_key_file }}"
    register: etcdkeeper_key_file
  when:
    - inventory_hostname == groups['etcd'][0]

- name: show key contents
  debug:
    var: hostvars[groups['etcd'][0]].etcdkeeper_cacert_file.stdout

- name: Autoops | Create addon dir
  file:
    path: "{{ item }}"
    state: directory
    owner: root
    group: root
    mode: 0755
    recurse: yes
  with_items:
    - "{{ kube_config_dir }}/addons/etcdkeeper"
  when:
    - inventory_hostname == groups['kube_control_plane'][0]

- name: Autoops | Create manifests
  template: 
    src: "{{ item }}.yml.j2" 
    dest: "{{ kube_config_dir }}/addons/etcdkeeper/{{ item }}.yml"
  vars:
    etcdkeeper_cacert: "{{ hostvars[groups['etcd'][0]].etcdkeeper_cacert_file.stdout }}"
    etcdkeeper_cert: "{{ hostvars[groups['etcd'][0]].etcdkeeper_cert_file.stdout }}"
    etcdkeeper_key: "{{ hostvars[groups['etcd'][0]].etcdkeeper_key_file.stdout }}"
  with_items:
    - etcdkeeper_ns
    - etcdkeeper_cm
    - etcdkeeper_deploy
    - etcdkeeper_svc
  register: etcdkeeper_manifests
  when:
    - inventory_hostname == groups['kube_control_plane'][0]

- name: Autoops | Apply manifests
  shell: "{{ bin_dir }}/kubectl apply -f {{ kube_config_dir }}/addons/etcdkeeper/{{ item.item }}.yml"
  with_items: "{{ etcdkeeper_manifests.results }}"
  when:
    - inventory_hostname == groups['kube_control_plane'][0]

- block:

- name: get remote file contents

command: "cat {{ etcdkeeper_cacert_file }}"

- name: get remote file contents

command: "cat {{ etcdkeeper_cert_file }}"

- name: get remote file contents

command: "cat {{ etcdkeeper_key_file }}"

when:

- inventory_hostname == groups['etcd'][0]

- name: show key contents

debug:

var: hostvars[groups['etcd'][0]].etcdkeeper_cacert_file.stdout

- name: Autoops | Create addon dir

file:

path: "{{ item }}"

state: directory

owner: root

group: root

mode: 0755

recurse: yes

with_items:

- "{{ kube_config_dir }}/addons/etcdkeeper"

when:

- inventory_hostname == groups['kube_control_plane'][0]

- name: Autoops | Create manifests

template:

src: "{{ item }}.yml.j2"

dest: "{{ kube_config_dir }}/addons/etcdkeeper/{{ item }}.yml"

vars:

etcdkeeper_cacert: "{{ hostvars[groups['etcd'][0]].etcdkeeper_cacert_file.stdout }}"

etcdkeeper_cert: "{{ hostvars[groups['etcd'][0]].etcdkeeper_cert_file.stdout }}"

etcdkeeper_key: "{{ hostvars[groups['etcd'][0]].etcdkeeper_key_file.stdout }}"

with_items:

- etcdkeeper_ns

- etcdkeeper_cm

- etcdkeeper_deploy

- etcdkeeper_svc

when:

- inventory_hostname == groups['kube_control_plane'][0]

- name: Autoops | Apply manifests

shell: "{{ bin_dir }}/kubectl apply -f {{ kube_config_dir }}/addons/etcdkeeper/{{ item.item }}.yml"

with_items: "{{ etcdkeeper_manifests.results }}"

when:

- inventory_hostname == groups['kube_control_plane'][0]

结果发现, 生成的配置里, 换行符都被换成了空格尝试使用slurp模块:

- block:
  - name: Etcdkeeper | get remote ca file contents
    slurp:
      src: "{{ etcdkeeper_cacert_file }}"
    register: etcdkeeper_cacert_file

  - name: Etcdkeeper | get remote cert file contents
    slurp:
      src: "{{ etcdkeeper_cert_file }}"
    register: etcdkeeper_cert_file

  - name: Etcdkeeper | get remote key file contents
    slurp:
      src: "{{ etcdkeeper_key_file }}"
    register: etcdkeeper_key_file
  when:
    - inventory_hostname == groups['etcd'][0]


- name: Etcdkeeper | Create addon dir
  file:
    path: "{{ item }}"
    state: directory
    owner: root
    group: root
    mode: 0755
    recurse: yes
  with_items:
    - "{{ kube_config_dir }}/addons/etcdkeeper"
  when:
    - inventory_hostname == groups['kube_control_plane'][0]

- name: Etcdkeeper | Create manifests
  template: 
    src: "{{ item }}.yml.j2" 
    dest: "{{ kube_config_dir }}/addons/etcdkeeper/{{ item }}.yml"
  vars:
    etcdkeeper_cacert: "{{ hostvars[groups['etcd'][0]].etcdkeeper_cacert_file.content | b64decode  | regex_replace('\n', '\n    ') }}"
    etcdkeeper_cert: "{{ hostvars[groups['etcd'][0]].etcdkeeper_cert_file.content | b64decode  | regex_replace('\n', '\n    ') }}"
    etcdkeeper_key: "{{ hostvars[groups['etcd'][0]].etcdkeeper_key_file.content | b64decode  | regex_replace('\n', '\n    ') }}"
  with_items:
    - etcdkeeper_ns
    - etcdkeeper_cm
    - etcdkeeper_deploy
    - etcdkeeper_svc
  register: etcdkeeper_manifests
  when:
    - inventory_hostname == groups['kube_control_plane'][0]

- block:

- name: Etcdkeeper | get remote ca file contents

slurp:

src: "{{ etcdkeeper_cacert_file }}"

- name: Etcdkeeper | get remote cert file contents

slurp:

src: "{{ etcdkeeper_cert_file }}"

- name: Etcdkeeper | get remote key file contents

slurp:

src: "{{ etcdkeeper_key_file }}"

when:

- inventory_hostname == groups['etcd'][0]

- name: Etcdkeeper | Create addon dir

file:

path: "{{ item }}"

state: directory

owner: root

group: root

mode: 0755

recurse: yes

with_items:

- "{{ kube_config_dir }}/addons/etcdkeeper"

when:

- inventory_hostname == groups['kube_control_plane'][0]

- name: Etcdkeeper | Create manifests

template:

src: "{{ item }}.yml.j2"

dest: "{{ kube_config_dir }}/addons/etcdkeeper/{{ item }}.yml"

vars:

etcdkeeper_cacert: "{{ hostvars[groups['etcd'][0]].etcdkeeper_cacert_file.content | b64decode | regex_replace('\n', '\n ') }}"

etcdkeeper_cert: "{{ hostvars[groups['etcd'][0]].etcdkeeper_cert_file.content | b64decode | regex_replace('\n', '\n ') }}"

etcdkeeper_key: "{{ hostvars[groups['etcd'][0]].etcdkeeper_key_file.content | b64decode | regex_replace('\n', '\n ') }}"

with_items:

- etcdkeeper_ns

- etcdkeeper_cm

- etcdkeeper_deploy

- etcdkeeper_svc

when:

- inventory_hostname == groups['kube_control_plane'][0]

结果换行符都保留了下来, 格式正确.

By sean, 3 years11/07/2021 ago

Kubernetes高可用性的考虑高可用性考虑因素本文档包含了社区提供的关于设置高可用性Kubernetes集群的注意事项。如果有什么地方不完整、不清楚或者需要更多的信息，请随时留言。概述当创建生产集群时，高可用性是必须的（集群在某些控制平面或工作节点失效时仍能保持运行的能力）。对于工作节点，假设有足够多的节点。也要在规划和设置集群时，需要考虑到控制平面节点和etcd实例的冗余。 kubeadm支持设置多个控制平面和多etcd集群。但仍有一些方面需要考虑和设置，这些方面并不是Kubernetes本身的一部分，因此项目文档中没有涉及。本文档提供了一些额外的信息和例子，在用kubeadm规划和引导HA集群时很有用。软件负载均衡的选项当创建一个具有多个控制平面的集群时，可以将API Server实例放在负载均衡后面，可以在运行kubeadm init时使用–control-plane-endpoint选项让新集群使用它来实现更高的可用性。当然，负载均衡器本身也应该是高度可用的。这通常是通过给负载均衡器增加冗余来实现的。为此，设置一个管理虚拟IP的主机集群，每台主机运行一个负载均衡器的实例，这样在其他主机处于待机状态时，总是使用当前持有vIP的主机上的负载均衡器。在某些环境中，例如在具有专用负载均衡组件（例如由某些云提供商提供）的数据中心中，该功能可能已经可用。如果没有，可以使用用户管理的负载均衡。在这种情况下，在启动集群之前需要做一些准备工作。由于这不是Kubernetes或kubeadm的一部分，所以必须单独处理。在下面的章节中，我们给出了一些例子，当然也有可能是其他几十种可能的配置。 keepalived 和 haproxy 对于从虚拟IP提供负载均衡，keepalived和haproxy的组合已经存在了很长时间，可以说是众所周知、久经考验。 keepalived提供了一个由可配置的健康检查管理的虚拟IP。由于虚拟IP的实施方式，协商虚拟IP的所有主机必须在同一IP子网中。 haproxy服务可以配置为简单的基于流的负载平衡，从而允许TLS终止由其后面的API服务器实例处理。这种组合既可以作为操作系统上的服务运行，也可以作为控制平面主机上的静态Pods运行。两种情况下的服务配置是相同的。 keepalived配置 keepalived配置由两个文件组成：服务配置文件和健康检查脚本，该脚本将定期被调用，以验证持有虚拟IP的节点是否仍在运行。这些文件位于/etc/keepalived目录中。但请注意，有些 Linux 发行版可能会把它们放在其他地方。下面的配置已经成功地用于keepalived1.3.5版本。

! /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
    router_id LVS_DEVEL
}
vrrp_script check_apiserver {
  script "/etc/keepalived/check_apiserver.sh"
  interval 3
  weight -2
  fall 10
  rise 2
}

vrrp_instance VI_1 {
    state ${STATE}
    interface ${INTERFACE}
    virtual_router_id ${ROUTER_ID}
    priority ${PRIORITY}
    authentication {
        auth_type PASS
        auth_pass ${AUTH_PASS}
    }
    virtual_ipaddress {
        ${APISERVER_VIP}
    }
    track_script {
        check_apiserver
    }
}

! /etc/keepalived/keepalived.conf

! Configuration File for keepalived

global_defs {

router_id LVS_DEVEL

}

vrrp_script check_apiserver {

script "/etc/keepalived/check_apiserver.sh"

interval 3

weight -2

fall 10

rise 2

}

vrrp_instance VI_1 {

state ${STATE}

interface ${INTERFACE}

virtual_router_id ${ROUTER_ID}

priority ${PRIORITY}

authentication {

auth_type PASS

auth_pass ${AUTH_PASS}

}

virtual_ipaddress {

${APISERVER_VIP}

}

track_script {

check_apiserver

}

bash变量样式中有一些占位符需要填写： ${STATE}设置一个主机是MASTER，其他主机是BACKUP，因此虚拟IP最初将分配给MASTER。 ${INTERFACE}是参与协商虚拟IP的网络接口，例如eth0。 ${ROUTER_ID}对于所有keepalived集群主机来说，应该是相同的，同时在同一子网的所有集群中是唯一的。许多发行版将其值预先配置为51。 ${PRIORITY} master上的优先级应高于backups。因此，101和100就足够了。 ${AUTH_PASS} 对所有keepalived集群主机而言，应该是相同的，例如42。 ${APISERVER_VIP}是keepalived集群主机之间协商的虚拟IP地址。上面的 keepalived 配置使用了一个健康检查脚本/etc/keepalived/check_apiserver.sh，负责确保在持有虚拟IP的节点上，API服务器是可用的。这个脚本可以是这样的。

#!/bin/sh

errorExit() {
    echo "*** $*" 1>&2
    exit 1
}

curl --silent --max-time 2 --insecure https://localhost:${APISERVER_DEST_PORT}/ -o /dev/null || errorExit "Error GET https://localhost:${APISERVER_DEST_PORT}/"
if ip addr | grep -q ${APISERVER_VIP}; then
    curl --silent --max-time 2 --insecure https://${APISERVER_VIP}:${APISERVER_DEST_PORT}/ -o /dev/null || errorExit "Error GET https://${APISERVER_VIP}:${APISERVER_DEST_PORT}/"
fi

#!/bin/sh

errorExit() {

echo "*** $*" 1>&2

exit 1

}

curl --silent --max-time 2 --insecure https://localhost:${APISERVER_DEST_PORT}/ -o /dev/null || errorExit "Error GET https://localhost:${APISERVER_DEST_PORT}/"

if ip addr | grep -q ${APISERVER_VIP}; then

curl --silent --max-time 2 --insecure https://${APISERVER_VIP}:${APISERVER_DEST_PORT}/ -o /dev/null || errorExit "Error GET https://${APISERVER_VIP}:${APISERVER_DEST_PORT}/"

bash变量样式中有一些占位符需要填写： ${APISERVER_VIP}是keepalived集群主机之间协商的虚拟IP地址。 ${APISERVER_DEST_PORT} Kubernetes与API服务器对话的端口。 haproxy配置 haproxy配置由一个文件组成：服务配置文件，它在/etc/haproxy目录中。但请注意，有些Linux发行版可能会把它们放在其他地方。以下配置已经成功地用于haproxy2.1.4版本

# /etc/haproxy/haproxy.cfg
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    log /dev/log local0
    log /dev/log local1 notice
    daemon

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 1
    timeout http-request    10s
    timeout queue           20s
    timeout connect         5s
    timeout client          20s
    timeout server          20s
    timeout http-keep-alive 10s
    timeout check           10s

#---------------------------------------------------------------------
# apiserver frontend which proxys to the masters
#---------------------------------------------------------------------
frontend apiserver
    bind *:${APISERVER_DEST_PORT}
    mode tcp
    option tcplog
    default_backend apiserver

#---------------------------------------------------------------------
# round robin balancing for apiserver
#---------------------------------------------------------------------
backend apiserver
    option httpchk GET /healthz
    http-check expect status 200
    mode tcp
    option ssl-hello-chk
    balance     roundrobin
        server ${HOST1_ID} ${HOST1_ADDRESS}:${APISERVER_SRC_PORT} check
        # [...]

# /etc/haproxy/haproxy.cfg

#---------------------------------------------------------------------

# Global settings

#---------------------------------------------------------------------

global

log /dev/log local0

log /dev/log local1 notice

daemon

#---------------------------------------------------------------------

# common defaults that all the 'listen' and 'backend' sections will

# use if not designated in their block

#---------------------------------------------------------------------

defaults

mode http

log global

option httplog

option dontlognull

option http-server-close

option forwardfor except 127.0.0.0/8

option redispatch

retries 1

timeout http-request 10s

timeout queue 20s

timeout connect 5s

timeout client 20s

timeout server 20s

timeout http-keep-alive 10s

timeout check 10s

#---------------------------------------------------------------------

# apiserver frontend which proxys to the masters

#---------------------------------------------------------------------

frontend apiserver

bind *:${APISERVER_DEST_PORT}

mode tcp

option tcplog

default_backend apiserver

#---------------------------------------------------------------------

# round robin balancing for apiserver

#---------------------------------------------------------------------

backend apiserver

option httpchk GET /healthz

http-check expect status 200

mode tcp

option ssl-hello-chk

balance roundrobin

server ${HOST1_ID} ${HOST1_ADDRESS}:${APISERVER_SRC_PORT} check

# [...]

同样，在bash变量样式中有一些占位符需要替换： ${APISERVER_DEST_PORT} Kubernetes与API服务器对话的端口。 ${APISERVER_SRC_PORT} API服务器实例使用的端口。 ${HOST1_ID}第一个负载均衡的API服务器主机的符号名称。 ${HOST1_ADDRESS}第一个负载均衡API服务器主机的可解析地址（DNS名、IP地址）。 Read more…

By sean, 3 years11/06/2021 ago

KUBERNETES

使用kube-vip做kubespray的apiserver的HA

使用kubespray搭建k8s集群, 如果没有外部LB做高可用的话, 对于node节点, kubespray默认为通过选择nginx或haproxy做apiserver的HA, 在master节点上, 是用127.0.0.1:6443来访问本机的apiserver, 而且程序调用apiserver时也没有做HA, 传统HA方式会搭建keepalived和haproxy, 本次是通过kube-vip来做HA 这是kube-vip官方对于ha的架构文档: https://github.com/kube-vip/kube-vip/blob/main/kubernetes-control-plane.md https://kube-vip.io/architecture/ 尝试着按照官方方法来运行: https://kube-vip.io/install_static/ , 始终失败, 由于docker run –network host –rm ghcr.io/kube-vip/kube-vip:v0.3.8 manifest pod的方式生成的配置默认是用的/etc/kubernetes/admin.conf来做backend的, 但是kubespray init集群时也是用的admin.conf来做认证的, 这就陷入了死循环. 后来通过直接配置文件的方式来做成功了. 参考: https://github.com/kube-vip/kube-vip/blob/main/kubernetes-control-plane.md https://www.codeleading.com/article/58065570523/ ansible如下: 注: 其中,templates/kube-vip.yaml.j2这个文件是通过这个来生成的: docker run –network host –rm 172.20.48.169:81/yks/kube-vip:v0.3.9 sample config | sudo tee /etc/kubernetes/manifests/kube-vip.yaml kubespray定义的默认k8s apiserver监听端口为6443(变量: kube_apiserver_port), 这里kube-vip定义的vip绑定的域名是k8s.apiserver.io, Read more…

By sean, 3 years11/06/2021 ago

M	T	W	T	F	S	S
1	2	3	4	5	6	7
8	9	10	11	12	13	14
15	16	17	18	19	20	21
22	23	24	25	26	27	28
29	30

未分类

通过ansible获取文件的几种方式

KUBERNETES

Kubernetes高可用性的考虑

KUBERNETES

使用kube-vip做kubespray的apiserver的HA