yq介绍
在开始使用yq之前，我们首先需要安装它。当你在谷歌上搜索yq时，你会发现两个项目/存储库。
首先，在https://github.com/kislyuk/yq是jq（JSON处理器）的包装器。如果你已经熟悉jq，你可能想抓住这个，使用你已经知道的语法。
不过，在本文中，我们将使用https://github.com/mikefarah/yq 这个版本，这个版本没有100%匹配jq语法，但它的优点是它没有依赖性（不依赖于jq），有关差异的更多上下文，请参阅下面的GitHub问题。
https://github.com/mikefarah/yq/issues/193
mikefarah版本的yq的具体用法参见文档：https://mikefarah.gitbook.io/yq/
工作实践
业务需求
由于公司产品需要部署在无网环境，所以需要制作适配各个组件的离线安装源，包括rpm/apt之类的系统相关软件包，以及各个二进制/压缩包之类的离线文件，原来是做法是统一将所有的离线源统一整合到一个yml文件当中去，现在的做法是将一个名为packages.yaml的文件放到各个组件的工程里，通过git下载然后进行聚合生成最后的全量列表。
聚合之前，通过生成一个config.yaml来进行选择哪些组件的聚合，下面是全量的配置：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
--- downMode: git project: - name: offline-packages url: https://git.xxx.com/PaaS/offline-packages.git branch: autoupdate-git item: - name: yks url: https://git.xxx.com/PaaS/paas-installer.git branch: develop item: - name: middleware url: https://git.xxx.com/PaaS/middleware.git branch: develop item: - redis - nginx - elasticsearch - kafka - kibana - license-server - minio |
通过流水线传参，生成以分号为分隔符的参数，来自定义生成config_custom.yaml:
|
1 2 3 4 5 6 7 8 9 10 11 12 |
# 生成聚合用的config.yml if [ "$middleware_names" != "all" ]; then middleware_pattern="middleware|$(echo ${middleware_names} |sed 's/;/\|/g')" # 只保留选择的大类，如'yks, middleware, offline-packages' yq 'del(.project[] | select(.name | test("'$middleware_pattern'")|not))' config.yml > ./config_custom_tmp.yml # 再次筛选, 只保留middleware大类下的小类,如'redis, nginx' yq 'del(.project[].item.[] | select(. |test("'$middleware_pattern'")|not) )' config_custom_tmp.yml > ./config_custom.yml config_file=config_custom.yml rm -rf config_custom_tmp.yml else config_file=config.yml fi |
解析：
由于yq的contains不支持多个匹配，所以在这里用到了test来测试是否包含以‘|’分隔的多个字符
del的用法：
用del删除之前，先要能查询打印出需要删除的部分，才能进一步使用del删除，如：
如果选择了只生成middleware和paas-installer, 则先查出不包含middleware和paas-installer的部分：
|
1 2 3 4 5 |
[root@jenkins1-iuap-hb2-ali offline-packages]# cat config.yml |yq '.project[]| select(.name | test("middleware|yks")|not)' name: offline-packages url: https://git.xxx.com/paas/offline-packages.git branch: develop item: |
然后调用del删除：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
[root@jenkins1 offline-packages]# cat config.yml |yq 'del(.project[]| select(.name | test("middleware|yks")|not))' --- downMode: git project: - name: paas url: https://git.xxx.com/PaaS/paas-installer.git branch: develop item: - name: middleware url: https://git.xxx.com/PaaS/middleware.git branch: develop item: - redis - nginx - elasticsearch - kafka - kibana - license-server - minio |
如果只选择了paas，nginx和redis,则需再次筛选, 只保留middleware大类下的小类,如’redis, nginx’:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
[root@jenkins1 offline-packages]# yq 'del(.project[].item.[] | select(. |test("redis|nginx")|not) )' config_custom_tmp.yml --- downMode: git project: - name: yks url: https://git.xxx.com/PaaS/paas-installer.git branch: develop item: [] - name: middleware url: https://git.xxx.com/PaaS/middleware.git branch: develop item: - redis - nginx |
osInfo=$(awk ‘BEGIN{split(“‘${os_names}'”, arr, “,”);for(key in arr) if(arr[key] ~ “ubuntu”) print arr[key]”:apt”;else print arr[key]”:yum”}’)
最后生成的格式如下：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 |
os: kylin version: v10-sp3 package: - name: - telnet - curl - wget - name: - docker-ce-20.10.17 - docker-ce-cli-20.10.17 - containerd.io-1.6.28 arch: amd64 repo: - https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo - http://mirrors.aliyun.com/repo/Centos-7.repo replace: - item: repo from: $releasever to: "7" - name: - docker-ce-20.10.17 - docker-ce-cli-20.10.17 - containerd.io-1.6.28 arch: arm64 repo: - https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo - http://mirrors.aliyun.com/repo/Centos-altarch-7.repo replace: - item: repo from: $releasever to: "7" file: - name: cni-plugins arch: amd64 src: http://bucket.oss-cn-beijing.aliyuncs.com/download/nexus2/raw-apis/containernetworking/plugins/releases/download/v1.1.1/cni-plugins-linux-amd64-v1.1.1.tgz dest: raw-apis/containernetworking/plugins/releases/download/v1.1.1/cni-plugins-linux-amd64-v1.1.1.tgz |
需要再次通过yq来进行提取过滤:
- 提取不需要配repo的以及带arch=amd64/x86_64或不带key为arch的列表：
|
1 |
yq -o=j -I=0 '.package[] | select(. | has("repo")|not) | select(.arch == "amd64" or .arch == "x86_64" or has("arch")|not)' packages.yaml |
其中-o=j表示输出是json格式，-I=0代表sets indent level for output为0，默认是2，输出结果是：
|
1 2 3 4 5 6 7 8 |
{ "name": [ "telnet", "curl", "wget", ... ] } |
- 提取需要配repo的(带key为repo的)以及带arch=amd64/x86_64或不带key为arch的列表：
|
1 |
yq -o=j -I=0 '.package[] | select(. | contains({"repo":""})) | select(.arch == "amd64" or .arch == "x86_64" or has("arch")|not) ' packages.yaml |
因为rhel 8的docker repo不同，聚合生成了好几个带repo的docker安装列表：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 |
os: rhel version: "8.8" package: - name: - telnet - name: - docker-ce-20.10.17 - docker-ce-cli-20.10.17 - containerd.io-1.6.28 arch: amd64 repo: - https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo - http://mirrors.aliyun.com/repo/Centos-7.repo replace: - item: repo from: $releasever to: "7" - name: - docker-ce-20.10.17 - docker-ce-cli-20.10.17 - containerd.io-1.6.28 arch: arm64 repo: - https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo - http://mirrors.aliyun.com/repo/Centos-altarch-7.repo replace: - item: repo from: $releasever to: "7" - name: - ntp - ntpdate - libselinux-python repo: - https://mirrors.aliyun.com/repo/Centos-7.repo replace: - item: repo from: $releasever to: "7" - name: - docker-ce-25.0.3 - docker-ce-cli-25.0.3 - containerd.io-1.6.28 arch: amd64 repo: - https://mirrors.aliyun.com/docker-ce/linux/rhel/docker-ce.repo - http://mirrors.aliyun.com/repo/Centos-8.repo replace: - item: repo from: $releasever to: "8" |
需要只提取rhel安装用的：
|
1 |
yq 'del( .package[] | select(. | contains({"repo":""})) | select(.name[] =="*docker*") | select(.repo[] == "*/linux/centos/*") )' all_packages.yaml |
下面是构建过程中用的Dockerfile:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 |
ARG FROM="" FROM $FROM as packages ARG FROM="" WORKDIR /packages # download packages RUN set -x \ && ARCH=$(uname -m); echo "ARCH=$ARCH" > .info \ && ARCH_ALIAS=$(echo $ARCH | sed 's/x86_64/amd64/;s/aarch64/arm64/'); echo "ARCH_ALIAS=$ARCH_ALIAS" >> .info \ && BASE_URL=http://bucket.oss.aliyuncs.com/download \ && image=$(echo $FROM|awk -F'/' '{print $NF}') \ && image_name=$(echo $image|awk -F':' '{print $1}') \ && image_tag=$(echo $image|awk -F':' '{print $2}') \ && os_name=${image_name} \ && os_version=$(echo $image_tag|sed 's/-arm64//;s/-amd64//') \ && echo "os_name=$os_name" >> .info \ && echo "image_name=$image_name" >> .info \ && echo "image_tag=$image_tag" >> .info \ && echo "os_version=$os_version" >> .info # && curl -o /usr/bin/yq $BASE_URL/binary/yq_linux_$ARCH && chmod +x /usr/bin/yq COPY all_packages.yaml . RUN set -x; echo "download package with no custom repo firstly" \ && source ./.info \ && declare -A packages \ && packages=$(yq -o=j -I=0 '.package[] | select(. | has("repo")|not) | select(.arch == "'$ARCH_ALIAS'" or .arch == "'$ARCH'" or has("arch")|not)' all_packages.yaml) \ && for pkg in ${packages}; do \ if (which rpm &> /dev/null); then \ echo "$pkg"|yq '.name[]' | sort -u | xargs repotrack -d 9 --destdir "$os_name/$os_version/os/$ARCH/Packages"; \ elif (which apt &> /dev/null); then \ echo $(echo "$pkg"|yq '.name[]') $(dpkg --get-selections | grep -v deinstall | cut -f1 | cut -d ':' -f1) | sed -e 's/ntp\|ntpdate/ /g' | \ sort -u | xargs apt-get install --reinstall --print-uris | awk -F "'" '{print $2}' | grep -v '^$' > packages_1.urls; \ apt-get install --reinstall --print-uris ntp ntpdate | awk -F "'" '{print $2}' | grep -v '^$' >> packages_1.urls; \ wget -q -x -P $os_name/$os_version -i packages_1.urls; \ fi; \ done RUN set -x; echo "download package with custom repo sencondly" \ && source ./.info \ && declare -A packagesWithRepo \ && packagesWithRepo=$(yq -o=j -I=0 '.package[] | select(. | contains({"repo":""})) | select(.arch == "'$ARCH_ALIAS'" or .arch == "'$ARCH'" or has("arch")|not) ' all_packages.yaml) \ && for pkg in ${packagesWithRepo}; do \ repo=$(echo "$pkg"|yq '.repo[]'); \ replace=$(echo "$pkg"|yq -o=j -I=0 '.replace[]'); \ echo "$repo"|while read ro; do \ if (which rpm &> /dev/null); then \ repo_name=${ro##*/}; \ curl -k -o /etc/yum.repos.d/${repo_name} ${ro}; \ for r in ${replace}; do \ from=$(echo "$r"|yq '.from'); \ to=$(echo "$r"|yq '.to'); \ sed -i "s@$from@$to@g" /etc/yum.repos.d/${repo_name}; \ done; \ elif (which apt &> /dev/null); then \ eval echo "$ro" > /etc/apt/sources.list.d/custom.list; \ fi; \ done; \ if (which rpm &> /dev/null); then \ yum makecache; \ echo "$pkg"|yq '.name[]' | sort -u | xargs repotrack -d 9 --destdir "$os_name/$os_version/os/$ARCH/Packages"; \ elif (which apt &> /dev/null); then \ echo "$pkg"|yq '.name[]' | xargs apt-get --allow-unauthenticated install --reinstall --print-uris | awk -F "'" '{print $2}' | grep -v '^$' | sort -u > packages_2.urls; \ wget -q -x -P $os_name/$os_version -i packages_2.urls; \ fi; \ done # create repo RUN set -x \ && source ./.info \ && if (which rpm &> /dev/null); then \ createrepo -d "$os_name/$os_version/os/$ARCH"; \ if [ "$image_name" == "rhel" -o "$image_name" == "nfs" ]; then \ # 修复：No available modular metadata for modular package repo2module -s stable $os_name/$os_version/os/$ARCH modules.yaml \ && modifyrepo_c --mdtype=modules modules.yaml $os_name/$os_version/os/$ARCH/repodata/ \ && rm -rf modules.yaml; \ fi; \ printf '\ ['$os_name''$os_version'] \n\ name='$os_name''$os_version' \n\ baseurl=http://{{ nexus_access_address|default(127.0.0.1) }}/nexus/content/repositories/'$os_name'/'$os_version'/os/$basearch/ \n\ enabled=1 \n\ gpgcheck=0 \n\ sslverify=0 \n\ ' > ${os_name}.repo.j2; \ elif (which apt &> /dev/null); then \ dpkg-scanpackages $os_name/$os_version | gzip -9c > $os_name/$os_version/Packages.gz; \ printf '\ deb [trusted=yes] http://{{ nexus_access_address|default(127.0.0.1) }}/nexus/content/repositories/'$os_name'/'$ARCH_ALIAS' '$os_version'/ \n\ ' > ${os_name}.list.j2; \ fi \ \ # clean finaly && rm -rf packages_1.urls packages_2.urls # download files RUN set -x;source ./.info \ && item=$(yq -o=j -I=0 '.file[] | select(.arch == "'$ARCH_ALIAS'" or .arch == "'$ARCH'" or has("arch")|not)' all_packages.yaml) \ && if [ "X$item" != "X" ]; then \ for i in $item; do \ src=$(echo "$i"|yq '.src'); \ dest=$(echo "$i"|yq '.dest'); \ dest_path=${dest%/*}; \ # dest_file=${dest##*/}; \ mkdir -p files/$dest_path; \ wget -c -q -O files/$dest "$src"; \ done; \ fi FROM scratch COPY --from=packages /packages ./packages |
0 Comments