sean

webmin安装及模块proftpd

在arm kylin服务器上安装webmin 1 官网下载最新的webmin-2.013.tar.gz，并解压到/usr/local/webmin/ 2 运行/usr/local/webmin/setup.sh 安装proftpd：下载和编译安装proftpd:

wget ftp://ftp.proftpd.org/distrib/source/proftpd-1.3.8.tar.gz
tar -zxvf proftpd-1.3.8.tar.gz
cd proftpd-1.3.8
./configure --prefix=/usr/local/proftpd --sysconfdir=/usr/local/proftpd
make  && make install

1

2

3

4

5

wget ftp://ftp.proftpd.org/distrib/source/proftpd-1.3.8.tar.gz

tar -zxvf proftpd-1.3.8.tar.gz

cd proftpd-1.3.8

./configure --prefix=/usr/local/proftpd --sysconfdir=/usr/local/proftpd

make && make install

编写service启动文件：

[root@installer55 proftpd-1.3.8]# cat /usr/lib/systemd/system/proftpd.service 
[Unit]
Description = ProFTPD FTP Server
After = network.target nss-lookup.target local-fs.target remote-fs.target

[Service]
Type = forking
PIDFile = /usr/local/proftpd/var/proftpd.pid
Environment = PROFTPD_OPTIONS=
EnvironmentFile = -/etc/sysconfig/proftpd
ExecStart = /usr/sbin/proftpd $PROFTPD_OPTIONS
ExecReload = /bin/kill -HUP $MAINPID

[Install]
WantedBy = multi-user.target

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

[root@installer55 proftpd-1.3.8]# cat /usr/lib/systemd/system/proftpd.service

[Unit]

Description = ProFTPD FTP Server

After = network.target nss-lookup.target local-fs.target remote-fs.target

[Service]

Type = forking

PIDFile = /usr/local/proftpd/var/proftpd.pid

Environment = PROFTPD_OPTIONS=

EnvironmentFile = -/etc/sysconfig/proftpd

ExecStart = /usr/sbin/proftpd $PROFTPD_OPTIONS

ExecReload = /bin/kill -HUP $MAINPID

[Install]

WantedBy = multi-user.target

启动和错误处理

1、启动
  systemctl start proftpd
2、启动会报错
  这里报错：- Fatal: Group: Unknown group &#39;nogroup&#39;.
3、处理修改配置文件
   vi /usr/local/proftpd/proftpd.conf
   --修改前
   # Set the user and group under which the server will run.
  User                             nobody
  Group                            nogroup
  --修改后
  # Set the user and group under which the server will run.
  User                             nobody
  Group                            nobody
4、再次启动即可

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

1、启动

systemctl start proftpd

2、启动会报错

这里报错：- Fatal: Group: Unknown group 'nogroup'.

3、处理修改配置文件

vi /usr/local/proftpd/proftpd.conf

--修改前

# Set the user and group under which the server will run.

User nobody

Group nogroup

--修改后

# Set the user and group under which the server will run.

User nobody

Group nobody

4、再次启动即可

配置用户及目录

1、建目录和用户
  建立目录 mkdir -p /data/
  建立用户 useradd -d /data/ftp -s /sbin/nologin ftp_user
  配置密码 echo ftp_user|passwd --stdin ftp_user
 修改权限 chown -R ftp_user:ftp_user /data/ftp
2、添加配置文件，在配置文件末尾添加如下代码
 vi /usr/local/proftpd/proftpd.conf
  <Directory /data/ftp>
     HideNoAccess off
      <Limit STOR MKD>
       Order deny,allow
       allowUser  ftp_user
      </Limit>
  </Directory>

3、重启服务 

4、登录ftp查看
可以在windows文件管理器里打开ftp://x.x.x.x/
可以用wget命令下载：
wget -nH -m -r --ftp-user=ftp_user --ftp-password='ftp_user'  ftp://x.x.x.x/aarch64-kylin/20230224/a.txt

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

1、建目录和用户

建立目录 mkdir -p /data/

建立用户 useradd -d /data/ftp -s /sbin/nologin ftp_user

配置密码 echo ftp_user|passwd --stdin ftp_user

修改权限 chown -R ftp_user:ftp_user /data/ftp

2、添加配置文件，在配置文件末尾添加如下代码

vi /usr/local/proftpd/proftpd.conf

HideNoAccess off

Order deny,allow

allowUser ftp_user

</Limit>

</Directory>

3、重启服务

4、登录ftp查看

可以在windows文件管理器里打开ftp://x.x.x.x/

可以用wget命令下载：

wget -nH -m -r --ftp-user=ftp_user --ftp-password='ftp_user' ftp://x.x.x.x/aarch64-kylin/20230224/a.txt

登录报错： Invalid shell: ‘/usr/sbin/nologin’ 解决： vim /etc/shells 结尾添加： /usr/sbin/nologin /sbin/nologin 下载安装proftpd webmin模块： 2 wget https://download.webmin.com/download/modules/proftpd.wbm.gz 2 打开webmin页面上的模块管理，从页面上选择服务器路径进行模块安装

By sean, 2 years02/24/2023 ago

OS

WSL报：The attempted operation is not supported for the type of object referenced

在windows上打开wsl ubuntu的时候，报错： The attempted operation is not supported for the type of object referenced. 解决方式：需要在注册表添加一个key 创建一个名为wsl2.reg（或任何其他带有 .reg 后缀的文件名）的文件在中打开此文件notepad.exe，将下面的文本粘贴到其中。保存存档。双击这个文件，给它Administrator权限打开wsl.exe wsl2.reg 代码，我从这个问题复制

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\WinSock2\Parameters\AppId_Catalog\0408F7A3]
"AppFullPath"="C:\\\\windows\\\\system32\\\\wsl.exe"
"PermittedLspCategories"=dword:80000000

1

2

3

4

5

6

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\WinSock2\Parameters\AppId_Catalog\0408F7A3]

"AppFullPath"="C:\\\\windows\\\\system32\\\\wsl.exe"

"PermittedLspCategories"=dword:80000000

或者你可以wsl2.reg直接下载我的： https://baijifeilong.github.io/wsl2.reg 参考： https://jon.sprig.gs/blog/post/2140 https://github.com/microsoft/WSL/issues/4177

By sean, 2 years02/23/2023 ago

OS

vim使用动图

By sean, 2 years ago

MIDDLEWARE

Jenkinsfile配置submodule

最近在搞Jenkinsfile进行项目自动构建，有一个项目配置了远程子模块依赖，遇到了很多问题，如下是尝试的步骤： 1, 利用checkbox

    stages {
        stage('拉取代码'){
            steps {
                  checkout([
                    $class: 'GitSCM', 
                    branches: [[name: "${params.xxx_branch}"]], 
                    doGenerateSubmoduleConfigurations: false, 
                    extensions: [[
                      $class: 'SubmoduleOption',
                      disableSubmodules: false,
                      parentCredentials: true,
                      recursiveSubmodules: true,
                      reference: '',
                      trackingSubmodules: false
                  ]],
                    submoduleCfg: [], 
                    userRemoteConfigs: [[credentialsId: "${git_auth}", url: "${xxx_git_address}"]]
                  ])
            }
        }

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

stages {

stage('拉取代码'){

steps {

checkout([

$class: 'GitSCM',

branches: [[name: "${params.xxx_branch}"]],

doGenerateSubmoduleConfigurations: false,

extensions: [[

$class: 'SubmoduleOption',

disableSubmodules: false,

parentCredentials: true,

recursiveSubmodules: true,

reference: '',

trackingSubmodules: false

]],

submoduleCfg: [],

userRemoteConfigs: [[credentialsId: "${git_auth}", url: "${xxx_git_address}"]]

])

}

报错：

Caused: java.io.IOException: Could not perform submodule update

1	Caused: java.io.IOException: Could not perform submodule update

2. 尝试配置公钥认证

stage('构建'){
  steps {
      container (JOB_NAME) {
        withCredentials([
          sshUserPrivateKey(credentialsId: 'jenkins_server_pri_key', keyFileVariable: 'SSH_KEY')
          ]) {
          sh 'GIT_SSH_COMMAND="ssh -i $SSH_KEY" git submodule update --init --recursive'

1

2

3

4

5

6

7

stage('构建'){

steps {

container (JOB_NAME) {

withCredentials([

sshUserPrivateKey(credentialsId: 'jenkins_server_pri_key', keyFileVariable: 'SSH_KEY')

]) {

sh 'GIT_SSH_COMMAND="ssh -i $SSH_KEY" git submodule update --init --recursive'

报错：

+ GIT_SSH_COMMAND=ssh -i **** git submodule update --init --recursive
fatal: detected dubious ownership in repository at '/home/jenkins/agent/workspace/xxx-nginx-downloads'
To add an exception for this directory, call:

	git config --global --add safe.directory /home/jenkins/agent/workspace/xxx-nginx-downloads

1

2

3

4

5

+ GIT_SSH_COMMAND=ssh -i **** git submodule update --init --recursive

fatal: detected dubious ownership in repository at '/home/jenkins/agent/workspace/xxx-nginx-downloads'

To add an exception for this directory, call:

git config --global --add safe.directory /home/jenkins/agent/workspace/xxx-nginx-downloads

需要增加 git config –global –add safe.directory ${WORKSPACE} 报错：

+ GIT_SSH_COMMAND='ssh -i ****'
+ git config --global --add safe.directory /home/jenkins/agent/workspace/plugin-xxx-cluster-nginx-downloads
+ git submodule update --init --recursive
Submodule 'xx-plugin' (http://git.xxx.com/xxx/xxx-plugin) registered for path 'xxx-plugin'
Submodule 'xxx-downloader' (http://git.xxx.com/xxx/xxx-downloader.git) registered for path 'xxx-downloader'
Cloning into '/home/jenkins/agent/workspace/plugin-xxx-cluster-nginx-downloads/xxx-plugin'...
fatal: could not read Username for 'http://git.xxx.com': No such device or address
fatal: clone of 'http://git.xxx.com/iUAP_gPaaS/xxx-plugin' into submodule path '/home/jenkins/agent/workspace/plugin-xxx-cluster-nginx-downloads/xxx-plugin' failed
Failed to clone 'xxx-plugin'. Retry scheduled

1

2

3

4

5

6

7

8

9

+ GIT_SSH_COMMAND='ssh -i ****'

+ git config --global --add safe.directory /home/jenkins/agent/workspace/plugin-xxx-cluster-nginx-downloads

+ git submodule update --init --recursive

Submodule 'xx-plugin' (http://git.xxx.com/xxx/xxx-plugin) registered for path 'xxx-plugin'

Submodule 'xxx-downloader' (http://git.xxx.com/xxx/xxx-downloader.git) registered for path 'xxx-downloader'

Cloning into '/home/jenkins/agent/workspace/plugin-xxx-cluster-nginx-downloads/xxx-plugin'...

fatal: could not read Username for 'http://git.xxx.com': No such device or address

fatal: clone of 'http://git.xxx.com/iUAP_gPaaS/xxx-plugin' into submodule path '/home/jenkins/agent/workspace/plugin-xxx-cluster-nginx-downloads/xxx-plugin' failed

Failed to clone 'xxx-plugin'. Retry scheduled

3. 重新添加覆盖 .gitmodules

stage('构建'){
  steps {
      container (JOB_NAME) {
        withCredentials([
          usernamePassword(credentialsId: "${git_auth}", passwordVariable: 'git_password', usernameVariable: 'git_username'),
          ]) {
          sh '''#!/bin/bash
            set -ex
            git config --global --add safe.directory ${WORKSPACE}
            git submodule add -f http://$git_username:$git_password@git.xxx.com/xxx/xxx-plugin xxx-plugin || true
            git submodule add -f http://$git_username:$git_password@git.xxx.com/xxx/api-station.git api-station || true
            git submodule add -f http://$git_username:$git_password@git.xxx.com/xxx/xxx-downloader.git xxx-downloader || true
            cat .gitmodules
            git submodule update --init --recursive

1

2

3

4

5

6

7

8

9

10

11

12

13

14

stage('构建'){

steps {

container (JOB_NAME) {

withCredentials([

usernamePassword(credentialsId: "${git_auth}", passwordVariable: 'git_password', usernameVariable: 'git_username'),

]) {

sh '''#!/bin/bash

set -ex

git config --global --add safe.directory ${WORKSPACE}

git submodule add -f http://$git_username:$git_password@git.xxx.com/xxx/xxx-plugin xxx-plugin || true

git submodule add -f http://$git_username:$git_password@git.xxx.com/xxx/api-station.git api-station || true

git submodule add -f http://$git_username:$git_password@git.xxx.com/xxx/xxx-downloader.git xxx-downloader || true

cat .gitmodules

git submodule update --init --recursive

报错： fatal: could not read Username for ‘http://git.xxx.com‘: No such device or address 4. 修改.gitmodules 下面是原来的

$ cat .gitmodules
[submodule "xxx-plugin"]
        path = xxx-plugin
        url = http://git.xxx.com/xxx/xxx-plugin
[submodule "api-station"]
        path = api-station
        url = http://git.xxx.com/xxx/api-station.git
[submodule "xxx-downloader"]
        path = xxx-downloader
        url = http://git.xxx.com/xxx/xxx-downloader.git

1

2

3

4

5

6

7

8

9

10

$ cat .gitmodules

[submodule "xxx-plugin"]

path = xxx-plugin

url = http://git.xxx.com/xxx/xxx-plugin

[submodule "api-station"]

path = api-station

url = http://git.xxx.com/xxx/api-station.git

[submodule "xxx-downloader"]

path = xxx-downloader

url = http://git.xxx.com/xxx/xxx-downloader.git

通过sed进行修改

stage('构建'){
      steps {
          container (JOB_NAME) {
            withCredentials([
              usernamePassword(credentialsId: "${git_auth}", passwordVariable: 'git_password', usernameVariable: 'git_username'),
              usernamePassword(credentialsId: "${webmin_auth}", passwordVariable: 'webmin_password', usernameVariable: 'webmin_username')
              ]) {
              sh '''#!/bin/bash
                set -ex
                git config --global --add safe.directory ${WORKSPACE}
                git config --global --add safe.directory ${WORKSPACE}/xxx-plugin
                git config --global --add safe.directory ${WORKSPACE}/api-station
                git config --global --add safe.directory ${WORKSPACE}/xxx-downloader
                sed -i -E "s/(^\\s*.url = http:\\/\\/)(.*)$/\\1$git_username:$git_password@\\2/g" .gitmodules
                cat .gitmodules
                git submodule update --init --recursive

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

stage('构建'){

steps {

container (JOB_NAME) {

withCredentials([

usernamePassword(credentialsId: "${git_auth}", passwordVariable: 'git_password', usernameVariable: 'git_username'),

usernamePassword(credentialsId: "${webmin_auth}", passwordVariable: 'webmin_password', usernameVariable: 'webmin_username')

]) {

sh '''#!/bin/bash

set -ex

git config --global --add safe.directory ${WORKSPACE}

git config --global --add safe.directory ${WORKSPACE}/xxx-plugin

git config --global --add safe.directory ${WORKSPACE}/api-station

git config --global --add safe.directory ${WORKSPACE}/xxx-downloader

sed -i -E "s/(^\\s*.url = http:\\/\\/)(.*)$/\\1$git_username:$git_password@\\2/g" .gitmodules

cat .gitmodules

git submodule update --init --recursive

最后拉取成功。有时间再细细研究原因

By sean, 2 years02/21/2023 ago

HANDBOOK

调用webmin上传下载文件

需求：构建的包需要上传到文件服务器上，需要从文件服务器上进行下载可以搭建一个webmin服务，既有UI界面，又可以通过模拟调用接口进行上传下载操作一安装通过yum安装：

# 安装依赖
yum install perl-Crypt-Blowfish perl-Crypt-CBC perl-Crypt-Rijndael perl-Digest-SHA perl-Expect perl-Gtk2 perl-IO-Stty perl-Net-ARP perl-YAML perl-Cairo perl-Glib perl-IO-Tty perl-Pango uuid-perl
curl -o setup-repos.sh https://raw.githubusercontent.com/webmin/webmin/master/setup-repos.sh
bash setup-repos.sh
yum install webmin

1

2

3

4

5

# 安装依赖

yum install perl-Crypt-Blowfish perl-Crypt-CBC perl-Crypt-Rijndael perl-Digest-SHA perl-Expect perl-Gtk2 perl-IO-Stty perl-Net-ARP perl-YAML perl-Cairo perl-Glib perl-IO-Tty perl-Pango uuid-perl

curl -o setup-repos.sh https://raw.githubusercontent.com/webmin/webmin/master/setup-repos.sh

bash setup-repos.sh

yum install webmin

也可以通过下载安装包安装，适用于国产操作系统，因为官方安装脚本不支持kylin等系统

wget https://jaist.dl.sourceforge.net/project/webadmin/webmin/2.013/webmin-2.013.tar.gz
tar -zxPf webmin-2.013.tar.gz -C /usr/local/webmin --strip-components 1
/usr/local/webmin/setup.sh

1

2

3

wget https://jaist.dl.sourceforge.net/project/webadmin/webmin/2.013/webmin-2.013.tar.gz

tar -zxPf webmin-2.013.tar.gz -C /usr/local/webmin --strip-components 1

/usr/local/webmin/setup.sh

配置：

/etc/webmin/config：
referers_none=0  # 可以不需要传referers而访问
gotomodule=filemin # 默认进filemin模块
gotoone=1 # 当用户默认只有一个模块时，直接进入

1

2

3

4

/etc/webmin/config：

referers_none=0 # 可以不需要传referers而访问

gotomodule=filemin # 默认进filemin模块

gotoone=1 # 当用户默认只有一个模块时，直接进入

上传下载：

server=http://x.x.x.x:10000
user='xxx'
pass='xxx'
file=/data/test.tar.gz
# 生成cookie，用来登录
curl -c 'cookie.txt' -k $server/session_login.cgi -d user=$user -d pass=$pass -H 'Cookie: testing=1'
#下载方式1：
curl -O -b 'cookie.txt' --progress-bar "$server/updown/fetch.cgi$file"
# 下载方式2：
curl -b 'cookie.txt' --progress-bar "$server/filemin/download.cgi?file=/z-md5.txt&path=/aarch64-kylin/20220430/" -o z-md5.txt
# 可以加下载失败重试：
export ec=18; while [ $ec -eq 18 ]; do curl -b 'cookie.txt' --progress-bar "$server/filemin/download.cgi?file=/z-md5.txt&path=/aarch64-kylin/20220430/" -o z-md5.txt; export ec=$?; done

# 上传
curl -X POST -b 'cookie.txt' --progress-bar -F 'filename=@/tmp/sean.txt'  "$server/filemin/upload.cgi?path=/"

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

server=http://x.x.x.x:10000

user='xxx'

pass='xxx'

file=/data/test.tar.gz

# 生成cookie，用来登录

curl -c 'cookie.txt' -k $server/session_login.cgi -d user=$user -d pass=$pass -H 'Cookie: testing=1'

#下载方式1：

curl -O -b 'cookie.txt' --progress-bar "$server/updown/fetch.cgi$file"

# 下载方式2：

curl -b 'cookie.txt' --progress-bar "$server/filemin/download.cgi?file=/z-md5.txt&path=/aarch64-kylin/20220430/" -o z-md5.txt

# 可以加下载失败重试：

export ec=18; while [ $ec -eq 18 ]; do curl -b 'cookie.txt' --progress-bar "$server/filemin/download.cgi?file=/z-md5.txt&path=/aarch64-kylin/20220430/" -o z-md5.txt; export ec=$?; done

# 上传

curl -X POST -b 'cookie.txt' --progress-bar -F 'filename=@/tmp/sean.txt' "$server/filemin/upload.cgi?path=/"

By sean, 2 years02/13/2023 ago

KUBERNETES

使用 GitHub Actions 编译 kubernetes 组件

在使用 kubernetes 过程中由于某些需求往往要修改一下 k8s 官方的源码，然后重新编译才行。本文就以修改 kubeadm 生成证书为默认 10 年为例，来讲解如何使用 GitHub Actions 来编译和发布生成的二进制文件。构建 clone repo 将 kubernetes 官方源码 fork 到自己的 repo 中

$ git clone https://github.com/rainingwalk/kubernetes.git 
$ cd kubernetes 
$ git remote add upstream https://github.com/kubernetes/kubernetes.git 
$ git fetch --all 
$ git checkout upstream/release-1.21 
$ git checkout -B kubeadm-1.21

1

2

3

4

5

6

$ git clone https://github.com/rainingwalk/kubernetes.git

$ cd kubernetes

$ git remote add upstream https://github.com/kubernetes/kubernetes.git

$ git fetch --all

$ git checkout upstream/release-1.21

$ git checkout -B kubeadm-1.21

workflow .github/workflows/kubeadm.yaml

---
name: Build kubeadm binary

on:
  push:
    tag:
      - 'v*'
jobs:
  build:
    runs-on: ubuntu-20.04
    # 使此git action在release时有写权限，不然会报GitHub release failed with status: 403 undefined
    permissions:
      contents: write
    # 这里我们选择以 tag 的方式触发 job 的运行
    if: startsWith(github.ref, 'refs/tags/')
    steps:
      - name: Checkout
        uses: actions/checkout@v2

      # 运行 build/run.sh 构建脚本来编译相应平台上的二进制文件
      - name: Build kubeadm binary
        shell: bash
        run: |
          bash -x build/run.sh make kubeadm KUBE_BUILD_PLATFORMS=linux/amd64
          bash -x build/run.sh make kubeadm KUBE_BUILD_PLATFORMS=linux/arm64

      # 构建好的二进制文件存放在 _output/dockerized/bin/ 中
      # 我们根据二进制目标文件的系统名称+CPU体系架构名称进行命名
      - name: Prepare for upload
        shell: bash
        run: |
          mv _output/dockerized/bin/linux/amd64/kubeadm kubeadm-linux-amd64
          mv _output/dockerized/bin/linux/arm64/kubeadm kubeadm-linux-arm64
          sha256sum kubeadm-linux-{amd64,arm64} > sha256sum.txt
          
      # 使用 softprops/action-gh-release 来将构建产物上传到 GitHub release 当中
      - name: Release and upload packages
        uses: softprops/action-gh-release@v1
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        with:
          files: |
            sha256sum.txt
            kubeadm-linux-amd64
            kubeadm-linux-arm64

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

---

name: Build kubeadm binary

on:

push:

tag:

- 'v*'

jobs:

build:

runs-on: ubuntu-20.04

# 使此git action在release时有写权限，不然会报GitHub release failed with status: 403 undefined

permissions:

contents: write

# 这里我们选择以 tag 的方式触发 job 的运行

if: startsWith(github.ref, 'refs/tags/')

steps:

- name: Checkout

uses: actions/checkout@v2

# 运行 build/run.sh 构建脚本来编译相应平台上的二进制文件

- name: Build kubeadm binary

shell: bash

run: |

bash -x build/run.sh make kubeadm KUBE_BUILD_PLATFORMS=linux/amd64

bash -x build/run.sh make kubeadm KUBE_BUILD_PLATFORMS=linux/arm64

# 构建好的二进制文件存放在 _output/dockerized/bin/ 中

# 我们根据二进制目标文件的系统名称+CPU体系架构名称进行命名

- name: Prepare for upload

shell: bash

run: |

mv _output/dockerized/bin/linux/amd64/kubeadm kubeadm-linux-amd64

mv _output/dockerized/bin/linux/arm64/kubeadm kubeadm-linux-arm64

sha256sum kubeadm-linux-{amd64,arm64} > sha256sum.txt

# 使用 softprops/action-gh-release 来将构建产物上传到 GitHub release 当中

- name: Release and upload packages

uses: softprops/action-gh-release@v1

env:

GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

with:

files: |

sha256sum.txt

kubeadm-linux-amd64

kubeadm-linux-arm64

build/run.sh : Run a command in a build docker container. Common invocations: build/run.sh make: Build just linux binaries in the container. Pass options and Read more…

By sean, 2 years02/06/2023 ago

KUBERNETES

为什么 kubernetes 环境要求开启 bridge-nf-call-iptables

为什么 kubernetes 环境要求开启 bridge-nf-call-iptables ? 背景 Kubernetes 环境中，很多时候都要求节点内核参数开启 bridge-nf-call-iptables:

sysctl -w net.bridge.bridge-nf-call-iptables=1

1 2	sysctl -w net.bridge.bridge-nf-call-iptables=1

参考官方文档 Network Plugin Requirements 如果不开启或中途因某些操作导致参数被关闭了，就可能造成一些奇奇怪怪的网络问题，排查起来非常麻烦。为什么要开启呢？本文就来跟你详细掰扯下。基于网桥的容器网络 Kubernetes 集群网络有很多种实现，有很大一部分都用到了 Linux 网桥: 每个 Pod 的网卡都是 veth 设备，veth pair 的另一端连上宿主机上的网桥。由于网桥是虚拟的二层设备，同节点的 Pod 之间通信直接走二层转发，跨节点通信才会经过宿主机 eth0。 Service 同节点通信问题不管是 iptables 还是 ipvs 转发模式，Kubernetes 中访问 Service 都会进行 DNAT，将原本访问 ClusterIP:Port 的数据包 DNAT 成 Service 的某个 Endpoint Read more…

By sean, 2 years01/12/2023 ago

NETWORK

calico网络问题排查

问题：今天同事安装了一套k8s环境, 结果发现calico网络互相访问不通，同台机器上的pod之间的calico ip都ping不通分析解决： calico支持三种网络模式，可通过修过calico.yaml进行配置: overlay之ipip overlay之vxlan underlay之BGP overlay — ipip traffic flow overlay — vxlan traffic flow underlay — BGP traffic flow 本环境用的是CrossSubnet且都在同网段，用的是BGP 下面是node信息：

[root@localhost ~]# kubectl get no -owide
NAME                    STATUS   ROLES                                      AGE   VERSION    INTERNAL-IP    EXTERNAL-IP   OS-IMAGE                     KERNEL-VERSION            CONTAINER-RUNTIME
yks1                    Ready    control-plane,ingress,master,node,system   22h   v1.20.15   10.16.245.46   <none>        NFSChina Server 4.0 (G196)   4.19.113-14.nfs4.x86_64   docker://20.10.8
yks2                    Ready    control-plane,ingress,master,node,system   22h   v1.20.15   10.16.245.47   <none>        NFSChina Server 4.0 (G196)   4.19.113-14.nfs4.x86_64   docker://20.10.8
yks3                    Ready    ingress,node,system                        22h   v1.20.15   10.16.245.65   <none>        NFSChina Server 4.0 (G196)   4.19.113-14.nfs4.x86_64   docker://20.10.8

1

2

3

4

5

[root@localhost ~]# kubectl get no -owide

NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME

yks1 Ready control-plane,ingress,master,node,system 22h v1.20.15 10.16.245.46 <none> NFSChina Server 4.0 (G196) 4.19.113-14.nfs4.x86_64 docker://20.10.8

yks2 Ready control-plane,ingress,master,node,system 22h v1.20.15 10.16.245.47 <none> NFSChina Server 4.0 (G196) 4.19.113-14.nfs4.x86_64 docker://20.10.8

yks3 Ready ingress,node,system 22h v1.20.15 10.16.245.65 <none> NFSChina Server 4.0 (G196) 4.19.113-14.nfs4.x86_64 docker://20.10.8

查看yks2上的路由：

[root@yks3 ~]# ip r
default via 10.16.245.1 dev ens3 proto dhcp metric 100 
10.16.245.0/24 dev ens3 proto kernel scope link src 10.16.245.65 metric 100 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
172.23.27.0/24 via 10.16.245.47 dev ens3 proto bird 
172.23.39.0/24 via 10.16.245.34 dev ens3 proto bird 
blackhole 172.23.119.0/24 proto bird 
172.23.119.19 dev cali112980e2900 scope link 
172.23.119.20 dev cali7ad105f6052 scope link 
172.23.119.21 dev cali850bd215908 scope link 
172.23.119.22 dev cali014e534f978 scope link 
172.23.119.23 dev cali90505cacde3 scope link 
172.23.119.24 dev cali1e38a050268 scope link 
172.23.119.25 dev cali55015378d77 scope link 
172.23.119.26 dev caliad0c2e2531c scope link 
172.23.255.0/24 via 10.16.245.46 dev ens3 proto bird

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

[root@yks3 ~]# ip r

default via 10.16.245.1 dev ens3 proto dhcp metric 100

10.16.245.0/24 dev ens3 proto kernel scope link src 10.16.245.65 metric 100

172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown

172.23.27.0/24 via 10.16.245.47 dev ens3 proto bird

172.23.39.0/24 via 10.16.245.34 dev ens3 proto bird

blackhole 172.23.119.0/24 proto bird

172.23.119.19 dev cali112980e2900 scope link

172.23.119.20 dev cali7ad105f6052 scope link

172.23.119.21 dev cali850bd215908 scope link

172.23.119.22 dev cali014e534f978 scope link

172.23.119.23 dev cali90505cacde3 scope link

172.23.119.24 dev cali1e38a050268 scope link

172.23.119.25 dev cali55015378d77 scope link

172.23.119.26 dev caliad0c2e2531c scope link

172.23.255.0/24 via 10.16.245.46 dev ens3 proto bird

可以看到， 172.23.27.0/24 via 10.16.245.47 dev ens3 proto bird 这条路由，172.23.27.0/24这个地址是直接到yks2主机上的

yks             nfs-server-fc7d676c9-8dkq5                 1/1     Running   0          21h     172.23.119.4    yks3 
kube-system     coredns-6d7c6648db-4stmp                   1/1 Running       0          23h     172.23.27.2     yks2

1 2	yks nfs-server-fc7d676c9-8dkq5 1/1 Running 0 21h 172.23.119.4 yks3 kube-system coredns-6d7c6648db-4stmp 1/1 Running 0 23h 172.23.27.2 yks2

在yks2上ping coredns:

[root@yks1 ~]# ping 172.23.27.2
PING 172.23.27.2 (172.23.27.2) 56(84) bytes of data.

1 2	[root@yks1 ~]# ping 172.23.27.2 PING 172.23.27.2 (172.23.27.2) 56(84) bytes of data.

上主机yks2上抓包：首先抓宿主机网卡：

[root@yks2 ~]# tcpdump -i ens3 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes

15:42:16.286242 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 20, length 64
15:42:17.310198 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 21, length 64
15:42:18.334192 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 22, length 64
15:42:19.358197 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 23, length 64
15:42:20.382236 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 24, length 64
15:42:21.410206 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 25, length 64
15:42:22.430279 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 26, length 64
15:42:23.454333 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 27, length 64
15:42:24.478138 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 28, length 64
15:42:25.502182 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 29, length 64
15:42:26.526196 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 30, length 64
15:42:27.550195 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 31, length 64
^C
12 packets captured
13 packets received by filter
0 packets dropped by kernels

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

[root@yks2 ~]# tcpdump -i ens3 icmp

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes

15:42:16.286242 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 20, length 64

15:42:17.310198 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 21, length 64

15:42:18.334192 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 22, length 64

15:42:19.358197 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 23, length 64

15:42:20.382236 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 24, length 64

15:42:21.410206 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 25, length 64

15:42:22.430279 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 26, length 64

15:42:23.454333 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 27, length 64

15:42:24.478138 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 28, length 64

15:42:25.502182 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 29, length 64

15:42:26.526196 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 30, length 64

15:42:27.550195 IP yks1.cluster.local > 172.23.27.2: ICMP echo request, id 29208, seq 31, length 64

^C

12 packets captured

13 packets received by filter

0 packets dropped by kernels

可以接收到包看下yks2的路由：

[root@yks2 ~]# ip r
default via 10.16.245.1 dev ens3 proto dhcp metric 100 
10.16.245.0/24 dev ens3 proto kernel scope link src 10.16.245.47 metric 100 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 
172.18.0.0/16 dev br-34de7dda2079 proto kernel scope link src 172.18.0.1 
blackhole 172.23.27.0/24 proto bird 
172.23.27.1 dev calid894dc87e68 scope link 
172.23.27.2 dev calif945d263abc scope link 
172.23.27.3 dev califc5d5492c2d scope link 
172.23.27.5 dev cali420874a7614 scope link 
172.23.27.7 dev calif2ef5efbec8 scope link 
172.23.27.8 dev calif508601809c scope link 
172.23.27.9 dev calida55188c614 scope link 
172.23.27.10 dev cali3e6ad545e92 scope link 
172.23.27.11 dev cali2c5cdc71fac scope link 
172.23.27.12 dev cali1be0e00985c scope link 
172.23.27.13 dev calif00b7a22fc0 scope link 
172.23.27.14 dev cali56eb38c7099 scope link 
172.23.39.0/24 via 10.16.245.34 dev ens3 proto bird 
172.23.119.0/24 via 10.16.245.65 dev ens3 proto bird 
172.23.255.0/24 via 10.16.245.46 dev ens3 proto bird

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

[root@yks2 ~]# ip r

default via 10.16.245.1 dev ens3 proto dhcp metric 100

10.16.245.0/24 dev ens3 proto kernel scope link src 10.16.245.47 metric 100

172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1

172.18.0.0/16 dev br-34de7dda2079 proto kernel scope link src 172.18.0.1

blackhole 172.23.27.0/24 proto bird

172.23.27.1 dev calid894dc87e68 scope link

172.23.27.2 dev calif945d263abc scope link

172.23.27.3 dev califc5d5492c2d scope link

172.23.27.5 dev cali420874a7614 scope link

172.23.27.7 dev calif2ef5efbec8 scope link

172.23.27.8 dev calif508601809c scope link

172.23.27.9 dev calida55188c614 scope link

172.23.27.10 dev cali3e6ad545e92 scope link

172.23.27.11 dev cali2c5cdc71fac scope link

172.23.27.12 dev cali1be0e00985c scope link

172.23.27.13 dev calif00b7a22fc0 scope link

172.23.27.14 dev cali56eb38c7099 scope link

172.23.39.0/24 via 10.16.245.34 dev ens3 proto bird

172.23.119.0/24 via 10.16.245.65 dev ens3 proto bird

172.23.255.0/24 via 10.16.245.46 dev ens3 proto bird

SECURITY

squid代理上网

squid代理及常见的代理上网本文中涉及两台服务器，这两台服务器均有内网ip地址，分别为

A: 192.168.0.200 
B: 192.168.0.100 
C: 192.168.0.101

1

2

3

A: 192.168.0.200

B: 192.168.0.100

C: 192.168.0.101

其中A具有公网访问能力，B和C不具备公网访问能力。 1、安装squid服务

sudo yum install -y squid

1	sudo yum install -y squid

1、默认安装的squid服务只需要配置一点，默认是拒绝所有服务器连接，只需要修改成允许所有服务器连接即可

# sudo vim /etc/squid/squid.conf
http_access allow all

1 2	# sudo vim /etc/squid/squid.conf http_access allow all

2、启动服务并设置成开机启动

# sudo systemctl start squid.service
# sudo systemctl enable squid.service

1 2	# sudo systemctl start squid.service # sudo systemctl enable squid.service

3、配置受信这样配置之后，squid代理服务器就可以使用了，默认的端口是3128，但是为了安全，只让受信的服务器连接，通常还需要对squid配置账号验证授权使用，通过httpd-tools生成密码文件，安装

# yum install httpd-tools -y

1	# yum install httpd-tools -y

生成密码文件

# sudo mkdir /etc/squid3/
# 生成密码文件，指定文件路径，其中squid是用户名
# sudo htpasswd -cd /etc/squid3/passwords squid
#提示输入密码，不能超过8个字符，输入密码123456

1

2

3

4

# sudo mkdir /etc/squid3/

# 生成密码文件，指定文件路径，其中squid是用户名

# sudo htpasswd -cd /etc/squid3/passwords squid

#提示输入密码，不能超过8个字符，输入密码123456

测试密码文件

# /usr/lib64/squid/basic_ncsa_auth /etc/squid3/passwords                
squid 123456
OK 
# 测试完成，crtl + c 打断

1

2

3

4

# /usr/lib64/squid/basic_ncsa_auth /etc/squid3/passwords

squid 123456

OK

# 测试完成，crtl + c 打断

配置squid使用验证，修改配置文件

# sudo vim /etc/squid/squid.conf
# And finally deny all other access to this proxy
auth_param basic program /usr/lib64/squid/basic_ncsa_auth /etc/squid3/passwords #账户密码文件
auth_param basic realm proxy
auth_param basic children 50 #最多 50 个账户同时运行
auth_param basic realm CoolTube Proxy Server #密码框描述
auth_param basic credentialsttl 2 hours #认证持续时间
acl authenticated proxy_auth REQUIRED #对 authenticated 进行外部认证
http_access allow authenticated #允许 authenticated 中的成员访问
http_access deny all #拒绝所有其他访问
visible_hostname squid.CoolTube #代理机名字

# 这里是端口号，可以按需修改
# http_port 3128 这样写会同时监听 ipv6 和 ipv4 的端口，推荐适应下面的配置方法。
http_port 0.0.0.0:3128

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

# sudo vim /etc/squid/squid.conf

# And finally deny all other access to this proxy

auth_param basic program /usr/lib64/squid/basic_ncsa_auth /etc/squid3/passwords #账户密码文件

auth_param basic realm proxy

auth_param basic children 50 #最多 50 个账户同时运行

auth_param basic realm CoolTube Proxy Server #密码框描述

auth_param basic credentialsttl 2 hours #认证持续时间

acl authenticated proxy_auth REQUIRED #对 authenticated 进行外部认证

http_access allow authenticated #允许 authenticated 中的成员访问

http_access deny all #拒绝所有其他访问

visible_hostname squid.CoolTube #代理机名字

# 这里是端口号，可以按需修改

# http_port 3128 这样写会同时监听 ipv6 和 ipv4 的端口，推荐适应下面的配置方法。

http_port 0.0.0.0:3128

初始化服务并重新启动

sudo squid -z
sudo systemctl restart squid.service

1 2	sudo squid -z sudo systemctl restart squid.service

如果开启了防火墙，还要在防火墙中允许端口3128的策略，以CentOS7为例

# 把 3128 端口加入防火墙过滤掉
sudo firewall-cmd --permanent --zone=public --add-port=3128/tcp
# 重启防火墙
sudo firewall-cmd --reload

1

2

3

4

# 把 3128 端口加入防火墙过滤掉

sudo firewall-cmd --permanent --zone=public --add-port=3128/tcp

# 重启防火墙

sudo firewall-cmd --reload

2、配置代理 2.1、配置全局代理修改环境变量文件并使其生效

sudo vim /etc/profile
#在最后加入（有认证的情况）
export http_proxy="http://squid:123456@192.168.0.200:3128"
export https_proxy="http://squid:123456@192.168.0.200:3128"
如果没有开启认证
http_proxy=http://192.168.0.200:3128
https_proxy=http://192.168.0.200:3128
# source /etc/profile

1

2

3

4

5

6

7

8

sudo vim /etc/profile

#在最后加入（有认证的情况）

export http_proxy="http://squid:123456@192.168.0.200:3128"

export https_proxy="http://squid:123456@192.168.0.200:3128"

如果没有开启认证

http_proxy=http://192.168.0.200:3128

https_proxy=http://192.168.0.200:3128

# source /etc/profile

2.2、配置wget代理新增wget的环境变量文件

# sudo vim ~/.wgetrc
http_proxy=http://192.168.0.200:3128
https_proxy=http://192.168.0.200:3128
use_proxy = on
wait = 30
# 验证，如果返回值为0表示wget可用
# sudo wget --spider -T 5 -q -t 2 www.baidu.com | echo $?
0

1

2

3

4

5

6

7

8

# sudo vim ~/.wgetrc

http_proxy=http://192.168.0.200:3128

https_proxy=http://192.168.0.200:3128

use_proxy = on

wait = 30

# 验证，如果返回值为0表示wget可用

# sudo wget --spider -T 5 -q -t 2 www.baidu.com | echo $?

0

2.3、配置yum代理编辑yum的配置文件，新增下面的内容

# vim /etc/yum.conf
# 有认证的情况
# Proxy
proxy=http://squid:123456@192.168.0.200:3128
# 没有认证
# Proxy
proxy=http://192.168.0.200:3128

1

2

3

4

5

6

7

# vim /etc/yum.conf

# 有认证的情况

# Proxy

proxy=http://squid:123456@192.168.0.200:3128

# 没有认证

# Proxy

proxy=http://192.168.0.200:3128

2.4、配置docker代理 docker配置代理是为了保证docker能从外网pull镜像，在安装了docker之后，在docker配置文件下新增配置并重启docker

# mkdir -p /etc/systemd/system/docker.service.d
# vim /etc/systemd/system/docker.service.d/http-proxy.conf
[Service]
Environment="HTTP_PROXY=squid:123456@192.168.0.200:3128" 
# systemctl daemon-reload
# systemctl restart docker

1

2

3

4

5

6

# mkdir -p /etc/systemd/system/docker.service.d

# vim /etc/systemd/system/docker.service.d/http-proxy.conf

[Service]

Environment="HTTP_PROXY=squid:123456@192.168.0.200:3128"

# systemctl daemon-reload

# systemctl restart docker

TOOLS

webmin安装及模块proftpd

OS

WSL报：The attempted operation is not supported for the type of object referenced

OS

vim使用动图

MIDDLEWARE

Jenkinsfile配置submodule

HANDBOOK

调用webmin上传下载文件

KUBERNETES

使用 GitHub Actions 编译 kubernetes 组件

KUBERNETES

为什么 kubernetes 环境要求开启 bridge-nf-call-iptables

NETWORK

calico网络问题排查

SECURITY

squid代理上网

HANDBOOK

最新Namecom优惠码和技巧心得