下午有个集群要做升级验证， 同事把developer-center的namespaces下的应用都删除重建了，后来发现paas平台打不开，查询nginx日志，应该是nginx-ingress出了问题
首先查看ingress-nginx-cotroller日志：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
[root@dciuap1 nginx]# kubectl logs -n ingress-nginx nginx-ingress-controller-qdfq5 ------------------------------------------------------------------------------- NGINX Ingress controller Release: 0.26.1 Build: git-2de5a893a Repository: https://github.com/kubernetes/ingress-nginx nginx version: openresty/1.15.8.2 。。。 。。。 W0325 00:28:50.153058 6 controller.go:811] Error obtaining Endpoints for Service "developer-center/k8s-api": no object matching key "developer-ce W0325 00:28:50.153070 6 controller.go:811] Error obtaining Endpoints for Service "developer-center/servmeta": no object matching key "developer-c W0325 00:28:50.153080 6 controller.go:811] Error obtaining Endpoints for Service "developer-center/runtime-log": no object matching key "develope W0325 00:28:50.153095 6 controller.go:811] Error obtaining Endpoints for Service "developer-center/res-remote-manager": no object matching key "d local store W0325 00:28:50.153108 6 controller.go:811] Error obtaining Endpoints for Service "developer-center/res-pool-manager": no object matching key "deval store I0325 00:28:50.365637 6 controller.go:150] Backend successfully reloaded. W0325 00:28:53.486108 6 controller.go:920] Service "developer-center/app-apply" does not have any active Endpoint. W0325 00:28:53.486154 6 controller.go:920] Service "developer-center/aiops" does not have any active Endpoint. W0325 00:28:53.486175 6 controller.go:920] Service "developer-center/confcenter" does not have any active Endpoint. W0325 00:28:53.486189 6 controller.go:920] Service "developer-center/collect" does not have any active Endpoint. |
可以修改deployment，使用 flag –v=XX来修改日志级别
|
1 2 3 4 5 6 7 |
$ kubectl get deploy -n <namespace-of-ingress-controller> NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE default-http-backend 1 1 1 1 35m nginx-ingress-controller 1 1 1 1 35m $ kubectl edit deploy -n <namespace-of-ingress-controller> nginx-ingress-controller # Add --v=X to "- args", where X is an integer |
--v=2shows details usingdiffabout the changes in the configuration in nginx--v=3shows details about the service, Ingress rule, endpoint changes and it dumps the nginx configuration in JSON format--v=5configures NGINX in debug mode
到api server的认证逻辑：
|
1 2 3 4 5 |
+-------------+ service +------------+ | | authentication | | + apiserver +<-------------------+ ingress | | | | controller | +-------------+ +------------+ |
得确定下看是否是服务认证或kubeconfig导致的问题
ingress需要从 apiserver获得信息，是需要认证的，有两种认证方式：
- Service Account: This is recommended, because nothing has to be configured. The Ingress controller will use information provided by the system to communicate with the API server. See ‘Service Account’ section for details.
- Kubeconfig file: In some Kubernetes environments service accounts are not available. In this case a manual configuration is required. The Ingress controller binary can be started with the –kubeconfig flag. The value of the flag is a path to a file specifying how to connect to the API server. Using the –kubeconfig does not requires the flag –apiserver-host. The format of the file is identical to ~/.kube/config which is used by kubectl to connect to the API server. See ‘kubeconfig’ section for details.
- Using the flag –apiserver-host: Using this flag –apiserver-host=http://localhost:8080 it is possible to specify an unsecured API server or reach a remote kubernetes cluster using kubectl proxy. Please do not use this approach in production.
下面是从前端到后端完整的认证流程.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
Kubernetes Workstation +---------------------------------------------------+ +------------------+ | | | | | +-----------+ apiserver +------------+ | | +------------+ | | | | proxy | | | | | | | | | apiserver | | ingress | | | | ingress | | | | | | controller | | | | controller | | | | | | | | | | | | | | | | | | | | | | | | | service account/ | | | | | | | | | | kubeconfig | | | | | | | | | +<-------------------+ | | | | | | | | | | | | | | | | | +------+----+ kubeconfig +------+-----+ | | +------+-----+ | | |<--------------------------------------------------------| | | | | | +---------------------------------------------------+ +------------------+ |
Service Account：
如果使用Service Account连接API server, ingress-controller需要存在/var/run/secrets/kubernetes.io/serviceaccount/token文件，来提供与API server认证所需要的secret token
可通过如下命令验证：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 |
# start a container that contains curl $ kubectl run test --image=tutum/curl -- sleep 10000 # check that container is running $ kubectl get pods NAME READY STATUS RESTARTS AGE test-701078429-s5kca 1/1 Running 0 16s # check if secret exists $ kubectl exec test-701078429-s5kca -- ls /var/run/secrets/kubernetes.io/serviceaccount/ ca.crt namespace token # get service IP of master $ kubectl get services NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes 10.0.0.1 <none> 443/TCP 1d # check base connectivity from cluster inside $ kubectl exec test-701078429-s5kca -- curl -k https://10.0.0.1 Unauthorized # connect using tokens $ TOKEN_VALUE=$(kubectl exec test-701078429-s5kca -- cat /var/run/secrets/kubernetes.io/serviceaccount/token) $ echo $TOKEN_VALUE eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3Mi....9A $ kubectl exec test-701078429-s5kca -- curl --cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt -H "Authorization: Bearer $TOKEN_VALUE" https://10.0.0.1 { "paths": [ "/api", "/api/v1", "/apis", "/apis/apps", "/apis/apps/v1alpha1", "/apis/authentication.k8s.io", "/apis/authentication.k8s.io/v1beta1", "/apis/authorization.k8s.io", "/apis/authorization.k8s.io/v1beta1", "/apis/autoscaling", "/apis/autoscaling/v1", "/apis/batch", "/apis/batch/v1", "/apis/batch/v2alpha1", "/apis/certificates.k8s.io", "/apis/certificates.k8s.io/v1alpha1", "/apis/networking", "/apis/networking/v1beta1", "/apis/policy", "/apis/policy/v1alpha1", "/apis/rbac.authorization.k8s.io", "/apis/rbac.authorization.k8s.io/v1alpha1", "/apis/storage.k8s.io", "/apis/storage.k8s.io/v1beta1", "/healthz", "/healthz/ping", "/logs", "/metrics", "/swaggerapi/", "/ui/", "/version" ] } |
如下是我的验证过程：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 |
[root@dciuap1 nginx]# kubectl run test --image=tutum/curl -- sleep 10000 [root@dciuap1 nginx]# TOKEN_VALUE=$(kubectl exec test-701078429-s5kca -- cat /var/run/secrets/kubernetes.io/serviceaccount/token) [root@dciuap1 nginx]# kubectl exec test-8656bc94b4-fqlsg -- curl -k https://10.96.0.1 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 165 100 165 0 0 1627 0 --:--:-- --:--:-- --:--:-- 1650 { "kind": "Status", "apiVersion": "v1", "metadata": { }, "status": "Failure", "message": "Unauthorized", "reason": "Unauthorized", "code": 401 } # 通过admin-user用户没问题 [root@dciuap1 nginx]# TOKEN_VALUE=$(echo $(kubectl -n kube-system get secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}') -o go-template='{{ .daase64 -d)) [root@dciuap1 nginx]# kubectl exec test-8656bc94b4-fqlsg -- curl -k https://10.96.0.1 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 4918 0 4918 0 0 342k 0 --:--:-- --:--:-- --:--:-- 369k { "paths": [ "/api", "/api/v1", "/apis", "/apis/", "/apis/admissionregistration.k8s.io", "/apis/admissionregistration.k8s.io/v1", "/apis/admissionregistration.k8s.io/v1beta1", "/apis/apiextensions.k8s.io", "/apis/apiextensions.k8s.io/v1", "/apis/apiextensions.k8s.io/v1beta1", "/apis/apiregistration.k8s.io", "/apis/apiregistration.k8s.io/v1", "/apis/apiregistration.k8s.io/v1beta1", "/apis/apps", "/apis/apps/v1", "/apis/authentication.k8s.io", "/apis/authentication.k8s.io/v1", "/apis/authentication.k8s.io/v1beta1", "/apis/authorization.k8s.io", "/apis/authorization.k8s.io/v1", "/apis/authorization.k8s.io/v1beta1", "/apis/autoscaling", "/apis/autoscaling/v1", "/apis/autoscaling/v2beta1", 。。。 。。。 |
If it is not working, there are two possible reasons:
- The contents of the tokens are invalid. Find the secret name with
kubectl get secrets | grep service-accountand delete it withkubectl delete secret <name>. It will automatically be recreated.- You have a non-standard Kubernetes installation and the file containing the token may not be present. The API server will mount a volume containing this file, but only if the API server is configured to use the ServiceAccount admission controller. If you experience this error, verify that your API server is using the ServiceAccount admission controller. If you are configuring the API server by hand, you can set this with the
--admission-controlparameter.Note that you should use other admission controllers as well. Before configuring this option, you should read about admission controllers.
解决方式：重装ingress-nginx-controller
参考： https://kubernetes.github.io/ingress-nginx/troubleshooting/
0 Comments