您现在的位置是：博客 > 文章详情文章详情

k8s "command failed" err="failed to parse kubelet flag: unknown flag: --network-plugin"

2023/09/02 00:44:38 发布预览数量

1445

分类： docker k8s 标签： k8s

一、问题描述

服务器夜间异常断电后在k8s Master主节点发现挂了。并且所有的容器都起不来，乍一看头皮发麻，没办法只能一步一步的排查问题了。

[root@t224yuntu1 ~]# kubectl get node
E0901 09:29:47.011209    9042 memcache.go:265] couldn't get current server API group list: Get "https://10.1.20.224:6443/api?timeout=32s": dial tcp 10.1.20.224:6443: connect: connection refused
E0901 09:29:47.011679    9042 memcache.go:265] couldn't get current server API group list: Get "https://10.1.20.224:6443/api?timeout=32s": dial tcp 10.1.20.224:6443: connect: connection refused
E0901 09:29:47.013172    9042 memcache.go:265] couldn't get current server API group list: Get "https://10.1.20.224:6443/api?timeout=32s": dial tcp 10.1.20.224:6443: connect: connection refused
E0901 09:29:47.014891    9042 memcache.go:265] couldn't get current server API group list: Get "https://10.1.20.224:6443/api?timeout=32s": dial tcp 10.1.20.224:6443: connect: connection refused
E0901 09:29:47.015849    9042 memcache.go:265] couldn't get current server API group list: Get "https://10.1.20.224:6443/api?timeout=32s": dial tcp 10.1.20.224:6443: connect: connection refused
The connection to the server 10.1.20.224:6443 was refused - did you specify the right host or port?

二、解决思路

1、检查环境变量情况（正常）

env | grep -i kub

检查环境变量情况

2、检查docker服务（正常）

systemctl status docker.service

检查docker服务

3、检查kubelet服务（异常）

systemctl status kubelet.service

检查kubelet服务

发现 kubelet.service起不来，如下：

[root@t224yuntu1 ~]# systemctl status kubelet.service
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: activating (auto-restart) (Result: exit-code) since 五 2023-09-01 09:36:28 CST; 2s ago
     Docs: https://kubernetes.io/docs/
  Process: 9544 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS (code=exited, status=1/FAILURE)
 Main PID: 9544 (code=exited, status=1/FAILURE)

9月 01 09:36:28 t224yuntu1 systemd[1]: kubelet.service: main process exited, code=exited, status=1/FAILURE
9月 01 09:36:28 t224yuntu1 systemd[1]: Unit kubelet.service entered failed state.
9月 01 09:36:28 t224yuntu1 systemd[1]: kubelet.service failed.

我们尝试通过 journalctl -xeu kubelet 来查询日志，日志内容如下：

[root@t224yuntu1 ~]# journalctl -xeu kubelet
-- 
-- The start-up result is done.
9月 01 09:36:48 t224yuntu1 kubelet[9571]: E0901 09:36:48.727154    9571 run.go:74] "command failed" err="failed to parse kubelet flag: unknown flag: --network-plugin"
9月 01 09:36:48 t224yuntu1 systemd[1]: kubelet.service: main process exited, code=exited, status=1/FAILURE
9月 01 09:36:48 t224yuntu1 systemd[1]: Unit kubelet.service entered failed state.
9月 01 09:36:48 t224yuntu1 systemd[1]: kubelet.service failed.
9月 01 09:36:58 t224yuntu1 systemd[1]: kubelet.service holdoff time over, scheduling restart.
9月 01 09:36:58 t224yuntu1 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
-- Subject: Unit kubelet.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit kubelet.service has finished shutting down.
9月 01 09:36:58 t224yuntu1 systemd[1]: Started kubelet: The Kubernetes Node Agent.
-- Subject: Unit kubelet.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit kubelet.service has finished starting up.
-- 
-- The start-up result is done.
9月 01 09:36:58 t224yuntu1 kubelet[9585]: E0901 09:36:58.980725    9585 run.go:74] "command failed" err="failed to parse kubelet flag: unknown flag: --network-plugin"
9月 01 09:36:58 t224yuntu1 systemd[1]: kubelet.service: main process exited, code=exited, status=1/FAILURE
9月 01 09:36:58 t224yuntu1 systemd[1]: Unit kubelet.service entered failed state.
9月 01 09:36:58 t224yuntu1 systemd[1]: kubelet.service failed.
9月 01 09:37:09 t224yuntu1 systemd[1]: kubelet.service holdoff time over, scheduling restart.
9月 01 09:37:09 t224yuntu1 systemd[1]: Stopped kubelet: The Kubernetes Node Agent.
-- Subject: Unit kubelet.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit kubelet.service has finished shutting down.
9月 01 09:37:09 t224yuntu1 systemd[1]: Started kubelet: The Kubernetes Node Agent.
-- Subject: Unit kubelet.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit kubelet.service has finished starting up.
-- 
-- The start-up result is done.
9月 01 09:37:09 t224yuntu1 kubelet[9596]: E0901 09:37:09.237987    9596 run.go:74] "command failed" err="failed to parse kubelet flag: unknown flag: --network-plugin"
9月 01 09:37:09 t224yuntu1 systemd[1]: kubelet.service: main process exited, code=exited, status=1/FAILURE
9月 01 09:37:09 t224yuntu1 systemd[1]: Unit kubelet.service entered failed state.
9月 01 09:37:09 t224yuntu1 systemd[1]: kubelet.service failed.

三、原因分析

根据日志可以初步判断，应该是网络模块除了问题，kebelet重启后，启动网络插件的命令无法执行。然后我又分别在master和node节点上查看下kubelet的版本，结果发现了问题：
原因分析

即master节点的 kubeadm、kubelet、kubernetes-cni、kubectl和node节点的版本不一致，且master节点使用的版本的高于node节点。

四、解决办法

我尝试将 master 节点和 node 节点的版本统一一下。

yum downgrade   kubeadm-1.23.0-0.x86_64 kubectl-1.23.0-0.x86_64 kubelet-1.23.0-0.x86_64

完成后重启kubelet

systemctl restart kubelet.service
systemctl daemon-reload

最后我们在看下k8s 服务的状态
K8s服务运行状态

大功告成！

原文链接：https://www.converts.cn/article/3443750.html

暂无评论