4.1
证书续期场景
找到k8s版本边界，区分证书续期的不同命令；
1.19/1.20
k8s集群和单点证书续期差异；
k8s集群master与node，单点master与node证书续期对比；
默认一年证书如何改为十年甚至更长时间;
场景01-k8s集群证书续期
集群证书检查
// k8s集群版本信息
[root@devopsecs004 ~]# kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2", GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean", BuildDate:"2020-09-16T13:41:02Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2", GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean", BuildDate:"2020-09-16T13:32:58Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}
// 三台集群环境证书状态检查
[root@devopsecs004 ~]# kubeadm alpha certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Jul 03, 2024 09:44 UTC 58d no
apiserver Jul 03, 2024 09:44 UTC 58d ca no
apiserver-etcd-client Jul 03, 2024 09:44 UTC 58d etcd-ca no
apiserver-kubelet-client Jul 03, 2024 09:44 UTC 58d ca no
controller-manager.conf Jul 03, 2024 09:44 UTC 58d no
etcd-healthcheck-client Jul 03, 2024 09:44 UTC 58d etcd-ca no
etcd-peer Jul 03, 2024 09:44 UTC 58d etcd-ca no
etcd-server Jul 03, 2024 09:44 UTC 58d etcd-ca no
front-proxy-client Jul 03, 2024 09:44 UTC 58d front-proxy-ca no
scheduler.conf Jul 03, 2024 09:44 UTC 58d no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Jul 01, 2033 09:44 UTC 9y no
etcd-ca Jul 01, 2033 09:44 UTC 9y no
front-proxy-ca Jul 01, 2033 09:44 UTC 9y no
[root@devopsecs005 ~]# kubeadm alpha certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Jul 03, 2024 09:44 UTC 58d no
apiserver Jul 03, 2024 10:15 UTC 58d ca no
apiserver-etcd-client Jul 03, 2024 10:15 UTC 58d etcd-ca no
apiserver-kubelet-client Jul 03, 2024 10:15 UTC 58d ca no
controller-manager.conf Jul 03, 2024 10:15 UTC 58d no
etcd-healthcheck-client Jul 03, 2024 10:15 UTC 58d etcd-ca no
etcd-peer Jul 03, 2024 10:15 UTC 58d etcd-ca no
etcd-server Jul 03, 2024 10:15 UTC 58d etcd-ca no
front-proxy-client Jul 03, 2024 10:15 UTC 58d front-proxy-ca no
scheduler.conf Jul 03, 2024 10:15 UTC 58d no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Jul 01, 2033 09:44 UTC 9y no
etcd-ca Jul 01, 2033 09:44 UTC 9y no
front-proxy-ca Jul 01, 2033 09:44 UTC 9y no
[root@devopsecs005 ~]#
[root@devopsecs006 ~]# kubeadm alpha certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Jul 03, 2024 09:44 UTC 58d no
apiserver Jul 03, 2024 10:10 UTC 58d ca no
apiserver-etcd-client Jul 03, 2024 10:10 UTC 58d etcd-ca no
apiserver-kubelet-client Jul 03, 2024 10:10 UTC 58d ca no
controller-manager.conf Jul 03, 2024 10:10 UTC 58d no
etcd-healthcheck-client Jul 03, 2024 10:10 UTC 58d etcd-ca no
etcd-peer Jul 03, 2024 10:10 UTC 58d etcd-ca no
etcd-server Jul 03, 2024 10:10 UTC 58d etcd-ca no
front-proxy-client Jul 03, 2024 10:10 UTC 58d front-proxy-ca no
scheduler.conf Jul 03, 2024 10:10 UTC 58d no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Jul 01, 2033 09:44 UTC 9y no
etcd-ca Jul 01, 2033 09:44 UTC 9y no
front-proxy-ca Jul 01, 2033 09:44 UTC 9y no
集群备份
// 安全起见，三台集群都进行备份
# cp -r /etc/kubernetes/ /etc/kubernetes_240506bak
# cp -r /var/lib/etcd /var/lib/etcd_240506bak
// 重点说明，上面只是目录备份，一定要进行etcd的快照备份
// etcdctl 客户端可以自己下载安装，也可以从容器拷贝到本地
// 快照备份，下面参考IP，替换为你实际的IP地址
# ETCDCTL_API=3 etcdctl --endpoints=https://10.1.6.104:2379 --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key --cacert=/etc/kubernetes/pki/etcd/ca.crt snapshot save /tmp/104etcdback.db
// 快照恢复
# ETCDCTL_API=3 etcdctl snapshot restore /tmp/104etcdback.db --data-dir=/var/lib/etcd
// etcd集群状态查看
# ETCDCTL_API=3 etcdctl --endpoints=https://10.2.6.103:2379,https://10.2.6.104:2379,https://10.2.6.105:2379 --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key --cacert=/etc/kubernetes/pki/etcd/ca.crt endpoint health -w table
// etcd单节点状态查看
# ETCDCTL_API=3 etcdctl --endpoints=https://10.1.6.104:2379 --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key --cacert=/etc/kubernetes/pki/etcd/ca.crt endpoint health -w table
集群证书升级
如下操作，在其中一个master节点上进行操作；
待证书升级完成后，将变动的证书拷贝到其他节点即可。
获取集群信息
kubeadm config view > cluster.yaml
# cat cluster.yaml
apiServer:
extraArgs:
authorization-mode: Node,RBAC
timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: {{cluster-keepalived-VIP}}:16443
controllerManager: {}
dns:
type: CoreDNS
etcd:
local:
dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.19.2
networking:
dnsDomain: cluster.local
podSubnet: 10.244.0.0/16
serviceSubnet: 10.96.0.0/12
scheduler: {}
证书更新
# kubeadm alpha certs renew all --config=/root/wnh/certs/cluster.yaml
查看各个证书过期时间
for item in `find /etc/kubernetes/pki -maxdepth 2 -name "*.crt"`;do echo ======================$item===============;openssl x509 -in $item -text -noout| grep Not;done
新生成的admin.conf 覆盖老的
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
POD重启
docker ps |grep -E 'k8s_kube-apiserver|k8s_kube-controller-manager|k8s_kube-scheduler|k8s_etcd_etcd' | awk -F ' ' '{print $1}' |xargs docker restart
检查证书有效期
# kubeadm alpha certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf May 06, 2025 08:26 UTC 364d no
apiserver May 06, 2025 08:26 UTC 364d ca no
apiserver-etcd-client May 06, 2025 08:26 UTC 364d etcd-ca no
apiserver-kubelet-client May 06, 2025 08:26 UTC 364d ca no
controller-manager.conf May 06, 2025 08:26 UTC 364d no
etcd-healthcheck-client May 06, 2025 08:26 UTC 364d etcd-ca no
etcd-peer May 06, 2025 08:26 UTC 364d etcd-ca no
etcd-server May 06, 2025 08:26 UTC 364d etcd-ca no
front-proxy-client May 06, 2025 08:26 UTC 364d front-proxy-ca no
scheduler.conf May 06, 2025 08:26 UTC 364d no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Nov 26, 2032 09:13 UTC 8y no
etcd-ca Nov 26, 2032 09:13 UTC 8y no
front-proxy-ca Nov 26, 2032 09:13 UTC 8y no
证书同步
# mkdir -p /etc/kubernetes/pki/etcd
scp /etc/kubernetes/pki/ca.* root@devopsecs005:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/sa.* root@devopsecs005:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/front-proxy-ca.* root@devopsecs005:/etc/kubernetes/pki/
scp /etc/kubernetes/pki/etcd/ca.* root@devopsecs005:/etc/kubernetes/pki/etcd/
# pwd
/etc/kubernetes
# ll
总用量 36
-rw------- 1 root root 5562 5月 6 16:26 admin.conf
-rw------- 1 root root 5598 5月 6 16:26 controller-manager.conf
-rw------- 1 root root 1907 11月 29 2022 kubelet.conf
drwx------ 2 root root 113 6月 16 2023 manifests
drwxr-xr-x 3 root root 4096 11月 29 2022 pki
-rw------- 1 root root 5550 5月 6 16:26 scheduler.conf
# pwd
/etc/kubernetes/pki
# ll
总用量 56
-rw-r--r-- 1 root root 1273 5月 6 16:26 apiserver.crt
-rw-r--r-- 1 root root 1135 5月 6 16:26 apiserver-etcd-client.crt
-rw------- 1 root root 1675 5月 6 16:26 apiserver-etcd-client.key
-rw------- 1 root root 1679 5月 6 16:26 apiserver.key
-rw-r--r-- 1 root root 1143 5月 6 16:26 apiserver-kubelet-client.crt
-rw------- 1 root root 1675 5月 6 16:26 apiserver-kubelet-client.key
-rw-r--r-- 1 root root 1066 11月 29 2022 ca.crt
-rw------- 1 root root 1675 11月 29 2022 ca.key
drwxr-xr-x 2 root root 162 11月 29 2022 etcd
-rw-r--r-- 1 root root 1078 11月 29 2022 front-proxy-ca.crt
-rw------- 1 root root 1679 11月 29 2022 front-proxy-ca.key
-rw-r--r-- 1 root root 1103 5月 6 16:26 front-proxy-client.crt
-rw------- 1 root root 1675 5月 6 16:26 front-proxy-client.key
-rw------- 1 root root 1675 11月 29 2022 sa.key
-rw------- 1 root root 451 11月 29 2022 sa.pub
# pwd
/etc/kubernetes/pki/etcd
# ll
总用量 32
-rw-r--r-- 1 root root 1058 11月 29 2022 ca.crt
-rw------- 1 root root 1679 11月 29 2022 ca.key
-rw-r--r-- 1 root root 1139 5月 6 16:26 healthcheck-client.crt
-rw------- 1 root root 1675 5月 6 16:26 healthcheck-client.key
-rw-r--r-- 1 root root 1180 5月 6 16:26 peer.crt
-rw------- 1 root root 1679 5月 6 16:26 peer.key
-rw-r--r-- 1 root root 1180 5月 6 16:26 server.crt
-rw------- 1 root root 1679 5月 6 16:26 server.key
其他集群证书节点状态检查
检查集群所有POD运行状态
参考配置-本次升级无需此命令
# kubectl get cm -o yaml -n kube-system kubeadm-config > kubeadm.yaml
# cat kubeadm.yaml
apiVersion: v1
data:
ClusterConfiguration: |
apiServer:
extraArgs:
authorization-mode: Node,RBAC
timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta2
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: {{cluster-keepalived-VIP}}:16443
controllerManager: {}
dns:
type: CoreDNS
etcd:
local:
dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers
kind: ClusterConfiguration
kubernetesVersion: v1.19.2
networking:
dnsDomain: cluster.local
podSubnet: 10.244.0.0/16
serviceSubnet: 10.96.0.0/12
scheduler: {}
ClusterStatus: |
apiEndpoints:
devopsecs004:
advertiseAddress: {{master1-IP}}
bindPort: 6443
devopsecs005:
advertiseAddress: {{master2-IP}}
bindPort: 6443
devopsecs006:
advertiseAddress: {{master3-IP}}
bindPort: 6443
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterStatus
kind: ConfigMap
metadata:
creationTimestamp: "2023-07-04T09:44:42Z"
managedFields:
- apiVersion: v1
fieldsType: FieldsV1
fieldsV1:
f:data:
.: {}
f:ClusterConfiguration: {}
f:ClusterStatus: {}
manager: kubeadm
operation: Update
time: "2023-07-04T10:16:04Z"
name: kubeadm-config
namespace: kube-system
resourceVersion: "5109"
selfLink: /api/v1/namespaces/kube-system/configmaps/kubeadm-config
uid: ba020ed3-2205-40b5-b60d-03efd70933bf
排错
// 配置文件不正确导致，不能通过上面方法生成的yaml文件执行
// 网上参考说直接通过# kubectl get cm -o yaml -n kube-system kubeadm-config > kubeadm.yaml
// 执行会报错
# kubeadm alpha certs renew all --config=/root/wnh/certs/kubeadm.yaml
invalid configuration for GroupVersionKind /v1, Kind=ConfigMap: kind and apiVersion is mandatory information that must be specified
To see the stack trace of this error execute with --v=5 or higher
场景02-自定义证书时间
环境说明
// 10.1.6.106
// 节点环境
# kubectl get node
NAME STATUS ROLES AGE VERSION
master Ready control-plane,master 2y59d v1.20.6
node1 Ready <none> 389d v1.20.6
// 版本及平台依赖环境等
# kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.6", GitCommit:"8a62859e515889f07e3e3be6a1080413f17cf2c3", GitTreeState:"clean", BuildDate:"2021-04-15T03:28:42Z", GoVersion:"go1.15.10", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.6", GitCommit:"8a62859e515889f07e3e3be6a1080413f17cf2c3", GitTreeState:"clean", BuildDate:"2021-04-15T03:19:55Z", GoVersion:"go1.15.10", Compiler:"gc", Platform:"linux/amd64"}
// 证书有效期
# kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Oct 07, 2024 07:35 UTC 152d no
apiserver Oct 07, 2024 07:36 UTC 152d ca no
apiserver-etcd-client Oct 07, 2024 07:36 UTC 152d etcd-ca no
apiserver-kubelet-client Oct 07, 2024 07:36 UTC 152d ca no
controller-manager.conf Oct 07, 2024 07:36 UTC 152d no
etcd-healthcheck-client Oct 07, 2024 07:36 UTC 152d etcd-ca no
etcd-peer Oct 07, 2024 07:36 UTC 152d etcd-ca no
etcd-server Oct 07, 2024 07:36 UTC 152d etcd-ca no
front-proxy-client Oct 07, 2024 07:36 UTC 152d front-proxy-ca no
scheduler.conf Oct 07, 2024 07:36 UTC 152d no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Mar 07, 2032 02:17 UTC 7y no
etcd-ca Mar 07, 2032 02:17 UTC 7y no
front-proxy-ca Mar 07, 2032 02:17 UTC 7y no
k8s源码下载
// page=21这里会随着版本迭代page会变动，参考位置
https://github.com/kubernetes/kubernetes/releases?page=21
https://github.com/kubernetes/kubernetes/archive/refs/tags/v1.20.6.zip
https://github.com/kubernetes/kubernetes/archive/refs/tags/v1.20.6.tar.gz
当前版本源码与Go版本关系
https://github.com/kubernetes/kubernetes/blob/release-1.20/go.mod
kubernetes/go.mod
go 1.15
说明：测试时只针对大版本（release-1.20）进行了编译即可完成证书的更新，严谨的话建议选择具体版本v1.20.6
Go在Linux环境安装
// 下载、解压、移动到目录、配置环境变量
wget https://go.dev/dl/go1.15.linux-amd64.tar.gz
tar xvf go1.15.linux-amd64.tar.gz
mv go /usr/local/
export PATH=$PATH:/usr/local/go/bin
编译操作
cd kubernetes-1.20
vim cmd/kubeadm/app/constants/constants.go
// 这里是修改10年（* 10）
CertificateValidity为CertificateValidity = time.Hour * 24 * 365 * 10
// 重新编译文件
make WHAT=cmd/kubeadm GOFLAGS=-v
// 备份
cp /usr/bin/kubeadm /usr/bin/kubeadm.old
cp _output/bin/kubeadm /usr/bin/
// 增加执行权限
chmod a+x /usr/bin/kubeadm
k8s证书更新
备份操作
# cp -r /etc/kubernetes/ /etc/kubernetes_240508bak
# cp -r /var/lib/etcd /var/lib/etcd_240508bak
// 重点说明，上面只是目录备份，一定要进行etcd的快照备份
// etcdctl 客户端可以自己下载安装，也可以从容器拷贝到本地
// 快照备份
# ETCDCTL_API=3 etcdctl --endpoints=https://10.1.6.104:2379 --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key --cacert=/etc/kubernetes/pki/etcd/ca.crt snapshot save /tmp/104etcdback.db
// 快照恢复
# ETCDCTL_API=3 etcdctl snapshot restore /tmp/104etcdback.db --data-dir=/var/lib/etcd
// etcd集群状态查看
# ETCDCTL_API=3 etcdctl --endpoints=https://10.2.6.103:2379,https://10.2.6.104:2379,https://10.2.6.105:2379 --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key --cacert=/etc/kubernetes/pki/etcd/ca.crt endpoint health -w table
// etcd单节点状态查看
# ETCDCTL_API=3 etcdctl --endpoints=https://10.1.6.104:2379 --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key --cacert=/etc/kubernetes/pki/etcd/ca.crt endpoint health -w table
重新生成证书
// 这里命令和1.19的区别kubeadm certs与kubeadm alpha certs
# kubeadm certs renew all --config=/home/opt/cluster.yaml
certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed
Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.
检查证书
# kubeadm certs check-expiration
[check-expiration] Reading configuration from the cluster...
[check-expiration] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf May 06, 2034 02:53 UTC 9y ca no
apiserver May 06, 2034 02:53 UTC 9y ca no
apiserver-etcd-client May 06, 2034 02:53 UTC 9y etcd-ca no
apiserver-kubelet-client May 06, 2034 02:53 UTC 9y ca no
controller-manager.conf May 06, 2034 02:53 UTC 9y ca no
etcd-healthcheck-client May 06, 2034 02:53 UTC 9y etcd-ca no
etcd-peer May 06, 2034 02:53 UTC 9y etcd-ca no
etcd-server May 06, 2034 02:53 UTC 9y etcd-ca no
front-proxy-client May 06, 2034 02:53 UTC 9y front-proxy-ca no
scheduler.conf May 06, 2034 02:53 UTC 9y ca no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Mar 07, 2032 02:17 UTC 7y no
etcd-ca Mar 07, 2032 02:17 UTC 7y no
front-proxy-ca Mar 07, 2032 02:17 UTC 7y no
查看证书时间
# for item in `find /etc/kubernetes/pki -maxdepth 2 -name "*.crt"`;do echo ======================$item===============;openssl x509 -in $item -text -noout| grep Not;done
重启相关pod
# docker ps |grep -E 'k8s_kube-apiserver|k8s_kube-controller-manager|k8s_kube-scheduler|k8s_etcd_etcd' | awk -F ' ' '{print $1}' |xargs docker restart
引申思考
ca
etcd-ca
front-proxy-ca
证书该如何修改
参考文档
// 证书更新
https://blog.csdn.net/Mr_rsq/article/details/116703659
https://www.cnblogs.com/kuku0223/p/12978716.html
// 自定义证书
https://www.cnblogs.com/zhangmingcheng/p/15423770.html
https://www.cnblogs.com/zhangmingcheng/p/14317470.html
// k8s源码
https://github.com/kubernetes/kubernetes/tree/release-1.20