1.1
主旨
- Helm 安装配置使用
- Helm 系统进阶学习整理
- Helm渲染过程
说明
环境说明
// 10.2.21.47
# helm version
version.BuildInfo{Version:"v3.11.2", GitCommit:"912ebc1cd10d38d340f048efaf0abda047c3468e", GitTreeState:"clean", GoVersion:"go1.18.10"}
// 10.2.21.47（如下版本已经被替换）
# helm version
version.BuildInfo{Version:"v3.10.0", GitCommit:"ce66412a723e4d89555dc67217607c6579ffcb21", GitTreeState:"clean", GoVersion:"go1.18.6"}
版本对应关系
Helm 版本 支持的 Kubernetes 版本
3.11.x 1.26.x - 1.23.x
3.10.x 1.25.x - 1.22.x
3.9.x 1.24.x - 1.21.x
3.8.x 1.23.x - 1.20.x
3.7.x 1.22.x - 1.19.x
3.6.x 1.21.x - 1.18.x
3.5.x 1.20.x - 1.17.x
3.4.x 1.19.x - 1.16.x
安装
helm-v3.10.0
tar xvf helm-v3.10.0-linux-amd64.tar.gz
mv linux-amd64/helm /usr/local/bin/helm
官网安装文档地址：
https://docs.helm.sh/docs/intro/install/
github下载：
https://github.com/helm/helm/releases
https://get.helm.sh/helm-v3.11.2-linux-arm.tar.gz
下载安装版本：
wget https://get.helm.sh/helm-v3.10.0-linux-amd64.tar.gz
wget https://get.helm.sh/helm-v3.11.2-linux-amd64.tar.gz
配置仓库源
helm repo add aliyun https://kubernetes.oss-cn-hangzhou.aliyuncs.com/charts
helm repo update
helm repo list
helm repo remove aliyun
官方链接
helm中文doc
helm英文doc
目录结构
# ls
charts Chart.yaml templates values.yaml
# tree
.
├── charts
├── Chart.yaml
├── templates
│ ├── deployment.yaml
│ ├── _helpers.tpl
│ ├── hpa.yaml
│ ├── ingress.yaml
│ ├── NOTES.txt
│ ├── serviceaccount.yaml
│ ├── service.yaml
│ └── tests
│ └── test-connection.yaml
└── values.yaml
3 directories, 10 files
Chart.yaml：用于描述这个 Chart 的基本信息，包括名字、描述信息以及版本等。
values.yaml：用于存储 templates 目录中模板文件中用到变量的值。
Templates：目录里面存放所有 yaml 模板文件。
charts：目录里存放这个 chart 依赖的所有子 chart。
NOTES.txt ：用于介绍 Chart 帮助信息，helm install 部署后展示给用户。例如：如何使用这个 Chart、 列出缺省的设置等。
_helpers.tpl：放置模板助手的地方，可以在整个 chart 中重复使用。
目录文件详解
Chart.yaml
# cat Chart.yaml
apiVersion: v2
name: mychart
description: A Helm chart for Kubernetes
# A chart can be either an 'application' or a 'library' chart.
#
# Application charts are a collection of templates that can be packaged into versioned archives
# to be deployed.
#
# Library charts provide useful utilities or functions for the chart developer. They're included as
# a dependency of application charts to inject those utilities and functions into the rendering
# pipeline. Library charts do not define any templates and therefore cannot be deployed.
type: application
# This is the chart version. This version number should be incremented each time you make changes
# to the chart and its templates, including the app version.
# Versions are expected to follow Semantic Versioning (https://semver.org/)
version: 0.1.0
# This is the version number of the application being deployed. This version number should be
# incremented each time you make changes to the application. Versions are not expected to
# follow Semantic Versioning. They should reflect the version the application is using.
# It is recommended to use it with quotes.
appVersion: "1.16.0"
values.yaml
# cat values.yaml
# Default values for mychart.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.
replicaCount: 1
image:
repository: nginx
pullPolicy: IfNotPresent
# Overrides the image tag whose default is the chart appVersion.
tag: ""
imagePullSecrets: []
nameOverride: ""
fullnameOverride: ""
serviceAccount:
# Specifies whether a service account should be created
create: true
# Annotations to add to the service account
annotations: {}
# The name of the service account to use.
# If not set and create is true, a name is generated using the fullname template
name: ""
podAnnotations: {}
podSecurityContext: {}
# fsGroup: 2000
securityContext: {}
# capabilities:
# drop:
# - ALL
# readOnlyRootFilesystem: true
# runAsNonRoot: true
# runAsUser: 1000
service:
type: ClusterIP
port: 80
ingress:
enabled: false
className: ""
annotations: {}
# kubernetes.io/ingress.class: nginx
# kubernetes.io/tls-acme: "true"
hosts:
- host: chart-example.local
paths:
- path: /
pathType: ImplementationSpecific
tls: []
# - secretName: chart-example-tls
# hosts:
# - chart-example.local
resources: {}
# We usually recommend not to specify default resources and to leave this as a conscious
# choice for the user. This also increases chances charts run on environments with little
# resources, such as Minikube. If you do want to specify resources, uncomment the following
# lines, adjust them as necessary, and remove the curly braces after 'resources:'.
# limits:
# cpu: 100m
# memory: 128Mi
# requests:
# cpu: 100m
# memory: 128Mi
autoscaling:
enabled: false
minReplicas: 1
maxReplicas: 100
targetCPUUtilizationPercentage: 80
# targetMemoryUtilizationPercentage: 80
nodeSelector: {}
tolerations: []
affinity: {}
templates/deployment.yaml
# cat deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ include "mychart.fullname" . }}
labels:
{{- include "mychart.labels" . | nindent 4 }}
spec:
{{- if not .Values.autoscaling.enabled }}
replicas: {{ .Values.replicaCount }}
{{- end }}
selector:
matchLabels:
{{- include "mychart.selectorLabels" . | nindent 6 }}
template:
metadata:
{{- with .Values.podAnnotations }}
annotations:
{{- toYaml . | nindent 8 }}
{{- end }}
labels:
{{- include "mychart.selectorLabels" . | nindent 8 }}
spec:
{{- with .Values.imagePullSecrets }}
imagePullSecrets:
{{- toYaml . | nindent 8 }}
{{- end }}
serviceAccountName: {{ include "mychart.serviceAccountName" . }}
securityContext:
{{- toYaml .Values.podSecurityContext | nindent 8 }}
containers:
- name: {{ .Chart.Name }}
securityContext:
{{- toYaml .Values.securityContext | nindent 12 }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
ports:
- name: http
containerPort: 80
protocol: TCP
livenessProbe:
httpGet:
path: /
port: http
readinessProbe:
httpGet:
path: /
port: http
resources:
{{- toYaml .Values.resources | nindent 12 }}
{{- with .Values.nodeSelector }}
nodeSelector:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.affinity }}
affinity:
{{- toYaml . | nindent 8 }}
{{- end }}
{{- with .Values.tolerations }}
tolerations:
{{- toYaml . | nindent 8 }}
{{- end }}
templates/_helpers.tpl
# cat _helpers.tpl
{{/*
Expand the name of the chart.
*/}}
{{- define "mychart.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
{{- end }}
{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
If release name contains chart name it will be used as a full name.
*/}}
{{- define "mychart.fullname" -}}
{{- if .Values.fullnameOverride }}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- $name := default .Chart.Name .Values.nameOverride }}
{{- if contains $name .Release.Name }}
{{- .Release.Name | trunc 63 | trimSuffix "-" }}
{{- else }}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" }}
{{- end }}
{{- end }}
{{- end }}
{{/*
Create chart name and version as used by the chart label.
*/}}
{{- define "mychart.chart" -}}
{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" }}
{{- end }}
{{/*
Common labels
*/}}
{{- define "mychart.labels" -}}
helm.sh/chart: {{ include "mychart.chart" . }}
{{ include "mychart.selectorLabels" . }}
{{- if .Chart.AppVersion }}
app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
{{- end }}
app.kubernetes.io/managed-by: {{ .Release.Service }}
{{- end }}
{{/*
Selector labels
*/}}
{{- define "mychart.selectorLabels" -}}
app.kubernetes.io/name: {{ include "mychart.name" . }}
app.kubernetes.io/instance: {{ .Release.Name }}
{{- end }}
{{/*
Create the name of the service account to use
*/}}
{{- define "mychart.serviceAccountName" -}}
{{- if .Values.serviceAccount.create }}
{{- default (include "mychart.fullname" .) .Values.serviceAccount.name }}
{{- else }}
{{- default "default" .Values.serviceAccount.name }}
{{- end }}
{{- end }}
templates/hpa.yaml
# cat hpa.yaml
{{- if .Values.autoscaling.enabled }}
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: {{ include "mychart.fullname" . }}
labels:
{{- include "mychart.labels" . | nindent 4 }}
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: {{ include "mychart.fullname" . }}
minReplicas: {{ .Values.autoscaling.minReplicas }}
maxReplicas: {{ .Values.autoscaling.maxReplicas }}
metrics:
{{- if .Values.autoscaling.targetCPUUtilizationPercentage }}
- type: Resource
resource:
name: cpu
targetAverageUtilization: {{ .Values.autoscaling.targetCPUUtilizationPercentage }}
{{- end }}
{{- if .Values.autoscaling.targetMemoryUtilizationPercentage }}
- type: Resource
resource:
name: memory
targetAverageUtilization: {{ .Values.autoscaling.targetMemoryUtilizationPercentage }}
{{- end }}
{{- end }}
templates/ingress.yaml
# cat ingress.yaml
{{- if .Values.ingress.enabled -}}
{{- $fullName := include "mychart.fullname" . -}}
{{- $svcPort := .Values.service.port -}}
{{- if and .Values.ingress.className (not (semverCompare ">=1.18-0" .Capabilities.KubeVersion.GitVersion)) }}
{{- if not (hasKey .Values.ingress.annotations "kubernetes.io/ingress.class") }}
{{- $_ := set .Values.ingress.annotations "kubernetes.io/ingress.class" .Values.ingress.className}}
{{- end }}
{{- end }}
{{- if semverCompare ">=1.19-0" .Capabilities.KubeVersion.GitVersion -}}
apiVersion: networking.k8s.io/v1
{{- else if semverCompare ">=1.14-0" .Capabilities.KubeVersion.GitVersion -}}
apiVersion: networking.k8s.io/v1beta1
{{- else -}}
apiVersion: extensions/v1beta1
{{- end }}
kind: Ingress
metadata:
name: {{ $fullName }}
labels:
{{- include "mychart.labels" . | nindent 4 }}
{{- with .Values.ingress.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
spec:
{{- if and .Values.ingress.className (semverCompare ">=1.18-0" .Capabilities.KubeVersion.GitVersion) }}
ingressClassName: {{ .Values.ingress.className }}
{{- end }}
{{- if .Values.ingress.tls }}
tls:
{{- range .Values.ingress.tls }}
- hosts:
{{- range .hosts }}
- {{ . | quote }}
{{- end }}
secretName: {{ .secretName }}
{{- end }}
{{- end }}
rules:
{{- range .Values.ingress.hosts }}
- host: {{ .host | quote }}
http:
paths:
{{- range .paths }}
- path: {{ .path }}
{{- if and .pathType (semverCompare ">=1.18-0" $.Capabilities.KubeVersion.GitVersion) }}
pathType: {{ .pathType }}
{{- end }}
backend:
{{- if semverCompare ">=1.19-0" $.Capabilities.KubeVersion.GitVersion }}
service:
name: {{ $fullName }}
port:
number: {{ $svcPort }}
{{- else }}
serviceName: {{ $fullName }}
servicePort: {{ $svcPort }}
{{- end }}
{{- end }}
{{- end }}
{{- end }}
templates/NOTES.txt
]# cat NOTES.txt
1. Get the application URL by running these commands:
{{- if .Values.ingress.enabled }}
{{- range $host := .Values.ingress.hosts }}
{{- range .paths }}
http{{ if $.Values.ingress.tls }}s{{ end }}://{{ $host.host }}{{ .path }}
{{- end }}
{{- end }}
{{- else if contains "NodePort" .Values.service.type }}
export NODE_PORT=$(kubectl get --namespace {{ .Release.Namespace }} -o jsonpath="{.spec.ports[0].nodePort}" services {{ include "mychart.fullname" . }})
export NODE_IP=$(kubectl get nodes --namespace {{ .Release.Namespace }} -o jsonpath="{.items[0].status.addresses[0].address}")
echo http://$NODE_IP:$NODE_PORT
{{- else if contains "LoadBalancer" .Values.service.type }}
NOTE: It may take a few minutes for the LoadBalancer IP to be available.
You can watch the status of by running 'kubectl get --namespace {{ .Release.Namespace }} svc -w {{ include "mychart.fullname" . }}'
export SERVICE_IP=$(kubectl get svc --namespace {{ .Release.Namespace }} {{ include "mychart.fullname" . }} --template "{{"{{ range (index .status.loadBalancer.ingress 0) }}{{.}}{{ end }}"}}")
echo http://$SERVICE_IP:{{ .Values.service.port }}
{{- else if contains "ClusterIP" .Values.service.type }}
export POD_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ include "mychart.name" . }},app.kubernetes.io/instance={{ .Release.Name }}" -o jsonpath="{.items[0].metadata.name}")
export CONTAINER_PORT=$(kubectl get pod --namespace {{ .Release.Namespace }} $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
echo "Visit http://127.0.0.1:8080 to use your application"
kubectl --namespace {{ .Release.Namespace }} port-forward $POD_NAME 8080:$CONTAINER_PORT
{{- end }}
templates/serviceaccount.yaml
# cat serviceaccount.yaml
{{- if .Values.serviceAccount.create -}}
apiVersion: v1
kind: ServiceAccount
metadata:
name: {{ include "mychart.serviceAccountName" . }}
labels:
{{- include "mychart.labels" . | nindent 4 }}
{{- with .Values.serviceAccount.annotations }}
annotations:
{{- toYaml . | nindent 4 }}
{{- end }}
{{- end }}
templates/service.yaml
# cat service.yaml
apiVersion: v1
kind: Service
metadata:
name: {{ include "mychart.fullname" . }}
labels:
{{- include "mychart.labels" . | nindent 4 }}
spec:
type: {{ .Values.service.type }}
ports:
- port: {{ .Values.service.port }}
targetPort: http
protocol: TCP
name: http
selector:
{{- include "mychart.selectorLabels" . | nindent 4 }}
templates/tests/test-connection.yaml
cat tests/test-connection.yaml
apiVersion: v1
kind: Pod
metadata:
name: "{{ include "mychart.fullname" . }}-test-connection"
labels:
{{- include "mychart.labels" . | nindent 4 }}
annotations:
"helm.sh/hook": test
spec:
containers:
- name: wget
image: busybox
command: ['wget']
args: ['{{ include "mychart.fullname" . }}:{{ .Values.service.port }}']
restartPolicy: Never
查看仓库示例
// 查看仓库中的包
# helm search repo tomcat
NAME CHART VERSION APP VERSION DESCRIPTION
bitnami/tomcat 10.9.1 10.1.9 Apache Tomcat is an open-source web server desi...
// 将包下载到本地
# helm pull bitnami/tomcat
示例演示
示例一
创建一个模版demo-chart包
# helm create demo-chart
Creating demo-chart
# tree demo-chart/
demo-chart/
├── charts
├── Chart.yaml
├── templates
│ ├── deployment.yaml
│ ├── _helpers.tpl
│ ├── hpa.yaml
│ ├── ingress.yaml
│ ├── NOTES.txt
│ ├── serviceaccount.yaml
│ ├── service.yaml
│ └── tests
│ └── test-connection.yaml
└── values.yaml
3 directories, 10 files
将demo-chart包部署到Kubernetes上
// helm install 命令可以从多个来源进行安装，参考命令如下：
chart 的仓库（如上所述）
本地 chart 压缩包（helm install foo foo-0.1.1.tgz）
解压后的 chart 目录（helm install foo path/to/foo）
完整的 URL（helm install foo https://example.com/charts/foo-1.2.3.tgz）
# helm install demo-chart --generate-name
NAME: demo-chart-1684750433
LAST DEPLOYED: Mon May 22 18:13:54 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=demo-chart,app.kubernetes.io/instance=demo-chart-1684750433" -o jsonpath="{.items[0].metadata.name}")
export CONTAINER_PORT=$(kubectl get pod --namespace default $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
echo "Visit http://127.0.0.1:8080 to use your application"
kubectl --namespace default port-forward $POD_NAME 8080:$CONTAINER_PORT
# kubectl get pod
NAME READY STATUS RESTARTS AGE
demo-chart-1684750433-68dbb4654b-vzrn9 1/1 Running 0 41s
rbd-provisioner-76f6bc6669-9l5c6 1/1 Running 4 174d
# kubectl get svc --all-namespaces
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default demo-chart-1684750433 ClusterIP 10.1.55.124 <none> 80/TCP 52s
k8s上查看部署的应用
# kubectl get svc --all-namespaces -o wide
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
default demo-chart-1684750433 ClusterIP 10.1.55.124 <none> 80/TCP 15h app.kubernetes.io/instance=demo-chart-1684750433,app.kubernetes.io/name=demo-chart
helm卸载应用
# helm uninstall demo-chart-1684750433
release "demo-chart-1684750433" uninstalled
示例二
# helm create nginx
Creating nginx
# tree nginx/
nginx/
├── charts
├── Chart.yaml
├── templates
│ ├── deployment.yaml
│ ├── _helpers.tpl
│ ├── hpa.yaml
│ ├── ingress.yaml
│ ├── NOTES.txt
│ ├── serviceaccount.yaml
│ ├── service.yaml
│ └── tests
│ └── test-connection.yaml
└── values.yaml
3 directories, 10 files
# helm install -f values.yaml nginx .
Error: INSTALLATION FAILED: open values.yaml: no such file or directory
# cd nginx/
# helm install -f values.yaml nginx .
Error: INSTALLATION FAILED: Kubernetes cluster unreachable: Get "http://localhost:8080/version": dial tcp [::1]:8080: connect: connection refused
// 解决方法
将k8s集群的证书文件放在和helm所在的服务器即可
临时测试：
# export KUBECONFIG=/etc/kubernetes/admin.conf
长期有效：
# vi /etc/profile
此文件末尾添加
export KUBECONFIG=/etc/kubernetes/admin.conf
# source /etc/profile
// helm安装应用
# helm install -f values.yaml nginx .
NAME: nginx
LAST DEPLOYED: Mon Oct 23 17:03:56 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=nginx,app.kubernetes.io/instance=nginx" -o jsonpath="{.items[0].metadata.name}")
export CONTAINER_PORT=$(kubectl get pod --namespace default $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
echo "Visit http://127.0.0.1:8080 to use your application"
kubectl --namespace default port-forward $POD_NAME 8080:$CONTAINER_PORT
// 获取部署应用的pod名称
# kubectl get pods --namespace default -l "app.kubernetes.io/name=nginx,app.kubernetes.io/instance=nginx" -o jsonpath="{.items[0].metadata.name}"
nginx-5cd949c459-d8frv
// 获取容器端口
# kubectl get pod --namespace default $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}"
80
// Forwarding from 127.0.0.1:8080 -> 80将端口暴露方便外部访问
# kubectl --namespace default port-forward $POD_NAME 8080:$CONTAINER_PORT
Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80
Handling connection for 8080
// 新开一个终端查看
# curl http://127.0.0.1:8080
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
// 打包应用示例
# pwd
/tmp
# ls
nginx
# helm package nginx
Successfully packaged chart and saved it to: /tmp/nginx-0.1.0.tgz
# ls
nginx nginx-0.1.0.tgz
示例三
[root@master helm]# helm search repo tomcat
NAME CHART VERSION APP VERSION DESCRIPTION
bitnami/tomcat 10.9.1 10.1.9 Apache Tomcat is an open-source web server desi...
[root@master helm]# helm install my-tomcat bitnami/tomcat
NAME: my-tomcat
LAST DEPLOYED: Tue May 23 10:39:34 2023
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: tomcat
CHART VERSION: 10.9.1
APP VERSION: 10.1.9
** Please be patient while the chart is being deployed **
1. Get the Tomcat URL by running:
NOTE: It may take a few minutes for the LoadBalancer IP to be available.
Watch the status with: 'kubectl get svc --namespace default -w my-tomcat'
export SERVICE_IP=$(kubectl get svc --namespace default my-tomcat --template "{{ range (index .status.loadBalancer.ingress 0) }}{{ . }}{{ end }}")
echo "Tomcat URL: http://$SERVICE_IP/"
echo "Tomcat Management URL: http://$SERVICE_IP/manager"
2. Login with the following credentials
echo Username: user
echo Password: $(kubectl get secret --namespace default my-tomcat -o jsonpath="{.data.tomcat-password}" | base64 -d)
[root@master helm]# kubectl get secret --namespace default my-tomcat -o jsonpath="{.data.tomcat-password}" | base64 -d
B4Lqi3TK7o
[root@master helm]#
自定义chart包
- 基于示例能够制作自己的业务chart包
- 将自己的chart包能够部署到k8s环境中
- 详细了解各个模版中各个文件各个参数意义
- 尽可能适配不同的k8s版本、k8s集群等
打包自定义包
// 基于文件夹打包
# helm package nginx
Successfully packaged chart and saved it to: /home/opt/helm/tmp/nginx-0.1.0.tgz
基于自定义打包安装
helm install foo nginx-0.1.0.tgz
如下示例说明
历史版本查看
[root@master helm]# helm ls
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
myapp default 1 2023-05-24 13:37:21.930444942 +0800 CST failed myapp-0.1.0 1.16.0
mynginxx default 2 2023-05-24 13:44:34.1743329 +0800 CST deployed nginx-0.1.0 1.16.0
[root@master helm]# helm history mynginxx
REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION
1 Wed May 24 13:38:20 2023 superseded nginx-0.1.0 1.16.0 Install complete
2 Wed May 24 13:44:34 2023 deployed nginx-0.1.0 1.16.0 Upgrade complete
升级回滚
// 升级操作
[root@master helm]# helm upgrade mynginxx ./nginx/
Release "mynginxx" has been upgraded. Happy Helming!
NAME: mynginxx
LAST DEPLOYED: Wed May 24 13:44:34 2023
NAMESPACE: default
STATUS: deployed
REVISION: 2
NOTES:
1. Get the application URL by running these commands:
export POD_NAME=$(kubectl get pods --namespace default -l "app.kubernetes.io/name=nginx,app.kubernetes.io/instance=mynginxx" -o jsonpath="{.items[0].metadata.name}")
export CONTAINER_PORT=$(kubectl get pod --namespace default $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
echo "Visit http://127.0.0.1:8080 to use your application"
kubectl --namespace default port-forward $POD_NAME 8080:$CONTAINER_PORT
[root@master helm]#
// 之前是一个副本，现在升级后变成两个副本
[root@master helm]# kubectl get pod
NAME READY STATUS RESTARTS AGE
mynginxx-78c6558bbb-hz9zx 1/1 Running 0 6m43s
mynginxx-78c6558bbb-vm487 1/1 Running 0 30s
rbd-provisioner-76f6bc6669-9l5c6 1/1 Running 4 176d
[root@master helm]#
// 执行回滚操作
[root@master helm]# helm rollback mynginxx 1 -n default
Rollback was a success! Happy Helming!
[root@master helm]#
// 历史版本查看
[root@master helm]# helm history mynginxx
REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION
1 Wed May 24 13:38:20 2023 superseded nginx-0.1.0 1.16.0 Install complete
2 Wed May 24 13:44:34 2023 superseded nginx-0.1.0 1.16.0 Upgrade complete
3 Wed May 24 13:49:19 2023 deployed nginx-0.1.0 1.16.0 Rollback to 1
[root@master helm]#
# chartrepo，固定参数，bigdata自定义项目
helm repo add local-harbor --username=admin --password=Harbor12345 https://myharbor.com/chartrepo/bigdata/ --ca-file /opt/k8s/helm/ca.crt
helm repo add local-harbor --username=admin --password=Harbor12345 https://myharbor.zznode.com/helm
helm install myrelease oci://container-registry.com/container-registry/harbor --version 1.7.4
推送helm到harbor
// 检查推送插件
[root@master helm]# helm plugin list
NAME VERSION DESCRIPTION
push 0.9.0 Push chart package to ChartMuseum
[root@master helm]#
// 若无则执行
[root@master helm]# helm plugin install https://github.com/chartmuseum/helm-push
异常排查
隐藏配置文件
/root/.cache/helm/repository
/root/.config/helm/
# helm search repo nginx
WARNING: Repo "myrepo" is corrupt or missing. Try 'helm repo update'.
WARNING: open /root/.cache/helm/repository/myrepo-index.yaml: no such file or directory
NAME CHART VERSION APP VERSION DESCRIPTION
aliyun/nginx-ingress 0.9.5 0.10.2 An nginx Ingress controller that uses ConfigMap...
aliyun/nginx-lego 0.3.1 Chart for nginx-ingress-controller and kube-lego
bitnami/nginx 14.2.2 1.24.0 NGINX Open Source is a web server that can be a...
WARNING: Repo "myrepo" is corrupt or missing. Try 'helm repo update'.
WARNING: open /root/.cache/helm/repository/myrepo-index.yaml: no such file or directory
# ll /root/.cache/helm/repository/
总用量 9120
-rw-r--r-- 1 root root 1400 5月 23 09:31 aliyun-charts.txt
-rw-r--r-- 1 root root 296570 5月 23 09:31 aliyun-index.yaml
-rw-r--r-- 1 root root 1216 5月 23 09:31 bitnami-charts.txt
-rw-r--r-- 1 root root 5569455 5月 23 09:31 bitnami-index.yaml
-rw-r--r-- 1 root root 93 5月 23 09:31 crossplane-alpha-charts.txt
-rw-r--r-- 1 root root 21249 5月 23 09:31 crossplane-alpha-index.yaml
-rw-r--r-- 1 root root 163 5月 23 09:31 gitlab-charts.txt
-rw-r--r-- 1 root root 708593 5月 23 09:31 gitlab-index.yaml
-rw-r--r-- 1 root root 7 5月 23 09:31 harbor-charts.txt
-rw-r--r-- 1 root root 61363 5月 23 09:31 harbor-index.yaml
-rw-r--r-- 1 root root 19 5月 23 09:31 kube-state-metrics-charts.txt
-rw-r--r-- 1 root root 8603 5月 23 09:31 kube-state-metrics-index.yaml
-rw-r--r-- 1 root root 963 5月 23 09:32 prometheus-community-charts.txt
-rw-r--r-- 1 root root 2630167 5月 23 09:32 prometheus-community-index.yaml
# ll /root/.config/helm/
总用量 8
-rw------- 1 root root 105 2月 7 2021 registry.json
-rw------- 1 root root 0 8月 28 2020 repositories.lock
-rw-r--r-- 1 root root 1683 5月 22 16:21 repositories.yaml
# cat /root/.config/helm/repositories.yaml | grep myrepo
name: myrepo
url: https://myharbor.zznode.com/chartrepo/myrepo
# cat /root/.config/helm/registry.json
{
"auths": {
"myharbor.zznode.com/chartrepo/library": {
"auth": "YWRtaW46SGFyYm9yMTIzNDU="
}
}
}
# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
10.2.21.47 myharbor.zznode.com
## 删除掉省略之间的行，重新执行helm search repo list则不报错
# vi /root/.config/helm/repositories.yaml
……省略……
- caFile: ""
certFile: ""
insecure_skip_tls_verify: false
keyFile: ""
name: myrepo
pass_credentials_all: false
password: ""
url: https://myharbor.zznode.com/chartrepo/myrepo
username: ""
……省略……
示例二异常排查
// 暂时无法访问
[root@master tomcat]# kubectl get pod
NAME READY STATUS RESTARTS AGE
my-tomcat-66d759f5cc-wlfd4 0/1 Pending 0 41m
rbd-provisioner-76f6bc6669-9l5c6 1/1 Running 4 174d
[root@master tomcat]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.1.0.1 <none> 443/TCP 439d
my-tomcat LoadBalancer 10.1.155.6 <pending> 80:15773/TCP 41m
[root@master tomcat]#
[root@master tomcat]# kubectl describe pod my-tomcat-66d759f5cc-wlfd4
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 40m default-scheduler 0/2 nodes are available: 2 pod has unbound immediate PersistentVolumeClaims.
Warning FailedScheduling 40m default-scheduler 0/2 nodes are available: 2 pod has unbound immediate PersistentVolumeClaims.
[root@master tomcat]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-cd243181-e011-4e92-88ce-1844214d8b93 10Gi RWO Delete Bound default/maven-cache ceph-rdb 396d
pvc-eb920709-7cd1-4c57-8f93-b47a53f57562 10Gi RWO Delete Bound default/pkg ceph-rdb 394d
[root@master tomcat]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
maven-cache Bound pvc-cd243181-e011-4e92-88ce-1844214d8b93 10Gi RWO ceph-rdb 396d
my-tomcat Pending 43m
pkg Bound pvc-eb920709-7cd1-4c57-8f93-b47a53f57562 10Gi RWO ceph-rdb 394d
[root@master tomcat]#
[root@master tomcat]# kubectl describe pvc my-tomcat
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal FailedBinding 3m31s (x162 over 43m) persistentvolume-controller no persistent volumes available for this claim and no storage class is set
[root@master tomcat]#
// 变量剖析
[root@master templates]# cat ingress.yaml
{{- if .Values.ingress.enabled }}
apiVersion: {{ include "common.capabilities.ingress.apiVersion" . }}
kind: Ingress
metadata:
name: {{ template "common.names.fullname" . }}
namespace: {{ .Release.Namespace }}
[root@master templates]# helm ls
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
my-tomcat default 1 2023-05-23 10:39:34.100611768 +0800 CST deployed tomcat-10.9.1 10.1.9
[root@master templates]#
如上的release是来自于helm ls中对应的变量值
渲染过程展示
[root@master templates]# helm install my-tomcat bitnami/tomcat --dry-run --debug
install.go:192: [debug] Original chart version: ""
install.go:209: [debug] CHART PATH: /root/.cache/helm/repository/tomcat-10.9.1.tgz
NAME: my-tomcat
LAST DEPLOYED: Tue May 23 13:59:17 2023
NAMESPACE: default
STATUS: pending-install
REVISION: 1
TEST SUITE: None
USER-SUPPLIED VALUES:
{}
COMPUTED VALUES:
affinity: {}
args: []
catalinaOpts: ""
clusterDomain: cluster.local
command: []
common:
exampleValue: common-chart
global:
imagePullSecrets: []
imageRegistry: ""
storageClass: ""
commonAnnotations: {}
commonLabels: {}
containerExtraPorts: []
containerPorts:
http: 8080
containerSecurityContext:
enabled: true
runAsNonRoot: true
runAsUser: 1001
customLivenessProbe: {}
customReadinessProbe: {}
customStartupProbe: {}
deployment:
type: deployment
extraDeploy: []
extraEnvVars: []
extraEnvVarsCM: ""
extraEnvVarsSecret: ""
extraPodSpec: {}
extraVolumeClaimTemplates: []
extraVolumeMounts: []
extraVolumes: []
fullnameOverride: ""
global:
imagePullSecrets: []
imageRegistry: ""
storageClass: ""
hostAliases: []
image:
debug: false
digest: ""
pullPolicy: IfNotPresent
pullSecrets: []
registry: docker.io
repository: bitnami/tomcat
tag: 10.1.9-debian-11-r0
ingress:
annotations: {}
apiVersion: ""
enabled: false
extraHosts: []
extraPaths: []
extraRules: []
extraTls: []
hostname: tomcat.local
ingressClassName: ""
path: /
pathType: ImplementationSpecific
secrets: []
selfSigned: false
tls: false
initContainers: []
kubeVersion: ""
lifecycleHooks: {}
livenessProbe:
enabled: true
failureThreshold: 6
initialDelaySeconds: 120
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
metrics:
jmx:
catalinaOpts: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=5555
-Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.local.only=true
config: |
jmxUrl: service:jmx:rmi:///jndi/rmi://127.0.0.1:5555/jmxrmi
startDelaySecs: 120
ssl: false
lowercaseOutputName: true
lowercaseOutputLabelNames: true
attrNameSnakeCase: true
containerSecurityContext:
enabled: true
runAsNonRoot: true
runAsUser: 1001
enabled: false
existingConfigmap: ""
image:
digest: ""
pullPolicy: IfNotPresent
pullSecrets: []
registry: docker.io
repository: bitnami/jmx-exporter
tag: 0.18.0-debian-11-r22
ports:
metrics: 5556
resources:
limits: {}
requests: {}
podMonitor:
additionalLabels: {}
enabled: false
interval: 30s
namespace: ""
podTargetLabels: []
relabelings: []
scheme: http
scrapeTimeout: 30s
tlsConfig: {}
prometheusRule:
additionalLabels: {}
enabled: false
namespace: ""
rules: []
nameOverride: ""
networkPolicy:
allowExternal: true
enabled: false
explicitNamespacesSelector: {}
nodeAffinityPreset:
key: ""
type: ""
values: []
nodeSelector: {}
persistence:
accessModes:
- ReadWriteOnce
annotations: {}
enabled: true
existingClaim: ""
selectorLabels: {}
size: 8Gi
storageClass: ""
podAffinityPreset: ""
podAnnotations: {}
podAntiAffinityPreset: soft
podLabels: {}
podManagementPolicy: ""
podSecurityContext:
enabled: true
fsGroup: 1001
readinessProbe:
enabled: true
failureThreshold: 3
initialDelaySeconds: 30
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 3
replicaCount: 1
resources:
limits: {}
requests:
cpu: 300m
memory: 512Mi
schedulerName: ""
service:
annotations: {}
clusterIP: ""
externalTrafficPolicy: Cluster
extraPorts: []
headless:
annotations: {}
loadBalancerIP: ""
loadBalancerSourceRanges: []
nodePorts:
http: ""
ports:
http: 80
sessionAffinity: None
sessionAffinityConfig: {}
type: LoadBalancer
sidecars: []
startupProbe:
enabled: false
failureThreshold: 3
initialDelaySeconds: 30
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 3
tolerations: []
tomcatAllowRemoteManagement: 0
tomcatPassword: ""
tomcatUsername: user
topologySpreadConstraints: []
updateStrategy:
type: RollingUpdate
volumePermissions:
enabled: false
image:
digest: ""
pullPolicy: IfNotPresent
pullSecrets: []
registry: docker.io
repository: bitnami/bitnami-shell
tag: 11-debian-11-r118
resources:
limits: {}
requests: {}
HOOKS:
MANIFEST:
---
# Source: tomcat/templates/secrets.yaml
apiVersion: v1
kind: Secret
metadata:
name: my-tomcat
namespace: default
labels:
app.kubernetes.io/name: tomcat
helm.sh/chart: tomcat-10.9.1
app.kubernetes.io/instance: my-tomcat
app.kubernetes.io/managed-by: Helm
type: Opaque
data:
tomcat-password: "Z0xvSDNnUDZ6cQ=="
---
# Source: tomcat/templates/pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: my-tomcat
namespace: default
labels:
app.kubernetes.io/name: tomcat
helm.sh/chart: tomcat-10.9.1
app.kubernetes.io/instance: my-tomcat
app.kubernetes.io/managed-by: Helm
spec:
accessModes:
- "ReadWriteOnce"
resources:
requests:
storage: "8Gi"
---
# Source: tomcat/templates/svc.yaml
apiVersion: v1
kind: Service
metadata:
name: my-tomcat
namespace: default
labels:
app.kubernetes.io/name: tomcat
helm.sh/chart: tomcat-10.9.1
app.kubernetes.io/instance: my-tomcat
app.kubernetes.io/managed-by: Helm
spec:
type: LoadBalancer
externalTrafficPolicy: "Cluster"
sessionAffinity: None
ports:
- name: http
port: 80
targetPort: http
selector:
app.kubernetes.io/name: tomcat
app.kubernetes.io/instance: my-tomcat
---
# Source: tomcat/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-tomcat
namespace: default
labels:
app.kubernetes.io/name: tomcat
helm.sh/chart: tomcat-10.9.1
app.kubernetes.io/instance: my-tomcat
app.kubernetes.io/managed-by: Helm
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: tomcat
app.kubernetes.io/instance: my-tomcat
strategy:
type: RollingUpdate
template:
metadata:
labels:
app.kubernetes.io/name: tomcat
helm.sh/chart: tomcat-10.9.1
app.kubernetes.io/instance: my-tomcat
app.kubernetes.io/managed-by: Helm
spec:
affinity:
podAffinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchLabels:
app.kubernetes.io/name: tomcat
app.kubernetes.io/instance: my-tomcat
topologyKey: kubernetes.io/hostname
weight: 1
nodeAffinity:
securityContext:
fsGroup: 1001
initContainers:
containers:
- name: tomcat
image: docker.io/bitnami/tomcat:10.1.9-debian-11-r0
imagePullPolicy: "IfNotPresent"
securityContext:
runAsNonRoot: true
runAsUser: 1001
env:
- name: BITNAMI_DEBUG
value: "false"
- name: TOMCAT_USERNAME
value: "user"
- name: TOMCAT_PASSWORD
valueFrom:
secretKeyRef:
name: my-tomcat
key: tomcat-password
- name: TOMCAT_ALLOW_REMOTE_MANAGEMENT
value: "0"
ports:
- name: http
containerPort: 8080
livenessProbe:
httpGet:
path: /
port: http
failureThreshold: 6
initialDelaySeconds: 120
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
readinessProbe:
httpGet:
path: /
port: http
failureThreshold: 3
initialDelaySeconds: 30
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 3
resources:
limits: {}
requests:
cpu: 300m
memory: 512Mi
volumeMounts:
- name: data
mountPath: /bitnami/tomcat
volumes:
- name: data
persistentVolumeClaim:
claimName: my-tomcat
NOTES:
CHART NAME: tomcat
CHART VERSION: 10.9.1
APP VERSION: 10.1.9
** Please be patient while the chart is being deployed **
1. Get the Tomcat URL by running:
NOTE: It may take a few minutes for the LoadBalancer IP to be available.
Watch the status with: 'kubectl get svc --namespace default -w my-tomcat'
export SERVICE_IP=$(kubectl get svc --namespace default my-tomcat --template "{{ range (index .status.loadBalancer.ingress 0) }}{{ . }}{{ end }}")
echo "Tomcat URL: http://$SERVICE_IP/"
echo "Tomcat Management URL: http://$SERVICE_IP/manager"
2. Login with the following credentials
echo Username: user
echo Password: $(kubectl get secret --namespace default my-tomcat -o jsonpath="{.data.tomcat-password}" | base64 -d)
[root@master templates]#
[root@master ~]# kubectl get pv pvc-cd243181-e011-4e92-88ce-1844214d8b93 -o yaml
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 10Gi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: maven-cache
namespace: default
resourceVersion: "6600512"
uid: cd243181-e011-4e92-88ce-1844214d8b93
persistentVolumeReclaimPolicy: Delete
rbd:
image: kubernetes-dynamic-pvc-2b048fd1-c1d8-11ec-922d-3e8f89442688
keyring: /etc/ceph/keyring
monitors:
- 10.2.6.63:6789
- 10.2.6.64:6789
- 10.2.6.65:6789
pool: k8s
secretRef:
name: ceph-user-secret
user: kube
storageClassName: ceph-rdb
volumeMode: Filesystem
status:
phase: Bound
# cat testpv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: my-tomcat
spec:
capacity:
storage: 8Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Delete
storageClassName: ceph-rdb
rbd:
monitors:
- 10.2.6.63:6789
- 10.2.6.64:6789
- 10.2.6.65:6789
pool: k8s
image: rbda
user: kube
secretRef:
name: ceph-user-secret
[root@master ~]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
my-tomcat 8Gi RWO Delete Available ceph-rdb 8s
再次执行创建：
报同样的错误，修改storageClass: "ceph-rdb"
[root@master helm]# cat tomcat/values.yaml | grep storageClass
## Current available global Docker image parameters: imageRegistry, imagePullSecrets and storageClass
## @param global.storageClass Global StorageClass for Persistent Volume(s)
storageClass: ""
## @param persistence.storageClass PVC Storage Class for Tomcat volume
## If defined, storageClassName: <storageClass>
## If set to "-", storageClassName: "", which disables dynamic provisioning
## If undefined (the default) or set to null, no storageClassName spec is
storageClass: "ceph-rdb
[root@master helm]# kubectl get pod
NAME READY STATUS RESTARTS AGE
my-tomcat-66d759f5cc-rrj5w 0/1 ContainerCreating 0 4m2s
rbd-provisioner-76f6bc6669-9l5c6 1/1 Running 4 175d
[root@master helm]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
my-tomcat 8Gi RWO Delete Bound default/my-tomcat ceph-rdb 40m
pvc-cd243181-e011-4e92-88ce-1844214d8b93 10Gi RWO Delete Bound default/maven-cache ceph-rdb 396d
pvc-eb920709-7cd1-4c57-8f93-b47a53f57562 10Gi RWO Delete Bound default/pkg ceph-rdb 395d
[root@master helm]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
maven-cache Bound pvc-cd243181-e011-4e92-88ce-1844214d8b93 10Gi RWO ceph-rdb 396d
my-tomcat Bound my-tomcat 8Gi RWO ceph-rdb 4m8s
pkg Bound pvc-eb920709-7cd1-4c57-8f93-b47a53f57562 10Gi RWO ceph-rdb 395d
[root@master helm]#
[root@master helm]# kubectl describe pod my-tomcat-66d759f5cc-rrj5w
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 4m58s default-scheduler Successfully assigned default/my-tomcat-66d759f5cc-rrj5w to node1
Normal SuccessfulAttachVolume 4m59s attachdetach-controller AttachVolume.Attach succeeded for volume "my-tomcat"
Warning FailedMount 40s (x2 over 2m56s) kubelet Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[default-token-b8dgk data]: timed out waiting for the condition
[root@master helm]# kubectl describe pod my-tomcat-66d759f5cc-w5gl9
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 29m default-scheduler Successfully assigned default/my-tomcat-66d759f5cc-w5gl9 to node1
Warning FailedAttachVolume 29m attachdetach-controller Multi-Attach error for volume "my-tomcat" Volume is already exclusively attached to one node and can't be attached to another
Normal SuccessfulAttachVolume 29m attachdetach-controller AttachVolume.Attach succeeded for volume "my-tomcat"
Warning FailedMount 2m40s (x2 over 20m) kubelet Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[default-token-b8dgk data]: timed out waiting for the condition
Warning FailedMount 22s (x11 over 27m) kubelet Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[data default-token-b8dgk]: timed out waiting for the condition
// 目前还是失败，到此结束，不再继续剖析
解决因为PVC创建失败的问题
Warning FailedScheduling 40m default-scheduler 0/2 nodes are available: 2 pod has unbound immediate PersistentVolumeClaims.
[root@master helm]# cat jenkins/values.yaml | grep -A 2 Persistence
Persistence:
Enabled: false
## A manually managed Persistent Volume and Claim
## Requires Persistence.Enabled: true
## If defined, PVC must be created manually before volume will be bound
# ExistingClaim:
[root@master helm]#
将 Enabled: 默认true改为false
参考文档
helm3自定义chart编写：
https://www.cnblogs.com/fengzi7314/p/14872326.html
官网参考链接：（需要深入学习）
https://helm.sh/zh/docs/
历史事件
2023-10-23｜新增高版本安装，调整 目录层级
2023-05-22｜补充安装小节
2023-05-19｜创建文档