k8s v1.28.15安装---test--ok---20250819
#现在主要是外网限制，引起的各种安装麻烦，如果有梯子，应该很简单。
#环境：centos7.9 + Docker version 26.1.4 + k8s-v1.28.2+containerd 1.6.33
#原则是：全使用yum安装和直接安装，简单点
#k8s的repo源是https://mirrors.aliyun.com ,注意地址说明
#aliyun老地址最高支持k8s-v1.28.15 新地址支持k8s-v1.33.4
#k8s-v1.28.15-install-docker+containerd+flannel--20250826--ok
#参考教程：
# https://zhuanlan.zhihu.com/p/699063977
# https://zhuanlan.zhihu.com/p/675796199
# https://blog.csdn.net/dyj095/category_12997521.html
# https://blog.csdn.net/dyj095/article/details/130512120
# https://developer.aliyun.com/mirror/kubernetes?spm=a2c6h.13651102.0.0.25971b11TFhCoA
#kubectl和webUI:客户端。
#etcd:K-V结构的分布式数据库，保存了整个集群的状态和配置信息，是集群的持久化机制，遇到故障可以使集群快速恢复。该组件发生故障，整个K8S就会瘫痪。 apiserver:所有服务访问和资源操作的统一入口，并提供认证、授权、访问控制、API注册和发现等机制;
#replication controller:它能够保证Pod持续运行，并且在任何时候都有指定数量的Pod副本，在此基础上提供一些高级特性，比如滚动升级和弹性伸缩
#scheduler:负责资源的调度，按照预定的调度策略将Pod调度到相应的机器上;
#kubelet:负责维护容器的生命周期,通俗理解就是对于容器的创建和销毁就是kubelet来控制的。
#kube-proxy:负责为Service提供cluster内部的服务发现和负载均衡;
#pod:在K8S中，最小的管理元素不是一个个独立的容器，而是Pod,Pod是最小的，管理，创建，计划的最小单元。
#Pod的设计理念是支持多个容器在一个Pod中共享网络地址和文件系统。
##注意事项
从 Kubernetes 1.22 开始，如果未明确指定 cgroupDriver，kubeadm 会默认使用 systemd
Kubernetes 1.28 引入了 cgroup 驱动自动检测功能（Alpha 阶段）
修改配置时，建议逐个节点操作，确保工作负载平稳迁移
不同容器运行时（Docker、containerd 等）的 systemd 驱动配置方式可能不同
#master-192.168.9.10
#node1-192.168.9.11
#node2-192.168.9.12
#harbor-192.168.9.13
#kuboard-192.168.9.14
#端口使用：
kubelet 10250 10248
kube-controller 127.0.0.1:10257
kube-schedule 10259
kube-proxy 10256 127.0.0.1:10249
kube-apiserver 6443
etcd 2379 2380 2381
#===所有主机====================
#初始配置（所有主机）
#主机配置主机名
hostnamectl set-hostname master
hostnamectl set-hostname node1
hostnamectl set-hostname node2
#修改hosts文件
cat >> /etc/hosts << EOF
192.168.9.10 master
192.168.9.11 node1
192.168.9.12 node2
192.168.9.13 harbor
EOF
echo "br_netfilter" | sudo tee /etc/modules-load.d/br_netfilter.conf
#永久关闭
sed -ri 's/.*swap.*/#&/' /etc/fstab
#允许 iptables 检查桥接流量
cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
sysctl -p
#常用配置
yum install -y bash-completion wget vim-enhanced net-tools ntpdate
#ntpdate ntp1.aliyun.com
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
#固定IP
sed -i 's/dhcp/static/g' /etc/sysconfig/network-scripts/ifcfg-ens33
echo "IPADDR=192.168.9.12
NETMASK=255.255.255.0
GATEWAY=192.168.9.2
DNS1=192.168.9.2" >>/etc/sysconfig/network-scripts/ifcfg-ens33
systemctl restart network
#添加docker的阿里云yum源
curl -o /etc/yum.repos.d/docker-ce.repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum install -y docker-ce docker-compose
systemctl enable --now docker
#配置docker镜像源
cat >> /etc/docker/daemon.json << EOF
{
"registry-mirrors": [
"https://2a6bf1988cb6428c877f723ec7530dbc.mirror.swr.myhuaweicloud.com",
"https://docker.m.daocloud.io",
"https://hub-mirror.c.163.com",
"https://mirror.baidubce.com",
"https://your_preferred_mirror",
"https://dockerhub.icu",
"https://docker.registry.cyou",
"https://docker-cf.registry.cyou",
"https://dockercf.jsdelivr.fyi",
"https://docker.jsdelivr.fyi",
"https://dockertest.jsdelivr.fyi",
"https://mirror.aliyuncs.com",
"https://dockerproxy.com",
"https://mirror.baidubce.com",
"https://docker.m.daocloud.io",
"https://docker.nju.edu.cn",
"https://docker.mirrors.sjtug.sjtu.edu.cn",
"https://docker.mirrors.ustc.edu.cn",
"https://mirror.iscas.ac.cn",
"https://docker.rainbond.cc"
],
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"data-root": "/data/var/lib/docker",
"log-opts": {
"max-size": "100m"
},
"insecure-registries": ["http://192.168.9.13"],
"storage-driver": "overlay2"
}
EOF
#添加k8s的阿里云yum 源
cat > /etc/yum.repos.d/kubernetes.repo << EOF
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
# 安装,本文安装1.28.15版本，可以指定其他版本进行安装
#yum install -y kubeadm-1.18.8 kubelet-1.18.8 kubectl-1.18.8 ipvsadm
yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes --nogpgcheck
Kubectl命令补全
kubectl completion bash > ~/.kube/completion.bash.inc
echo source '$HOME/.kube/completion.bash.inc' >> ~/.bashrc
source ~/.bashrc
kubeadm completion bash > /etc/bash_completion.d/kubeadm
kubelet completion bash > /etc/bash_completion.d/kubelet
source <(kubectl completion bash)
source <(kubeadm completion bash)
source <(kubelet completion bash)
echo "source <(kubectl completion bash)" >> ~/.bashrc
echo "source <(kubeadm completion bash)" >> ~/.bashrc
echo "source <(kubelet completion bash)" >> ~/.bashrc
#生成/etc/crictl.yaml
crictl config runtime-endpoint unix:///run/containerd/containerd.sock
#说明：
#kubeadm init：可以直接执行，自动去拉取镜像，再安装。主要是外网限制问题
#k8s集群安装，主要是镜像源，默认地址为国外，
#查看当前系统需要的k8s相关镜像版本：
#[root@master ~]# kubeadm config images list
#I0818 11:54:00.561459 4041 version.go:256] remote version is much newer: v1.33.4; falling back to: stable-1.28
#registry.k8s.io/kube-apiserver:v1.28.15
#registry.k8s.io/kube-controller-manager:v1.28.15
#registry.k8s.io/kube-scheduler:v1.28.15
#registry.k8s.io/kube-proxy:v1.28.15
#registry.k8s.io/pause:3.9 #这里检测是3.9，但是container默认下载3.6,好奇怪。暂时不明白
#registry.k8s.io/etcd:3.5.9-0
#registry.k8s.io/coredns/coredns:v1.10.1
#====前置环境配置完成========================
#提前处理containerd，后面也有处理步骤
#sed -i 's/"cri"//' /etc/containerd/config.toml
#推荐，使用下面的方法，上面的方法在下载pause组件时，会报错，无法连接
containerd config default >/etc/containerd/config.toml
sed -i 's#k8s.io/pause:3.6#aliyuncs.com/google_containers/pause:3.9#' /etc/containerd/config.toml
systemctl restart containerd && systemctl status containerd
systemctl enable kubelet
#配置crictl
echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile
#检测安装环境
#kubeadm init --dry-run
#查看镜像：ctr -n k8s.io i ls
#查看镜像：ctr -n k8s.io i ls | awk 'NR>=1{print $1}' | grep registry | grep -v sha256
#===所有主机,基础配置，完成====================
#====开始安装集群的master========================
#直接安装master
## 自启，先别启动
#systemctl enable kubelet
#官方文档中明确指出，必须在kubeadm init的时候设置 --pod-network-cidr=10.244.0.0/16
kubeadm init --image-repository registry.aliyuncs.com/google_containers \
--pod-network-cidr=10.244.0.0/16
kubeadm init --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers \
--pod-network-cidr=10.244.0.0/16
#查看执行日志：
#journalctl -fxe
#报错:
#[ERROR CRI]: container runtime is not running
#Containerd服务：在安装docker时，已经安装了，主要是配置问题
#解决方法：https://blog.csdn.net/qq_42257666/article/details/143894958
#vim /etc/containerd/config.toml 这个文件默认看不到pause:3.6，需要使用
#containerd config default |grep image
#containerd config default >/etc/containerd/config.toml
sed -i 's/registry.k8s.io\/pause:3.6/registry.aliyuncs.com\/google_containers\/pause:3.9/g' /etc/containerd/config.toml
#将原先的disabled_plugins = [“cri”]复制粘贴后注释掉，然后删除中括号内的"cri"，保存并退出。
sed -i 's/"cri"//' /etc/containerd/config.toml
systemctl restart containerd
#报错:/proc/sys/net/bridge/bridge-nf-call-iptables does not exist
echo "br_netfilter" | sudo tee /etc/modules-load.d/br_netfilter.conf
#查看下载的镜像
crictl config runtime-endpoint unix:///run/containerd/containerd.sock
crictl images
ctr -n k8s.io i ls | awk 'NR>=1{print $1}' | grep google_containers | grep -v sha256
#报错：
#[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
#[kubelet-check] Initial timeout of 40s passed.
#需要提前下载好pause:3.6，不然一直卡在上面：
#containerd config default
#a==docker+container
docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6
docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6 registry.k8s.io/pause:3.6
docker save -o pause.tar registry.k8s.io/pause:3.6
ctr -n k8s.io images import pause.tar
#b==直接container--推荐
ctr -n=k8s.io images pull registry.aliyuncs.com/google_containers/pause:3.6
ctr -n=k8s.io images tag registry.aliyuncs.com/google_containers/pause:3.6 registry.k8s.io/pause:3.6
# ctr -n k8s.io i ls | awk 'NR>=1{print $1}' | grep registry | grep -v sha256
#c==直接crictl---不推荐，不能修改tag
crictl pull registry.aliyuncs.com/google_containers/pause:3.6
ctr -n=k8s.io images pull registry.aliyuncs.com/google_containers/pause:3.6
ctr -n=k8s.io images tag registry.aliyuncs.com/google_containers/pause:3.6 registry.k8s.io/pause:3.6
#重置
kubeadm reset -f
#问题七: kubelet报错/var/lib/kubelet/config.yaml不存在，
#在执行kubeadm init 或kubeadm join之前，会发现启动的kubelet日志报错，读不到配置文件/var/lib/kubelet/config.yaml。不用担心，执行完kubeadm init/join之后，会自动生成配置文件。
#再次执行
kubeadm init --image-repository registry.aliyuncs.com/google_containers \
--pod-network-cidr=10.244.0.0/16
#-成功安装后提示：------------------------------------------
#Your Kubernetes control-plane has initialized successfully!
#
#To start using your cluster, you need to run the following as a regular user:
#
# mkdir -p $HOME/.kube
# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
# sudo chown $(id -u):$(id -g) $HOME/.kube/config
#
#Alternatively, if you are the root user, you can run:
#
# export KUBECONFIG=/etc/kubernetes/admin.conf
#
#You should now deploy a pod network to the cluster.
#Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
# https://kubernetes.io/docs/concepts/cluster-administration/addons/
#
#Then you can join any number of worker nodes by running the following on each as root:
#
kubeadm join 192.168.9.10:6443 --token 4o0p04.8g10um9uwar36hbl \
--discovery-token-ca-cert-hash sha256:985e98adfe692b5b39e5fd08008d0a828ef5490d68dcad769ad72911c945f680
#------------------------------------------
##为了在工作节点上也能使用kubectl，而kubectl命令需要使用kubernetes-admin来运行，
#因此我们需要将主节点中的【/etc/kubernetes/admin.conf】文件拷贝到工作节点相同目录下，这里推荐使用scp进行复制
#参考：https://cloud.tencent.com/developer/article/1479987
#拷贝授权文件，用于管理K8S集群，2种方法：
#master节点配置：
#方法一：
# mkdir -p $HOME/.kube
# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
# sudo chown $(id -u):$(id -g) $HOME/.kube/config
#方法二：
export KUBECONFIG=/etc/kubernetes/admin.conf
echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile
#node节点配置：
scp /etc/kubernetes/admin.conf 192.168.9.13:/etc/kubernetes/admin.conf
scp /etc/kubernetes/admin.conf 192.168.9.13:/etc/kubernetes/admin.conf
#设置kubeconfig文件
export KUBECONFIG=/etc/kubernetes/admin.conf
echo "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile
#实操：
#拷贝授权文件，到其他节点，用于管理K8S集群
[root@master ~]# scp .kube/ root@node1:/root/.kube/config
#-master成功安装后查询：------------------------------------------
[root@master ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
master NotReady control-plane 10m v1.28.2
[root@node1 ~]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 45m
#===node加入集群======================================================
#先：node加入集群,后在master安装网络插件，自动会安装在所有节点上。
echo "br_netfilter" | sudo tee /etc/modules-load.d/br_netfilter.conf
kubeadm join 192.168.9.10:6443 --token 4o0p04.8g10um9uwar36hbl \
--discovery-token-ca-cert-hash sha256:985e98adfe692b5b39e5fd08008d0a828ef5490d68dcad769ad72911c945f680
#在节点上使用kubectl
scp -r master:/root/.kube .
#===安装网络插件flannel======================================================
#后：随便在哪个节点，安装网络插件flannel，自动会安装在所有节点上。包括master
#就是在每个节点上，自动创建一个flannel网卡，自动配置10.244.0.x/16网段，方便pod通信
#https://my.oschina.net/vinci321/blog/16513861
#https://github.com/flannel-io
wget https://github.com/flannel-io/flannel/releases/download/v0.24.3/kube-flannel.yml
#注意版本：
[root@master ~]# grep image kube-flannel.yml
image: ghcr.io/flannel-io/flannel:v0.27.2
image: ghcr.io/flannel-io/flannel-cni-plugin:v1.7.1-flannel1
image: ghcr.io/flannel-io/flannel:v0.27.2
#修改flannel源
sed -i 's#ghcr.io/flannel-io#m.daocloud.io/docker.io/flannel#g' kube-flannel.yml
#手动，先下载相应版本
ctr -n=k8s.io images pull m.daocloud.io/docker.io/flannel/flannel:v0.27.2
ctr -n=k8s.io images pull m.daocloud.io/docker.io/flannel/flannel-cni-plugin:v1.7.1-flannel1
ctr -n=k8s.io images tag m.daocloud.io/docker.io/flannel/flannel:v0.27.2 ghcr.io/flannel-io/flannel:v0.27.2
ctr -n=k8s.io images tag m.daocloud.io/docker.io/flannel/flannel-cni-plugin:v1.7.1-flannel1 ghcr.io/flannel-io/flannel-cni-plugin:v1.7.1-flannel1
kubectl apply -f kube-flannel.yml
#输出：并未成功
#namespace/kube-flannel unchanged 创建kube-flannel的命名空间
#serviceaccount/flannel unchanged 创建kube-flannel的sa
#clusterrole.rbac.authorization.k8s.io/flannel unchanged 创建kube-flannel的权限和绑定
#clusterrolebinding.rbac.authorization.k8s.io/flannel unchanged
#configmap/kube-flannel-cfg unchanged 创建kube-flannel的的configmap
#daemonset.apps/kube-flannel-ds configured 创建kube-flannel的控制器。模式为daemonset
#查看 pod
[root@master ~]# kubectl get pods -n kube-flannel -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-flannel-ds-x5mj4 0/1 CrashLoopBackOff 3 (44s ago) 95s 192.168.9.10 master <none> <none>
[root@node1 ~]# kubectl get pods -n kube-flannel -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-flannel-ds-4z5d4 1/1 Running 0 4m6s 192.168.9.10 master <none> <none>
kube-flannel-ds-mcqtw 0/1 Init:1/2 0 4m6s 192.168.9.11 node1 <none> <none>
[root@node1 ~]#
#发现 pod 没起来，看起来是拉取镜像失败。进一步查看信息
kubectl describe pod kube-flannel -n kube-flannel
kubectl logs -f -n kube-flannel kube-flannel-ds-x5mj4
#如果报错，直接删除后，再次执行
#删除flannel
kubectl delete -f kube-flannel.yml
kubectl apply -f kube-flannel.yml
#报错[root@node1 ~]# kubectl get pods -n kube-flannel -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-flannel-ds-4z5d4 1/1 Running 0 4m6s 192.168.9.10 master <none> <none>
kube-flannel-ds-mcqtw 0/1 Init:1/2 0 4m6s 192.168.9.11 node1 <none> <none>
#node上也要安装pause:3.6，安装完成后，等待1分钟，就变成Running，感觉好难受
ctr -n=k8s.io images pull registry.aliyuncs.com/google_containers/pause:3.6
ctr -n=k8s.io images tag registry.aliyuncs.com/google_containers/pause:3.6 registry.k8s.io/pause:3.6
#报错教程
# https://blog.csdn.net/weixin_52418790/article/details/144423279
# https://www.cnblogs.com/cc21216695/p/17628885.html
#==k8s集群安装，全部完成======================================
Master隔离（可选）
默认情况下，由于安全原因，集群并不会将pods部署在Master节点上。但是在开发环境下，我们可能就只有一个Master节点，这时可以使用下面的命令来解除这个限制：
$ sudo kubectl taint nodes --all node-role.kubernetes.io/master-
## 输出
node/ubuntu1 untainted
等待一会，我们可以在Master节点上使用kubectl get nodes命令来查看节点的状态：
$ sudo kubectl get nodes
NAME STATUS ROLES AGE VERSION
kube-node1 Ready master 3h3m v1.13.3
kube-node2 Ready <none> 118s v1.13.3
kube-node3 Ready <none> 95s v1.13.3
#为Work节点设置标签
[root@master ~]# kubectl label nodes node node-role.kubernetes.io/work=work
#常用 Kubernetes 命令整理
kubeadm config images list
kubectl get pods -A -o wide
kubectl -n kube-system get cm kubeadm-config -oyaml
#=====查看token===================================
#默认情况下 Token 过期是时间是24小时，如果 Token 过期以后，可以输入以下命令，生成新的 Token
kubeadm token list
#重新获取一下token
kubeadm token create --print-join-command
kubeadm token create
# cat /etc/kubernetes/pki/ca.crt
#生成新ca证书sha256编码hash值
openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
#=====token完成===================================
#查看集群状态
kubectl get nodes
kubectl get nodes -o wide
kubectl get pods -A
kubectl get pods -A
kubectl get pods -A -o wide -w
kubectl get all
kubectl get all -o wide
#节点管理
kubectl get nodes # 查看节点状态
kubectl describe node <node-name> # 查看节点详细信息
kubectl cordon <node-name> # 标记节点不可调度
kubectl uncordon <node-name> # 标记节点可调度
kubectl drain <node-name> # 驱逐节点上的 Pod，准备维护
# Pod 和服务管理
kubectl get pods -A # 查看所有命名空间的 Pod
kubectl get pods # 查看当前命名空间 Pod
kubectl describe pod <pod-name> # 查看 Pod 详细信息
kubectl logs <pod-name> # 查看 Pod 日志
kubectl exec -it <pod-name> -- bash # 进入 Pod 容器终端
#部署与管理应用
kubectl apply -f <filename.yaml> # 创建或更新资源
kubectl delete -f <filename.yaml> # 删除资源
kubectl rollout status deployment/<deployment-name> # 查看滚动更新状态
kubectl scale deployment/<name> --replicas=3 # 扩缩容
#集群信息查看
kubectl cluster-info # 查看集群信息
kubectl get namespaces # 查看命名空间列表
kubectl get svc # 查看服务
yum reinstall -y kubernetes-cni
# 获取节点
kubectl get nodes -o wide
# 实时查询nodes状态
watch kubectl get nodes -o wide
# 获取pod
kubectl get pods --all-namespaces -o wide
# 查看镜像列表
kubeadm config images list
# 节点加入集群
kubeadm token create --print-join-command
# 描述node
kubectl describe node k8s-master
# 描述pod
kubectl describe pod kube-flannel-ds-hs8bq --namespace=kube-system
#===另类安装，也不成功，主要是镜像源问题，默认地址为国外，=========================
#kubeadm config print init-defaults >> kubeadm.conf
#sed -i "s#imageRepository: .*#imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers#g" kubeadm.conf
#kubeadm config images pull --config kubeadm.conf
#kubeadm init --config kubeadm.conf
#kubeadm init \
# --apiserver-advertise-address=192.168.9.10 \
# --image-repository registry.aliyuncs.com/google_containers \
# --kubernetes-version v1.28.2 \
# --service-cidr=10.10.0.0/16 \
# --pod-network-cidr=10.244.0.0/16 \
# --ignore-preflight-errors=all
k8s1.20 搭建-抛弃docker使用containerd
#https://blog.csdn.net/xingzuo_1840/article/details/119579751
#crictl pull 命令拉取的镜像都是在k8s.io 这个namespace下。
#所以如果想在ctr中看到，需要指定namespace
ctr -n k8s.io image ls === crictl images
ctr -n k8s.io i ls | awk 'NR>=1{print $1}' | grep registry | grep -v sha256
ctr -n k8s.io image pull
ctr -n k8s.io image tag
ctr -n k8s.io image pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.9
#查询已下载的镜像
ctr -n k8s.io i ls | awk 'NR>=1{print $1}' | grep google_containers | grep -v sha256
#导出镜像
ctr -n k8s.io i export master-v1.24.17.tar.gz `ctr -n k8s.io i ls | awk 'NR>=1{print $1}' | grep google_containers | grep -v sha256`
#导入镜像
ctr -n k8s.io i import master-v1.24.17.tar.gz `ctr -n k8s.io i ls | awk 'NR>=1{print $1}' | grep google_containers | grep -v sha256`
#Kubernetes自动化部署脚本
#https://www.jianshu.com/p/1530cd0f5485
crictl config runtime-endpoint unix:///run/containerd/containerd.sock
#NFS 服务，请确保任意节点都可成功挂载该 NFS 服务。
mkdir /tmp/testnfs \
&& mount -t nfs : /tmp/testnfs \
&& echo "hello nfs" >> /tmp/testnfs/test.txt \
&& cat /tmp/testnfs/test.txt
#k8s-v1.28.15-install-docker+containerd+flannel--20250826--ok
#现在主要是限制外网，引起的各种安装麻烦，如果有梯子，应该很简单。
#参考教程：# https://zhuanlan.zhihu.com/p/699063977# https://zhuanlan.zhihu.com/p/675796199# https://blog.csdn.net/dyj095/category_12997521.html# https://blog.csdn.net/dyj095/article/details/130512120
#kubectl和webUI:客户端。#etcd:K-V结构的分布式数据库，保存了整个集群的状态和配置信息，是集群的持久化机制，遇到故障可以使集群快速恢复。该组件发生故障，整个K8S就会瘫痪。 apiserver:所有服务访问和资源操作的统一入口，并提供认证、授权、访问控制、API注册和发现等机制;#replication controller:它能够保证Pod持续运行，并且在任何时候都有指定数量的Pod副本，在此基础上提供一些高级特性，比如滚动升级和弹性伸缩 #scheduler:负责资源的调度，按照预定的调度策略将Pod调度到相应的机器上;#kubelet:负责维护容器的生命周期,通俗理解就是对于容器的创建和销毁就是kubelet来控制的。 #kube-proxy:负责为Service提供cluster内部的服务发现和负载均衡;#pod:在K8S中，最小的管理元素不是一个个独立的容器，而是Pod,Pod是最小的，管理，创建，计划的最小单元。#Pod的设计理念是支持多个容器在一个Pod中共享网络地址和文件系统。
##注意事项从 Kubernetes 1.22 开始，如果未明确指定 cgroupDriver，kubeadm 会默认使用 systemdKubernetes 1.28 引入了 cgroup 驱动自动检测功能（Alpha 阶段）修改配置时，建议逐个节点操作，确保工作负载平稳迁移不同容器运行时（Docker、containerd 等）的 systemd 驱动配置方式可能不同
#k8s v1.28.15安装---testok---20250819#环境：centos7.9 + Docker version 26.1.4 + k8s-v1.28.15#原则是：全使用yum安装和直接安装，简单点#k8s的repo源是https://mirrors.aliyun.com ,注意地址说明#https://developer.aliyun.com/mirror/kubernetes?spm=a2c6h.13651102.0.0.25971b11TFhCoA#aliyun老地址最高支持k8s-v1.28.15 新地址支持k8s-v1.33.4#机器和host准备：硬件配置：大于等于2GB+2个CPU,硬盘 >= 20G
#master-192.168.9.10#node1-192.168.9.11#node2-192.168.9.12#harbor-192.168.9.13#kuboard-192.168.9.14
#端口使用：kubelet 10250 10248kube-controller 127.0.0.1:10257kube-schedule 10259kube-proxy 10256 127.0.0.1:10249kube-apiserver 6443etcd 2379 2380 2381
#===所有主机====================#初始配置（所有主机）#主机配置主机名hostnamectl set-hostname masterhostnamectl set-hostname node1hostnamectl set-hostname node2#修改hosts文件cat >> /etc/hosts << EOF192.168.9.10 master192.168.9.11 node1192.168.9.12 node2192.168.9.13 harborEOF
echo "br_netfilter" | sudo tee /etc/modules-load.d/br_netfilter.conf
#永久关闭sed -ri 's/.*swap.*/#&/' /etc/fstab
#允许 iptables 检查桥接流量cat > /etc/sysctl.d/k8s.conf << EOFnet.bridge.bridge-nf-call-ip6tables = 1net.bridge.bridge-nf-call-iptables = 1net.ipv4.ip_forward = 1EOFsysctl -p
#常用配置yum install -y bash-completion wget vim-enhanced net-tools ntpdate#ntpdate ntp1.aliyun.comsed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
#固定IPsed -i 's/dhcp/static/g' /etc/sysconfig/network-scripts/ifcfg-ens33echo "IPADDR=192.168.9.12NETMASK=255.255.255.0GATEWAY=192.168.9.2DNS1=192.168.9.2" >>/etc/sysconfig/network-scripts/ifcfg-ens33systemctl restart network#添加docker的阿里云yum源curl -o /etc/yum.repos.d/docker-ce.repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repoyum install -y docker-ce docker-composesystemctl enable --now docker
#配置docker镜像源cat >> /etc/docker/daemon.json << EOF{ "registry-mirrors": [ "https://2a6bf1988cb6428c877f723ec7530dbc.mirror.swr.myhuaweicloud.com", "https://docker.m.daocloud.io", "https://hub-mirror.c.163.com", "https://mirror.baidubce.com", "https://your_preferred_mirror", "https://dockerhub.icu", "https://docker.registry.cyou", "https://docker-cf.registry.cyou", "https://dockercf.jsdelivr.fyi", "https://docker.jsdelivr.fyi", "https://dockertest.jsdelivr.fyi", "https://mirror.aliyuncs.com", "https://dockerproxy.com", "https://mirror.baidubce.com", "https://docker.m.daocloud.io", "https://docker.nju.edu.cn", "https://docker.mirrors.sjtug.sjtu.edu.cn", "https://docker.mirrors.ustc.edu.cn", "https://mirror.iscas.ac.cn", "https://docker.rainbond.cc" ], "exec-opts": ["native.cgroupdriver=systemd"], "log-driver": "json-file", "data-root": "/data/var/lib/docker", "log-opts": { "max-size": "100m" }, "insecure-registries": ["http://192.168.9.13"], "storage-driver": "overlay2"}EOF
#添加k8s的阿里云yum 源cat > /etc/yum.repos.d/kubernetes.repo << EOF[kubernetes]name=Kubernetesbaseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64enabled=1gpgcheck=0repo_gpgcheck=0gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpgEOF
# 安装,本文安装1.28.15版本，可以指定其他版本进行安装#yum install -y kubeadm-1.18.8 kubelet-1.18.8 kubectl-1.18.8 ipvsadm
yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes --nogpgcheckKubectl命令补全
kubectl completion bash > ~/.kube/completion.bash.incecho source '$HOME/.kube/completion.bash.inc' >> ~/.bashrcsource ~/.bashrc
kubeadm completion bash > /etc/bash_completion.d/kubeadmkubelet completion bash > /etc/bash_completion.d/kubelet
source <(kubectl completion bash)source <(kubeadm completion bash)source <(kubelet completion bash)echo "source <(kubectl completion bash)" >> ~/.bashrcecho "source <(kubeadm completion bash)" >> ~/.bashrcecho "source <(kubelet completion bash)" >> ~/.bashrc#生成/etc/crictl.yamlcrictl config runtime-endpoint unix:///run/containerd/containerd.sock
#说明：#kubeadm init：可以直接执行，自动去拉取镜像，再安装。主要是外网限制问题#k8s集群安装，主要是镜像源，默认地址为国外，#查看当前系统需要的k8s相关镜像版本：#[root@master ~]# kubeadm config images list#I0818 11:54:00.561459 4041 version.go:256] remote version is much newer: v1.33.4; falling back to: stable-1.28#registry.k8s.io/kube-apiserver:v1.28.15#registry.k8s.io/kube-controller-manager:v1.28.15#registry.k8s.io/kube-scheduler:v1.28.15#registry.k8s.io/kube-proxy:v1.28.15#registry.k8s.io/pause:3.9 #这里检测是3.9，但是container默认下载3.6,好奇怪。暂时不明白#registry.k8s.io/etcd:3.5.9-0#registry.k8s.io/coredns/coredns:v1.10.1#====前置环境配置完成========================
#提前处理containerd，后面也有处理步骤#sed -i 's/"cri"//' /etc/containerd/config.toml#推荐，使用下面的方法，上面的方法在下载pause组件时，会报错，无法连接containerd config default >/etc/containerd/config.tomlsed -i 's#k8s.io/pause:3.6#aliyuncs.com/google_containers/pause:3.9#' /etc/containerd/config.tomlsystemctl restart containerd && systemctl status containerdsystemctl enable kubelet#配置crictlecho "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile
#检测安装环境#kubeadm init --dry-run
#查看镜像：ctr -n k8s.io i ls#查看镜像：ctr -n k8s.io i ls | awk 'NR>=1{print $1}' | grep registry | grep -v sha256#===所有主机,基础配置，完成====================#====开始安装集群的master========================#直接安装master## 自启，先别启动#systemctl enable kubelet#官方文档中明确指出，必须在kubeadm init的时候设置 --pod-network-cidr=10.244.0.0/16
kubeadm init --image-repository registry.aliyuncs.com/google_containers \ --pod-network-cidr=10.244.0.0/16
kubeadm init --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers \ --pod-network-cidr=10.244.0.0/16 #查看执行日志：#journalctl -fxe
#报错:#[ERROR CRI]: container runtime is not running#Containerd服务：在安装docker时，已经安装了，主要是配置问题#解决方法：https://blog.csdn.net/qq_42257666/article/details/143894958#vim /etc/containerd/config.toml 这个文件默认看不到pause:3.6，需要使用#containerd config default |grep image#containerd config default >/etc/containerd/config.tomlsed -i 's/registry.k8s.io\/pause:3.6/registry.aliyuncs.com\/google_containers\/pause:3.9/g' /etc/containerd/config.toml
#将原先的disabled_plugins = [“cri”]复制粘贴后注释掉，然后删除中括号内的"cri"，保存并退出。sed -i 's/"cri"//' /etc/containerd/config.tomlsystemctl restart containerd
#报错:/proc/sys/net/bridge/bridge-nf-call-iptables does not existecho "br_netfilter" | sudo tee /etc/modules-load.d/br_netfilter.conf
#查看下载的镜像crictl config runtime-endpoint unix:///run/containerd/containerd.sockcrictl imagesctr -n k8s.io i ls | awk 'NR>=1{print $1}' | grep google_containers | grep -v sha256#报错：#[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s#[kubelet-check] Initial timeout of 40s passed.#需要提前下载好pause:3.6，不然一直卡在上面：#containerd config default#a==docker+containerdocker pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6docker tag registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6 registry.k8s.io/pause:3.6docker save -o pause.tar registry.k8s.io/pause:3.6ctr -n k8s.io images import pause.tar#b==直接container--推荐ctr -n=k8s.io images pull registry.aliyuncs.com/google_containers/pause:3.6 ctr -n=k8s.io images tag registry.aliyuncs.com/google_containers/pause:3.6 registry.k8s.io/pause:3.6# ctr -n k8s.io i ls | awk 'NR>=1{print $1}' | grep registry | grep -v sha256
#c==直接crictl---不推荐，不能修改tagcrictl pull registry.aliyuncs.com/google_containers/pause:3.6
ctr -n=k8s.io images pull registry.aliyuncs.com/google_containers/pause:3.6 ctr -n=k8s.io images tag registry.aliyuncs.com/google_containers/pause:3.6 registry.k8s.io/pause:3.6
#重置kubeadm reset -f
#问题七: kubelet报错/var/lib/kubelet/config.yaml不存在，#在执行kubeadm init 或kubeadm join之前，会发现启动的kubelet日志报错，读不到配置文件/var/lib/kubelet/config.yaml。不用担心，执行完kubeadm init/join之后，会自动生成配置文件。#再次执行kubeadm init --image-repository registry.aliyuncs.com/google_containers \ --pod-network-cidr=10.244.0.0/16
#-成功安装后提示：------------------------------------------#Your Kubernetes control-plane has initialized successfully!##To start using your cluster, you need to run the following as a regular user:## mkdir -p $HOME/.kube# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config# sudo chown $(id -u):$(id -g) $HOME/.kube/config##Alternatively, if you are the root user, you can run:## export KUBECONFIG=/etc/kubernetes/admin.conf##You should now deploy a pod network to the cluster.#Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:# https://kubernetes.io/docs/concepts/cluster-administration/addons/##Then you can join any number of worker nodes by running the following on each as root:#kubeadm join 192.168.9.10:6443 --token 4o0p04.8g10um9uwar36hbl \ --discovery-token-ca-cert-hash sha256:985e98adfe692b5b39e5fd08008d0a828ef5490d68dcad769ad72911c945f680
#------------------------------------------##为了在工作节点上也能使用kubectl，而kubectl命令需要使用kubernetes-admin来运行，#因此我们需要将主节点中的【/etc/kubernetes/admin.conf】文件拷贝到工作节点相同目录下，这里推荐使用scp进行复制#参考：https://cloud.tencent.com/developer/article/1479987#拷贝授权文件，用于管理K8S集群，2种方法：#master节点配置：#方法一：# mkdir -p $HOME/.kube# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config# sudo chown $(id -u):$(id -g) $HOME/.kube/config#方法二：export KUBECONFIG=/etc/kubernetes/admin.confecho "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile#node节点配置：scp /etc/kubernetes/admin.conf 192.168.9.13:/etc/kubernetes/admin.confscp /etc/kubernetes/admin.conf 192.168.9.13:/etc/kubernetes/admin.conf#设置kubeconfig文件export KUBECONFIG=/etc/kubernetes/admin.confecho "export KUBECONFIG=/etc/kubernetes/admin.conf" >> ~/.bash_profile#实操：#拷贝授权文件，到其他节点，用于管理K8S集群[root@master ~]# scp .kube/ root@node1:/root/.kube/config#-master成功安装后查询：------------------------------------------[root@master ~]# kubectl get nodesNAME STATUS ROLES AGE VERSIONmaster NotReady control-plane 10m v1.28.2
[root@node1 ~]# kubectl get svcNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGEkubernetes ClusterIP 10.96.0.1 <none> 443/TCP 45m
#===node加入集群======================================================#先：node加入集群,后在master安装网络插件，自动会安装在所有节点上。
echo "br_netfilter" | sudo tee /etc/modules-load.d/br_netfilter.conf
kubeadm join 192.168.9.10:6443 --token 4o0p04.8g10um9uwar36hbl \ --discovery-token-ca-cert-hash sha256:985e98adfe692b5b39e5fd08008d0a828ef5490d68dcad769ad72911c945f680#在节点上使用kubectlscp -r master:/root/.kube .
#===安装网络插件flannel======================================================#后：随便在哪个节点，安装网络插件flannel，自动会安装在所有节点上。包括master#就是在每个节点上，自动创建一个flannel网卡，自动配置10.244.0.x/16网段，方便pod通信#https://my.oschina.net/vinci321/blog/16513861#https://github.com/flannel-iowget https://github.com/flannel-io/flannel/releases/download/v0.24.3/kube-flannel.yml
#注意版本：[root@master ~]# grep image kube-flannel.yml image: ghcr.io/flannel-io/flannel:v0.27.2 image: ghcr.io/flannel-io/flannel-cni-plugin:v1.7.1-flannel1 image: ghcr.io/flannel-io/flannel:v0.27.2#修改flannel源sed -i 's#ghcr.io/flannel-io#m.daocloud.io/docker.io/flannel#g' kube-flannel.yml
#手动，先下载相应版本ctr -n=k8s.io images pull m.daocloud.io/docker.io/flannel/flannel:v0.27.2ctr -n=k8s.io images pull m.daocloud.io/docker.io/flannel/flannel-cni-plugin:v1.7.1-flannel1ctr -n=k8s.io images tag m.daocloud.io/docker.io/flannel/flannel:v0.27.2 ghcr.io/flannel-io/flannel:v0.27.2ctr -n=k8s.io images tag m.daocloud.io/docker.io/flannel/flannel-cni-plugin:v1.7.1-flannel1 ghcr.io/flannel-io/flannel-cni-plugin:v1.7.1-flannel1
kubectl apply -f kube-flannel.yml#输出：并未成功#namespace/kube-flannel unchanged 创建kube-flannel的命名空间#serviceaccount/flannel unchanged 创建kube-flannel的sa#clusterrole.rbac.authorization.k8s.io/flannel unchanged 创建kube-flannel的权限和绑定#clusterrolebinding.rbac.authorization.k8s.io/flannel unchanged#configmap/kube-flannel-cfg unchanged 创建kube-flannel的的configmap#daemonset.apps/kube-flannel-ds configured 创建kube-flannel的控制器。模式为daemonset
#查看 pod[root@master ~]# kubectl get pods -n kube-flannel -o wideNAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATESkube-flannel-ds-x5mj4 0/1 CrashLoopBackOff 3 (44s ago) 95s 192.168.9.10 master <none> <none>[root@node1 ~]# kubectl get pods -n kube-flannel -o wideNAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATESkube-flannel-ds-4z5d4 1/1 Running 0 4m6s 192.168.9.10 master <none> <none>kube-flannel-ds-mcqtw 0/1 Init:1/2 0 4m6s 192.168.9.11 node1 <none> <none>[root@node1 ~]#
#发现 pod 没起来，看起来是拉取镜像失败。进一步查看信息kubectl describe pod kube-flannel -n kube-flannel
kubectl logs -f -n kube-flannel kube-flannel-ds-x5mj4 #如果报错，直接删除后，再次执行#删除flannelkubectl delete -f kube-flannel.yml kubectl apply -f kube-flannel.yml
#报错[root@node1 ~]# kubectl get pods -n kube-flannel -o wideNAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATESkube-flannel-ds-4z5d4 1/1 Running 0 4m6s 192.168.9.10 master <none> <none>kube-flannel-ds-mcqtw 0/1 Init:1/2 0 4m6s 192.168.9.11 node1 <none> <none>
#node上也要安装pause:3.6，安装完成后，等待1分钟，就变成Running，感觉好难受ctr -n=k8s.io images pull registry.aliyuncs.com/google_containers/pause:3.6 ctr -n=k8s.io images tag registry.aliyuncs.com/google_containers/pause:3.6 registry.k8s.io/pause:3.6#报错教程# https://blog.csdn.net/weixin_52418790/article/details/144423279# https://www.cnblogs.com/cc21216695/p/17628885.html#==k8s集群安装，全部完成======================================Master隔离（可选）默认情况下，由于安全原因，集群并不会将pods部署在Master节点上。但是在开发环境下，我们可能就只有一个Master节点，这时可以使用下面的命令来解除这个限制：
$ sudo kubectl taint nodes --all node-role.kubernetes.io/master-## 输出node/ubuntu1 untainted等待一会，我们可以在Master节点上使用kubectl get nodes命令来查看节点的状态：
$ sudo kubectl get nodesNAME STATUS ROLES AGE VERSIONkube-node1 Ready master 3h3m v1.13.3kube-node2 Ready <none> 118s v1.13.3kube-node3 Ready <none> 95s v1.13.3
#为Work节点设置标签[root@master ~]# kubectl label nodes node node-role.kubernetes.io/work=work
#常用 Kubernetes 命令整理kubeadm config images listkubectl get pods -A -o widekubectl -n kube-system get cm kubeadm-config -oyaml
#=====查看token===================================#默认情况下 Token 过期是时间是24小时，如果 Token 过期以后，可以输入以下命令，生成新的 Tokenkubeadm token list#重新获取一下tokenkubeadm token create --print-join-command
kubeadm token create
# cat /etc/kubernetes/pki/ca.crt#生成新ca证书sha256编码hash值openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'#=====token完成===================================
#查看集群状态kubectl get nodeskubectl get nodes -o widekubectl get pods -Akubectl get pods -A kubectl get pods -A -o wide -wkubectl get all kubectl get all -o wide#节点管理kubectl get nodes # 查看节点状态kubectl describe node <node-name> # 查看节点详细信息kubectl cordon <node-name> # 标记节点不可调度kubectl uncordon <node-name> # 标记节点可调度kubectl drain <node-name> # 驱逐节点上的 Pod，准备维护# Pod 和服务管理kubectl get pods -A # 查看所有命名空间的 Podkubectl get pods # 查看当前命名空间 Podkubectl describe pod <pod-name> # 查看 Pod 详细信息kubectl logs <pod-name> # 查看 Pod 日志kubectl exec -it <pod-name> -- bash # 进入 Pod 容器终端#部署与管理应用kubectl apply -f <filename.yaml> # 创建或更新资源kubectl delete -f <filename.yaml> # 删除资源kubectl rollout status deployment/<deployment-name> # 查看滚动更新状态kubectl scale deployment/<name> --replicas=3 # 扩缩容#集群信息查看kubectl cluster-info # 查看集群信息kubectl get namespaces # 查看命名空间列表kubectl get svc # 查看服务yum reinstall -y kubernetes-cni
# 获取节点kubectl get nodes -o wide# 实时查询nodes状态watch kubectl get nodes -o wide# 获取podkubectl get pods --all-namespaces -o wide# 查看镜像列表kubeadm config images list# 节点加入集群kubeadm token create --print-join-command# 描述nodekubectl describe node k8s-master# 描述podkubectl describe pod kube-flannel-ds-hs8bq --namespace=kube-system
#===另类安装，也不成功，主要是镜像源问题，默认地址为国外，=========================#kubeadm config print init-defaults >> kubeadm.conf#sed -i "s#imageRepository: .*#imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers#g" kubeadm.conf#kubeadm config images pull --config kubeadm.conf#kubeadm init --config kubeadm.conf
#kubeadm init \# --apiserver-advertise-address=192.168.9.10 \# --image-repository registry.aliyuncs.com/google_containers \# --kubernetes-version v1.28.2 \# --service-cidr=10.10.0.0/16 \# --pod-network-cidr=10.244.0.0/16 \# --ignore-preflight-errors=allk8s1.20 搭建-抛弃docker使用containerd#https://blog.csdn.net/xingzuo_1840/article/details/119579751#crictl pull 命令拉取的镜像都是在k8s.io 这个namespace下。#所以如果想在ctr中看到，需要指定namespacectr -n k8s.io image ls === crictl imagesctr -n k8s.io i ls | awk 'NR>=1{print $1}' | grep registry | grep -v sha256ctr -n k8s.io image pullctr -n k8s.io image tagctr -n k8s.io image pull registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.9#查询已下载的镜像ctr -n k8s.io i ls | awk 'NR>=1{print $1}' | grep google_containers | grep -v sha256#导出镜像ctr -n k8s.io i export master-v1.24.17.tar.gz `ctr -n k8s.io i ls | awk 'NR>=1{print $1}' | grep google_containers | grep -v sha256`#导入镜像ctr -n k8s.io i import master-v1.24.17.tar.gz `ctr -n k8s.io i ls | awk 'NR>=1{print $1}' | grep google_containers | grep -v sha256`
#Kubernetes自动化部署脚本#https://www.jianshu.com/p/1530cd0f5485
crictl config runtime-endpoint unix:///run/containerd/containerd.sock
#NFS 服务，请确保任意节点都可成功挂载该 NFS 服务。mkdir /tmp/testnfs \&& mount -t nfs : /tmp/testnfs \&& echo "hello nfs" >> /tmp/testnfs/test.txt \&& cat /tmp/testnfs/test.txt