k8s 1.29 一键安装脚本, 丝滑致极

在这里插入图片描述
博客原文

文章目录

- 集群配置
- - 配置清单
  - 集群规划
  - 集群网络规划
- 环境初始化
- - 主机配置
- 安装脚本
- - 需要魔法的脚本
  - 不需要魔法的脚本
  - 配置自动补全
  - 加入其余节点
- 验证集群

高可用版本: 高可用 k8s 1.29 一键安装脚本

集群配置

配置清单

OS： ubuntu 20.04
kubernetes： 1.29.1
Container Runtime：Containerd 1.7.11
CRI: runc 1.10
CNI: cni-plugin 1.4

集群规划

IP	Hostname	配置
192.168.254.130	master01	2C 4G 30G
192.168.254.131	node01	2C 4G 30G
192.168.254.132	node02	2C 4G 30G

集群网络规划

Pod 网络: 10.244.0.0/16
Service 网络: 10.96.0.0/12
Node 网络: 11.0.1.0/24

环境初始化

主机配置

ssh-keygen
ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.254.131
ssh-copy-id -i /root/.ssh/id_rsa.pub root@192.168.254.132

# 将节点加入 hosts
cat << EOF >> /etc/hosts
192.168.254.130 master01
192.168.254.131 node01
192.168.254.132 node02
EOF

安装脚本

**前置条件: ** 脚本中存在拉取国外资源, 需要你配置代理 ==> [如何让虚拟机拥有愉快网络环境](https://ai-feier.github.io/p/%E5%A6%82%E4%BD%95%E8%AE%A9%E8%99%9A%E6%8B%9F%E6%9C%BA%E6%8B%A5%E6%9C%89%E6%84%89%E5%BF%AB%E7%BD%91%E7%BB%9C%E7%8E%AF%E5%A2%83/)

需要:

虚拟机代理
apt 下载代理

在所有节点执行以下脚本

脚本功能:

时间同步
关闭 swap
启用内核模块
安装 ipvs 并启用内核参数
安装 containerd, runc, cni
更改 containerd 沙箱镜像和 cgroup 并且配置镜像加速
安装最新 kubelet, kubeadm, kubectl

注意: 请先通过export name=master01方式设置当前 node 的 hostname

需要魔法的脚本

install.sh:

export name=master01  # 改为你 hostname 的名称, 脚本中删除该行
#!/bin/bash

hostnamectl set-hostname $name

# 阿里源
mv /etc/apt/sources.list /etc/apt/sources.list.bak
cat <<EOF > /etc/apt/sources.list
deb https://mirrors.aliyun.com/ubuntu/ focal main restricted universe multiverse
deb-src https://mirrors.aliyun.com/ubuntu/ focal main restricted universe multiverse
deb https://mirrors.aliyun.com/ubuntu/ focal-security main restricted universe multiverse
deb-src https://mirrors.aliyun.com/ubuntu/ focal-security main restricted universe multiverse
deb https://mirrors.aliyun.com/ubuntu/ focal-updates main restricted universe multiverse
deb-src https://mirrors.aliyun.com/ubuntu/ focal-updates main restricted universe multiversedeb https://mirrors.aliyun.com/ubuntu/ focal-backports main restricted universe multiverse
deb-src https://mirrors.aliyun.com/ubuntu/ focal-backports main restricted universe multiverse
EOF
apt update


# 时间同步
timedatectl set-timezone Asia/Shanghai
#安装chrony，联网同步时间
apt install chrony -y && systemctl enable --now chronyd

# 禁用 swap
sudo swapoff -a && sed -i '/swap/s/^/#/' /etc/fstab

# 安装 ipvs
apt install -y ipset ipvsadm

# 配置需要的内核模块
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

# 启动模块
sudo modprobe overlay
sudo modprobe br_netfilter

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

# 是 sysctl 参数生效
sudo sysctl --system
# 检验是否配置成功
#lsmod | grep br_netfilter
#lsmod | grep overlay
#sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-ip6tables net.ipv4.ip_forward


# 配置 ipvs 内核参数
cat <<EOF | sudo tee /etc/modules-load.d/ipvs.conf
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack
EOF

# 内核加载 ipvs
sudo modprobe ip_vs
sudo modprobe ip_vs_rr
sudo modprobe ip_vs_wrr
sudo modprobe ip_vs_sh
sudo modprobe nf_conntrack
# 确认ipvs模块加载
#lsmod |grep -e ip_vs -e nf_conntrack


# 安装 Containerd
wget -c https://github.com/containerd/containerd/releases/download/v1.7.11/containerd-1.7.11-linux-amd64.tar.gz
tar -xzvf containerd-1.7.11-linux-amd64.tar.gz
#解压出来一个bin目录,containerd可执行文件都在bin目录里面
mv bin/* /usr/local/bin/
rm -rf bin

#使用systemcd来管理containerd
cat << EOF > /usr/lib/systemd/system/containerd.service
[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target local-fs.target

[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/containerd

Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5

# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity

# Comment TasksMax if your systemd version does not supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
OOMScoreAdjust=-999

[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload && systemctl enable --now containerd 
#systemctl  status containerd


# 安装 runc
#runc是容器运行时，runc实现了容器的init，run，create，ps...我们在运行容器所需要的cmd：
curl -LO https://github.com/opencontainers/runc/releases/download/v1.1.10/runc.amd64 && \
install -m 755 runc.amd64 /usr/local/sbin/runc

# 安装 CNI plugins
wget -c https://github.com/containernetworking/plugins/releases/download/v1.4.0/cni-plugins-linux-amd64-v1.4.0.tgz
#根据官网的安装步骤来，创建一个目录用于存放cni插件
mkdir -p /opt/cni/bin
tar -xzvf  cni-plugins-linux-amd64-v1.4.0.tgz -C /opt/cni/bin/

# 修改 Containd 配置
#修改containerd的配置，因为containerd默认从k8s官网拉取镜像
#创建一个目录用于存放containerd的配置文件
mkdir -p /etc/containerd
#把containerd配置导出到文件
containerd config default | sudo tee /etc/containerd/config.toml

# 修改沙箱镜像
sed -i 's#sandbox_image = "registry.k8s.io/pause:.*"#sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.9"#' /etc/containerd/config.toml
# 修改 cgroup 为 systemd
sed -i 's#SystemdCgroup = false#SystemdCgroup = true#' /etc/containerd/config.toml
# 配置镜像加速
sed -i 's#config_path = ""#config_path = "/etc/containerd/certs.d"#' /etc/containerd/config.toml

# 配置 Containerd 镜像源
# docker hub镜像加速
mkdir -p /etc/containerd/certs.d/docker.io
cat > /etc/containerd/certs.d/docker.io/hosts.toml << EOF
server = "https://docker.io"
[host."https://dockerproxy.com"]
  capabilities = ["pull", "resolve"]

[host."https://docker.m.daocloud.io"]
  capabilities = ["pull", "resolve"]

[host."https://reg-mirror.qiniu.com"]
  capabilities = ["pull", "resolve"]

[host."https://registry.docker-cn.com"]
  capabilities = ["pull", "resolve"]

[host."http://hub-mirror.c.163.com"]
  capabilities = ["pull", "resolve"]

EOF

# k8s.gcr.io镜像加速
mkdir -p /etc/containerd/certs.d/k8s.gcr.io
tee /etc/containerd/certs.d/k8s.gcr.io/hosts.toml << 'EOF'
server = "https://k8s.gcr.io"

[host."https://k8s-gcr.m.daocloud.io"]
  capabilities = ["pull", "resolve", "push"]
EOF

#重启containerd
systemctl restart containerd 
#systemctl status containerd

# 安装 kubeadm、kubelet、kubectl
# 安装依赖
sudo systemctl restart containerd
sudo apt-get update -y
sudo apt-get install -y apt-transport-https ca-certificates curl gpg 

mkdir -p /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.29/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.29/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update -y
sudo apt-get install -y kubelet kubeadm kubectl 
sudo apt-mark hold kubelet kubeadm kubectl

# kubelet 开机自启
systemctl enable --now kubelet

# 配置 crictl socket
crictl config  runtime-endpoint unix:///run/containerd.sock
crictl config image-endpoint unix:///run/containerd/containerd.sock

不需要魔法的脚本

前置:

下载我下载好的资源包

阿里云 OSS
CSDN 资源

资源列表:

Containerd: - Container Runtime：Containerd 1.7.11 - CRI: runc 1.10 - CNI: cni-plugin 1.4 calico 3.27: - tigera-operator.yaml - custom-resources.yaml

资源	原始地址
Container Runtime：Containerd 1.7.11	https://github.com/containerd/containerd/releases/download/v1.7.11/containerd-1.7.11-linux-amd64.tar.gz
CRI: runc 1.10	https://github.com/opencontainers/runc/releases/download/v1.1.10/runc.amd64
CNI: cni-plugin 1.4	https://github.com/containernetworking/plugins/releases/download/v1.4.0/cni-plugins-linux-amd64-v1.4.0.tgz
calico 3.27 : tigera-operator.yaml	https://raw.githubusercontent.com/projectcalico/calico/v3.27.0/manifests/tigera-operator.yaml
calico 3.27 : custom-resources.yaml	https://raw.githubusercontent.com/projectcalico/calico/v3.27.0/manifests/custom-resources.yaml

下载资源:

wget -O - https://blog-source-mkt.oss-cn-chengdu.aliyuncs.com/resources/k8s/kubeadm%20init/k8s1.29.tar.gz | tar xzvf -
cd workdir

export name=master01  # 改为你 hostname 的名称

install.sh

#!/bin/bash

hostnamectl set-hostname $name

# 阿里源
mv /etc/apt/sources.list /etc/apt/sources.list.bak
cat <<EOF > /etc/apt/sources.list
deb https://mirrors.aliyun.com/ubuntu/ focal main restricted universe multiverse
deb-src https://mirrors.aliyun.com/ubuntu/ focal main restricted universe multiverse
deb https://mirrors.aliyun.com/ubuntu/ focal-security main restricted universe multiverse
deb-src https://mirrors.aliyun.com/ubuntu/ focal-security main restricted universe multiverse
deb https://mirrors.aliyun.com/ubuntu/ focal-updates main restricted universe multiverse
deb-src https://mirrors.aliyun.com/ubuntu/ focal-updates main restricted universe multiversedeb https://mirrors.aliyun.com/ubuntu/ focal-backports main restricted universe multiverse
deb-src https://mirrors.aliyun.com/ubuntu/ focal-backports main restricted universe multiverse
EOF
apt update


# 时间同步
timedatectl set-timezone Asia/Shanghai
#安装chrony，联网同步时间
apt install chrony -y && systemctl enable --now chronyd

# 禁用 swap
sudo swapoff -a && sed -i '/swap/s/^/#/' /etc/fstab

# 安装 ipvs
apt install -y ipset ipvsadm

# 配置需要的内核模块
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF

# 启动模块
sudo modprobe overlay
sudo modprobe br_netfilter

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF

# 是 sysctl 参数生效
sudo sysctl --system
# 检验是否配置成功
#lsmod | grep br_netfilter
#lsmod | grep overlay
#sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-ip6tables net.ipv4.ip_forward


# 配置 ipvs 内核参数
cat <<EOF | sudo tee /etc/modules-load.d/ipvs.conf
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack
EOF

# 内核加载 ipvs
sudo modprobe ip_vs
sudo modprobe ip_vs_rr
sudo modprobe ip_vs_wrr
sudo modprobe ip_vs_sh
sudo modprobe nf_conntrack
# 确认ipvs模块加载
#lsmod |grep -e ip_vs -e nf_conntrack


# 安装 Containerd
#wget -c https://github.com/containerd/containerd/releases/download/v1.7.11/containerd-1.7.11-linux-amd64.tar.gz
tar -xzvf containerd-1.7.11-linux-amd64.tar.gz
#解压出来一个bin目录,containerd可执行文件都在bin目录里面
mv bin/* /usr/local/bin/
rm -rf bin

#使用systemcd来管理containerd
cat << EOF > /usr/lib/systemd/system/containerd.service
[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target local-fs.target

[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/containerd

Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5

# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity

# Comment TasksMax if your systemd version does not supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
OOMScoreAdjust=-999

[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload && systemctl enable --now containerd 
#systemctl  status containerd


# 安装 runc
#runc是容器运行时，runc实现了容器的init，run，create，ps...我们在运行容器所需要的cmd：
#curl -LO https://github.com/opencontainers/runc/releases/download/v1.1.10/runc.amd64 && \
install -m 755 runc.amd64 /usr/local/sbin/runc

# 安装 CNI plugins
#wget -c https://github.com/containernetworking/plugins/releases/download/v1.4.0/cni-plugins-linux-amd64-v1.4.0.tgz
#根据官网的安装步骤来，创建一个目录用于存放cni插件
mkdir -p /opt/cni/bin
tar -xzvf  cni-plugins-linux-amd64-v1.4.0.tgz -C /opt/cni/bin/

# 修改 Containd 配置
#修改containerd的配置，因为containerd默认从k8s官网拉取镜像
#创建一个目录用于存放containerd的配置文件
mkdir -p /etc/containerd
#把containerd配置导出到文件
containerd config default | sudo tee /etc/containerd/config.toml

# 修改沙箱镜像
sed -i 's#sandbox_image = "registry.k8s.io/pause:.*"#sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.9"#' /etc/containerd/config.toml
# 修改 cgroup 为 systemd
sed -i 's#SystemdCgroup = false#SystemdCgroup = true#' /etc/containerd/config.toml
# 配置镜像加速
sed -i 's#config_path = ""#config_path = "/etc/containerd/certs.d"#' /etc/containerd/config.toml

# 配置 Containerd 镜像源
# docker hub镜像加速
mkdir -p /etc/containerd/certs.d/docker.io
cat > /etc/containerd/certs.d/docker.io/hosts.toml << EOF
server = "https://docker.io"
[host."https://dockerproxy.com"]
  capabilities = ["pull", "resolve"]

[host."https://docker.m.daocloud.io"]
  capabilities = ["pull", "resolve"]

[host."https://reg-mirror.qiniu.com"]
  capabilities = ["pull", "resolve"]

[host."https://registry.docker-cn.com"]
  capabilities = ["pull", "resolve"]

[host."http://hub-mirror.c.163.com"]
  capabilities = ["pull", "resolve"]

EOF

# k8s.gcr.io镜像加速
mkdir -p /etc/containerd/certs.d/k8s.gcr.io
tee /etc/containerd/certs.d/k8s.gcr.io/hosts.toml << 'EOF'
server = "https://k8s.gcr.io"

[host."https://k8s-gcr.m.daocloud.io"]
  capabilities = ["pull", "resolve", "push"]
EOF

#重启containerd
systemctl restart containerd 
#systemctl status containerd

# 安装 kubeadm、kubelet、kubectl
# 安装依赖
sudo systemctl restart containerd
sudo apt-get update -y
sudo apt-get install -y apt-transport-https ca-certificates curl gpg 

mkdir -p /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.29/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.29/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list

sudo apt-get update -y
sudo apt-get install -y kubelet kubeadm kubectl 
sudo apt-mark hold kubelet kubeadm kubectl

# kubelet 开机自启
systemctl enable --now kubelet

# 配置 crictl socket
crictl config  runtime-endpoint unix:///run/containerd/containerd.sock
crictl config image-endpoint unix:///run/containerd/containerd.sock

chmod +x install.sh
./install.sh

在你的主节点初始化集群(同样在 workdir/ 下)

export POD_CIDR=10.244.0.0/16
export SERVICE_CIDR=10.96.0.0/12
export APISERVER_MASTER01=192.168.254.130

kubeadm init \
--apiserver-advertise-address=$APISERVER_MASTER01 \
--apiserver-bind-port=6443 \
--image-repository registry.aliyuncs.com/google_containers \
--kubernetes-version v1.29.1 \
--service-cidr=$SERVICE_CIDR \
--pod-network-cidr=$POD_CIDR  --upload-certs

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

# 安装 calico
sed -i 's#cidr.*#cidr: '$POD_CIDR'#' custom-resources.yaml
kubectl create -f tigera-operator.yaml
kubectl create -f custom-resources.yaml

此时, 你的 k8s 集群就已经搭建成功了, 但是, kube-proxy 使用的依然是 iptables, 我们需要将其手动更换为 ipvs

kubectl -n kube-system edit cm kube-proxy

找到: 
mode: ""
更改为: ==>  
mode: "ipvs"

k8s 集群就万事大吉了, 如果你想把已经搭建好的集群升级为高可用集群 | 传送: [keepalived+nginx实现高可用apiserver](https://ai-feier.github.io/p/keepalived-nginx%E5%AE%9E%E7%8E%B0%E9%AB%98%E5%8F%AF%E7%94%A8apiserver)

配置自动补全

apt install bash-completion -y
cat << EOF >> ~/.profile
alias k='kubectl'
source <(kubectl completion bash)
complete -F __start_kubectl k
EOF

source ~/.profile

加入其余节点

在其余节点执行

$ kubeadm join 192.168.254.130:6443 --token ub130l.4i7hcdhk9c0g5nz6 \
        --discovery-token-ca-cert-hash sha256:56c8eafbd4c8c37ea88dd4690f4e7b38b5773c3b64b97a1165f5961b0450b0ac

验证集群

$ k get po -A
NAMESPACE         NAME                                       READY   STATUS              RESTARTS   AGE
calico-system     calico-kube-controllers-5c7b4b46d6-hd5mv   0/1     Pending             0          2m5s
calico-system     calico-node-4nkcf                          0/1     Init:1/2            0          2m5s
calico-system     calico-node-gkwpf                          0/1     Init:1/2            0          60s
calico-system     calico-node-xb222                          0/1     Init:1/2            0          56s
calico-system     calico-typha-599c7784cf-kkmpf              0/1     ContainerCreating   0          2m6s
calico-system     calico-typha-599c7784cf-ktrdr              0/1     ContainerCreating   0          47s
calico-system     csi-node-driver-2cfhm                      0/2     ContainerCreating   0          60s
calico-system     csi-node-driver-rwtnq                      0/2     ContainerCreating   0          56s
calico-system     csi-node-driver-wq486                      0/2     ContainerCreating   0          2m5s
kube-system       coredns-857d9ff4c9-fz2z6                   0/1     Pending             0          2m26s
kube-system       coredns-857d9ff4c9-j6247                   0/1     Pending             0          2m26s
kube-system       etcd-master01                              1/1     Running             0          2m39s
kube-system       kube-apiserver-master01                    1/1     Running             0          2m39s
kube-system       kube-controller-manager-master01           1/1     Running             0          2m41s
kube-system       kube-proxy-nxtgf                           1/1     Running             0          60s
kube-system       kube-proxy-sps8j                           1/1     Running             0          2m26s
kube-system       kube-proxy-vgh2g                           1/1     Running             0          56s
kube-system       kube-scheduler-master01                    1/1     Running             0          2m39s
tigera-operator   tigera-operator-55585899bf-sfvkf           1/1     Running             0          2m26s

参考:

https://kubernetes.io/zh-cn/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
https://ai-feier.github.io/p/keepalived-nginx%E5%AE%9E%E7%8E%B0%E9%AB%98%E5%8F%AF%E7%94%A8apiserver/
https://blog.csdn.net/m0_51964671/article/details/135256571