Kubernetes集群自动化部署

1.1 实验介绍

1.1.1 关于本实验

1.1.2 实验目的

1.2 环境准备

步骤 1 设置节点名

步骤 2 配置 hosts 节点名解析

步骤 3 配置免密登录

步骤 4 清空 iptables、关闭防火墙并禁用 selinux

步骤 5 关闭交换分区

步骤 6 开启 ipvs

步骤 7 设置时间同步

步骤 8 配置 yum 源

步骤 9 安装 docker

步骤 10 配置内核转发及网桥过滤

步骤 11 安装cri-docker

步骤 12 启动cri-docker服务

步骤 13 cri-dockerd设置国内镜像加速

1.3 Master 节点部署高可用

步骤 1 安装 nginx 和 Keepalived

步骤 2 修改 nginx 配置文件

步骤 3 修改 Keepalived 配置文件

步骤 4 启动 nginx 和 Keepalived

步骤 5 结果验证

步骤 6 高可用切换验证

1.4 部署 Kubernetes 集群

步骤 1 K8S软件安装

1.5 K8S集群初始化

1.6 K8S集群工作节点加入

步骤 1 其他 master 节点加入集群

步骤 2 node 节点加入集群

步骤 3 安装网络插件 Calico

步骤 4 查看节点状态

1.7 结果验证

步骤 1 创建 deployment

步骤 2 访问 service

步骤 3 访问 pod

步骤 4 删除 deployment 和 service

1.1 实验介绍

1.1.1 关于本实验

本实验通过使用 kubeadm 实现自动化部署 Kubernetes 集群，并用 nginx+Keepalived 实现高

可用。

1.1.2 实验目的

理解 Kubernetes 集群的架构。

理解高可用集群的搭建方式。

掌握 kubeadm 搭建 Kubernetes 集群的方法。

1.2 环境准备

步骤 1 设置节点名

Hostname	Kubernetes集群IP	VIP
k8smaster1	192.168.186.100	192.168.186.50
k8smaster2	192.168.186.101	192.168.186.50
k8smaster3	192.168.186.102	192.168.186.50
k8snode1	192.168.186.103
k8snode2	192.168.186.104
k8snode3	192.168.186.105

依次在 6 台节点中执行以下命令设置节点名，节点名分别为 k8smaster1、k8smaster2、k8smaster3、k8snode1、k8snode2、k8snode3：

[root@localhost ~]# hostnamectl hostname k8smaster1 
[root@localhost ~]# bash


Welcome to 5.10.0-182.0.0.95.oe2203sp3.x86_64

System information as of time: 	2024年 04月 09日 星期二 10:17:46 CST

System load: 	0.00
Processes: 	154
Memory used: 	5.6%
Swap used: 	0%
Usage On: 	9%
IP address: 	192.168.186.100
Users online: 	2


[root@k8smaster1 ~]#

步骤 2 配置 hosts 节点名解析

所有节点执行以下命令

[root@k8smaster1 ~]# cat <<EOF>>/etc/hosts
> 192.168.186.100  k8smaster1 
> 192.168.186.101  k8smaster2
> 192.168.186.102  k8smaster3 
> 192.168.186.103  k8snode1 
> 192.168.186.104  k8snode2
> 192.168.186.105  k8snode3 
> EOF
[root@k8smaster1 ~]#

步骤 3 配置免密登录

在 k8smaster1 节点执行以下命令，然后一直敲击回车，生成秘钥文件：

[root@k8smaster1 ~]#  ssh-keygen

在 k8smaster1 节点拷贝秘钥到其他 5 台节点，例如 k8smaster2：

[root@k8smaster1 ~]#  ssh-copy-id k8smaster2
[root@k8smaster1 ~]#  ssh-copy-id k8smaster3
[root@k8smaster1 ~]#  ssh-copy-id k8snode1
[root@k8smaster1 ~]#  ssh-copy-id k8snode2
[root@k8smaster1 ~]#  ssh-copy-id k8snode3

步骤 4 清空 iptables、关闭防火墙并禁用 selinux

所有节点执行以下命令：

iptables -F
setenforce 0
sed -i 's/^SELINUX=.*/SELINUX=disabled/g' /etc/selinux/config
systemctl stop firewalld && systemctl disable firewalld

步骤 5 关闭交换分区

所有节点执行以下命令：

[root@k8smaster1 ~]# swapoff -a
[root@k8smaster1 ~]# sed -i 's/.*swap.*/#&/' /etc/fstab

步骤 6 开启 ipvs

所有节点执行以下命令：

cat > /etc/sysconfig/modules/ipvs.modules <<END
#!/bin/bash
ipvs_modules="ip_vs ip_vs_lc ip_vs_wlc ip_vs_rr ip_vs_wrr ip_vs_lblc ip_vs_lblcr ip_vs_dh ip_vs_sh ip_vs_nq 
ip_vs_sed ip_vs_ftp nf_conntrack"
for kernel_module in ${ipvs_modules}; do
/sbin/modinfo -F filename ${kernel_module} > /dev/null 2>&1
if [ 0 -eq 0 ]; then
/sbin/modprobe ${kernel_module}
fi
done
END
chmod 755 /etc/sysconfig/modules/ipvs.modules
bash /etc/sysconfig/modules/ipvs.modules

步骤 7 设置时间同步

所有节点执行以下命令：

yum install -y chrony
systemctl enable --now chronyd
chronyc sources

步骤 8 配置 yum 源

所有节点执行以下命令，配置 kubernetes 的 yum 源：

cat <<EOF | tee /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://pkgs.k8s.io/core:/stable:/v1.28/rpm/
enabled=1
gpgcheck=1
gpgkey=https://pkgs.k8s.io/core:/stable:/v1.28/rpm/repodata/repomd.xml.key
#exclude=kubelet kubeadm kubectl cri-tools kubernetes-cni
EOF

步骤 9 安装 docker

所有节点执行以下命令，安装启动 docker：

1、下载官方repo
cd /etc/yum.repos.d/
curl -O https://download.docker.com/linux/centos/docker-ce.repo
sed -i 's/$releasever/8/g' docker-ce.repo
2. 安装docker
yum install -y docker-ce

# 设置国内镜像加速
mkdir -p /etc/docker
tee /etc/docker/daemon.json <<-'EOF'
{
    "registry-mirrors": [
        "https://hub-mirror.c.163.com",
        "https://mirror.baidubce.com",
        "https://ccr.ccs.tencentyun.com"
    ]
}
EOF

设置docker开机启动并启动
# systemctl enable --now docker


查看docker版本
# docker version

# 所有节点执行以下命令，将 docker 的 CgroupDriver 改成 systemd，在/etc/docker/daemon.json 中添加配置：

{
    "exec-opts": ["native.cgroupdriver=systemd"],
    "registry-mirrors": [
        "https://hub-mirror.c.163.com",
        "https://mirror.baidubce.com",
        "https://ccr.ccs.tencentyun.com"
    ]
}

# 所有节点执行以下命令，重启 docker：
systemctl daemon-reload
systemctl restart docker

步骤 10 配置内核转发及网桥过滤

添加网桥过滤及内核转发配置文件
# cat > /etc/sysctl.d/k8s.conf << EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
vm.swappiness = 0
EOF

加载br_netfilter模块
# modprobe br_netfilter

查看是否加载
# lsmod | grep br_netfilter
br_netfilter           22256  0
bridge                151336  1 br_netfilter

使用新添加配置文件生效
# sysctl -p /etc/sysctl.d/k8s.conf

步骤 11 安装cri-docker

wget -c http://github.com/Mirantis/cri-dockerd/releases/download/v0.3.4/cri-dockerd-0.3.4-3.el8.x86_64.rpm
yum install cri-dockerd-0.3.4-3.el8.x86_64.rpm

步骤 12 启动cri-docker服务

systemctl enable cri-docker

步骤 13 cri-dockerd设置国内镜像加速

$ vi /usr/lib/systemd/system/cri-docker.service # 找到第10行ExecStart= 

# 修改为ExecStart=/usr/bin/cri-dockerd --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.9
# 重启Docker组件
$ systemctl daemon-reload && systemctl restart docker cri-docker.socket cri-docker 
# 检查Docker组件状态
$ systemctl status docker cir-docker.socket cri-docker

1.3 Master 节点部署高可用

步骤 1 安装 nginx 和 Keepalived

3 台 master 节点执行以下命令，安装 nginx 和 Keepalived：

yum install -y nginx keepalived nginx-all-modules

步骤 2 修改 nginx 配置文件

3 台 master 节点修改/etc/nginx/nginx.conf 配置文件，在如下位置添加加粗部分内容：

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;
include /usr/share/nginx/modules/*.conf;
events {
 worker_connections 1024;
}
stream {
 log_format main '$remote_addr $upstream_addr - [$time_local] $status 
$upstream_bytes_sent';
 access_log /var/log/nginx/k8s-access.log main;
 upstream k8s-apiserver {
 server 192.168.186.100:6443;
 server 192.168.186.101:6443;
 server 192.168.186.102:6443; 
 }
 server {
 listen 16443;
 proxy_pass k8s-apiserver;
 }
}

步骤 3 修改 Keepalived 配置文件

覆盖修改 k8smaster1 节点配置文件/etc/keepalived/keepalived.conf：

cat > /etc/keepalived/keepalived.conf<<EOF
! Configuration File for keepalived
global_defs {
 router_id master1
}
vrrp_instance Nginx {
 state MASTER
 interface ens33
 virtual_router_id 51
 priority 200
 advert_int 1
 authentication {
 auth_type PASS
 auth_pass Huawei@1
 }
 virtual_ipaddress {
 192.168.186.50/24
 }
}
EOF

cat > /etc/keepalived/keepalived.conf<<EOF
! Configuration File for keepalived
global_defs {
 router_id master1
}
vrrp_instance Nginx {
 state BACKUP
 interface ens33
 virtual_router_id 51
 priority 150
 advert_int 1
 authentication {
 auth_type PASS
 auth_pass Huawei@1
 }
 virtual_ipaddress {
 192.168.186.50/24
 }
}
EOF

cat > /etc/keepalived/keepalived.conf<<EOF
! Configuration File for keepalived
global_defs {
 router_id master1
}
vrrp_instance Nginx {
 state BACKUP
 interface ens33
 virtual_router_id 51
 priority 100
 advert_int 1
 authentication {
 auth_type PASS
 auth_pass Huawei@1
 }
 virtual_ipaddress {
 192.168.186.50/24
 }
}
EOF

k8smaster2 和 k8smaster3 配置如上，注意字段 state 修改为 BACKUP，降低 priority，例如k8smaster2 的 priority 值为 150，k8smaster3 的 priority 值为 100。

步骤 4 启动 nginx 和 Keepalived

3 台 master 节点执行以下命令：

systemctl enable nginx --now
systemctl enable keepalived --now

步骤 5 结果验证

3 台 master 节点执行以下命令，查看 nginx 和 keepalived 状态为 active(running)：

systemctl status nginx
systemctl status keepalived

查看 k8smaster1 节点 IP 地址中显示 192.168.1.10 虚拟地址：

步骤 6 高可用切换验证

在 k8smaster1 停止 keepalived 服务，虚拟 IP 地址 192.168.1.10 切换到 k8smaster2：

1.4 部署 Kubernetes 集群

步骤 1 K8S软件安装

安装kubelet、kubeadm、kubectl、kubernetes-cni
yum install -y kubelet kubeadm kubectl kubernetes-cni
systemctl enable kubelet.service

1.5 K8S集群初始化

[root@k8smaster1 ~]# kubeadm config print init-defaults > kubeadm-init.yaml
[root@k8smaster1 ~]# vim kubeadm-init.yaml 
修改如下配置：

- advertiseAddress：为控制平面地址，（ Master 主机 IP ）
	advertiseAddress: 1.2.3.4
修改为 advertiseAddress: 172.16.100.21
- criSocket：为 containerd 的 socket 文件地址
	criSocket: unix:///var/run/containerd/containerd.sock
修改为 criSocket: unix:///var/run/cri-dockerd.sock
- name: node 修改node为 k8s-master01
	name: node
修改为 name: k8s-master01
clusterName下面添加 VIP和端口
controlPlaneEndpoint: 172.16.100.20:16443

- imageRepository：阿里云镜像代理地址，否则拉取镜像会失败
	imageRepository: registry.k8s.io
修改为：imageRepository: registry.aliyuncs.com/google_containers
- kubernetesVersion：为 k8s 版本
	kubernetesVersion: 1.28.0
修改为：kubernetesVersion: 1.28.5
注意：一定要配置镜像代理，否则会由于防火墙问题导致集群安装失败
文件末尾增加启用ipvs功能
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs


# 根据配置文件启动 kubeadm 初始化 k8s
$ kubeadm init --config=kubeadm-init.yaml --upload-certs --v=6

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join 192.168.186.50:16443 --token abcdef.0123456789abcdef \
	--discovery-token-ca-cert-hash sha256:74dc5e7f26f041e65ddc085b8eb9d05233a9394e50a8c19374f9bf5e198a3e5e \
	--control-plane --certificate-key fbe0f58aeb0d2e0640f4ddcfcda3afb56ee04ea9a90dab22502511a315843509

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.186.50:16443 --token abcdef.0123456789abcdef \
	--discovery-token-ca-cert-hash sha256:74dc5e7f26f041e65ddc085b8eb9d05233a9394e50a8c19374f9bf5e198a3e5e 

[root@k8smaster1 ~]# mkdir -p $HOME/.kube
[root@k8smaster1 ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
[root@k8smaster1 ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config
[root@k8smaster1 ~]# kubectl get node
NAME         STATUS     ROLES           AGE     VERSION
k8smaster1   NotReady   control-plane   3m37s   v1.28.9

1.6 K8S集群工作节点加入

步骤 1 其他 master 节点加入集群

所有的工作节点加入集群
注意：加入集群时需要添加 --cri-socket unix:///var/run/cri-dockerd.sock
[root@k8smaster2 ~]# kubeadm join 192.168.186.50:16443 --token abcdef.0123456789abcdef \> --discovery-token-ca-cert-hash sha256:74dc5e7f26f041e65ddc085b8eb9d05233a9394e50a8c19374f9bf5e198a3e5e \
> --cri-socket unix:///var/run/cri-dockerd.sock

[root@k8smaster3 ~]# kubeadm join 192.168.186.50:16443 --token abcdef.0123456789abcdef \
> --discovery-token-ca-cert-hash sha256:74dc5e7f26f041e65ddc085b8eb9d05233a9394e50a8c19374f9bf5e198a3e5e \
> --cri-socket unix:///var/run/cri-dockerd.sock

步骤 2 node 节点加入集群

[root@k8snode1 ~]# kubeadm join 192.168.186.50:16443 --token abcdef.0123456789abcdef \
> --discovery-token-ca-cert-hash sha256:74dc5e7f26f041e65ddc085b8eb9d05233a9394e50a8c19374f9bf5e198a3e5e \
> --cri-socket unix:///var/run/cri-dockerd.sock

[root@k8snode2 ~]# kubeadm join 192.168.186.50:16443 --token abcdef.0123456789abcdef \
> --discovery-token-ca-cert-hash sha256:74dc5e7f26f041e65ddc085b8eb9d05233a9394e50a8c19374f9bf5e198a3e5e \
> --cri-socket unix:///var/run/cri-dockerd.sock

[root@k8snode3 ~]# kubeadm join 192.168.186.50:16443 --token abcdef.0123456789abcdef \
> --discovery-token-ca-cert-hash sha256:74dc5e7f26f041e65ddc085b8eb9d05233a9394e50a8c19374f9bf5e198a3e5e \
> --cri-socket unix:///var/run/cri-dockerd.sock

步骤 3 安装网络插件 Calico

在 k8smaster1 节点执行以下命令：

[root@k8smaster1 ~]#  curl -O https://docs.projectcalico.org/archive/v3.23/manifests/calico.yaml
[root@k8smaster1 ~]# vim calico.yaml 
...
        - name: CALICO_IPV4POOL_CIDR
          value: "10.244.0.0/16"
[root@master ~]# kubectl apply -f calico.yaml

步骤 4 查看节点状态

在 k8smaster1 执行以下命令，查看所有节点状态 Ready，需要在部署 calico 完成后等待几分钟

至此，kubernetes的集群环境搭建完成

1.7 结果验证

步骤 1 创建 deployment

在 k8smaster1 节点中执行以下命令，在集群中创建一个 deployment，验证是否正常运行：

[root@k8smaster1 ~]# kubectl create deployment nginx --image=nginx
deployment.apps/nginx created
[root@k8smaster1 ~]# kubectl expose deployment nginx --port=80 --type=NodePort
service/nginx exposed

步骤 2 访问 service

在 k8smaster1 节点执行以下命令，查看 service/nginx 的 IP 地址：

 [root@k8smaster1 ~]# kubectl get pod,svc

在任意节点执行以下命令，使用 curl 访问 service，注意 IP 地址为上述命令查看到的 IP 地址，不能直接复制以下命令：

[root@k8smaster1 ~]# curl 10.103.233.15

步骤 3 访问 pod

在 k8smaster1 节点执行以下命令，查看 pod 的 IP 地址：

root@k8smaster1 ~]# kubectl get pods -o wide

在任意节点执行以下命令，使用 curl 访问 pod，注意 IP 地址为上述命令查看到的 IP 地址，不能直接复制以下命令：

[root@k8smaster1 ~]# curl 172.16.185.193

步骤 4 删除 deployment 和 service

[root@k8smaster1 ~]# kubectl delete deployment nginx
deployment.apps "nginx" deleted
[root@k8smaster1 ~]# kubectl delete service nginx
service "nginx" deleted
[root@k8smaster1 ~]#