Nacos 集群 On K8s 实践服务注册发现、服务动态配置

一、K8s 部署 Nacos 集群

安装规划

组件	replicas	类型
mysql	1	StatefulSet
nacos	3	StatefulSet

使用 k8s 版本为：v1.18.0 。

本次使用 OpenEBS 来作为存储引擎，OpenEBS 是一个开源的、可扩展的存储平台，它提供了一种简单的方式来创建和管理持久化存储卷。它支持各种存储后端，包括但不限于 ZFS、Btrfs、XFS 等。同时，OpenEBS 具有高度的可扩展性和可配置性，可以满足不同的存储需求。

安装 OpenEBS ：

kubectl apply -f https://openebs.github.io/charts/openebs-operator.yaml

验证OpenEBS是否正确安装：

kubectl get pods -n openebs

在这里插入图片描述

所有的 OpenEBS pods 都处于Running状态表示正常。

首先创建一个 nacos 命名空间，下面所有的服务都放在该命名空间下：

kubectl create ns nacos

本次 Nacos 使用 MySQL 存储数据，首先部署 MySQL 服务。

1.1 部署 MySQL 服务

vi mysql.yml

# headless service 
apiVersion: v1
kind: Service
metadata:
  name: mysql
  namespace: nacos
  labels:
    app: mysql
spec:
  clusterIP: None
  ports:
  - name: mysql-port
    port: 3306
  selector:
    app: mysql

---
# NodePort service 
apiVersion: v1
kind: Service
metadata:
  name: mysql-np
  namespace: nacos
  labels:
    app: mysql-np
spec:
  clusterIP: 
  ports:
  - name: mysql-port
    port: 3306
    nodePort: 31306
    targetPort: 3306
  selector:
    app: mysql
  type: NodePort
  target-port:
  externalTrafficPolicy: Cluster 
  
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mysql
  namespace: nacos
spec:
  serviceName: "mysql"
  replicas: 1
  selector: 
    matchLabels: 
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      containers:
      - name: mysql
        image: mysql:5.7
        ports:
        - containerPort: 3306
          name: mysql-port
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: "root"
        - name: TZ
          value: "Asia/Shanghai"
        volumeMounts:                           
        - name: mysql-data
          mountPath: /var/lib/mysql 
  volumeClaimTemplates:
    - metadata:
        name: mysql-data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: openebs-hostpath
        resources:
          requests:
            storage: 5Gi

kubectl apply -f mysql.yml

查看 Pod ：

kubectl get pods -n nacos

在这里插入图片描述

1.2 部署 nacos 集群

nacos 版本这里使用 2.0.3 版本，在开始前先下载官方的安装包，使用其中的SQL文件初始化数据库。

https://github.com/alibaba/nacos/releases/tag/2.0.3

下载解压后，可以看到如下结构：

在这里插入图片描述

在上面部署的 MySQL 中创建 nacos 数据库，并执行 config 目录下的 nacos-mysql.sql 文件：

在这里插入图片描述

下面开始在 k8s 中启动 nacos 服务：

vi nacos.yml

apiVersion: v1
kind: Service
metadata:
  name: nacos-headless
  namespace: nacos
  labels:
    app: nacos-headless
spec:
  type: ClusterIP
  clusterIP: None
  ports:
    - port: 8848
      name: server
      targetPort: 8848
    - port: 9848
      name: client-rpc
      targetPort: 9848
    - port: 9849
      name: raft-rpc
      targetPort: 9849
    - port: 7848
      name: old-raft-rpc
      targetPort: 7848
  selector:
    app: nacos
---
# NodePort service
apiVersion: v1
kind: Service
metadata:
  name: nacos-np
  namespace: nacos
  labels:
    app: nacos-np
spec:
  clusterIP:
  ports:
  - name: nacos-port
    port: 8848
    nodePort: 31848
    targetPort: 8848
  selector:
    app: nacos
  type: NodePort
  target-port:
  externalTrafficPolicy: Cluster
---
# mysql 信息
apiVersion: v1
kind: ConfigMap
metadata:
  name: nacos-cm
  namespace: nacos
data:
  mysql.host: "mysql"
  mysql.db.name: "nacos"
  mysql.port: "3306"
  mysql.user: "root"
  mysql.password: "root"
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nacos
  namespace: nacos
spec:
  serviceName: nacos-headless
  replicas: 3
  template:
    metadata:
      labels:
        app: nacos
      annotations:
        pod.alpha.kubernetes.io/initialized: "true"
  selector:
    matchLabels:
      app: nacos
    spec:
      containers:
        - name: nacos
          image: nacos/nacos-server:v2.0.3
          ports:
            - containerPort: 8848
              name: client
            - containerPort: 9848
              name: client-rpc
            - containerPort: 9849
              name: raft-rpc
            - containerPort: 7848
              name: old-raft-rpc
          env:
            - name: NACOS_REPLICAS
              value: "3"
            - name: MYSQL_SERVICE_HOST
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: mysql.host
            - name: MYSQL_SERVICE_DB_NAME
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: mysql.db.name
            - name: MYSQL_SERVICE_PORT
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: mysql.port
            - name: MYSQL_SERVICE_USER
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: mysql.user
            - name: MYSQL_SERVICE_PASSWORD
              valueFrom:
                configMapKeyRef:
                  name: nacos-cm
                  key: mysql.password
            - name: SPRING_DATASOURCE_PLATFORM
              value: "mysql"
            - name: NACOS_SERVER_PORT
              value: "8848"
            - name: NACOS_APPLICATION_PORT
              value: "8848"
            - name: PREFER_HOST_MODE
              value: "hostname"
            - name: NACOS_SERVERS
              value: "nacos-0.nacos-headless.nacos.svc.cluster.local:8848 nacos-1.nacos-headless.nacos.svc.cluster.local:8848 nacos-2.nacos-headless.nacos.svc.cluster.local:8848"

启动服务：

kubectl apply -f nacos.yml

查看 Pod ：

kubectl get pods -n nacos

在这里插入图片描述

查看启动日志：

kubectl logs nacos-0 -n nacos

在这里插入图片描述

可以看到集群已经初始化了。

在浏览器访问 http://{node ip}:31848/nacos 进到 nacos 控制台：

在这里插入图片描述

默认用户名密码都是 nacos ，查看集群情况：

在这里插入图片描述

三个节点都是 UP 状态。

二、Nacos 服务注册发现、服务动态配置

这里使用两个 SpringBoot 服务，分别作为服务提供者、服务消费者，并使用上面搭建的 nacos 作为注册中心和配置信息。

首先创建一个 Maven 项目，在 pom 中增加依赖，声明SpringBoot、 SpringCloud Alibaba、StringCloud 的版本：

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example</groupId>
    <artifactId>nacos-parent</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>pom</packaging>

    <name>service-parent</name>
    <url>http://maven.apache.org</url>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <spring-boot.version>2.3.7.RELEASE</spring-boot.version>
        <spring-cloud.version>Hoxton.SR9</spring-cloud.version>
    </properties>

    <modules>
        <module>provider-service</module>
        <module>consumer-service</module>
    </modules>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>com.alibaba.cloud</groupId>
                <artifactId>spring-cloud-alibaba-dependencies</artifactId>
                <version>2.2.6.RELEASE</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
            <dependency>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-dependencies</artifactId>
                <version>${spring-boot.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
            <dependency>
                <groupId>org.springframework.cloud</groupId>
                <artifactId>spring-cloud-dependencies</artifactId>
                <version>${spring-cloud.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

</project>

2.1 服务提供者搭建

新建 SpringBoot model ，修改 pom 内容如下：

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.example</groupId>
    <artifactId>provider-service</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>provider-service</name>
    <description>provider-service</description>
    <parent>
        <groupId>org.example</groupId>
        <artifactId>nacos-parent</artifactId>
        <version>1.0-SNAPSHOT</version>
    </parent>
    <properties>
        <java.version>1.8</java.version>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>com.alibaba.cloud</groupId>
            <artifactId>spring-cloud-starter-alibaba-nacos-discovery</artifactId>
        </dependency>

    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.8.1</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                    <encoding>UTF-8</encoding>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <version>${spring-boot.version}</version>
                <configuration>
                    <mainClass>com.example.providerservice.ProviderServiceApplication</mainClass>
                </configuration>
                <executions>
                    <execution>
                        <id>repackage</id>
                        <goals>
                            <goal>repackage</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

</project>

修改 application.yml ：

server:
  port: 8091

spring:
  application:
    name: provider
  cloud:
    nacos:
      discovery:
        server-addr: ${NACOS_SERVER:nacos:8848}

这里将 nacos 集群地址通过环境变量传递进来。

启动类增加 @EnableDiscoveryClient 注解：

@EnableDiscoveryClient
@SpringBootApplication
public class ProviderServiceApplication {

    public static void main(String[] args) {
        SpringApplication.run(ProviderServiceApplication.class, args);
    }

}

创建一个测试接口，后面在服务消费端通过 OpenFeign 调用：

@RestController
public class ProviderController {

    @GetMapping("/providerData")
    public String providerData() {
        return "provider from : " + getCurrentHostName();
    }

    private String getCurrentHostName(){
        try {
            InetAddress inetAddress = InetAddress.getLocalHost();
            return inetAddress.getHostName();
        } catch (UnknownHostException e) {
            e.printStackTrace();
        }
        return null;
    }

}

建立 Dockerfile 文件，内容如下：

FROM java:8
MAINTAINER bxc
WORKDIR /app
ADD target/provider-service-0.0.1-SNAPSHOT.jar /app/app.jar
CMD ["java", "-jar", "app.jar"]

在这里插入图片描述

2.2 服务消费者搭建

消费者端通过OpenFeign 调用服务提供者端的接口信息，并且配置 nacos 动态配置。

新建 SpringBoot model ，修改 pom 内容如下：

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.example</groupId>
    <artifactId>consumer-service</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>consumer-service</name>
    <description>consumer-service</description>
    <parent>
        <groupId>org.example</groupId>
        <artifactId>nacos-parent</artifactId>
        <version>1.0-SNAPSHOT</version>
    </parent>

    <properties>
        <java.version>1.8</java.version>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>com.alibaba.cloud</groupId>
            <artifactId>spring-cloud-starter-alibaba-nacos-discovery</artifactId>
        </dependency>

        <dependency>
            <groupId>com.alibaba.cloud</groupId>
            <artifactId>spring-cloud-starter-alibaba-nacos-config</artifactId>
        </dependency>

        <!--openfeign-->
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-openfeign</artifactId>
        </dependency>

    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.8.1</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                    <encoding>UTF-8</encoding>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <version>${spring-boot.version}</version>
                <configuration>
                    <mainClass>com.example.consumerservice.ConsumerServiceApplication</mainClass>
                </configuration>
                <executions>
                    <execution>
                        <id>repackage</id>
                        <goals>
                            <goal>repackage</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

</project>

修改配置文件，注意不要写在application.yml 中，新建一个bootstrap.yml文件，用来在程序引导时执行：

server:
  port: 8080

spring:
  application:
    name: consumer
  cloud:
    nacos:
      discovery:
        server-addr: ${NACOS_SERVER:nacos:8848}
      config:
        server-addr: ${NACOS_SERVER:nacos:8848}
        file-extension: yaml

这里同样将 nacos 集群地址通过环境变量传递进来。

启动类增加 @EnableDiscoveryClient 和 @EnableFeignClients 注解：

@EnableFeignClients
@EnableDiscoveryClient
@SpringBootApplication
public class ConsumerServiceApplication {

    public static void main(String[] args) {
        SpringApplication.run(ConsumerServiceApplication.class, args);
    }

}

使用 FeignClient 声明服务提供者的接口：

@Component
@FeignClient(value = "provider")
public interface ProviderClient {

    @GetMapping("/providerData")
    String providerData();

}

创建测试接口，测试服务提供者接口调用和动态配置：

@RestController
@RefreshScope
public class ConsumerController {

    @Resource
    ProviderClient providerClient;

    @Value("${param.param1}")
    private String param;

    @GetMapping("/consumerData")
    public String consumerData() {
        String providerData = providerClient.providerData();
        return "sunncess: consumer from : " + getCurrentHostName() + " ; " + providerData;
    }

    @GetMapping("/param")
    public String getParam() {
        return "success: current param : " + param;
    }

    private String getCurrentHostName() {
        try {
            InetAddress inetAddress = InetAddress.getLocalHost();
            return inetAddress.getHostName();
        } catch (UnknownHostException e) {
            e.printStackTrace();
        }
        return null;
    }

}

最后在 nacos 中创建 consumer.yaml 配置文件，内容如下：

param:
  param1: abc

在这里插入图片描述

建立 Dockerfile 文件，内容如下：

FROM java:8
MAINTAINER bxc
WORKDIR /app
ADD target/consumer-service-0.0.1-SNAPSHOT.jar /app/app.jar
CMD ["java", "-jar", "app.jar"]

在这里插入图片描述

2.3 打包上传镜像

这里我将镜像打包后上传至私有的 harbor 仓库中，如果没有 harbor 仓库也可以将镜像上传至每个 k8s node 节点中，然后使用 docker load 到本地镜像仓库中。

# 打包成 jar 包
mvn clean package
# 构建镜像
cd consumer-service
docker build -t consumer-service:1.0 .
# 上传至 harbor
docker tag consumer-service:1.0 11.0.1.150/image/consumer-service:1.0 .
docker push 11.0.1.150/image/consumer-service:1.0

# 构建镜像
cd ../provider-service
docker build -t provider-service:1.0 .
# 上传至 harbor
docker tag provider-service:1.0 11.0.1.150/image/provider-service:1.0 .
docker push 11.0.1.150/image/provider-service:1.0

2.4 k8s 启动服务

每个服务启动 2 个副本。

vi nacos-test-demo.yml

apiVersion: v1
kind: Service
metadata:
  name: consumer
  namespace: nacos
  labels:
    app: consumer
spec:
  type: NodePort
  ports:
  - port: 8080
    name: client
    nodePort: 31880
    targetPort: 8080
  selector:
    app: consumer

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: consumer
  namespace: nacos
spec:
  replicas: 2
  selector:
    matchLabels:
      app: consumer
  template:
    metadata:
      labels:
        app: consumer
    spec:
      containers:
      - name: consumer
        image: 11.0.1.150/image/consumer-service:1.0
        ports:
        - containerPort: 8080
          name: server
        env:
        - name: NACOS_SERVER
          value: nacos-0.nacos-headless.nacos.svc.cluster.local:8848,nacos-1.nacos-headless.nacos.svc.cluster.local:8848,nacos-2.nacos-headless.nacos.svc.cluster.local:8848

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: provider
  namespace: nacos
spec:
  replicas: 2
  selector:
    matchLabels:
      app: provider
  template:
    metadata:
      labels:
        app: provider
    spec:
      containers:
      - name: provider
        image: 11.0.1.150/image/provider-service:1.0
        ports:
        - containerPort: 8091
          name: server
        env:
        - name: NACOS_SERVER
          value: nacos-0.nacos-headless.nacos.svc.cluster.local:8848,nacos-1.nacos-headless.nacos.svc.cluster.local:8848,nacos-2.nacos-headless.nacos.svc.cluster.local:8848

注意：如果想覆盖某个配置，可以直接以环境变量的方式覆盖，SpringBoot 会自动读取，例如上面的nacos地址，也可以这样传递：

apiVersion: v1
kind: Service
metadata:
  name: consumer
  namespace: nacos
  labels:
    app: consumer
spec:
  type: NodePort
  ports:
  - port: 8080
    name: client
    nodePort: 31880
    targetPort: 8080
  selector:
    app: consumer

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: consumer
  namespace: nacos
spec:
  replicas: 2
  selector:
    matchLabels:
      app: consumer
  template:
    metadata:
      labels:
        app: consumer
    spec:
      containers:
      - name: consumer
        image: 11.0.1.150/image/consumer-service:1.0
        ports:
        - containerPort: 8080
          name: server
        env:
        - name: spring.cloud.nacos.discovery.server-addr
          value: nacos-0.nacos-headless.nacos.svc.cluster.local:8848,nacos-1.nacos-headless.nacos.svc.cluster.local:8848,nacos-2.nacos-headless.nacos.svc.cluster.local:8848
        - name: spring.cloud.nacos.config.server-addr
          value: nacos-0.nacos-headless.nacos.svc.cluster.local:8848,nacos-1.nacos-headless.nacos.svc.cluster.local:8848,nacos-2.nacos-headless.nacos.svc.cluster.local:8848

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: provider
  namespace: nacos
spec:
  replicas: 2
  selector:
    matchLabels:
      app: provider
  template:
    metadata:
      labels:
        app: provider
    spec:
      containers:
      - name: provider
        image: 11.0.1.150/image/provider-service:1.0
        ports:
        - containerPort: 8091
          name: server
        env:
        - name: spring.cloud.nacos.discovery.server-addr
          value: nacos-0.nacos-headless.nacos.svc.cluster.local:8848,nacos-1.nacos-headless.nacos.svc.cluster.local:8848,nacos-2.nacos-headless.nacos.svc.cluster.local:8848