云原生之深入解析Flink on k8s的运行模式与实战操作

一、概述

  • Flink 核心是一个流式的数据流执行引擎，并且能够基于同一个 Flink 运行时，提供支持流处理和批处理两种类型应用。其针对数据流的分布式计算提供了数据分布，数据通信及容错机制等功能。
  • Flink 官网
  • 不同版本的文档
  • flink on k8s 官方文档
  • GitHub 地址

二、Flink 运行模式

  • 官方文档
  • Flink on yarn 有三种运行模式：
    • yarn-session 模式（Seesion Mode）
    • yarn-cluster 模式（Per-Job Mode）
    • Application 模式（Application Mode）

在这里插入图片描述

  • 注意：Per-Job 模式（已弃用），Per-job 模式仅由 YARN 支持，并已在 Flink 1.15 中弃用，它将被丢弃在 FLINK-26000 中。

三、Flink on k8s 实战操作

① flink 下载

  • 下载地址，如下所示：
wget https://dlcdn.apache.org/flink/flink-1.14.6/flink-1.14.6-bin-scala_2.12.tgz

② 构建基础镜像

docker pull apache/flink:1.14.6-scala_2.12
docker tag apache/flink:1.14.6-scala_2.12 myharbor.com/bigdata/flink:1.14.6-scala_2.12
docker push myharbor.com/bigdata/flink:1.14.6-scala_2.12

③ session 模式

  • Flink Session 集群作为长时间运行的 Kubernetes Deployment 执行，可以在一个 Session 集群上运行多个 Flink 作业，每个作业都需要在集群部署完成后提交到集群。
  • Kubernetes 中的 Flink Session 集群部署至少包含三个组件：
    • 运行 JobManager 的部署；
    • TaskManagers 池的部署；
    • 暴露JobManager 的 REST 和 UI 端口的服务。

（A）Native Kubernetes 模式

  • 参数配置：
https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/config/#kubernetes-namespace
  • 构建镜像 Dockerfile：
FROM myharbor.com/bigdata/flink:1.14.6-scala_2.12
RUN rm -f /etc/localtime && ln -sv /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && echo "Asia/Shanghai" > /etc/timezone
RUN export LANG=zh_CN.UTF-8
  • 开始构建镜像：
docker build -t myharbor.com/bigdata/flink-session:1.14.6-scala_2.12 . --no-cache

# 上传镜像
docker push myharbor.com/bigdata/flink-session:1.14.6-scala_2.12
  • 创建命名空间和 serviceaccount：
# 创建namespace
kubectl create ns flink
# 创建serviceaccount
kubectl create serviceaccount flink-service-account -n flink
# 用户授权
kubectl create clusterrolebinding flink-role-binding-flink --clusterrole=edit --serviceaccount=flink:flink-service-account
  • 创建 flink 集群：
./bin/kubernetes-session.sh \
-Dkubernetes.cluster-id=my-first-flink-cluster  \
-Dkubernetes.container.image=myharbor.com/bigdata/flink-session:1.14.6-scala_2.12 \
-Dkubernetes.namespace=flink \
-Dkubernetes.jobmanager.service-account=flink-service-account \
-Dkubernetes.rest-service.exposed.type=NodePort

在这里插入图片描述
在这里插入图片描述

  • 提交任务（注意 jdk 版本，目前 jdk8 是正常的）：
./bin/flink run \
--target kubernetes-session \
-Dkubernetes.cluster-id=my-first-flink-cluster \
-Dkubernetes.namespace=flink \
-Dkubernetes.jobmanager.service-account=flink-service-account \
./examples/streaming/TopSpeedWindowing.jar

#   参数配置
./examples/streaming/WordCount.jar
-Dkubernetes.taskmanager.cpu=2000m \
-Dexternal-resource.limits.kubernetes.cpu=4000m \
-Dexternal-resource.limits.kubernetes.memory=10Gi \
-Dexternal-resource.requests.kubernetes.cpu=2000m \
-Dexternal-resource.requests.kubernetes.memory=8Gi \
-Dkubernetes.taskmanager.cpu=2000m \
  • 查看：
kubectl get pods -n flink
kubectl logs -f my-first-flink-cluster-taskmanager-1-1

在这里插入图片描述
在这里插入图片描述

  • 删除 flink 集群：
kubectl delete deployment/my-first-flink-cluster -n flink
kubectl delete ns flink --force

（B）Standalone 模式

  • 构建镜像：默认用户是 flink 用户，这里换成 admin，根据企业需要更换用户，脚本可以通过上面运行的 pod 拿到。启动脚本 docker-entrypoint.sh：
#!/usr/bin/env bash

###############################################################################
#  Licensed to the Apache Software Foundation (ASF) under one
#  or more contributor license agreements.  See the NOTICE file
#  distributed with this work for additional information
#  regarding copyright ownership.  The ASF licenses this file
#  to you under the Apache License, Version 2.0 (the
#  "License"); you may not use this file except in compliance
#  with the License.  You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
#  Unless required by applicable law or agreed to in writing, software
#  distributed under the License is distributed on an "AS IS" BASIS,
#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#  See the License for the specific language governing permissions and
# limitations under the License.
###############################################################################

COMMAND_STANDALONE="standalone-job"
COMMAND_HISTORY_SERVER="history-server"

# If unspecified, the hostname of the container is taken as the JobManager address
JOB_MANAGER_RPC_ADDRESS=${JOB_MANAGER_RPC_ADDRESS:-$(hostname -f)}
CONF_FILE="${FLINK_HOME}/conf/flink-conf.yaml"

drop_privs_cmd() {
    if [ $(id -u) != 0 ]; then
        # Don't need to drop privs if EUID != 0
        return
    elif [ -x /sbin/su-exec ]; then
        # Alpine
        echo su-exec admin
    else
        # Others
        echo gosu admin
    fi
}

copy_plugins_if_required() {
  if [ -z "$ENABLE_BUILT_IN_PLUGINS" ]; then
    return 0
  fi

  echo "Enabling required built-in plugins"
  for target_plugin in $(echo "$ENABLE_BUILT_IN_PLUGINS" | tr ';' ' '); do
    echo "Linking ${target_plugin} to plugin directory"
    plugin_name=${target_plugin%.jar}

    mkdir -p "${FLINK_HOME}/plugins/${plugin_name}"
    if [ ! -e "${FLINK_HOME}/opt/${target_plugin}" ]; then
      echo "Plugin ${target_plugin} does not exist. Exiting."
      exit 1
    else
      ln -fs "${FLINK_HOME}/opt/${target_plugin}" "${FLINK_HOME}/plugins/${plugin_name}"
      echo "Successfully enabled ${target_plugin}"
    fi
  done
}

set_config_option() {
  local option=$1
  local value=$2

  # escape periods for usage in regular expressions
  local escaped_option=$(echo ${option} | sed -e "s/\./\\\./g")

  # either override an existing entry, or append a new one
  if grep -E "^${escaped_option}:.*" "${CONF_FILE}" > /dev/null; then
        sed -i -e "s/${escaped_option}:.*/$option: $value/g" "${CONF_FILE}"
  else
        echo "${option}: ${value}" >> "${CONF_FILE}"
  fi
}

prepare_configuration() {
    set_config_option jobmanager.rpc.address ${JOB_MANAGER_RPC_ADDRESS}
    set_config_option blob.server.port 6124
    set_config_option query.server.port 6125

    if [ -n "${TASK_MANAGER_NUMBER_OF_TASK_SLOTS}" ]; then
        set_config_option taskmanager.numberOfTaskSlots ${TASK_MANAGER_NUMBER_OF_TASK_SLOTS}
    fi

    if [ -n "${FLINK_PROPERTIES}" ]; then
        echo "${FLINK_PROPERTIES}" >> "${CONF_FILE}"
    fi
    envsubst < "${CONF_FILE}" > "${CONF_FILE}.tmp" && mv "${CONF_FILE}.tmp" "${CONF_FILE}"
}

maybe_enable_jemalloc() {
    if [ "${DISABLE_JEMALLOC:-false}" == "false" ]; then
        JEMALLOC_PATH="/usr/lib/$(uname -m)-linux-gnu/libjemalloc.so"
        JEMALLOC_FALLBACK="/usr/lib/x86_64-linux-gnu/libjemalloc.so"
        if [ -f "$JEMALLOC_PATH" ]; then
            export LD_PRELOAD=$LD_PRELOAD:$JEMALLOC_PATH
        elif [ -f "$JEMALLOC_FALLBACK" ]; then
            export LD_PRELOAD=$LD_PRELOAD:$JEMALLOC_FALLBACK
        else
            if [ "$JEMALLOC_PATH" = "$JEMALLOC_FALLBACK" ]; then
                MSG_PATH=$JEMALLOC_PATH
            else
                MSG_PATH="$JEMALLOC_PATH and $JEMALLOC_FALLBACK"
            fi
            echo "WARNING: attempted to load jemalloc from $MSG_PATH but the library couldn't be found. glibc will be used instead."
        fi
    fi
}

maybe_enable_jemalloc

copy_plugins_if_required

prepare_configuration

args=("$@")
if [ "$1" = "help" ]; then
    printf "Usage: $(basename "$0") (jobmanager|${COMMAND_STANDALONE}|taskmanager|${COMMAND_HISTORY_SERVER})\n"
    printf "    Or $(basename "$0") help\n\n"
    printf "By default, Flink image adopts jemalloc as default memory allocator. This behavior can be disabled by setting the 'DISABLE_JEMALLOC' environment variable to 'true'.\n"
    exit 0
elif [ "$1" = "jobmanager" ]; then
    args=("${args[@]:1}")

    echo "Starting Job Manager"

    exec $(drop_privs_cmd) "$FLINK_HOME/bin/jobmanager.sh" start-foreground "${args[@]}"
elif [ "$1" = ${COMMAND_STANDALONE} ]; then
    args=("${args[@]:1}")

    echo "Starting Job Manager"

    exec $(drop_privs_cmd) "$FLINK_HOME/bin/standalone-job.sh" start-foreground "${args[@]}"
elif [ "$1" = ${COMMAND_HISTORY_SERVER} ]; then
    args=("${args[@]:1}")

    echo "Starting History Server"

    exec $(drop_privs_cmd) "$FLINK_HOME/bin/historyserver.sh" start-foreground "${args[@]}"
elif [ "$1" = "taskmanager" ]; then
    args=("${args[@]:1}")

    echo "Starting Task Manager"

    exec $(drop_privs_cmd) "$FLINK_HOME/bin/taskmanager.sh" start-foreground "${args[@]}"
fi

args=("${args[@]}")

# Running command in pass-through mode
exec $(drop_privs_cmd) "${args[@]}"
  • 编排 Dockerfile：
FROM myharbor.com/bigdata/centos:7.9.2009

USER root

# 安装常用工具
RUN yum install -y vim tar wget curl rsync bzip2 iptables tcpdump less telnet net-tools lsof

# 设置时区，默认是UTC时区
RUN rm -f /etc/localtime && ln -sv /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && echo "Asia/Shanghai" > /etc/timezone

RUN mkdir -p /opt/apache

ADD jdk-8u212-linux-x64.tar.gz /opt/apache/

ADD flink-1.14.6-bin-scala_2.12.tgz  /opt/apache/

ENV FLINK_HOME /opt/apache/flink-1.14.6
ENV JAVA_HOME /opt/apache/jdk1.8.0_212
ENV PATH $JAVA_HOME/bin:$PATH

# 创建用户应用jar目录
RUN mkdir $FLINK_HOME/usrlib/

#RUN mkdir home
COPY docker-entrypoint.sh /opt/apache/
RUN chmod +x /opt/apache/docker-entrypoint.sh

RUN groupadd --system --gid=9999 admin && useradd --system --home-dir $FLINK_HOME --uid=9999 --gid=admin admin

RUN chown -R admin:admin /opt/apache

#设置的工作目录
WORKDIR $FLINK_HOME

# 对外暴露端口
EXPOSE 6123 8081

# 执行脚本，构建镜像时不执行，运行实例才会执行
ENTRYPOINT ["/opt/apache/docker-entrypoint.sh"]
CMD ["help"]
  • 开始构建镜像：
docker build -t myharbor.com/bigdata/flink-centos-admin:1.14.6-scala_2.12 . --no-cache

# 上传镜像
docker push myharbor.com/bigdata/flink-centos-admin:1.14.6-scala_2.12

# 删除镜像
docker rmi myharbor.com/bigdata/flink-centos-admin:1.14.6-scala_2.12
crictl rmi myharbor.com/bigdata/flink-centos-admin:1.14.6-scala_2.12
  • 创建命名空间和 serviceaccount：
# 创建namespace
kubectl create ns flink
# 创建serviceaccount
kubectl create serviceaccount flink-service-account -n flink
# 用户授权
kubectl create clusterrolebinding flink-role-binding-flink --clusterrole=edit --serviceaccount=flink:flink-service-account
  • 编排 yaml 文件：
    • flink-configuration-configmap.yaml：
apiVersion: v1
kind: ConfigMap
metadata:
  name: flink-config
  labels:
    app: flink
data:
  flink-conf.yaml: |+
    jobmanager.rpc.address: flink-jobmanager
    taskmanager.numberOfTaskSlots: 2
    blob.server.port: 6124
    jobmanager.rpc.port: 6123
    taskmanager.rpc.port: 6122
    queryable-state.proxy.ports: 6125
    jobmanager.memory.process.size: 3200m
    taskmanager.memory.process.size: 2728m
    taskmanager.memory.flink.size: 2280m
    parallelism.default: 2
  log4j-console.properties: |+
    # This affects logging for both user code and Flink
    rootLogger.level = INFO
    rootLogger.appenderRef.console.ref = ConsoleAppender
    rootLogger.appenderRef.rolling.ref = RollingFileAppender

    # Uncomment this if you want to _only_ change Flink's logging
    #logger.flink.name = org.apache.flink
    #logger.flink.level = INFO

    # The following lines keep the log level of common libraries/connectors on
    # log level INFO. The root logger does not override this. You have to manually
    # change the log levels here.
    logger.akka.name = akka
    logger.akka.level = INFO
    logger.kafka.name= org.apache.kafka
    logger.kafka.level = INFO
    logger.hadoop.name = org.apache.hadoop
    logger.hadoop.level = INFO
    logger.zookeeper.name = org.apache.zookeeper
    logger.zookeeper.level = INFO

    # Log all infos to the console
    appender.console.name = ConsoleAppender
    appender.console.type = CONSOLE
    appender.console.layout.type = PatternLayout
    appender.console.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

    # Log all infos in the given rolling file
    appender.rolling.name = RollingFileAppender
    appender.rolling.type = RollingFile
    appender.rolling.append = false
    appender.rolling.fileName = ${sys:log.file}
    appender.rolling.filePattern = ${sys:log.file}.%i
    appender.rolling.layout.type = PatternLayout
    appender.rolling.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n
    appender.rolling.policies.type = Policies
    appender.rolling.policies.size.type = SizeBasedTriggeringPolicy
    appender.rolling.policies.size.size=100MB
    appender.rolling.strategy.type = DefaultRolloverStrategy
    appender.rolling.strategy.max = 10

    # Suppress the irrelevant (wrong) warnings from the Netty channel handler
    logger.netty.name = org.jboss.netty.channel.DefaultChannelPipeline
    logger.netty.level = OFF
    • jobmanager-service.yaml可选服务，仅非 HA 模式需要：
apiVersion: v1
kind: Service
metadata:
  name: flink-jobmanager
spec:
  type: ClusterIP
  ports:
  - name: rpc
    port: 6123
  - name: blob-server
    port: 6124
  - name: webui
    port: 8081
  selector:
    app: flink
    component: jobmanager
    • jobmanager-rest-service.yaml 可选服务，将 jobmanager rest端口公开为公共 Kubernetes 节点的端口：
apiVersion: v1
kind: Service
metadata:
  name: flink-jobmanager-rest
spec:
  type: NodePort
  ports:
  - name: rest
    port: 8081
    targetPort: 8081
    nodePort: 30081
  selector:
    app: flink
    component: jobmanager
    • taskmanager-query-state-service.yaml 可选服务，公开 TaskManager 端口以访问可查询状态作为公共 Kubernetes 节点的端口：
apiVersion: v1
kind: Service
metadata:
  name: flink-taskmanager-query-state
spec:
  type: NodePort
  ports:
  - name: query-state
    port: 6125
    targetPort: 6125
    nodePort: 30025
  selector:
    app: flink
    component: taskmanager
  • 如下几个配置文件是公共的：
    • jobmanager-session-deployment-non-ha.yaml：
apiVersion: apps/v1
kind: Deployment
metadata:
  name: flink-jobmanager
spec:
  replicas: 1
  selector:
    matchLabels:
      app: flink
      component: jobmanager
  template:
    metadata:
      labels:
        app: flink
        component: jobmanager
    spec:
      containers:
      - name: jobmanager
        image: myharbor.com/bigdata/flink-centos-admin:1.14.6-scala_2.12
        args: ["jobmanager"]
        ports:
        - containerPort: 6123
          name: rpc
        - containerPort: 6124
          name: blob-server
        - containerPort: 8081
          name: webui
        livenessProbe:
          tcpSocket:
            port: 6123
          initialDelaySeconds: 30
          periodSeconds: 60
        volumeMounts:
        - name: flink-config-volume
          mountPath: /opt/apache/flink-1.14.6/conf/
        securityContext:
          runAsUser: 9999  # refers to user _flink_ from official flink image, change if necessary
      volumes:
      - name: flink-config-volume
        configMap:
          name: flink-config
          items:
          - key: flink-conf.yaml
            path: flink-conf.yaml
          - key: log4j-console.properties
            path: log4j-console.properties
    • taskmanager-session-deployment.yaml：
apiVersion: apps/v1
kind: Deployment
metadata:
  name: flink-taskmanager
spec:
  replicas: 2
  selector:
    matchLabels:
      app: flink
      component: taskmanager
  template:
    metadata:
      labels:
        app: flink
        component: taskmanager
    spec:
      containers:
      - name: taskmanager
        image: myharbor.com/bigdata/flink-centos-admin:1.14.6-scala_2.12
        args: ["taskmanager"]
        ports:
        - containerPort: 6122
          name: rpc
        - containerPort: 6125
          name: query-state
        livenessProbe:
          tcpSocket:
            port: 6122
          initialDelaySeconds: 30
          periodSeconds: 60
        volumeMounts:
        - name: flink-config-volume
          mountPath: /opt/apache/flink-1.14.6/conf/
        securityContext:
          runAsUser: 9999  # refers to user _flink_ from official flink image, change if necessary
      volumes:
      - name: flink-config-volume
        configMap:
          name: flink-config
          items:
          - key: flink-conf.yaml
            path: flink-conf.yaml
          - key: log4j-console.properties
            path: log4j-console.properties
  • 创建 flink 集群：
    • 如下所示：
kubectl create ns flink
# Configuration and service definition
kubectl create -f flink-configuration-configmap.yaml -n flink

# service
kubectl create -f jobmanager-service.yaml -n flink
kubectl create -f jobmanager-rest-service.yaml -n flink
kubectl create -f taskmanager-query-state-service.yaml -n flink

# Create the deployments for the cluster
kubectl create -f jobmanager-session-deployment-non-ha.yaml -n flink
kubectl create -f taskmanager-session-deployment.yaml -n flink
    • 镜像逆向解析 dockerfile：
alias whaler="docker run -t --rm -v /var/run/docker.sock:/var/run/docker.sock:ro pegleg/whaler"
whaler flink:1.14.6-scala_2.12
    • 查看：
kubectl get pods,svc -n flink -owide

在这里插入图片描述

    • Web UI 地址。
  • 提交任务：
./bin/flink run -m local-168-182-110:30081 ./examples/streaming/WordCount.jar

kubectl logs flink-taskmanager-54649bf96c-zjtkh -n flink

在这里插入图片描述
在这里插入图片描述

  • 删除 flink 集群：
kubectl delete -f jobmanager-service.yaml -n flink
kubectl delete -f flink-configuration-configmap.yaml -n flink
kubectl delete -f taskmanager-session-deployment.yaml -n flink
kubectl delete -f jobmanager-session-deployment.yaml -n flink
kubectl delete ns flink --force
  • 访问 flink web：
    • 端口就是 jobmanager-rest-service.yaml 文件中的 NodePort：
http://192.168.182.110:30081/#/overview

在这里插入图片描述

④ application 模式（推荐）

  • Kubernetes 中一个基本的 Flink Application 集群部署包含三个组件：
    • 运行 JobManager 的应用程序；
    • TaskManagers 池的部署；
    • 暴露 JobManager 的 REST 和 UI 端口的服务。

（A）Native Kubernetes 模式（常用）

  • 构建镜像 Dockerfile：
FROM myharbor.com/bigdata/flink:1.14.6-scala_2.12
RUN rm -f /etc/localtime && ln -sv /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && echo "Asia/Shanghai" > /etc/timezone
RUN export LANG=zh_CN.UTF-8
RUN mkdir -p $FLINK_HOME/usrlib
COPY  TopSpeedWindowing.jar $FLINK_HOME/usrlib/
  • 开始构建镜像：
docker build -t myharbor.com/bigdata/flink-application:1.14.6-scala_2.12 . --no-cache

# 上传镜像
docker push myharbor.com/bigdata/flink-application:1.14.6-scala_2.12

# 删除镜像
docker rmi myharbor.com/bigdata/flink-application:1.14.6-scala_2.12
crictl rmi myharbor.com/bigdata/flink-application:1.14.6-scala_2.12
  • 创建命名空间和 serviceacount：
# 创建namespace
kubectl create ns flink
# 创建serviceaccount
kubectl create serviceaccount flink-service-account -n flink
# 用户授权
kubectl create clusterrolebinding flink-role-binding-flink --clusterrole=edit --serviceaccount=flink:flink-service-account
  • 创建 flink 集群并提交任务：
./bin/flink run-application \
    --target kubernetes-application \
    -Dkubernetes.cluster-id=my-first-application-cluster  \
 -Dkubernetes.container.image=myharbor.com/bigdata/flink-application:1.14.6-scala_2.12 \
 -Dkubernetes.jobmanager.replicas=1 \
 -Dkubernetes.namespace=flink \
 -Dkubernetes.jobmanager.service-account=flink-service-account \
 -Dexternal-resource.limits.kubernetes.cpu=2000m \
 -Dexternal-resource.limits.kubernetes.memory=2Gi \
 -Dexternal-resource.requests.kubernetes.cpu=1000m \
 -Dexternal-resource.requests.kubernetes.memory=1Gi \
 -Dkubernetes.rest-service.exposed.type=NodePort \
 local:///opt/flink/usrlib/TopSpeedWindowing.jar
  • local 是应用模式中唯一支持的方案，local 代表本地环境，这里即 pod 或者容器环境，并非宿主机。查看：
kubectl get pods pods,svc -n flink

在这里插入图片描述

kubectl logs -f my-first-application-cluster-taskmanager-1-1 -n flink

在这里插入图片描述
在这里插入图片描述
在这里插入图片描述

  • 删除 flink 集群：

kubectl delete deployment/my-first-application-cluster -n flink
kubectl delete ns flink --force

（B）Standalone 模式

  • 构建镜像 Dockerfile，启动脚本 docker-entrypoint.sh：
#!/usr/bin/env bash

###############################################################################
#  Licensed to the Apache Software Foundation (ASF) under one
#  or more contributor license agreements.  See the NOTICE file
#  distributed with this work for additional information
#  regarding copyright ownership.  The ASF licenses this file
#  to you under the Apache License, Version 2.0 (the
#  "License"); you may not use this file except in compliance
#  with the License.  You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
#  Unless required by applicable law or agreed to in writing, software
#  distributed under the License is distributed on an "AS IS" BASIS,
#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#  See the License for the specific language governing permissions and
# limitations under the License.
###############################################################################

COMMAND_STANDALONE="standalone-job"
COMMAND_HISTORY_SERVER="history-server"

# If unspecified, the hostname of the container is taken as the JobManager address
JOB_MANAGER_RPC_ADDRESS=${JOB_MANAGER_RPC_ADDRESS:-$(hostname -f)}
CONF_FILE="${FLINK_HOME}/conf/flink-conf.yaml"

drop_privs_cmd() {
    if [ $(id -u) != 0 ]; then
        # Don't need to drop privs if EUID != 0
        return
    elif [ -x /sbin/su-exec ]; then
        # Alpine
        echo su-exec admin
    else
        # Others
        echo gosu admin
    fi
}

copy_plugins_if_required() {
  if [ -z "$ENABLE_BUILT_IN_PLUGINS" ]; then
    return 0
  fi

  echo "Enabling required built-in plugins"
  for target_plugin in $(echo "$ENABLE_BUILT_IN_PLUGINS" | tr ';' ' '); do
    echo "Linking ${target_plugin} to plugin directory"
    plugin_name=${target_plugin%.jar}

    mkdir -p "${FLINK_HOME}/plugins/${plugin_name}"
    if [ ! -e "${FLINK_HOME}/opt/${target_plugin}" ]; then
      echo "Plugin ${target_plugin} does not exist. Exiting."
      exit 1
    else
      ln -fs "${FLINK_HOME}/opt/${target_plugin}" "${FLINK_HOME}/plugins/${plugin_name}"
      echo "Successfully enabled ${target_plugin}"
    fi
  done
}

set_config_option() {
  local option=$1
  local value=$2

  # escape periods for usage in regular expressions
  local escaped_option=$(echo ${option} | sed -e "s/\./\\\./g")

  # either override an existing entry, or append a new one
  if grep -E "^${escaped_option}:.*" "${CONF_FILE}" > /dev/null; then
        sed -i -e "s/${escaped_option}:.*/$option: $value/g" "${CONF_FILE}"
  else
        echo "${option}: ${value}" >> "${CONF_FILE}"
  fi
}

prepare_configuration() {
    set_config_option jobmanager.rpc.address ${JOB_MANAGER_RPC_ADDRESS}
    set_config_option blob.server.port 6124
    set_config_option query.server.port 6125

    if [ -n "${TASK_MANAGER_NUMBER_OF_TASK_SLOTS}" ]; then
        set_config_option taskmanager.numberOfTaskSlots ${TASK_MANAGER_NUMBER_OF_TASK_SLOTS}
    fi

    if [ -n "${FLINK_PROPERTIES}" ]; then
        echo "${FLINK_PROPERTIES}" >> "${CONF_FILE}"
    fi
    envsubst < "${CONF_FILE}" > "${CONF_FILE}.tmp" && mv "${CONF_FILE}.tmp" "${CONF_FILE}"
}

maybe_enable_jemalloc() {
    if [ "${DISABLE_JEMALLOC:-false}" == "false" ]; then
        JEMALLOC_PATH="/usr/lib/$(uname -m)-linux-gnu/libjemalloc.so"
        JEMALLOC_FALLBACK="/usr/lib/x86_64-linux-gnu/libjemalloc.so"
        if [ -f "$JEMALLOC_PATH" ]; then
            export LD_PRELOAD=$LD_PRELOAD:$JEMALLOC_PATH
        elif [ -f "$JEMALLOC_FALLBACK" ]; then
            export LD_PRELOAD=$LD_PRELOAD:$JEMALLOC_FALLBACK
        else
            if [ "$JEMALLOC_PATH" = "$JEMALLOC_FALLBACK" ]; then
                MSG_PATH=$JEMALLOC_PATH
            else
                MSG_PATH="$JEMALLOC_PATH and $JEMALLOC_FALLBACK"
            fi
            echo "WARNING: attempted to load jemalloc from $MSG_PATH but the library couldn't be found. glibc will be used instead."
        fi
    fi
}

maybe_enable_jemalloc

copy_plugins_if_required

prepare_configuration

args=("$@")
if [ "$1" = "help" ]; then
    printf "Usage: $(basename "$0") (jobmanager|${COMMAND_STANDALONE}|taskmanager|${COMMAND_HISTORY_SERVER})\n"
    printf "    Or $(basename "$0") help\n\n"
    printf "By default, Flink image adopts jemalloc as default memory allocator. This behavior can be disabled by setting the 'DISABLE_JEMALLOC' environment variable to 'true'.\n"
    exit 0
elif [ "$1" = "jobmanager" ]; then
    args=("${args[@]:1}")

    echo "Starting Job Manager"

    exec $(drop_privs_cmd) "$FLINK_HOME/bin/jobmanager.sh" start-foreground "${args[@]}"
elif [ "$1" = ${COMMAND_STANDALONE} ]; then
    args=("${args[@]:1}")

    echo "Starting Job Manager"

    exec $(drop_privs_cmd) "$FLINK_HOME/bin/standalone-job.sh" start-foreground "${args[@]}"
elif [ "$1" = ${COMMAND_HISTORY_SERVER} ]; then
    args=("${args[@]:1}")

    echo "Starting History Server"

    exec $(drop_privs_cmd) "$FLINK_HOME/bin/historyserver.sh" start-foreground "${args[@]}"
elif [ "$1" = "taskmanager" ]; then
    args=("${args[@]:1}")

    echo "Starting Task Manager"

    exec $(drop_privs_cmd) "$FLINK_HOME/bin/taskmanager.sh" start-foreground "${args[@]}"
fi

args=("${args[@]}")

# Running command in pass-through mode
exec $(drop_privs_cmd) "${args[@]}"
  • 编排 Dockerfile：
FROM myharbor.com/bigdata/centos:7.9.2009

USER root

# 安装常用工具
RUN yum install -y vim tar wget curl rsync bzip2 iptables tcpdump less telnet net-tools lsof

# 设置时区，默认是UTC时区
RUN rm -f /etc/localtime && ln -sv /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && echo "Asia/Shanghai" > /etc/timezone

RUN mkdir -p /opt/apache

ADD jdk-8u212-linux-x64.tar.gz /opt/apache/

ADD flink-1.14.6-bin-scala_2.12.tgz  /opt/apache/

ENV FLINK_HOME /opt/apache/flink-1.14.6
ENV JAVA_HOME /opt/apache/jdk1.8.0_212
ENV PATH $JAVA_HOME/bin:$PATH

# 创建用户应用jar目录
RUN mkdir $FLINK_HOME/usrlib/

#RUN mkdir home
COPY docker-entrypoint.sh /opt/apache/

RUN groupadd --system --gid=9999 admin && useradd --system --home-dir $FLINK_HOME --uid=9999 --gid=admin admin

RUN chown -R admin:admin /opt/apache
RUN chmod +x ${FLINK_HOME}/docker-entrypoint.sh

#设置的工作目录
WORKDIR $FLINK_HOME

# 对外暴露端口
EXPOSE 6123 8081

# 执行脚本，构建镜像时不执行，运行实例才会执行
ENTRYPOINT ["/opt/apache/docker-entrypoint.sh"]
CMD ["help"]

docker build -t myharbor.com/bigdata/flink-centos-admin:1.14.6-scala_2.12 . --no-cache

# 上传镜像
docker push myharbor.com/bigdata/flink-centos-admin:1.14.6-scala_2.12

# 删除镜像
docker rmi myharbor.com/bigdata/flink-centos-admin:1.14.6-scala_2.12
  • 创建命名空间和 serviceacount：
# 创建namespace
kubectl create ns flink
# 创建serviceaccount
kubectl create serviceaccount flink-service-account -n flink
# 用户授权
kubectl create clusterrolebinding flink-role-binding-flink --clusterrole=edit --serviceaccount=flink:flink-service-account
  • 编排 yaml 文件：
    • flink-configuration-configmap.yaml：
apiVersion: v1
kind: ConfigMap
metadata:
  name: flink-config
  labels:
    app: flink
data:
  flink-conf.yaml: |+
    jobmanager.rpc.address: flink-jobmanager
    taskmanager.numberOfTaskSlots: 2
    blob.server.port: 6124
    jobmanager.rpc.port: 6123
    taskmanager.rpc.port: 6122
    queryable-state.proxy.ports: 6125
    jobmanager.memory.process.size: 3200m
    taskmanager.memory.process.size: 2728m
    taskmanager.memory.flink.size: 2280m
    parallelism.default: 2
  log4j-console.properties: |+
    # This affects logging for both user code and Flink
    rootLogger.level = INFO
    rootLogger.appenderRef.console.ref = ConsoleAppender
    rootLogger.appenderRef.rolling.ref = RollingFileAppender

    # Uncomment this if you want to _only_ change Flink's logging
    #logger.flink.name = org.apache.flink
    #logger.flink.level = INFO

    # The following lines keep the log level of common libraries/connectors on
    # log level INFO. The root logger does not override this. You have to manually
    # change the log levels here.
    logger.akka.name = akka
    logger.akka.level = INFO
    logger.kafka.name= org.apache.kafka
    logger.kafka.level = INFO
    logger.hadoop.name = org.apache.hadoop
    logger.hadoop.level = INFO
    logger.zookeeper.name = org.apache.zookeeper
    logger.zookeeper.level = INFO

    # Log all infos to the console
    appender.console.name = ConsoleAppender
    appender.console.type = CONSOLE
    appender.console.layout.type = PatternLayout
    appender.console.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n

    # Log all infos in the given rolling file
    appender.rolling.name = RollingFileAppender
    appender.rolling.type = RollingFile
    appender.rolling.append = false
    appender.rolling.fileName = ${sys:log.file}
    appender.rolling.filePattern = ${sys:log.file}.%i
    appender.rolling.layout.type = PatternLayout
    appender.rolling.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n
    appender.rolling.policies.type = Policies
    appender.rolling.policies.size.type = SizeBasedTriggeringPolicy
    appender.rolling.policies.size.size=100MB
    appender.rolling.strategy.type = DefaultRolloverStrategy
    appender.rolling.strategy.max = 10

    # Suppress the irrelevant (wrong) warnings from the Netty channel handler
    logger.netty.name = org.jboss.netty.channel.DefaultChannelPipeline
    logger.netty.level = OFF
    • jobmanager-service.yaml可选服务，仅非 HA 模式需要：
apiVersion: v1
kind: Service
metadata:
  name: flink-jobmanager
spec:
  type: ClusterIP
  ports:
  - name: rpc
    port: 6123
  - name: blob-server
    port: 6124
  - name: webui
    port: 8081
  selector:
    app: flink
    component: jobmanager
    • jobmanager-rest-service.yaml 可选服务，将 jobmanager rest 端口公开为公共 Kubernetes 节点的端口：
apiVersion: v1
kind: Service
metadata:
  name: flink-jobmanager-rest
spec:
  type: NodePort
  ports:
  - name: rest
    port: 8081
    targetPort: 8081
    nodePort: 30081
  selector:
    app: flink
    component: jobmanager
    • taskmanager-query-state-service.yaml 可选服务，公开 TaskManager 端口以访问可查询状态作为公共 Kubernetes 节点的端口：
apiVersion: v1
kind: Service
metadata:
  name: flink-taskmanager-query-state
spec:
  type: NodePort
  ports:
  - name: query-state
    port: 6125
    targetPort: 6125
    nodePort: 30025
  selector:
    app: flink
    component: taskmanager
    • jobmanager-application-non-ha.yaml，非高可用（注意这里的挂载 /mnt/bigdata/flink/usrlib，最好这里使用共享目录）：
apiVersion: batch/v1
kind: Job
metadata:
  name: flink-jobmanager
spec:
  template:
    metadata:
      labels:
        app: flink
        component: jobmanager
    spec:
      restartPolicy: OnFailure
      containers:
        - name: jobmanager
          image: myharbor.com/bigdata/flink-centos-admin:1.14.6-scala_2.12
          env:
          args: ["standalone-job", "--job-classname", "org.apache.flink.examples.java.wordcount.WordCount","--output","/tmp/result"]
          ports:
            - containerPort: 6123
              name: rpc
            - containerPort: 6124
              name: blob-server
            - containerPort: 8081
              name: webui
          livenessProbe:
            tcpSocket:
              port: 6123
            initialDelaySeconds: 30
            periodSeconds: 60
          volumeMounts:
            - name: flink-config-volume
              mountPath: /opt/apache/flink-1.14.6/conf
            - name: job-artifacts-volume
              mountPath: /opt/apache/flink-1.14.6/usrlib
          securityContext:
            runAsUser: 9999  # refers to user _flink_ from official flink image, change if necessary
      volumes:
        - name: flink-config-volume
          configMap:
            name: flink-config
            items:
              - key: flink-conf.yaml
                path: flink-conf.yaml
              - key: log4j-console.properties
                path: log4j-console.properties
        - name: job-artifacts-volume
          hostPath:
            path: /mnt/nfsdata/flink/application/job-artifacts
    • taskmanager-job-deployment.yaml：
apiVersion: apps/v1
kind: Deployment
metadata:
  name: flink-taskmanager
spec:
  replicas: 2
  selector:
    matchLabels:
      app: flink
      component: taskmanager
  template:
    metadata:
      labels:
        app: flink
        component: taskmanager
    spec:
      containers:
      - name: taskmanager
        image: myharbor.com/bigdata/flink-centos-admin:1.14.6-scala_2.12
        env:
        args: ["taskmanager"]
        ports:
        - containerPort: 6122
          name: rpc
        - containerPort: 6125
          name: query-state
        livenessProbe:
          tcpSocket:
            port: 6122
          initialDelaySeconds: 30
          periodSeconds: 60
        volumeMounts:
        - name: flink-config-volume
          mountPath: /opt/apache/flink-1.14.6/conf
        - name: job-artifacts-volume
          mountPath: /opt/apache/flink-1.14.6/usrlib
        securityContext:
          runAsUser: 9999  # refers to user _flink_ from official flink image, change if necessary
      volumes:
      - name: flink-config-volume
        configMap:
          name: flink-config
          items:
          - key: flink-conf.yaml
            path: flink-conf.yaml
          - key: log4j-console.properties
            path: log4j-console.properties
      - name: job-artifacts-volume
        hostPath:
          path: /mnt/nfsdata/flink/application/job-artifacts
  • 创建 flink 集群并提交任务：
kubectl create ns flink
# Configuration and service definition
kubectl create -f flink-configuration-configmap.yaml -n flink

# service
kubectl create -f jobmanager-service.yaml -n flink
kubectl create -f jobmanager-rest-service.yaml -n flink
kubectl create -f taskmanager-query-state-service.yaml -n flink

# Create the deployments for the cluster
kubectl create -f  jobmanager-application-non-ha.yaml -n flink
kubectl create -f  taskmanager-job-deployment.yaml -n flink
  • 查看：
kubectl get pods,svc -n flink

在这里插入图片描述

  • 删除 flink 集群：
kubectl delete -f flink-configuration-configmap.yaml -n flink
kubectl delete -f jobmanager-service.yaml -n flink
kubectl delete -f jobmanager-rest-service.yaml -n flink
kubectl delete -f taskmanager-query-state-service.yaml -n flink
kubectl delete -f jobmanager-application-non-ha.yaml -n flink
kubectl delete -f taskmanager-job-deployment.yaml -n flink

kubectl delete ns flink --force
  • 查看：
kubectl get pods,svc -n flink
kubectl exec -it flink-taskmanager-54cb7fc57c-g484q -n flink -- bash

在这里插入图片描述

本文来自互联网用户投稿，该文观点仅代表作者本人，不代表本站立场。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如若转载，请注明出处：/a/38223.html

如若内容造成侵权/违法违规/事实不符，请联系我们进行投诉反馈qq邮箱809451989@qq.com，一经查实，立即删除！

相关文章

优秀的 RocketMQ 可视化管理工具 GUI 客户端

优秀的 RocketMQ 可视化管理工具 GUI 客户端

优秀的 RocketMQ 可视化管理工具 GUI 客户端 官网地址&#xff1a;http://www.redisant.cn/rocketmq 快速查看所有 RocketMQ 集群&#xff0c;包括Brokers、Topics和Consumers查看消费者订阅了哪些主题&#xff0c;以及消息队列被分配给了哪些消费者&#xff1b;当出现消息积…
阅读更多...
黑客（网络安全）自学

黑客（网络安全）自学

建议一&#xff1a;黑客七个等级 黑客&#xff0c;对很多人来说充满诱惑力。很多人可以发现这门领域如同任何一门领域&#xff0c;越深入越敬畏&#xff0c;知识如海洋&#xff0c;黑客也存在一些等级&#xff0c;参考知道创宇 CEO ic&#xff08;世界顶级黑客团队 0x557 成员…
阅读更多...
(转载)极限学习机(extreme learning machine, ELM)的回归拟合及分类(matlab实现)

(转载)极限学习机(extreme learning machine, ELM)的回归拟合及分类(matlab实现)

单隐含层前馈神经网络(single-hidden layer feedforward neural network,SLFN)以其良好的学习能力在许多领域中得到了广泛的应用。然而&#xff0c;传统的学习算法(如BP算法等)固有的一些缺点&#xff0c;成为制约其发展的主要瓶颈。前馈神经网络大多采用梯度下降方法&#xff…
阅读更多...
MSA【1】：Segment Anything Model for Medical Image Analysis: an Experimental Study

MSA【1】：Segment Anything Model for Medical Image Analysis: an Experimental Study

文章目录 前言1. Abstraction & Introduction1.1. Abstraction1.2. Introduction1.2.1. What is SAM?1.2.2. How to segment medical images with SAM? 2. Methodology2.1. SAM is used in the process of segmentation of medical images2.1.1. Semi-automated annotati…
阅读更多...
【golang中的切片的相关知识点】[ ] slice

【golang中的切片的相关知识点】[ ] slice

golang-切片 切片的定义和初始化切片的内存分析切片的操作获取长度和容量追加元素复制切片 切片的遍历切片的特性总结 Golang中的切片是一种灵活且强大的数据结构&#xff0c;它可以动态地增长和缩小。切片是基于数组的抽象&#xff0c;它提供了更方便的操作和更灵活的内存管理…
阅读更多...
前端实现 DIV 高度只有100px，宽度只有100px ，我要在这个DIV放一个宽度200的DIV，左右拉动滚动条显示

前端实现 DIV 高度只有100px，宽度只有100px ，我要在这个DIV放一个宽度200的DIV，左右拉动滚动条显示

<!DOCTYPE html> <html> <head><title>点击监听两组span标签</title><style>.outer-div {width: 100px;height: 100px;overflow-x: scroll;background-color: #abc1ee;}.inner-div {width: 200px;}/* 自定义滚动条样式 */.outer-div::-web…
阅读更多...
MYSQL多表查询

MYSQL多表查询

创建学生表和分数表并插入相应数据 mysql> INSERT INTO student VALUES( 901,张老大, 男,1985,计算机系, 北京市海淀区); mysql> INSERT INTO student VALUES( 902,张老二, 男,1986,中文系, 北京市昌平区); mysql> INSERT INTO student VALUES( 903,张三, 女,1990,中…
阅读更多...
我是怎么把win11一步一步变成Mac的

我是怎么把win11一步一步变成Mac的

目录 【三指拖动】 【空格预览】 【切换Ctrl和Alt】 【使用Linux命令】 【其它】 之前很长一段时间在MacBook上面开发习惯了&#xff0c;然后因为一些原因现在换到了windows上面&#xff0c;不管是使用上还是系统上都很不习惯&#xff0c;因此做了一些改造&#xff0c;…
阅读更多...
C++-把字符串转换成整数

C++-把字符串转换成整数

题目来源&#xff1a;牛客网 题目描述&#xff1a; 将一个字符串转换成一个整数&#xff0c;要求不能使用字符串转换整数的库函数。 数值为 0 或者字符串不是一个合法的数值则返回 0 数据范围&#xff1a;字符串长度满足 0≤n≤100 进阶&#xff1a;空间复杂度 O(1) O(1) &…
阅读更多...
APP外包开发原生和H5的对比

APP外包开发原生和H5的对比

在开发APP的技术中&#xff0c;除了原生开发外也可以使用H5框架来开发。原生开发的特点是质量高&#xff0c;用户体验更好&#xff0c;但成本高&#xff0c;适用于对质量要求高的APP项目。H5框架的特点是通用性较强&#xff0c;对开发人员的要求相对较低&#xff0c;成本也低&a…
阅读更多...
Spring Boot中的@EnableWebSocketMessageBroker注解是什么，原理，以及如何使用

Spring Boot中的@EnableWebSocketMessageBroker注解是什么，原理，以及如何使用

Spring Boot中的EnableWebSocketMessageBroker注解是什么&#xff0c;原理&#xff0c;以及如何使用 WebSocket是一种在Web浏览器和Web服务器之间进行双向通信的技术。在传统的HTTP通信中&#xff0c;客户端向服务器发送请求&#xff0c;服务器响应请求&#xff0c;然后关闭连…
阅读更多...
ABAP调用阿里云接口-短信服务-HTTP协议及签名（abap版本）＜转载＞

ABAP调用阿里云接口-短信服务-HTTP协议及签名（abap版本）＜转载＞

原文链接&#xff1a;https://blog.csdn.net/xiefireworks/article/details/113037650 阿里云接口文档请参考官网地址 https://help.aliyun.com/document_detail/59210.html?spm5176.8195934.J_5834642020.5.11ba4378DLVi4O 此处仅介绍使用ABAP完成阿里云短信服务签名请求的…
阅读更多...
感知网络安全态势是什么？感知网络安全态势如何实施

感知网络安全态势是什么？感知网络安全态势如何实施

网络安全是当今社会中一个非常重要的话题。随着互联网的普及和信息技术的发展&#xff0c;网络安全问题日益突出。为了有效应对各种网络威胁和攻击&#xff0c;网络安全态势感知成为了一种关键的技术手段。 网络安全态势感知的定义 网络安全态势感知是指通过对网络环境中的各种…
阅读更多...
如何升级iOS17/iPadOS17公测版？iOS17公测版升级教程

如何升级iOS17/iPadOS17公测版？iOS17公测版升级教程

苹果官方发布了iOS 17/iPadOS 17系统首个公测版更新&#xff0c;其版本号及更新内容与iOS 17 beta 3一致&#xff0c;版本号为21A5277j。 对于想升级iOS 17/iPadOS 17 公测版的用户&#xff0c;可以参考本教程进行操作。 升级注意事项&#xff1a; 1. 为防止意外情况&#xf…
阅读更多...
MySql数据库的学习

MySql数据库的学习

MySQL 是最流行的关系型数据库管理系统&#xff0c;在 WEB 应用方面 MySQL 是最好的 RDBMS&#xff08;Relational Database Management System&#xff1a;关系数据库管理系统&#xff09;应用软件之一。 参考博客&#xff1a;MySQL 教程 | 菜鸟教程 (runoob.com) 一、什么是…
阅读更多...
Latex合并多个公式且居中

Latex合并多个公式且居中

要求&#xff1a;1&#xff1a;多个公式居中对齐 2&#xff1a;多个公式组合只有一个编号。 结果类似于这一种&#xff1a; 代码&#xff1a;使用gathered可以。 \begin{equation}\begin{gathered}\vspace{0.6em}{E} {A(I)}\\\vspace{0.6em}{F} Conv(\sum_{i1}^3{M_i}) \\{…
阅读更多...
微软开源了一个 助力开发LLM 加持的应用的 工具包 semantic-kernel

微软开源了一个 助力开发LLM 加持的应用的 工具包 semantic-kernel

在首席执行官萨蒂亚纳德拉&#xff08;Satya Nadella&#xff09;的支持下&#xff0c;微软似乎正在迅速转变为一家以人工智能为中心的公司。最近微软的众多产品线都采用GPT-4加持&#xff0c;从Microsoft 365等商业产品到“新必应”搜索引擎&#xff0c;再到低代码/无代码Powe…
阅读更多...
【RocketMQ】005-Docker 部署 RocketMQ

【RocketMQ】005-Docker 部署 RocketMQ

【RocketMQ】005-Docker 部署 RocketMQ 文章目录 【RocketMQ】005-Docker 部署 RocketMQ一、部署1、拉取镜像MQ 镜像可视化平台镜像 2、创建挂载目录创建 nameserver 挂载目录创建 broker 目录创建 broker 配置文件目录 3、编辑配置文件4、启动服务启动 nameserver启动 broker启…
阅读更多...
Electron + vue 搭建桌面客户端

Electron + vue 搭建桌面客户端

下载Electron 压缩包&#xff0c;放到本地 Electron 压缩包下载地址 cd ~/Library/Caches/electron
阅读更多...
9.Ceph部署

9.Ceph部署

文章目录 Ceph部署前期环境准备实验部署软件安装部署Ceph集群部署mon节点部署OSD存储节点部署mgr节点开启监控模块管理pool Ceph部署 前期环境准备 主机名public网络cluster网络角色admin192.168.242.69admin(管理节点)node01192.168.242.66192.168.242.100.11mon、mgr、osdn…
阅读更多...
最新文章

Copyright @ 2022~2023