Centos 7 系统 openGauss 3.1.0 一主两备集群安装部署指南

==========================================

现供职于某上市互联网公司担任DBA

Oracle & PG ACE称号，

拥有 Oracle OCM、AWS、以及部分国产数据库等产品认证。

喜欢技术分享，热爱交友，也热爱健身。

2019年加入墨天轮，目前已发表了一百多篇原创文章，曾多次获评"月度墨力之星"。

OceanBase、PG中国分会、IvorySQL等南京地区组织者，也是TeckTalk技术交流社区创始人。

注：本次测试环境所用三台服务器原操作系统为Centos 7.6,曾做过系统升级，操作系统变为 Centos 7.9,后将其调整内核版本为 3.10.0-693.el7.x86_64，操作系统为CentOS Linux release 7.9.2009 (Core)

一、安装环境设置

1.1 硬件环境

名称	最低配置	建议配置	测试配置
服务器数量	3	略	略
硬盘	* 至少1GB用于安装openGauss的应用程序。 * 每个主机需大约300MB用于元数据存储。 * 预留70%以上的磁盘剩余空间用于数据存储。	略	略
内存	>=2G	略	略
CPU	功能调试最小1×8核，2.0GHz	略	略
网络	300兆以上以太网，生产建议采用bond	略	略

1.2 软件环境

软件类型	信息描述
操作系统	Centos 7.6
Linux文件系统	剩余inode个数 > 15亿（推荐）
工具	bzip2
Python	* oepnEuler：支持Python 3.7.x * Centos 7.6: 支持Python 3.6.x
数据库版本	opengauss 3.1.0 企业版
数据库软件包名称	openGauss-3.1.0-CentOS-64bit-all.tar.gz

1.2.1 安装python 3.6

-- root用户(所有节点)
# 本次选择安装python 3.6.5版本
mkdir /usr/local/python3
cd /usr/local/python3
wget https://www.python.org/ftp/python/3.6.5/Python-3.6.5.tgz
tar -zxf Python-3.6.5.tgz
cd /usr/local/python3/Python-3.6.5
./configure --prefix=/usr/local/python3 --enable-shared CFLAGS=-fPIC  
make && make install
ln -s /usr/local/python3/bin/python3 /usr/bin/python3
ln -s /usr/local/python3/bin/pip3 /usr/bin/pip3

-- 设置环境变量
cat >>/etc/profile<<EOF
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/python3/bin
EOF
source /etc/profile
echo "/usr/local/python3/lib/" >> /etc/ld.so.conf
ldconfig

-- 如遇如下报错
python3: error while loading shared libraries: libpython3.6m.so.1.0: cannot open shared object file: No such file or directory
可执行 cp /usr/local/python3/Python-3.6.5/libpython3.6m.so.1.0 /usr/lib64/

1.2.2 安装软件依赖包

1.2.2.1 软件依赖要求

所需软件	建议版本
libaio-devel	建议版本：0.3.109-13
flex	要求版本：2.5.31 以上
bison	建议版本：2.7-4
ncurses-devel	建议版本：5.9-13.20130511
glibc-devel	建议版本：2.17-111
patch	建议版本：2.7.1-10
redhat-lsb-core	建议版本：4.1
readline-devel	建议版本：7.0-13
libnsl（openeuler+x86环境中）	建议版本：2.28-36

1.2.2.2 安装软件依赖包

-- root用户 【所有节点】
-- 安装依赖包
yum install -y libaio-devel flex bison ncurses-devel glibc-devel patch redhat-lsb-core readline-devel zlib readline gcc perl-ExtUtils-Embed readline-devel zlib-devel expect

-- 检查是否已安装
rpm -qa --queryformat "%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n" | grep -E "libaio-devel|flex|bison|ncurses-devel|glibc-devel|patch|redhat-lsb-core|readline-devel|zlib|readline|gcc|python|python-devel|perl-ExtUtils-Embed|readline-devel|zlib-devel|expect"

1.2.3 修改操作系统配置

1.2.3.1 修改操作系统参数

-- root用户 【所有节点】
-- 添加如下参数：
cat>>/etc/sysctl.conf <<EOF
net.ipv4.tcp_fin_timeout=60
net.ipv4.tcp_retries1=5
net.ipv4.tcp_syn_retries=5
net.sctp.path_max_retrans=10
net.sctp.max_init_retransmits=10
EOF

-- 生效参数
sysctl -p

## 执行sysctl -p 时有如下报错，此处选择忽略
sysctl: cannot stat /proc/sys/net/sctp/path_max_retrans: No such file or directory
sysctl: cannot stat /proc/sys/net/sctp/max_init_retransmits: No such file or directory

# 预安装过程根据系统参数配置会有相应提示，请按照提示对应系统参数

1.2.3.2 关闭透明大页

-- root用户 【所有节点】
-- openGauss默认关闭使用transparent_hugepage服务，并将关闭命令写入操作系统启动文件

cat >> /etc/rc.d/rc.local<<EOF
if test -f /sys/kernel/mm/transparent_hugepage/enabled;
	then
		echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/transparent_hugepage/defrag;
	then
		echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi
EOF

-- 查看是否关闭：
cat /sys/kernel/mm/transparent_hugepage/enabled
cat /sys/kernel/mm/transparent_hugepage/defrag

1.2.3.3 关闭防火墙

-- root用户 【所有节点】
systemctl disable firewalld.service
systemctl stop firewalld.service

1.2.3.4 关闭selinux

-- root用户 【所有节点】
sed -i  "s/SELINUX=enforcing/SELINUX=disabled/"  /etc/selinux/config
-- 检查
cat /etc/selinux/config | grep disabled

1.2.3.5 修改字符集

-- root用户 【所有节点】
cat>> /etc/profile<<EOF
export LANG=en_US.UTF-8
EOF
source /etc/profile
# 检查
cat /etc/profile | grep LANG
env |grep -i lang

1.2.3.6 修改时区

-- root用户 【所有节点】
cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
-- 查看
date

1.2.3.7 关闭SWAP

-- root用户 【所有节点】
-- 临时关闭
swapoff -a

-- 永久关闭
注释 swap 行
vim /etc/fstab

-- 查看
free -h 查看 swap一行全是 0 表示swap关闭

1.2.3.8 设置网卡MTU

-- root用户 【所有节点】
-- 设置网卡为1500
ifconfig 网卡名 mtu 1500

-- 查看
ifconfig -a |grep -i mtu

1.2.3.9 设置root用户远程登陆

-- root用户 【所有节点】
cat >>/etc/ssh/sshd_config<<EOF
PermitRootLogin yes
EOF
-- 检查
cat /etc/ssh/sshd_config |grep PermitRootLogin

1.2.3.10 修改 Banner 配置

-- root用户 【所有节点】
cat >>/etc/ssh/sshd_config<<EOF
Banner none
EOF
-- 检查
cat /etc/ssh/sshd_config | grep Banner

-- 重启生效
systemctl restart sshd.service

1.2.3.11 修改DNS配置

-- root用户 【所有节点】
-- 节点一
cat >>/etc/resolv.conf<<EOF
nameserver 192.168.17.113
EOF

-- 节点二
cat >>/etc/resolv.conf<<EOF
nameserver 192.168.17.139
EOF

-- 节点三
cat >>/etc/resolv.conf<<EOF
nameserver 192.168.17.159
EOF

1.2.3.12 添加hosts解析

-- root用户 【所有节点】
cat > /etc/hosts<<EOF
192.168.17.113 opengauss-db1
192.168.17.139 opengauss-db2
192.168.17.159 opengauss-db3
EOF

二、安装规划

2.1 主机名-IP地址-端口号规划

项目名称	描述说明	备注
主机名	opengauss-db1、opengauss-db2、opengauss-db3	主备主机名
IP地址	192.168.17.113、192.168.17.139、192.168.17.159	主备主机名及IP
端口号	5000、26000	cm监听端口、DBnode监听端口

2.2 用户名规划

项目名称	名称	所属类型	规划建议
用户名	omm	操作系统	参照官网
组名	dbgrp	操作系统	参照官网

2.3 软件目录规划

目录名称	对应名称	目录作用
/opt/software/openGauss	software	安装软件存放目录
/opt/gaussdb/install/app	gaussdbAppPath	数据库安装目录
/opt/gaussdb/log	gaussdbLogPath	日志目录
/opt/gaussdb/install/data/db1	dataNode1	主备节点数据存放目录
/opt/gaussdb/tmp	tmpMppdbPath	临时文件目录
/opt/gaussdb/gausstools/om	gaussdbToolPath	数据库工具目录
/opt/gaussdb/corefile	corePath	数据库core文件目录
/opt/gaussdb/install/data/cm	cmDir	CM数据目录

三、软件安装

3.1 下载软件包

打开浏览器登陆 https://www.opengauss.org/zh/download/ 下载 openGauss_3.1.0 Centos_x86_64 对应企业版

3.2 上传及解压软件包

-- 节点一 root 用户
-- 创建存放安装包目录
mkdir -p /opt/software/openGauss
chmod 755 -R /opt/software

-- 上传软件包
# 上传openGauss-3.1.0-CentOS-64bit-all.tar.gz软件至/opt/software/openGauss目录
# 解压软件包,解压openGauss-3.1.0-CentOS-64bit-all.tar.gz后需保留 .sha256文件，否则执行预检查时会报[GAUSS-50201] : The /opt/software/openGauss/openGauss-2.0.0-RedHat-64bit.sha256 does not exist.

tar -zxvf openGauss-3.1.0-CentOS-64bit-all.tar.gz

tar -zxvf openGauss-3.1.0-CentOS-64bit-om.tar.gz

3.3 创建XML配置文件

3.3.1 配置XML文件

-- 节点一 root 用户
-- 在/opt/software/openGauss 目录下创建 clusterconfig.xml配置文件
-- 执行操作如下
cat > clusterconfig.xml<<EOF
<?xml version="1.0" encoding="UTF-8"?>
<ROOT>
    <!-- openGauss整体信息 -->
    <CLUSTER>
    <!-- 数据库名称 -->
        <PARAM name="clusterName" value="openGSDB" />
    <!-- 数据库节点名称(hostname) -->
        <PARAM name="nodeNames" value="opengauss-db1,opengauss-db2,opengauss-db3" />
    <!-- 节点IP，与nodeNames一一对应 -->
        <PARAM name="backIp1s" value="192.168.17.113,192.168.17.139,192.168.17.159"/>
    <!-- 数据库安装目录-->
        <PARAM name="gaussdbAppPath" value="/opt/gaussdb/install/app" />
    <!-- 日志目录-->
        <PARAM name="gaussdbLogPath" value="/opt/gaussdb/log" />
    <!-- 临时文件目录-->
        <PARAM name="tmpMppdbPath" value="/opt/gaussdb/tmp"/>
    <!--数据库工具目录-->
        <PARAM name="gaussdbToolPath" value="/opt/gaussdb/gausstools/om" />
    <!--数据库core文件目录-->
        <PARAM name="corePath" value="/opt/gaussdb/corefile"/>
    <!-- openGauss类型，此处示例为单机类型，"single-inst"表示单机一主多备部署形态-->
        <PARAM name="clusterType" value="single-inst"/>
    </CLUSTER>
    <!-- 每台服务器上的节点部署信息 -->
    <DEVICELIST>
        <!-- opengauss-db1上的节点部署信息 -->
        <DEVICE sn="1000001">
        <!-- opengauss-db1的hostname -->
            <PARAM name="name" value="opengauss-db1"/>
        <!-- opengauss-db1所在的AZ及AZ优先级 -->
            <PARAM name="azName" value="AZ1"/>
            <PARAM name="azPriority" value="1"/>
        <!-- 如果服务器只有一个网卡可用，将backIP1和sshIP1配置成同一个IP -->
            <PARAM name="backIp1" value="192.168.17.113"/>
            <PARAM name="sshIp1" value="192.168.17.113"/>
            
        <!--CM-->
     <!--CM数据目录-->
            <PARAM name="cmDir" value="/opt/gaussdb/install/data/cm" />
            <PARAM name="cmsNum" value="1" />
      <!--CM监听端口-->
            <PARAM name="cmServerPortBase" value="5000" />
            <PARAM name="cmServerlevel" value="1" />
      <!--CM所有实例所在节点名及监听ip-->
            <PARAM name="cmServerListenIp1" value="192.168.17.113,192.168.17.139,192.168.17.159" />
            <PARAM name="cmServerRelation" value="opengauss-db1,opengauss-db2,opengauss-db3" />
            
      <!--dbnode-->
        <PARAM name="dataNum" value="1"/>
      <!--DBnode端口号-->
        <PARAM name="dataPortBase" value="26000"/>
      <!--DBnode主节点上数据目录，及备机数据目录-->
        <PARAM name="dataNode1" value="/opt/gaussdb/install/data/db1,opengauss-db2,/opt/gaussdb/install/data/db1,opengauss-db3,/opt/gaussdb/install/data/db1"/>
      <!--DBnode节点上设定同步模式的节点数-->
        <PARAM name="dataNode1_syncNum" value="0"/>
        </DEVICE>

        <!-- opengauss-db2上的节点部署信息，其中"name"的值配置为主机名称(hostname) -->
        <DEVICE sn="1000002">
            <PARAM name="name" value="opengauss-db2"/>
            <PARAM name="azName" value="AZ1"/>
            <PARAM name="azPriority" value="1"/>
            <!-- 如果服务器只有一个网卡可用，将backIP1和sshIP1配置成同一个IP -->
            <PARAM name="backIp1" value="192.168.17.139"/>
            <PARAM name="sshIp1" value="192.168.17.139"/>
            <PARAM name="cmDir" value="/opt/gaussdb/install/data/cm" />
        </DEVICE>

        <!-- opengauss-db3上的节点部署信息，其中"name"的值配置为主机名称(hostname) -->
        <DEVICE sn="1000003">
            <PARAM name="name" value="opengauss-db3"/>
            <PARAM name="azName" value="AZ1"/>
            <PARAM name="azPriority" value="1"/>
            <!-- 如果服务器只有一个网卡可用，将backIP1和sshIP1配置成同一个IP -->
            <PARAM name="backIp1" value="192.168.17.159"/>
            <PARAM name="sshIp1" value="192.168.17.159"/>
            <PARAM name="cmDir" value="/opt/gaussdb/install/data/cm" />
        </DEVICE>
    </DEVICELIST>
</ROOT>
EOF

3.3.2 参数简介

实例类型	参数	说明
整体信息	name	主机名称
azName	指定azName（Available Zone Name），字符串（不能含有特殊字符），例如AZ1、AZ2、AZ3。
azPriority	指定azPriority的优先级。
backIp1	主机在后端存储网络中的IP地址（内网IP）。所有openGauss主机使用后端存储网络通讯。
sshIp1	设置SSH可信通道IP地址（外网IP）。若无外网，则可以不设置该选项或者同backIp1设置相同IP。

3.4 初始化安装环境

# root 用户 节点一操作

-- 设置lib库
export LD_LIBRARY_PATH=/opt/software/openGauss/script/gspylib/clib:$LD_LIBRARY_PATH

-- 执行预执行
python3 /opt/software/openGauss/script/gs_preinstall -U omm -G dbgrp -X /opt/software/openGauss/clusterconfig.xml
# 预执行过程中，从日志来看是拷贝了software目录下文件到其它节点，将节点一的scripts下ssh-agent.sh拷贝到其它节点/root/.ssh/目录下，执行完并将其删除，创建了omm用户，每个节点/etc/profile下设置环境变量
# 预执行详细执行过程可查看 /opt/gaussdb/log/omm/om 目录下 

# 执行过程可查看/opt/gaussdb/log/omm/om 目录下gs_preinstall日志文件gs_preinstall_xxx.log文件

-- 预检查执行结果如下
Parsing the configuration file.
Successfully parsed the configuration file.
Installing the tools on the local node.
Successfully installed the tools on the local node.
Are you sure you want to create trust for root (yes/no)?yes  -- 输入 yes
Please enter password for root
Password:   输入 root 口令
Successfully created SSH trust for the root permission user.
Setting host ip env
Successfully set host ip env.
Distributing package.
Begin to distribute package to tool path.
Successfully distribute package to tool path.
Begin to distribute package to package path.
Successfully distribute package to package path.
Successfully distributed package.
Are you sure you want to create the user[omm] and create trust for it (yes/no)? yes -- 输入 yes
Please enter password for cluster user.
Password:  -- 设置 omm 口令
Please enter password for cluster user again.
Password:  -- 再次输入 omm 口令
Generate cluster user password files successfully.

Successfully created [omm] user on all nodes.
Preparing SSH service.
Successfully prepared SSH service.
Installing the tools in the cluster.
Successfully installed the tools in the cluster.
Checking hostname mapping.
Successfully checked hostname mapping.
Creating SSH trust for [omm] user.
Please enter password for current user[omm].  
Password:   -- 输入 omm 口令
Checking network information.
All nodes in the network are Normal.
Successfully checked network information.
Creating SSH trust.
Creating the local key file.
Successfully created the local key files.
Appending local ID to authorized_keys.
Successfully appended local ID to authorized_keys.
Updating the known_hosts file.
Successfully updated the known_hosts file.
Appending authorized_key on the remote node.
Successfully appended authorized_key on all remote node.
Checking common authentication file content.
Successfully checked common authentication content.
Distributing SSH trust file to all node.
Distributing trust keys file to all node successfully.
Successfully distributed SSH trust file to all node.
Verifying SSH trust on all hosts.
Successfully verified SSH trust on all hosts.
Successfully created SSH trust.
Successfully created SSH trust for [omm] user.
Checking OS software.
Successfully check os software.
Checking OS version.
Successfully checked OS version.
Creating cluster's path.
Successfully created cluster's path.
Set and check OS parameter.
Setting OS parameters.
Successfully set OS parameters.
Warning: Installation environment contains some warning messages.
Please get more details by "/opt/software/openGauss/script/gs_checkos -i A -h opengauss-db1,opengauss-db2,opengauss-db3 --detail".
Set and check OS parameter completed.
Preparing CRON service.
Successfully prepared CRON service.
Setting user environmental variables.
Successfully set user environmental variables.
Setting the dynamic link library.
Successfully set the dynamic link library.
Setting Core file
Successfully set core path.
Setting pssh path
Successfully set pssh path.
Setting Cgroup.
Successfully set Cgroup.
Set ARM Optimization.
No need to set ARM Optimization.
Fixing server package owner.
Setting finish flag.
Successfully set finish flag.
Preinstallation succeeded.

-- 查看预安装结果信息，并根据预检查调整参数
[root@opengauss-db1 ~]# /opt/software/openGauss/script/gs_checkos -i A -h opengauss-db1,opengauss-db2,opengauss-db3 --detail
-- 执行结果如下
# 本次测试环境三台服务器，操作系统为 Centos 7.9, 系统内核为 3.10.0-693.el7.x86_64  曾经做过7.6系统的升级
Checking items:
    A1. [ OS version status ]                                   : Normal     
        [opengauss-db3]
        centos_7.9.2009_64bit
        [opengauss-db2]
        centos_7.9.2009_64bit
        [opengauss-db1]
        centos_7.9.2009_64bit

    A2. [ Kernel version status ]                               : Warning    
        [opengauss-db1]
        3.10.0-957.27.2.el7.x86_64
        [opengauss-db2]
        3.10.0-1160.76.1.el7.x86_64
        [opengauss-db3]
        3.10.0-1160.76.1.el7.x86_64

    A3. [ Unicode status ]                                      : Normal     
        The values of all unicode are same. The value is "LANG=en_US.UTF-8".
    A4. [ Time zone status ]                                    : Normal     
        The informations about all timezones are same. The value is "+0800".
    A5. [ Swap memory status ]                                  : Normal     
        The value about swap memory is correct.            
    A6. [ System control parameters status ]                    : Warning    
        [opengauss-db1]
        Warning reason: variable 'net.ipv4.ip_local_port_range' RealValue '1024 65000' ExpectedValue '26000 65535'.
        Check_SysCtl_Parameter warning.

        [opengauss-db2]
        Warning reason: variable 'net.ipv4.ip_local_port_range' RealValue '1024 65000' ExpectedValue '26000 65535'.
        Check_SysCtl_Parameter warning.

        [opengauss-db3]
        Warning reason: variable 'net.ipv4.ip_local_port_range' RealValue '1024 65000' ExpectedValue '26000 65535'.
        Check_SysCtl_Parameter warning.


    A7. [ File system configuration status ]                    : Warning    
        [opengauss-db1]
        Warning reason: variable 'open files' RealValue '65536' ExpectedValue '1000000'
        Warning reason: variable 'max user processes' RealValue '69632' ExpectedValue 'unlimited'

        [opengauss-db2]
        Warning reason: variable 'open files' RealValue '65536' ExpectedValue '1000000'
        Warning reason: variable 'max user processes' RealValue '69632' ExpectedValue 'unlimited'

        [opengauss-db3]
        Warning reason: variable 'open files' RealValue '65536' ExpectedValue '1000000'
        Warning reason: variable 'max user processes' RealValue '69632' ExpectedValue 'unlimited'


    A8. [ Disk configuration status ]                           : Normal     
        The value about XFS mount parameters is correct.   
    A9. [ Pre-read block size status ]                          : Normal     
        The value about Logical block size is correct.     
    A10.[ IO scheduler status ]                                 : Normal     
        The value of IO scheduler is correct.              
    A11.[ Network card configuration status ]                   : Warning    
        [opengauss-db1]
BondMode Null
        Warning reason: network 'ens33' 'mtu' RealValue '1500' ExpectedValue '8192'   -- 可忽略

        [opengauss-db2]
BondMode Null
        Warning reason: network 'ens33' 'mtu' RealValue '1500' ExpectedValue '8192'   -- 可忽略

        [opengauss-db3]
BondMode Null
        Warning reason: network 'ens33' 'mtu' RealValue '1500' ExpectedValue '8192'   -- 可忽略


    A12.[ Time consistency status ]                             : Warning    
        [opengauss-db1]
        The NTPD not detected on machine and local time is "2022-11-09 11:53:23".
        [opengauss-db2]
        The NTPD not detected on machine and local time is "2022-11-09 11:53:23".
        [opengauss-db3]
        The NTPD not detected on machine and local time is "2022-11-09 11:53:24".

    A13.[ Firewall service status ]                             : Normal     
        The firewall service is stopped.                   
    A14.[ THP service status ]                                  : Normal     
        The THP service is stopped.                        
Total numbers:14. Abnormal numbers:0. Warning numbers:5.

=============================================================================
-- 根据执行detail命令结果对预检查告警进行调整
-- 预检查过程中遇到的问题可参照 后文 附录 部分

3.5 执行集群安装

3.5.1 修改属主

-- 务必确保已在节点一执行预检查
-- root 用户 节点一操作
chmod -R 755 /opt/software/openGauss/script/
chown -R omm:dbgrp /opt/software/openGauss/script/

3.5.2 执行安装

-- 节点一切换到omm用户执行
su - omm
-- 执行如下命令
gs_install -X /opt/software/openGauss/clusterconfig.xml

# 执行过程可查看/opt/gaussdb/log/omm/om及各节点/opt/gaussdb/log/omm/pg_log/ 下日志文件
# 

[omm@opengauss-db1 ~]$ gs_install -X /opt/software/openGauss/clusterconfig.xml --gsinit-parameter="--encoding=UTF8"

-- 执行结果如下
Parsing the configuration file.
Check preinstall on every node.
Successfully checked preinstall on every node.
Creating the backup directory.
Successfully created the backup directory.
begin deploy..
Installing the cluster.
begin prepare Install Cluster..
Checking the installation environment on all nodes.
begin install Cluster..
Installing applications on all nodes.
Successfully installed APP.
begin init Instance..
encrypt cipher and rand files for database.
Please enter password for database:     -- 设置数据库 口令 如 Passw0rd@1234
Please repeat for database:             -- 再次输入相同口令
begin to create CA cert files
The sslcert will be generated in /opt/gaussdb/install/app/share/sslcert/om
Create CA files for cm beginning.
Create CA files on directory [/opt/gaussdb/install/app_4e931f9a/share/sslcert/cm]. file list: ['client.key.rand', 'server.crt', 'cacert.pem', 'client.crt', 'server.key.rand', 'client.key', 'client.key.cipher', 'server.key', 'server.key.cipher']
Cluster installation is completed.
Configuring.
Deleting instances from all nodes.
Successfully deleted instances from all nodes.
Checking node configuration on all nodes.
Initializing instances on all nodes.
Updating instance configuration on all nodes.
Check consistence of memCheck and coresCheck on database nodes.
Successful check consistence of memCheck and coresCheck on all nodes.
Configuring pg_hba on all nodes.
Configuration is completed.
Starting cluster.
======================================================================
Successfully started primary instance. Wait for standby instance.
======================================================================
.
Successfully started cluster.
======================================================================
cluster_state      : Normal
redistributing     : No
node_count         : 3
Datanode State
    primary           : 1
    standby           : 2
    secondary         : 0
    cascade_standby   : 0
    building          : 0
    abnormal          : 0
    down              : 0

Successfully installed application.
end deploy..

3.5.3 数据库状态检查

-- omm用户
[omm@opengauss-db1 ~]$ gs_om -t status --detail
[  CMServer State   ]

node             node_ip         instance                                    state
------------------------------------------------------------------------------------
1  opengauss-db1 192.168.17.113  1    /opt/gaussdb/install/data/cm/cm_server Primary
2  opengauss-db2 192.168.17.139  2    /opt/gaussdb/install/data/cm/cm_server Standby
3  opengauss-db3 192.168.17.159  3    /opt/gaussdb/install/data/cm/cm_server Standby

[   Cluster State   ]

cluster_state   : Normal
redistributing  : No
balanced        : Yes
current_az      : AZ_ALL

[  Datanode State   ]

node             node_ip         instance                           state            
-------------------------------------------------------------------------------------
1  opengauss-db1 192.168.17.113  6001 /opt/gaussdb/install/data/db1 P Primary Normal
2  opengauss-db2 192.168.17.139  6002 /opt/gaussdb/install/data/db1 S Standby Normal
3  opengauss-db3 192.168.17.159  6003 /opt/gaussdb/install/data/db1 S Standby Normal

3.5.4 集群状态检查

-- omm 用户
[omm@opengauss-db1 ~]$ cm_ctl query -Cv
[  CMServer State   ]

node             instance state
---------------------------------
1  opengauss-db1 1        Primary
2  opengauss-db2 2        Standby
3  opengauss-db3 3        Standby

[   Cluster State   ]

cluster_state   : Normal
redistributing  : No
balanced        : Yes
current_az      : AZ_ALL

[  Datanode State   ]

node             instance state            | node             instance state            | node             instance state
---------------------------------------------------------------------------------------------------------------------------------------
1  opengauss-db1 6001     P Primary Normal | 2  opengauss-db2 6002     S Standby Normal | 3  opengauss-db3 6003     S Standby Normal

四、集群测试

4.1 主备数据验证测试

-- omm用户
-- 主节点
[omm@opengauss-db1 ~]$ gsql -d postgres -p 26000
gsql ((openGauss 3.1.0 build 4e931f9a) compiled at 2022-09-29 14:19:24 commit 0 last mr  )
Non-SSL connection (SSL connection is recommended when requiring high-security)
Type "help" for help.

openGauss=# create table tb1 (id int, name text);
CREATE TABLE
openGauss=# insert into tb1 (id,name) values (10,'Jacky');
INSERT 0 1
openGauss=# commit;
WARNING:  there is no transaction in progress
COMMIT
openGauss=# \echo :AUTOCOMMIT  -- 默认开启了自动提交
on
openGauss=#

-- 备节点一 
[omm@opengauss-db2 ~]$ gsql -d postgres -p 26000
gsql ((openGauss 3.1.0 build 4e931f9a) compiled at 2022-09-29 14:19:24 commit 0 last mr  )
Non-SSL connection (SSL connection is recommended when requiring high-security)
Type "help" for help.

openGauss=# select * from tb1;
 id | name  
----+-------
 10 | Jacky
(1 row)

openGauss=# insert into tb1 (id,name) values (11,'shlei');
ERROR:  cannot execute INSERT in a read-only transaction
openGauss=# 

-- 备节点二
[omm@opengauss-db3 ~]$ gsql -d postgres -p 26000
gsql ((openGauss 3.1.0 build 4e931f9a) compiled at 2022-09-29 14:19:24 commit 0 last mr  )
Non-SSL connection (SSL connection is recommended when requiring high-security)
Type "help" for help.

openGauss=# select * from tb1;
 id | name  
----+-------
 10 | Jacky
(1 row)

openGauss=# insert into tb1 (id,name) values (11,'shlei');
ERROR:  cannot execute INSERT in a read-only transaction
openGauss=#

4.2 集群启停测试

4.2.1 停止集群

-- omm 用户
-- 停止集群
[omm@opengauss-db1 ~]$ cm_ctl stop
cm_ctl: stop cluster. 
cm_ctl: stop nodeid: 1
cm_ctl: stop nodeid: 2
cm_ctl: stop nodeid: 3
..........
cm_ctl: stop cluster successfully.

-- 查看集群状态
[omm@opengauss-db1 ~]$ cm_ctl query -Cv
[  CMServer State   ]

node             instance state
---------------------------------
1  opengauss-db1 1        Down
2  opengauss-db2 2        Down
3  opengauss-db3 3        Down

cm_ctl: can't connect to cm_server.
Maybe cm_server is not running, or timeout expired. Please try again.

4.2.2 启动集群

-- omm 用户
-- 启动集群
[omm@opengauss-db1 ~]$ cm_ctl start
cm_ctl: checking cluster status.
cm_ctl: checking cluster status.
cm_ctl: checking finished in 901 ms.
cm_ctl: start cluster. 
cm_ctl: start nodeid: 1
cm_ctl: start nodeid: 2
cm_ctl: start nodeid: 3
.................
cm_ctl: start cluster successfully.


-- 查看集群状态
[omm@opengauss-db1 ~]$ cm_ctl query -Cv
[  CMServer State   ]

node             instance state
---------------------------------
1  opengauss-db1 1        Primary
2  opengauss-db2 2        Standby
3  opengauss-db3 3        Standby

[   Cluster State   ]

cluster_state   : Normal
redistributing  : No
balanced        : Yes
current_az      : AZ_ALL

[  Datanode State   ]

node             instance state            | node             instance state            | node             instance state
---------------------------------------------------------------------------------------------------------------------------------------
1  opengauss-db1 6001     P Primary Normal | 2  opengauss-db2 6002     S Standby Normal | 3  opengauss-db3 6003     S Standby Normal

五、补充

5.1 初始化报expect命令未找到

-- 如未安装expect包导致初始化预检查调用expect命令时报错，报错信息如下：
[2022-11-09 11:13:42.081711][2196828][gs_sshexkey][DEBUG]:add ssh id_rsa output:/root/.ssh/./ssh-agent.sh: line 18: expect: command not found
[2022-11-09 11:13:42.082355][2196828][gs_sshexkey][DEBUG]:Error: Failed to register other ssh-agent, output is [Failed to ssh-add perform.Error: /root/.ssh/./ssh-agent.sh: line 18: expect: command not found] for 2 times
[2022-11-09 11:13:42.082573][2196828][gs_sshexkey][ERROR]:Error: Failed to register other ssh-agent,output is [Failed to ssh-add perform.Error: /root/.ssh/./ssh-agent.sh: line 18: expect: command not found]
[2022-11-09 11:13:42.314059][2196716][gs_preinstall][ERROR]:[GAUSS-51632] : Failed to do gs_sshexkey.Error: Please enter password for current user[root].

-- ssh-agent.sh脚本是openGauss自带脚本，内容为：
#!/bin/bash
#############################################################################
# Copyright (c): 2021-2025, Huawei Tech. Co., Ltd.
# FileName     : ssh-agent
# Version      : V1.0.0
# Date         : 2020-01-13
#############################################################################

read -r secret
id_rsa_path="$1"
passwd=$secret
cmd="ssh-add ${id_rsa_path}"
func_remote_execute_cmd()
{
set timeout 5
echo "Access Method"
echo "ssh-agent command:$cmd"
expect << EOF
spawn $cmd
expect {
    "Enter passphrase for *" {send $passwd\n}
    }
expect {
    "Bad passphrase*" {puts \"failure\";exit 2}
    eof {puts \"success:eof\"\n }
    }
catch wait result
puts \$result
exit [lindex \$result 3]
EOF
}
func_remote_execute_cmd

-- 解决办法：使用yum 执行 yum install -y expect后重新执行预执行命令

5.2 初始化报libpython.so文件未找到

# 在预检查过程中，遇到如下一些报错

报错一：
[GAUSS-51400] : Failed to execute the command: rm -rf '/tmp/step_preinstall_file.dat'. Result:{'opengauss-node1': 'Failure', 'opengauss-node2': 'Failure'}.
Error:
[FAILURE] opengauss-node1:
[FAILURE] opengauss-node2:

[GAUSS-52200] : Unable to import module: libpython3.6m.so.1.0: cannot open shared object file: No such file or directory.

-- 解决办法：
重新编译python,或者在其它已编译该python版本环境将libpython3.6m.so.1.0拷贝到当前服务器/usr/lib64 目录下
并在/etc/profile里执行 export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/python3/bin，并source /etc/profile生效

5.3 重新执行预执行失败

-- 再次执行预执行时会因前一次预执行已创建了一些目录和文件，会导致失败。
-- 解决办法：可参照xml文件，将除/opt/software/openGauss之外的其它目录和文件，最好将创建的omm用户和dbgrp组删除，