ClickHouse 3节点集群安装

ClickHouse 简介

ClickHouse是一个用于联机分析(OLAP)的列式数据库管理系统(DBMS)。

在这里插入图片描述
官方网站：https://clickhouse.com/

项目地址：https://github.com/ClickHouse/ClickHouse

横向扩展集群介绍

此示例架构旨在提供可扩展性。它包括三个节点：两个组合的 ClickHouse 加协调（ClickHouse Keeper）服务器，以及第三个仅包含 ClickHouse Keeper 的服务器，以完成三个节点的法定人数。在这个示例中，我们将创建一个数据库、表和一个分布式表，以便能够查询两个节点上的数据。

基本概念

Replica

数据副本。 ClickHouse 始终至少拥有一份数据副本，因此副本的最小数量为 1。这是一个重要的细节，您可能不习惯将数据的原始副本算作副本，但这是 ClickHouse 代码和文档中使用的术语。添加数据的第二个副本可提供容错能力。

Shard

数据的子集。 ClickHouse 始终为您的数据提供至少一个分片，因此如果您不将数据拆分到多个服务器上，您的数据将存储在一个分片中。如果超出单个服务器的容量，可以使用将数据分片到多个服务器来划分负载。目标服务器由分片键决定，在创建分布式表时定义。分片密钥可以是随机的，也可以是哈希函数的输出。涉及分片的部署示例将使用rand()作为分片键，并将提供有关何时以及如何选择不同分片键的更多信息。

Distributed coordination

ClickHouse Keeper 提供数据复制和分布式 DDL 查询执行的协调系统。 ClickHouse Keeper 与 Apache ZooKeeper 兼容。

环境架构图

官方文档：https://clickhouse.com/docs/en/architecture/horizontal-scaling

我们将安装一个在 2 个节点上运行的 Clickhouse 集群，在这两台服务器上运行clickhouse-keeper，以及一个独立运行的 clickhouse-keeper。可以使用 apt install 进行在线安装。

整体部署架构如下图所示：
在这里插入图片描述

节点清单

主机名	节点IP	操作系统	节点配置	节点角色
chnode1	192.168.72.51	Ubuntu22.04	2C/4G/100G DISK	Clikhouse keeper , Clickhose server, client
chnode2	192.168.72.52	Ubuntu22.04	2C/4G/100G DISK	Clikhouse keeper , Clickhose server, client
chnode3	192.168.72.53	Ubuntu22.04	2C/4G/100G DISK	Clikhouse keeper

说明：

chnode3用于 ClickHouse Keeper 法定人数
在生产环境中，强烈建议 ClickHouse Keeper 在专用主机上运行。此基本配置在 ClickHouse 服务器进程中运行 Keeper 功能。官方安装文档中提供了独立部署 ClickHouse Keeper 的说明。

在所有节点上配置主机名

hostnamectl set-hostname chnode1
hostnamectl set-hostname chnode2
hostnamectl set-hostname chnode3

在所有节点上编辑 /etc/hosts 文件

cat >/etc/hosts<<EOF
192.168.72.51 chnode1 chnode1.example.com
192.168.72.52 chnode2 chnode2.example.com
192.168.72.53 chnode3 chnode3.example.com
EOF

安装clickhouse

官方文档：https://clickhouse.com/docs/en/install

chnode1和chnode2执行

节点1和节点2不需要安装ClickHouse Keeper，不能一起安装，如果要在服务器上使用keeper，可以通过config设置来完成。

sudo apt-get install -y apt-transport-https ca-certificates dirmngr
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 8919F6BD2B48D754
echo "deb https://packages.clickhouse.com/deb stable main" | sudo tee \
    /etc/apt/sources.list.d/clickhouse.list
sudo apt-get update
sudo apt-get install -y clickhouse-server clickhouse-client

说明：安装过程中注意为clickhouse-server默认用户default设置密码。

查看安装版本

root@chnode1:~# clickhouse --version
ClickHouse local version 24.9.2.42 (official build).

chnode3执行

在节点 3 上仅安装clickhose-keeper 。ClickHouse 服务器中有一个捆绑包，如果正在安装服务器，则无法单独安装 clickhouse-keeper，会收到冲突错误，但是，如果我们有仅用作 clickhouse-keeper 的服务器，这里只能安装clickhouse-keeper，无需安装服务器。

sudo apt-get install -y apt-transport-https ca-certificates dirmngr
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 8919F6BD2B48D754
echo "deb https://packages.clickhouse.com/deb stable main" | sudo tee \
    /etc/apt/sources.list.d/clickhouse.list
sudo apt-get update
sudo apt-get install -y clickhouse-keeper

检查安装版本

root@chnode3:~# clickhouse-keeper --version
ClickHouse keeper version 24.9.2.42 (official build).

创建clickhose-keeper相关目录

mkdir -p /etc/clickhouse-keeper/config.d
mkdir -p /var/log/clickhouse-keeper
mkdir -p /var/lib/clickhouse-keeper/coordination/log
mkdir -p /var/lib/clickhouse-keeper/coordination/snapshots
mkdir -p /var/lib/clickhouse-keeper/cores
chown -R clickhouse.clickhouse /etc/clickhouse-keeper /var/log/clickhouse-keeper /var/lib/clickhouse-keeper

编辑chnode1配置文件

对于chnode1，有五个配置文件。您可以选择将这些文件合并为一个文件，但为了文档的清晰性，单独查看它们可能会更简单。当你阅读配置文件时，你会发现 chnode1 和 chnode2 之间的大部分配置是相同的；差异将被突出显示。

网络和日志记录配置

这些值可以根据您的意愿进行定制。此示例配置为您提供了一个debug级别的调试日志，并且日志文件会在大小达到 1000M时触发轮转（roll over）。轮转的操作会保留三份之前的日志文件。ClickHouse 将在端口 8123 和 9000 上侦听 IPv4 网络，并将使用端口 9009 进行服务器间通信。

chnode1 上的 network-and-logging.xml

root@chnode1:~# cat /etc/clickhouse-server/config.d/network-and-logging.xml
<clickhouse>
        <logger>
                <level>debug</level>
                <log>/var/log/clickhouse-server/clickhouse-server.log</log>
                <errorlog>/var/log/clickhouse-server/clickhouse-server.err.log</errorlog>
                <size>1000M</size>
                <count>3</count>
        </logger>
        <display_name>clickhouse</display_name>
        <listen_host>0.0.0.0</listen_host>
        <http_port>8123</http_port>
        <tcp_port>9000</tcp_port>
        <interserver_http_port>9009</interserver_http_port>
        <interserver_listen_host>0.0.0.0</interserver_listen_host>
</clickhouse>

ClickHouse Keeper配置

ClickHouse Keeper 提供数据复制和分布式 DDL 查询执行的协调系统。 ClickHouse Keeper 与 Apache ZooKeeper 兼容。此配置在端口 9181 上启用 ClickHouse Keeper。突出显示的行指定该 Keeper 实例的server_id为 1。这是三台服务器的enable-keeper.xml文件中的唯一区别。 chnode2将server_id设置为2 ， chnode3将server_id设置为3 。所有三台服务器上的 raft 配置部分都是相同的，下面突出显示它以向您展示 raft 配置中server_id和server实例之间的关系。

说明：

如果出于任何原因更换或重建 Keeper 节点，请勿重复使用现有的server_id 。例如，如果重建了server_id为2的Keeper节点，则将其server_id设置为4或更高。

在 chnode1 上创建enable-keeper.xml

root@chnode1:~# cat /etc/clickhouse-server/config.d/enable-keeper.xml
<clickhouse>
  <keeper_server>
    <tcp_port>9181</tcp_port>
    <server_id>1</server_id>
    <log_storage_path>/var/lib/clickhouse/coordination/log</log_storage_path>
    <snapshot_storage_path>/var/lib/clickhouse/coordination/snapshots</snapshot_storage_path>

    <coordination_settings>
        <operation_timeout_ms>10000</operation_timeout_ms>
        <session_timeout_ms>30000</session_timeout_ms>
        <raft_logs_level>trace</raft_logs_level>
    </coordination_settings>

    <raft_configuration>
        <server>
            <id>1</id>
            <hostname>chnode1</hostname>
            <port>9234</port>
        </server>
        <server>
            <id>2</id>
            <hostname>chnode2</hostname>
            <port>9234</port>
        </server>
        <server>
            <id>3</id>
            <hostname>chnode3</hostname>
            <port>9234</port>
        </server>
    </raft_configuration>
  </keeper_server>
</clickhouse>

宏配置

宏shard和replica降低了分布式 DDL 的复杂性。配置的值会自动替换到您的 DDL 查询中，从而简化您的 DDL。此配置的宏指定每个节点的分片和副本数量。
在此 2 分片 1 副本示例中，chnode1 和 chnode2 上的副本宏均为replica_1 ，因为只有一个副本。 chnode1 上的分片宏为1 ，chnode2 上的分片宏为2 。

chnode1 上的 Macros.xml

root@chnode1:~# cat /etc/clickhouse-server/config.d/Macros.xml
<clickhouse>
  <macros>
    <shard>1</shard>
    <replica>replica_1</replica>
  </macros>
</clickhouse>

复制和分片配置

从顶部开始：

XML 的remote_servers部分指定环境中的每个集群。属性replace=true将默认ClickHouse配置中的示例remote_servers替换为此文件中指定的remote_servers配置。如果没有此属性，此文件中的远程服务器将被追加到默认的示例列表中。
在此示例中，有一个名为cluster_2S_1R的集群。
为名为cluster_2S_1R的集群创建一个机密，其值为mysecretphrase 。该秘密在环境中的所有远程服务器之间共享，以确保正确的服务器连接在一起。
集群cluster_2S_1R有两个分片，每个分片都有一个副本。查看本文档开头的架构图，并将其与下面 XML 中的两个shard定义进行比较。在每个分片定义中都有一个副本。该副本适用于该特定分片。该副本的主机和端口已指定。配置中第一个分片的副本存储在chnode1上，配置中第二个分片的副本存储在chnode2上。
分片的内部复制设置为 true。每个分片都可以在配置文件中定义internal_replication参数。如果该参数设置为true，则写操作会选择第一个健康的副本并向其写入数据。

chnode1 上的remote-servers.xml

root@chnode1:~# cat /etc/clickhouse-server/config.d/remote-servers.xml
<clickhouse>
  <remote_servers replace="true">
    <cluster_2S_1R>
    <secret>mysecretphrase</secret>
        <shard>
            <internal_replication>true</internal_replication>
            <replica>
                <host>chnode1</host>
                <port>9000</port>
            </replica>
        </shard>
        <shard>
            <internal_replication>true</internal_replication>
            <replica>
                <host>chnode2</host>
                <port>9000</port>
            </replica>
        </shard>
    </cluster_2S_1R>
  </remote_servers>
</clickhouse>

配置Keeper的使用

上面几个文件配置了ClickHouse Keeper。此配置文件use-keeper.xml将 ClickHouse Server 配置为使用 ClickHouse Keeper 来协调复制和分布式 DDL。该文件指定 ClickHouse Server 应在端口 9181 上的节点 chnode1 - 3 上使用 Keeper，并且该文件在chnode1和chnode2上相同。

在 chnode1 上创建enable-keeper.xml

root@chnode1:~# cat /etc/clickhouse-server/config.d/enable-keeper.xml
<clickhouse>
    <zookeeper>
        <node index="1">
            <host>chnode1</host>
            <port>9181</port>
        </node>
        <node index="2">
            <host>chnode2</host>
            <port>9181</port>
        </node>
        <node index="3">
            <host>chnode3</host>
            <port>9181</port>
        </node>
    </zookeeper>
</clickhouse>

chnode2配置

由于chnode1和chnode2上的配置非常相似，因此这里仅指出差异。

网络和日志记录配置

chnode2 上的network-and-logging.xml

root@chnode2:~# cat /etc/clickhouse-server/config.d/network-and-logging.xml
<clickhouse>
        <logger>
                <level>debug</level>
                <log>/var/log/clickhouse-server/clickhouse-server.log</log>
                <errorlog>/var/log/clickhouse-server/clickhouse-server.err.log</errorlog>
                <size>1000M</size>
                <count>3</count>
        </logger>
        <display_name>clickhouse</display_name>
        <listen_host>0.0.0.0</listen_host>
        <http_port>8123</http_port>
        <tcp_port>9000</tcp_port>
        <interserver_http_port>9009</interserver_http_port>
        <interserver_listen_host>0.0.0.0</interserver_listen_host>
</clickhouse>

ClickHouse Keeper配置

该文件包含 chnode1 和 chnode2 之间的两个差异之一。在 Keeper 配置中， server_id设置为2 。

在 chnode2 上创建enable-keeper.xml

root@chnode2:~# cat /etc/clickhouse-server/config.d/enable-keeper.xml
<clickhouse>
  <keeper_server>
    <tcp_port>9181</tcp_port>
    <server_id>2</server_id>
    <log_storage_path>/var/lib/clickhouse/coordination/log</log_storage_path>
    <snapshot_storage_path>/var/lib/clickhouse/coordination/snapshots</snapshot_storage_path>

    <coordination_settings>
        <operation_timeout_ms>10000</operation_timeout_ms>
        <session_timeout_ms>30000</session_timeout_ms>
        <raft_logs_level>trace</raft_logs_level>
    </coordination_settings>

    <raft_configuration>
        <server>
            <id>1</id>
            <hostname>chnode1</hostname>
            <port>9234</port>
        </server>
        <server>
            <id>2</id>
            <hostname>chnode2</hostname>
            <port>9234</port>
        </server>
        <server>
            <id>3</id>
            <hostname>chnode3</hostname>
            <port>9234</port>
        </server>
    </raft_configuration>
  </keeper_server>
</clickhouse>

宏配置

宏配置在 chnode1 和 chnode2 之间存在差异之一。该节点上的shard设置为2 。

chnode2 上的 Macros.xml

root@chnode2:~# cat /etc/clickhouse-server/config.d/Macros.xml
<clickhouse>
<macros>
    <shard>2</shard>
    <replica>replica_1</replica>
</macros>
</clickhouse>

复制和分片配置

chnode2 上的remote-servers.xml

root@chnode2:~# cat /etc/clickhouse-server/config.d/remote-servers.xml
<clickhouse>
  <remote_servers replace="true">
    <cluster_2S_1R>
    <secret>mysecretphrase</secret>
        <shard>
            <internal_replication>true</internal_replication>
            <replica>
                <host>chnode1</host>
                <port>9000</port>
            </replica>
        </shard>
            <shard>
            <internal_replication>true</internal_replication>
            <replica>
                <host>chnode2</host>
                <port>9000</port>
            </replica>
        </shard>
    </cluster_2S_1R>
  </remote_servers>
</clickhouse>

配置Keeper的使用

在 chnode2 上配置use-keeper.xml

root@chnode2:~# cat /etc/clickhouse-server/config.d/use-keeper.xml
<clickhouse>
    <zookeeper>
        <node index="1">
            <host>chnode1</host>
            <port>9181</port>
        </node>
        <node index="2">
            <host>chnode2</host>
            <port>9181</port>
        </node>
        <node index="3">
            <host>chnode3</host>
            <port>9181</port>
        </node>
    </zookeeper>
</clickhouse>

chnode3配置

由于 chnode3 不存储数据，仅用于 ClickHouse Keeper 提供仲裁中的第三个节点，因此 chnode3 只有两个配置文件，一个用于配置网络和日志记录，另一个用于配置 ClickHouse Keeper。

ClickHouse Keeper配置

在 chnode3 上创建enable-keeper.xml

root@chnode3:~# cat /etc/clickhouse-keeper/config.d/enable-keeper.xml
<clickhouse>
  <keeper_server>
    <tcp_port>9181</tcp_port>
    <server_id>3</server_id>
    <log_storage_path>/var/lib/clickhouse/coordination/log</log_storage_path>
    <snapshot_storage_path>/var/lib/clickhouse/coordination/snapshots</snapshot_storage_path>

    <coordination_settings>
        <operation_timeout_ms>10000</operation_timeout_ms>
        <session_timeout_ms>30000</session_timeout_ms>
        <raft_logs_level>trace</raft_logs_level>
    </coordination_settings>

    <raft_configuration>
        <server>
            <id>1</id>
            <hostname>chnode1</hostname>
            <port>9234</port>
        </server>
        <server>
            <id>2</id>
            <hostname>chnode2</hostname>
            <port>9234</port>
        </server>
        <server>
            <id>3</id>
            <hostname>chnode3</hostname>
            <port>9234</port>
        </server>
    </raft_configuration>
  </keeper_server>
  <listen_host>0.0.0.0</listen_host>
  <interserver_listen_host>0.0.0.0</interserver_listen_host>
</clickhouse>

启动服务

检查配置文件

最后运行并检查整个配置目录结构，除上述提到的配置，其他都为默认配置文件。
chnode1配置：

root@chnode1:~# apt install -y tree
root@chnode1:~# tree /etc/clickhouse-server/
/etc/clickhouse-server/
├── config.d
│   ├── enable-keeper.xml
│   ├── Macros.xml
│   ├── network-and-logging.xml
│   ├── remote-servers.xml
│   └── use-keeper.xml
├── config.xml
├── users.d
│   └── default-password.xml
└── users.xml

2 directories, 8 files

chnode2配置：

root@chnode2:~# apt install -y tree
root@chnode2:~# tree /etc/clickhouse-server/
/etc/clickhouse-server/
├── config.d
│   ├── enable-keeper.xml
│   ├── Macros.xml
│   ├── network-and-logging.xml
│   ├── remote-servers.xml
│   └── use-keeper.xml
├── config.xml
├── users.d
│   └── default-password.xml
└── users.xml

2 directories, 8 files

chnode3配置：

root@chnode3:~# apt install -y tree
root@chnode3:~# tree /etc/clickhouse-keeper/
/etc/clickhouse-keeper/
├── config.d
│   └── enable-keeper.xml
├── keeper_config-preprocessed.xml
└── keeper_config.xml

1 directory, 3 files

在chnode1和chnode2中启动服务

仅需启动clickhouse-server服务

systemctl enable --now clickhouse-server.service

查看chnode1 clickhouse-server服务运行状态

root@chnode1:~# systemctl status clickhouse-server.service 
● clickhouse-server.service - ClickHouse Server (analytic DBMS for big data)
     Loaded: loaded (/lib/systemd/system/clickhouse-server.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2024-10-25 23:49:59 CST; 9h ago
   Main PID: 2267 (clickhouse-serv)
      Tasks: 730 (limit: 4556)
     Memory: 1.1G
        CPU: 2h 37min 58.467s
     CGroup: /system.slice/clickhouse-server.service
             ├─2264 clickhouse-watchdog "" "" "" "" "" "" "" --config=/etc/clickhouse-server/config.xml --pid-file=/run/clickhouse-server/clickhouse-server.pid
             └─2267 /usr/bin/clickhouse-server --config=/etc/clickhouse-server/config.xml --pid-file=/run/clickhouse-server/clickhouse-server.pid

Oct 25 23:49:40 chnode1 clickhouse-server[2264]: Processing configuration file '/etc/clickhouse-server/config.xml'.
Oct 25 23:49:40 chnode1 clickhouse-server[2264]: Merging configuration file '/etc/clickhouse-server/config.d/Macros.xml'.
Oct 25 23:49:40 chnode1 clickhouse-server[2264]: Merging configuration file '/etc/clickhouse-server/config.d/enable-keeper.xml'.
Oct 25 23:49:40 chnode1 clickhouse-server[2264]: Merging configuration file '/etc/clickhouse-server/config.d/network-and-logging.xml'.
Oct 25 23:49:40 chnode1 clickhouse-server[2264]: Merging configuration file '/etc/clickhouse-server/config.d/remote-servers.xml'.
Oct 25 23:49:40 chnode1 clickhouse-server[2264]: Merging configuration file '/etc/clickhouse-server/config.d/use-keeper.xml'.
Oct 25 23:49:40 chnode1 clickhouse-server[2264]: Logging debug to /var/log/clickhouse-server/clickhouse-server.log
Oct 25 23:49:40 chnode1 clickhouse-server[2264]: Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
Oct 25 23:49:40 chnode1 systemd[1]: clickhouse-server.service: Supervising process 2267 which is not our child. We'll most likely not notice when it exits.
Oct 25 23:49:59 chnode1 systemd[1]: Started ClickHouse Server (analytic DBMS for big data).
root@chnode1:~#

查看chnode2 clickhouse-server服务运行状态

root@chnode2:~# systemctl status clickhouse-server.service
● clickhouse-server.service - ClickHouse Server (analytic DBMS for big data)
     Loaded: loaded (/lib/systemd/system/clickhouse-server.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2024-10-25 23:49:59 CST; 9h ago
   Main PID: 2149 (clickhouse-serv)
      Tasks: 733 (limit: 4556)
     Memory: 902.9M
        CPU: 2h 31min 49.440s
     CGroup: /system.slice/clickhouse-server.service
             ├─2146 clickhouse-watchdog "" "" "" "" "" "" "" --config=/etc/clickhouse-server/config.xml --pid-file=/run/clickhouse-server/clickhouse-server.pid
             └─2149 /usr/bin/clickhouse-server --config=/etc/clickhouse-server/config.xml --pid-file=/run/clickhouse-server/clickhouse-server.pid

Oct 25 23:49:56 chnode2 clickhouse-server[2146]: Processing configuration file '/etc/clickhouse-server/config.xml'.
Oct 25 23:49:56 chnode2 clickhouse-server[2146]: Merging configuration file '/etc/clickhouse-server/config.d/Macros.xml'.
Oct 25 23:49:56 chnode2 clickhouse-server[2146]: Merging configuration file '/etc/clickhouse-server/config.d/enable-keeper.xml'.
Oct 25 23:49:56 chnode2 clickhouse-server[2146]: Merging configuration file '/etc/clickhouse-server/config.d/network-and-logging.xml'.
Oct 25 23:49:56 chnode2 clickhouse-server[2146]: Merging configuration file '/etc/clickhouse-server/config.d/remote-servers.xml'.
Oct 25 23:49:56 chnode2 clickhouse-server[2146]: Merging configuration file '/etc/clickhouse-server/config.d/use-keeper.xml'.
Oct 25 23:49:56 chnode2 clickhouse-server[2146]: Logging debug to /var/log/clickhouse-server/clickhouse-server.log
Oct 25 23:49:56 chnode2 clickhouse-server[2146]: Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
Oct 25 23:49:56 chnode2 systemd[1]: clickhouse-server.service: Supervising process 2149 which is not our child. We'll most likely not notice when it exits.
Oct 25 23:49:59 chnode2 systemd[1]: Started ClickHouse Server (analytic DBMS for big data).
root@chnode2:~#

在chnode3中启动服务
仅需启动clickhouse-keeper服务

systemctl enable --now clickhouse-keeper.service

查看clickhouse-keeper服务运行状态

root@chnode3:~# systemctl status clickhouse-keeper.service 
● clickhouse-keeper.service - ClickHouse Keeper - zookeeper compatible distributed coordination server
     Loaded: loaded (/lib/systemd/system/clickhouse-keeper.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2024-10-25 23:55:58 CST; 9h ago
   Main PID: 3694 (clickhouse-keep)
      Tasks: 41 (limit: 4556)
     Memory: 70.6M
        CPU: 3min 6.234s
     CGroup: /system.slice/clickhouse-keeper.service
             └─3694 /usr/bin/clickhouse-keeper --config=/etc/clickhouse-keeper/keeper_config.xml --pid-file=/run/clickhouse-keeper/clickhouse-keeper.pid

Oct 25 23:55:58 chnode3 systemd[1]: Started ClickHouse Keeper - zookeeper compatible distributed coordination server.
Oct 25 23:55:58 chnode3 clickhouse-keeper[3694]: Processing configuration file '/etc/clickhouse-keeper/keeper_config.xml'.
Oct 25 23:55:58 chnode3 clickhouse-keeper[3694]: Logging trace to /var/log/clickhouse-keeper/clickhouse-keeper.log
Oct 25 23:55:58 chnode3 clickhouse-keeper[3694]: Logging errors to /var/log/clickhouse-keeper/clickhouse-keeper.err.log

我们来看看我们的Click Keeper服务，谁是leader？谁是follower？

对于 Node1；（追随者）

root@chnode1:~# echo mntr | nc localhost 9181
zk_version      v24.9.2.42-stable-de7c791a2eadce4093409574d6560d2332b0dd18
zk_avg_latency  0
zk_max_latency  0
zk_min_latency  0
zk_packets_received     0
zk_packets_sent 0
zk_num_alive_connections        0
zk_outstanding_requests 0
zk_server_state follower
zk_znode_count  21
zk_watch_count  0
zk_ephemerals_count     0
zk_approximate_data_size        4314
zk_key_arena_size       0
zk_latest_snapshot_size 0
zk_open_file_descriptor_count   64
zk_max_file_descriptor_count    500000

对于节点 2；如下所示，处于leader地位

root@chnode2:~# echo mntr | nc localhost 9181
zk_version      v24.9.2.42-stable-de7c791a2eadce4093409574d6560d2332b0dd18
zk_avg_latency  4
zk_max_latency  70
zk_min_latency  0
zk_packets_received     98
zk_packets_sent 102
zk_num_alive_connections        2
zk_outstanding_requests 0
zk_server_state leader
zk_znode_count  21
zk_watch_count  2
zk_ephemerals_count     0
zk_approximate_data_size        4314
zk_key_arena_size       0
zk_latest_snapshot_size 0
zk_open_file_descriptor_count   69
zk_max_file_descriptor_count    500000
zk_followers    2
zk_synced_followers     2

对于节点 3；追随者

root@chnode3:~# echo mntr | nc localhost 9181
zk_version      v24.9.2.42-stable-de7c791a2eadce4093409574d6560d2332b0dd18
zk_avg_latency  0
zk_max_latency  0
zk_min_latency  0
zk_packets_received     0
zk_packets_sent 0
zk_num_alive_connections        0
zk_outstanding_requests 0
zk_server_state standalone
zk_znode_count  4
zk_watch_count  0
zk_ephemerals_count     0
zk_approximate_data_size        578
zk_key_arena_size       0
zk_latest_snapshot_size 100
zk_open_file_descriptor_count   36
zk_max_file_descriptor_count    500000
zk_followers    0
zk_synced_followers     0

通过clickhouse-client客户端连接到clickhouse，密码为安装clickhouse时设置的密码

root@chnode1:~# clickhouse-client 
ClickHouse client version 24.9.2.42 (official build).
Connecting to localhost:9000 as user default.
Password for user (default): 
Connecting to localhost:9000 as user default.
Connected to ClickHouse server version 24.9.2.

Warnings:
 * Delay accounting is not enabled, OSIOWaitMicroseconds will not be gathered. You can enable it using `echo 1 > /proc/sys/kernel/task_delayacct` or by using sysctl.
 * Maximum number of threads is lower than 30000. There could be problems with handling a lot of simultaneous queries.

clickhouse :)

查看实例及副本，示例输出如下：

clickhouse :) SELECT
    host_name,
    host_address,
    replica_num
FROM system.clusters

SELECT
    host_name,
    host_address,
    replica_num
FROM system.clusters

Query id: 3c4fde97-583b-4819-ba47-e368c0e9dd8b

   ┌─host_name─┬─host_address──┬─replica_num─┐
1. │ chnode1   │ 192.168.72.51 │           1 │
2. │ chnode2   │ 192.168.72.52 │           1 │
   └───────────┴───────────────┴─────────────┘

2 rows in set. Elapsed: 0.002 sec.

测试集群

1、连接chnode1 ，验证上面配置的集群cluster_2S_1R是否存在

clickhouse :) SHOW CLUSTERS

SHOW CLUSTERS

Query id: 40f7932e-4ff8-4101-aa6b-d4378c22e744

   ┌─cluster───────┐
1. │ cluster_2S_1R │
   └───────────────┘

1 row in set. Elapsed: 0.003 sec. 

clickhouse :)

2、在集群上创建数据库

CREATE DATABASE db1 ON CLUSTER cluster_2S_1R

┌─host────┬─port─┬─status─┬─error─┬─num_hosts_remaining─┬─num_hosts_active─┐
│ chnode2 │ 9000 │      0 │       │                   1 │                0 │
│ chnode1 │ 9000 │      0 │       │                   0 │                0 │
└─────────┴──────┴────────┴───────┴─────────────────────┴──────────────────┘

3、在集群上使用MergeTree表引擎创建表。

笔记
我们不需要在表引擎上指定参数，因为这些参数将根据我们的宏自动定义

CREATE TABLE db1.table1 ON CLUSTER cluster_2S_1R
(
    `id` UInt64,
    `column1` String
)
ENGINE = MergeTree
ORDER BY id

┌─host────┬─port─┬─status─┬─error─┬─num_hosts_remaining─┬─num_hosts_active─┐
│ chnode1 │ 9000 │      0 │       │                   1 │                0 │
│ chnode2 │ 9000 │      0 │       │                   0 │                0 │
└─────────┴──────┴────────┴───────┴─────────────────────┴──────────────────┘

4、连接到chnode1并插入一行

INSERT INTO db1.table1 (id, column1) VALUES (1, 'abc');

5、连接到chnode2并插入一行

INSERT INTO db1.table1 (id, column1) VALUES (2, 'def');

6、连接到任一节点chnode1或chnode2 ，您将仅看到插入到该节点上的表中的行。例如，在chnode2上

SELECT * FROM db1.table1;

┌─id─┬─column1─┐
│  2 │ def     │
└────┴─────────┘

7、创建一个分布式表来查询两个节点上的两个分片。（在本例中， rand()函数被设置为分片键，以便它随机分布每个插入）

CREATE TABLE db1.table1_dist ON CLUSTER cluster_2S_1R
(
    `id` UInt64,
    `column1` String
)
ENGINE = Distributed('cluster_2S_1R', 'db1', 'table1', rand())

┌─host────┬─port─┬─status─┬─error─┬─num_hosts_remaining─┬─num_hosts_active─┐
│ chnode2 │ 9000 │      0 │       │                   1 │                0 │
│ chnode1 │ 9000 │      0 │       │                   0 │                0 │
└─────────┴──────┴────────┴───────┴─────────────────────┴──────────────────┘

8、连接到chnode1或chnode2并查询分布式表以查看这两行。

SELECT * FROM db1.table1_dist;

┌─id─┬─column1─┐
│  2 │ def     │
└────┴─────────┘
┌─id─┬─column1─┐
│  1 │ abc     │
└────┴─────────┘