Flink 高可用之StandAlone-HA模式

压缩包:

tar -xvzf flink-1.9.1-bin-scala_2.11.tgz -C /opt && cd /opt/flink-1.9.1

集群规划:

  1.集群规划

• - 服务器: node1(Master + Slave): JobManager + TaskManager
• - 服务器: node2(Master + Slave): JobManager + TaskManager
• - 服务器: node3(Slave): TaskManager

1. 启动ZooKeeper

     zkServer.sh status

      zkServer.sh stop

      zkServer.sh start

1. 启动HDFS

erves/hadoop/sbin/start-dfs.sh

1. 、修改flink-conf.yaml

jobmanager.rpc.address: bigdata1

jobmanager.rpc.port: 6123

jobmanager.heap.size: 2048m

taskmanager.heap.size: 4096m

taskmanager.numberOfTaskSlots: 4

parallelism.default: 1

#开启HA，使用文件系统作为快照存储

state.backend: filesystem

#启用检查点，可以将快照保存到HDFS

state.checkpoints.dir:hdfs://server2:8020/flink-checkpoints

#使用zookeeper搭建高可用

high-availability: zookeeper

#存储JobManager的元数据到HDFS

high-availability.storageDir: hdfs://server2:8020/flink/ha/

#配置ZK集群地址

high-availability.zookeeper.quorum: server1:2118,server2:2118,server3:2118

2）、修改masters

vim /usr/local/flink-1.13.5/conf/masters

server1:8081

server2:8081

3)、修改works文件

server1

server2

4)、分发(scp分发)

scp -r /usr/local/flink-1.13.5/conf/flink-conf.yaml server2:/usr/local/flink-1.13.5/conf/

scp -r /usr/local/flink-1.13.5/conf/flink-conf.yaml server3:/usr/local/flink-1.13.5/conf/

scp -r /usr/local/flink-1.13.5/conf/flink-conf.yaml server4:/usr/local/flink-1.13.5/conf/

scp -r /usr/local/flink-1.13.5/conf/masters server2:/usr/local/flink-1.13.5/conf/

scp -r /usr/local/flink-1.13.5/conf/masters server3:/usr/local/flink-1.13.5/conf/

scp -r /usr/local/flink-1.13.5/conf/masters server4:/usr/local/flink-1.13.5/conf/

5)、修改修改node2上的flink-conf.yaml

        Jobmanager.rpc.address: node2

1. 启动集群

   /bin/start-cluster.sh

  停止集群

   /bin/stop-cluster.sh

1. 查看进程     

 cat /export/server/flink/log/flink-root-standalonesession-0-node1.log

分发包:

for i in {2..3}; do scp -r flink-shaded-hadoop-2-uber-2.7.5-10.0.jar node$i:$PWD; done

注意:修改域名

vi hostname 

cat  hostname 

改完以后,添加到hosts中:

改完之后将每台服务器的对应信息添加到hosts文件

vim /etc/hosts

# 添加以下内容到文件中

192.168.188.135 node1-zookeeper

192.168.188.136 node2-zookeeper

192.168.188.137 node3-zookeeper

测试联通性: 比如: ping  node3-zookeeper