storm统计服务开启zookeeper、kafka 、Storm（sasl认证）

部署storm统计服务开启zookeeper、kafka 、Storm（sasl认证）

  • 当前测试验证结果：
    • 单独配置zookeeper 支持acl 设置用户和密码，在storm不修改代码情况下和kafka支持
    • 当kafka 开启ACL时，storm 和ccod模块不清楚配置用户和密码。
      • 使用python脚本连接kafka用户和密码是能成功发送消息。
  • 当前部署环境服务版本
服务IP模块版本信息
10.130.41.42zookeeperzookeeper-3.6.3
10.130.41.43kafkakafka_2.11-2.3.1
10.130.41.44stormapache-storm-1.2.4
zookeeper部署
  • 部署mongodb_1服务器的zookeeper
  • 传安装部署包和配置文件
[root@mongodb_1 ~]# su - storm
[storm@mongodb_1 ~]$ rz -be  ====>  mongodb_1 ~]$ rz -be  storm_node1.tar.gz
[storm@mongodb_1 ~]$ tar xvf storm_node1.tar.gz 
[storm@mongodb_1 ~]$ cd  storm_node1
[storm@mongodb_1 storm_node1]$ mv * .bash_profile  ../
[storm@mongodb_1 ~]$ source .bash_profile ;java -version 
java version "1.8.0_91"
Java(TM) SE Runtime Environment (build 1.8.0_91-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)
  • 修改zookeeper配置文件,然后在启动
[storm@mongodb_1 ~]$ cd zookeeper-3.6.3/conf/
[storm@mongodb_1 conf]$ vim zoo.cfg 
dataDir=/home/storm/zookeeper-3.6.3/data
dataLogDir=/home/storm/zookeeper-3.6.3/log
server.1:mongodb_1:2182:3181
server.2:mongodb_2:2182:3181
server.3:mongodb_3:2182:3181
#peerType=observer
autopurge.purgeInterval=1
autopurge.snapRetainCount=3
4lw.commands.whitelist=*
jaasLoginRenew=3600000
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
authProvider.2=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
authProvider.3=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
requireClientAuthScheme=sasl
[storm@mongodb_1 conf]$ vim  zk_server_jaas.conf
Server {
    org.apache.kafka.common.security.plain.PlainLoginModule required
    username="zk_cluster"
    password="zk_cluster_passwd"
    user_admin="Admin@123";
};

#拷贝kafka acl lib
[storm@mongodb_1 zookeeper-3.6.3]$  cd /home/storm/zookeeper-3.6.3
[storm@mongodb_1 zookeeper-3.6.3]$  mkdir zk_sasl_dependencies
[storm@mongodb_1 libs]$ cd /home/storm/kafka_2.11-2.3.1/libs
[storm@mongodb_1 libs]$ cp kafka-clients-2.3.1.jar  lz4-java-1.6.0.jar  slf4j-api-1.7.26.jar  snappy-java-1.1.7.3.jar  ~/zookeeper-3.6.3/lib/
[storm@mongodb_1 ~]$ mkdir -p /home/storm/zookeeper-3.6.3/log /home/storm/zookeeper-3.6.3/data
[storm@mongodb_1 ~]$ echo "1" >  /home/storm/zookeeper-3.6.3/data/myid 
[storm@mongodb_1 ~]$ cd ~/zookeeper-3.6.3/bin
[storm@mongodb_1 bin]$ vim zkEnv.sh  #最下面添加即可
#为zookeeper添加SASL支持
for i in ~/zookeeper-3.6.3/zk_sasl_dependencies/*.jar;
do
    CLASSPATH="$i:$CLASSPATH"
done
SERVER_JVMFLAGS=" -Djava.security.auth.login.config=$HOME/zookeeper-3.6.3/conf/zk_server_jaas.conf"
[storm@mongodb_1 bin]$ ./zkServer.sh  start 
ZooKeeper JMX enabled by default
Using config: /home/storm/zookeeper-3.6.3/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

#需要登录三台zookepper启动完成之后，在查看状态
[storm@mongodb_1 bin]$ ./zkServer.sh  status 
ZooKeeper JMX enabled by default
Using config: /home/storm/zookeeper-3.6.3/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: follower
  • 部署mongodb_2服务器的zookeeper
  • 上传安装部署包和配置文件
[root@mongodb_2 ~]# su - storm
[storm@mongodb_2 ~]$ rz -be  ====>  mongodb_1 ~]$ rz -be storm_node2.tar.gz
[storm@mongodb_2 ~]$ tar xvf storm_node2.tar.gz 
[storm@mongodb_2 ~]$ cd  storm_node2
[storm@mongodb_2 storm_node1]$ mv * .bash_profile  ../
[storm@mongodb_2 ~]$ source .bash_profile ;java -version 
java version "1.8.0_91"
Java(TM) SE Runtime Environment (build 1.8.0_91-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)
  • 修改zookeeper配置文件,然后在启动
[storm@mongodb_2 ~]$ cd zookeeper-3.6.3/conf/
[storm@mongodb_2 conf]$ vim zoo.cfg 
dataDir=/home/storm/zookeeper-3.6.3/data
dataLogDir=/home/storm/zookeeper-3.6.3/log
server.1:mongodb_2:2182:3181
server.2:mongodb_2:2182:3181
server.3:mongodb_3:2182:3181
#peerType=observer
autopurge.purgeInterval=1
autopurge.snapRetainCount=3
4lw.commands.whitelist=*
jaasLoginRenew=3600000
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
authProvider.2=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
authProvider.3=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
requireClientAuthScheme=sasl
[storm@mongodb_2 conf]$ vim  zk_server_jaas.conf
Server {
    org.apache.kafka.common.security.plain.PlainLoginModule required
    username="zk_cluster"
    password="zk_cluster_passwd"
    user_admin="Admin@123";
};

#拷贝kafka acl lib
[storm@mongodb_2 zookeeper-3.6.3]$  cd /home/storm/zookeeper-3.6.3
[storm@mongodb_2 zookeeper-3.6.3]$  mkdir zk_sasl_dependencies
[storm@mongodb_2 libs]$ cd /home/storm/kafka_2.11-2.3.1/libs
[storm@mongodb_2 libs]$ cp kafka-clients-2.3.1.jar  lz4-java-1.6.0.jar  slf4j-api-1.7.26.jar  snappy-java-1.1.7.3.jar  ~/zookeeper-3.6.3/lib/
[storm@mongodb_2 ~]$ mkdir -p /home/storm/zookeeper-3.6.3/log /home/storm/zookeeper-3.6.3/data
[storm@mongodb_2 ~]$ echo "1" >  /home/storm/zookeeper-3.6.3/data/myid 
[storm@mongodb_2 ~]$ cd ~/zookeeper-3.6.3/bin
[storm@mongodb_2 bin]$ vim zkEnv.sh  #最下面添加即可
#为zookeeper添加SASL支持
for i in ~/zookeeper-3.6.3/zk_sasl_dependencies/*.jar;
do
    CLASSPATH="$i:$CLASSPATH"
done
SERVER_JVMFLAGS=" -Djava.security.auth.login.config=$HOME/zookeeper-3.6.3/conf/zk_server_jaas.conf"
[storm@mongodb_2 bin]$ ./zkServer.sh  start 
ZooKeeper JMX enabled by default
Using config: /home/storm/zookeeper-3.6.3/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

#需要登录三台zookepper启动完成之后，在查看状态
[storm@mongodb_2 bin]$ ./zkServer.sh  status 
ZooKeeper JMX enabled by default
Using config: /home/storm/zookeeper-3.6.3/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: leader
  • 部署mongodb_3服务器的zookeeper
  • 上传安装部署包和配置文件
[root@mongodb_3 ~]# su - storm
[storm@mongodb_3 ~]$ rz -be  ====>  mongodb_1 ~]$ rz -be storm_node3.tar.gz
[storm@mongodb_3 ~]$ tar xvf storm_node3.tar.gz 
[storm@mongodb_3 ~]$ cd  storm_node3
[storm@mongodb_3 storm_node1]$ mv * .bash_profile  ../
[storm@mongodb_3 ~]$ source .bash_profile ;java -version 
java version "1.8.0_91"
Java(TM) SE Runtime Environment (build 1.8.0_91-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)
  • 修改zookeeper配置文件,然后在启动
[storm@mongodb_3 ~]$ cd zookeeper-3.6.3/conf/
[storm@mongodb_3 conf]$ vim zoo.cfg 
dataDir=/home/storm/zookeeper-3.6.3/data
dataLogDir=/home/storm/zookeeper-3.6.3/log
server.1:mongodb_3:2182:3181
server.2:mongodb_3:2182:3181
server.3:mongodb_3:2182:3181
#peerType=observer
autopurge.purgeInterval=1
autopurge.snapRetainCount=3
4lw.commands.whitelist=*
jaasLoginRenew=3600000
authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
authProvider.2=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
authProvider.3=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
requireClientAuthScheme=sasl
[storm@mongodb_3 conf]$ vim  zk_server_jaas.conf
Server {
    org.apache.zookeeper.server.auth.DigestLoginModule required
    admin="Admin@123";
};

#拷贝kafka acl lib
[storm@mongodb_3 zookeeper-3.6.3]$  cd /home/storm/zookeeper-3.6.3
[storm@mongodb_3 zookeeper-3.6.3]$  mkdir zk_sasl_dependencies
[storm@mongodb_3 libs]$ cd /home/storm/kafka_2.11-2.3.1/libs
[storm@mongodb_3 libs]$ cp kafka-clients-2.3.1.jar  lz4-java-1.6.0.jar  slf4j-api-1.7.26.jar  snappy-java-1.1.7.3.jar  ~/zookeeper-3.6.3/lib/
[storm@mongodb_3 ~]$ mkdir -p /home/storm/zookeeper-3.6.3/log /home/storm/zookeeper-3.6.3/data
[storm@mongodb_3 ~]$ echo "1" >  /home/storm/zookeeper-3.6.3/data/myid 
[storm@mongodb_3 ~]$ cd ~/zookeeper-3.6.3/bin
[storm@mongodb_3 bin]$ vim zkEnv.sh  #最下面添加即可
#为zookeeper添加SASL支持
for i in ~/zookeeper-3.6.3/zk_sasl_dependencies/*.jar;
do
    CLASSPATH="$i:$CLASSPATH"
done
SERVER_JVMFLAGS=" -Djava.security.auth.login.config=$HOME/zookeeper-3.6.3/conf/zk_server_jaas.conf"
[storm@mongodb_3 bin]$ ./zkServer.sh  start 
ZooKeeper JMX enabled by default
Using config: /home/storm/zookeeper-3.6.3/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

#需要登录三台zookepper启动完成之后，在查看状态
[storm@mongodb_3 bin]$ ./zkServer.sh  status 
ZooKeeper JMX enabled by default
Using config: /home/storm/zookeeper-3.6.3/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost. Client SSL: false.
Mode: follower
  • zookeeper设置用户密码，并设置连接zookeeper白名单。（如已经开启sals认证，则选acl认证跳过）
    • 当前新部署平台，zookeeper未做下面操作addauth
[storm@mongodb_2 bin]$ ./zkCli.sh   -server mongodb_2:2182,mongodb_1:2182,mongodb_3:2182

[zk: localhost:2181(CONNECTED) 0] addauth digest admin:123456  #admin为用户名, 123456为密码。
[zk: localhost:2181(CONNECTED) 1] setAcl / auth:admin:123456:cdrwa  
[zk: localhost:2181(CONNECTED) 2] getAcl /
'digest,'admin:0uek/hZ/V9fgiM35b0Z2226acMQ=
: cdrwa

[storm@mongodb_2 bin]$ ./zkCli.sh  #登录连接成功后每次连接需要输入   addauth digest admin:123456
[zk: localhost:2181(CONNECTED) 0] ls /
Insufficient permission : /
[zk: localhost:2181(CONNECTED) 1]  addauth digest admin:123456
[zk: localhost:2181(CONNECTED) 2] ls /
[test, zookeeper]

[zk: localhost:2181(CONNECTED) 8] setAcl / ip:10.130.41.42:cdrwa,ip:10.130.41.43:cdrwa,ip:10.130.41.44:cdrwa,ip:10.130.41.86:cdrwa,ip:10.130.41.119:cdrwa,ip:127.0.0.1:cdrwa,auth:kafka:cdrwa  #设置连接zookeeper白名单
初始化storm配置
  • 需要三个随便找一个zookeeper进行登录
[storm@mongodb_1 ~]$ cd ~/zookeeper-3.6.3/bin
[storm@mongodb_1 bin]$ ./zkCli.sh 
[zk: localhost:2181(CONNECTED) 2]   addauth digest admin:123456
create  /storm config
create  /storm/config 1
create /umg_cloud cluster
create /umg_cloud/cluster 1
create /umg_cloud/cfg  1 
create /umg_cloud/cfg/notify  1
create  /storm/config/ENT_CALL_RECORD_LIST 0103290030,0103290035,0103290023,0103290021,0103290022,0103290031
create  /storm/config/KAFKA_BROKER_LIST mongodb_1:9092,mongodb_2:9092,mongodb_3:9092
create  /storm/config/KAFKA_ZOOKEEPER_CONNECT mongodb_1:2181,mongodb_2:2181,mongodb_3:2181
create  /storm/config/MAX_SPOUT_PENDING 100
create  /storm/config/MONGO_ADMIN admin
create  /storm/config/MONGO_BLOCKSIZE 100
create  /storm/config/MONGO_HOST mongodb_1:30000;mongodb_2:30000;mongodb_3:30000
create  /storm/config/MONGO_NAME ""
create  /storm/config/MONGO_PASSWORD ""
create  /storm/config/MONGO_POOLSIZE 100
create  /storm/config/MONGO_SWITCH 0
create  /storm/config/OFF_ON_DTS 0
create  /storm/config/REDIS_HOST mongodb_3:27381
create  /storm/config/REDIS_PASSWORD  ""
create  /storm/config/SEND_MESSAGE 0
create  /storm/config/TOPO_COMPUBOLT_NUM 4
create  /storm/config/TOPO_MONGOBOLT_NUM 16
create  /storm/config/TOPO_SPOUT_NUM 3
create  /storm/config/TOPO_WORK_NUM 2
create  /storm/config/ENTID_MESSAGE  1
create /storm/config/MESSAGE_TOPIC  1
kafka部署
  • 部署mongodb_1服务器的kafka
时[storm@mongodb_1 ~]$ cd ~/kafka_2.11-2.3.1/config/
[storm@mongodb_1 config]$ cat server.properties
broker.id=1
listeners=SASL_PLAINTEXT://10.130.41.42:9092
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.enabled.mechanisms=PLAIN
security.inter.broker.protocol=SASL_PLAINTEXT
allow.everyone.if.no.acl.found=false
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
super.users=User:admin;User:kafkacluster
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/home/storm//kafka_2.11-2.3.1/logs
num.partitions=3
default.replication.factor=3
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=mongodb_1:2181,mongodb_2:2181,mongodb_3:2181
compression.type=producer
zookeeper.set.acl=true
zookeeper.defaultAcl=root;123456:rwcd
zookeeper.connection.timeout.ms=6000
zookeeper.digest.enabled=true
group.initial.rebalance.delay.ms=0

[storm@mongodb_1 config]$ more  sasl.conf
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="Admin@123";

[storm@mongodb_1 config]$ cat kafka_server_jaas.conf
KafkaServer {
 org.apache.kafka.common.security.plain.PlainLoginModule required
 username="kafkacluster"
 password="Admin@123"
 user_kafkacluster="Admin@123"
 user_admin="Admin@123"
};
 
Client{
 org.apache.kafka.common.security.plain.PlainLoginModule required  
 username="admin"  
 password="Admin@123";  
};

[storm@mongodb_1 ~]$mkdir /home/storm/kafka_2.11-2.3.1/log -p 
[storm@mongodb_1 ~]$cd  /home/storm/kafka_2.11-2.3.1/bin
[storm@mongodb_1 bin]$ vim   kafka-server-start.sh #把创建的文件 kafka_server_jaas.conf 加载到启动环境中
export KAFKA_HEAP_OPTS=$KAFKA_HEAP_OPTS" -Djava.security.auth.login.config=/home/storm/kafka_2.11-2.3.1/config/kafka_server_jaas.conf"

[storm@mongodb_1 ~]$ ./kafka-server-start.sh -daemon config/server.properties
  • 部署mongodb_2服务器的kafka
[storm@mongodb_2 ~]$ cd ~/kafka_2.11-2.3.1/config/
[storm@mongodb_2 config]$ cat server.properties
broker.id=1
listeners=SASL_PLAINTEXT://10.130.41.42:9092
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.enabled.mechanisms=PLAIN
security.inter.broker.protocol=SASL_PLAINTEXT
allow.everyone.if.no.acl.found=false
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
super.users=User:admin;User:kafkacluster
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/home/storm//kafka_2.11-2.3.1/logs
num.partitions=3
default.replication.factor=3
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=mongodb_2:2181,mongodb_2:2181,mongodb_3:2181
compression.type=producer
zookeeper.set.acl=true
zookeeper.defaultAcl=root;123456:rwcd
zookeeper.connection.timeout.ms=6000
zookeeper.digest.enabled=true
group.initial.rebalance.delay.ms=0

[storm@mongodb_2 config]$ more  sasl.conf
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="Admin@123";

[storm@mongodb_2 config]$ cat kafka_server_jaas.conf
KafkaServer {
 org.apache.kafka.common.security.plain.PlainLoginModule required
 username="kafkacluster"
 password="Admin@123"
 user_kafkacluster="Admin@123"
 user_admin="Admin@123"
};
 
Client{
 org.apache.kafka.common.security.plain.PlainLoginModule required  
 username="admin"  
 password="Admin@123";  
};

[storm@mongodb_2 ~]$mkdir /home/storm/kafka_2.11-2.3.1/log -p 
[storm@mongodb_2 ~]$cd  /home/storm/kafka_2.11-2.3.1/bin
[storm@mongodb_2 bin]$ vim   kafka-server-start.sh #把创建的文件 kafka_server_jaas.conf 加载到启动环境中
export KAFKA_HEAP_OPTS=$KAFKA_HEAP_OPTS" -Djava.security.auth.login.config=/home/storm/kafka_2.11-2.3.1/config/kafka_server_jaas.conf"

[storm@mongodb_2 ~]$ ./kafka-server-start.sh -daemon config/server.properties
  • 部署mongodb_3服务器的kafka
[storm@mongodb_3 ~]$ cd ~/kafka_2.11-2.3.1/config/
[storm@mongodb_3 config]$ cat server.properties
broker.id=1
listeners=SASL_PLAINTEXT://10.130.41.42:9092
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.enabled.mechanisms=PLAIN
security.inter.broker.protocol=SASL_PLAINTEXT
allow.everyone.if.no.acl.found=false
authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer
super.users=User:admin;User:kafkacluster
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/home/storm//kafka_2.11-2.3.1/logs
num.partitions=3
default.replication.factor=3
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=mongodb_3:2181,mongodb_2:2181,mongodb_3:2181
compression.type=producer
zookeeper.set.acl=true
zookeeper.defaultAcl=root;123456:rwcd
zookeeper.connection.timeout.ms=6000
zookeeper.digest.enabled=true
group.initial.rebalance.delay.ms=0

[storm@mongodb_3 config]$ more  sasl.conf
security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="Admin@123";

[storm@mongodb_3 config]$ cat kafka_server_jaas.conf
KafkaServer {
 org.apache.kafka.common.security.plain.PlainLoginModule required
 username="kafkacluster"
 password="Admin@123"
 user_kafkacluster="Admin@123"
 user_admin="Admin@123"
};
 
Client{
 org.apache.kafka.common.security.plain.PlainLoginModule required  
 username="admin"  
 password="Admin@123";  
};

[storm@mongodb_3 ~]$mkdir /home/storm/kafka_2.11-2.3.1/log -p 
[storm@mongodb_3 ~]$cd  /home/storm/kafka_2.11-2.3.1/bin
[storm@mongodb_3 bin]$ vim   kafka-server-start.sh #把创建的文件 kafka_server_jaas.conf 加载到启动环境中
export KAFKA_HEAP_OPTS=$KAFKA_HEAP_OPTS" -Djava.security.auth.login.config=/home/storm/kafka_2.11-2.3.1/config/kafka_server_jaas.conf"

[storm@mongodb_3 ~]$ ./kafka-server-start.sh -daemon config/server.properties
  • 初始化创建topics,在其中一台kafka上面操作即可
[storm@mongodb_1 kafka_2.11-2.3.1]$  cd kafka_2.11-2.3.1/

[storm@mongodb_1 kafka_2.11-2.3.1]$  bin/kafka-topics.sh --create --zookeeper mongodb_1:2181,mongodb_2:2181,mongodb_3:2181 --replication-factor 3 --partitions 1 --topic ent_record_fastdfs_url
  • 给对应topics配置acls权限
[storm@mongodb_1 bin]$ ./kafka-acls.sh --authorizer-properties zookeeper.connect=10.130.41.43:2181 --add --allow-principal User:admin --producer --topic ent_record_fastdfs_url
  • 查看那些topic设置alcs
[storm@mongodb_1 bin]$ ./kafka-acls.sh --authorizer-properties zookeeper.connect=10.130.41.43:2181 --list 
Current ACLs for resource `Topic:LITERAL:ent_record_fastdfs_url`: 
	User:admin has Allow permission for operations: Describe from hosts: *
	User:admin has Allow permission for operations: Create from hosts: *
	User:admin has Allow permission for operations: Write from hosts: *
storm部署

  • 当前版本不支持连接kafka支持 ACL， 需要修改storm jar包，待研发布支持版本。

  • 下面配置文件配置已支持zookeeper连接设置ACL

  • 部署mongodb_1服务器的storm

[storm@mongodb_1 ~]$ cd ~/apache-storm-1.2.4/conf/
[storm@mongodb_1 conf]$ vim storm.yaml 
storm.zookeeper.factory.class: "org.apache.storm.security.auth.SASLAuthenticationProvider"
storm.zookeeper.auth.provider.1: "org.apache.storm.security.auth.digest.DigestSaslServerCallbackHandler"
storm.zookeeper.auth.provider.2: "org.apache.zookeeper.server.auth.SASLAuthenticationProvider"
storm.zookeeper.superACL: "sasl:admin"
storm.zookeeper.servers:
    - "10.130.41.42"
    - "10.130.41.43"
    - "10.130.41.44"
nimbus.seeds: ["10.130.41.42","10.130.41.43","10.130.41.44"]
ui.port: 8888
storm.local.dir: "/home/storm/apache-storm-1.2.4/data"
supervisor.slots.ports:
   - 6700
   - 6701
   - 6702
   - 6703
   - 6704
   - 6705
   - 6706
storm.zookeeper.authProvider.1: org.apache.zookeeper.server.auth.DigestAuthenticationProvider
storm.zookeeper.auth.user: admin
storm.zookeeper.auth.password: Admin@123
storm.zookeeper.auth.digest.1: admin=Admin@123
java.security.auth.login.config: /home/storm/kafka_2.11-2.3.1/config/kafka_server_jaas.conf
worker.childopts: '-Djava.security.auth.login.config=/home/storm/kafka_2.11-2.3.1/config/kafka_server_jaas.conf'
worker.childopts: '-Djava.security.auth.login.config=/home/storm/zookeeper-3.6.3/conf/jaas.conf'
[storm@mongodb_1 ~]$  mkdir -p /home/storm/apache-storm-1.2.4/data
[storm@mongodb_1 ~]$  cd  /home/storm/apache-storm-1.2.4/bin
[storm@mongodb_1 bin]$  nohup ./storm nimbus &
[storm@mongodb_1 bin]$  nohup ./storm ui &
[storm@mongodb_1 bin]$ nohup ./storm supervisor &

[storm@mongodb_1 bin]$ jps  #查看是否正常运行
1808 worker
1792 LogWriter
1907 worker
29171 Supervisor
1891 LogWriter
20328 core
25224 Kafka
31867 ZooKeeperMain
2444 Jps
18957 QuorumPeerMain
32334 nimbus
  • 部署mongodb_2服务器的storm
[storm@mongodb_2 ~]$ cd ~/apache-storm-1.2.4/conf/
[storm@mongodb_2 conf]$ vim storm.yaml 
storm.zookeeper.factory.class: "org.apache.storm.security.auth.SASLAuthenticationProvider"
storm.zookeeper.auth.provider.1: "org.apache.storm.security.auth.digest.DigestSaslServerCallbackHandler"
storm.zookeeper.auth.provider.2: "org.apache.zookeeper.server.auth.SASLAuthenticationProvider"
storm.zookeeper.superACL: "sasl:admin"
storm.zookeeper.servers:
    - "10.130.41.42"
    - "10.130.41.43"
    - "10.130.41.44"
nimbus.seeds: ["10.130.41.42","10.130.41.43","10.130.41.44"]
ui.port: 8888
storm.local.dir: "/home/storm/apache-storm-1.2.4/data"
supervisor.slots.ports:
   - 6700
   - 6701
   - 6702
   - 6703
   - 6704
   - 6705
   - 6706
storm.zookeeper.authProvider.1: org.apache.zookeeper.server.auth.DigestAuthenticationProvider
storm.zookeeper.auth.user: admin
storm.zookeeper.auth.password: Admin@123
storm.zookeeper.auth.digest.1: admin=Admin@123
java.security.auth.login.config: /home/storm/kafka_2.11-2.3.1/config/kafka_server_jaas.conf
worker.childopts: '-Djava.security.auth.login.config=/home/storm/kafka_2.11-2.3.1/config/kafka_server_jaas.conf'
worker.childopts: '-Djava.security.auth.login.config=/home/storm/zookeeper-3.6.3/conf/jaas.conf'
[storm@mongodb_2 ~]$  mkdir -p /home/storm/apache-storm-1.2.4/data
[storm@mongodb_2 ~]$  cd  /home/storm/apache-storm-1.2.4/bin
[storm@mongodb_2 bin]$  nohup ./storm nimbus &
[storm@mongodb_2 bin]$  nohup ./storm ui &
[storm@mongodb_2 bin]$ nohup ./storm supervisor &

[storm@mongodb_2 bin]$ jps  #查看是否正常运行
1808 worker
1792 LogWriter
1907 worker
29171 Supervisor
1891 LogWriter
20328 core
25224 Kafka
31867 ZooKeeperMain
2444 Jps
18957 QuorumPeerMain
32334 nimbus
  • 部署mongodb_3服务器的storm
[storm@mongodb_3 ~]$ cd ~/apache-storm-1.2.4/conf/
[storm@mongodb_3 conf]$ vim storm.yaml 
storm.zookeeper.factory.class: "org.apache.storm.security.auth.SASLAuthenticationProvider"
storm.zookeeper.auth.provider.1: "org.apache.storm.security.auth.digest.DigestSaslServerCallbackHandler"
storm.zookeeper.auth.provider.2: "org.apache.zookeeper.server.auth.SASLAuthenticationProvider"
storm.zookeeper.superACL: "sasl:admin"
storm.zookeeper.servers:
    - "10.130.41.42"
    - "10.130.41.43"
    - "10.130.41.44"
nimbus.seeds: ["10.130.41.42","10.130.41.43","10.130.41.44"]
ui.port: 8888
storm.local.dir: "/home/storm/apache-storm-1.2.4/data"
supervisor.slots.ports:
   - 6700
   - 6701
   - 6702
   - 6703
   - 6704
   - 6705
   - 6706
storm.zookeeper.authProvider.1: org.apache.zookeeper.server.auth.DigestAuthenticationProvider
storm.zookeeper.auth.user: admin
storm.zookeeper.auth.password: Admin@123
storm.zookeeper.auth.digest.1: admin=Admin@123
java.security.auth.login.config: /home/storm/kafka_2.11-2.3.1/config/kafka_server_jaas.conf
worker.childopts: '-Djava.security.auth.login.config=/home/storm/kafka_2.11-2.3.1/config/kafka_server_jaas.conf'
worker.childopts: '-Djava.security.auth.login.config=/home/storm/zookeeper-3.6.3/conf/jaas.conf'
[storm@mongodb_3 ~]$  mkdir -p /home/storm/apache-storm-1.2.4/data
[storm@mongodb_3 ~]$  cd  /home/storm/apache-storm-1.2.4/bin
[storm@mongodb_3 bin]$  nohup ./storm nimbus &
[storm@mongodb_3 bin]$  nohup ./storm ui &
[storm@mongodb_3 bin]$ nohup ./storm supervisor &

[storm@mongodb_3 bin]$ jps  #查看是否正常运行
1808 worker
1792 LogWriter
1907 worker
29171 Supervisor
1891 LogWriter
20328 core
25224 Kafka
31867 ZooKeeperMain
2444 Jps
18957 QuorumPeerMain
32334 nimbus
  • storm的主节点nimbus在第一台上启动拓扑图
  • 启动拓扑图时需要指定stormStatics.jar 包，如果使用新storm安装包没有，请自行找测试部要stormStatics.jar包
[storm@mongodb_1 bin]$ cd /home/storm/apache-storm-1.2.4/bin

[storm@mongodb_1 bin]$ ./storm jar ../lib/stormStatics.jar com.channelsoft.stormStatics.topology.FastdfsUrlTopology FastdfsUrlTopology mongodb_1,mongodb_2,mongodb_3 2181 ent_record_fastdfs_url /brokers
  • 使用storm-ui查看刚启动拓扑是否正常
  • 访问第一台的服务器ip端口是8888

本文由mdnice多平台发布

本文来自互联网用户投稿，该文观点仅代表作者本人，不代表本站立场。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如若转载，请注明出处：/a/332271.html

如若内容造成侵权/违法违规/事实不符，请联系我们进行投诉反馈qq邮箱809451989@qq.com，一经查实，立即删除！

相关文章

【分享】MathWorks中国汽车年会：“软件定义汽车”

【分享】MathWorks中国汽车年会：“软件定义汽车”

从软件赋能到软件定义，汽车行业不仅需要解决诸如错误发现滞后带来的高昂代价、功能融合所需的跨学科知识、功能安全与实施成本之间的权衡等老问题，也面临着新的挑战：软件复杂度的不断提升、利用数据驱动创造价值、人工智能的引入和实现、数字…
阅读更多...
PyCharm 快捷键（Ctrl + R）正则表达式批量替换

PyCharm 快捷键（Ctrl + R）正则表达式批量替换

目录 一、使用快捷键CtrlR，打开替换界面 二、输入替换格式 三、点击全部替换 一、使用快捷键CtrlR，打开替换界面 二、输入替换格式 在第一个框输入 (.*): (.*) 第二个框输入 $1:$2, 三、点击全部替换
阅读更多...
【JVM】并发的可达性分析详细解释

【JVM】并发的可达性分析详细解释

​ 🍎个人博客：个人主页 🏆个人专栏：JVM ⛳️ 功不唐捐，玉汝于成 ​ 目录 前言 正文 可达性分析的基本原理： 根集合（Root Set）： 对象引用关系： 标记…
阅读更多...
Java导出Excel并合并单元格

Java导出Excel并合并单元格

需求：需要在导出excel时合并指定的单元格 ruoyi excel 项目基于若伊框架二次开发，本着能用现成的就不自己写的原则，先是尝试了Excel注解中needMerge属性 /*** 是否需要纵向合并单元格,应对需求:含有list集合单元格)*/public boolean needMer…
阅读更多...
记一次 .NET某道闸收费系统 内存溢出分析

记一次 .NET某道闸收费系统 内存溢出分析

一：背景 1. 讲故事 前些天有位朋友找到我，说他的程序几天内存就要爆一次，不知道咋回事，找不出原因，让我帮忙看一下，这种问题分析dump是最简单粗暴了，拿到dump后接下来就是一顿分析。 二&…
阅读更多...
11.1 pcl_ros的点云学习

11.1 pcl_ros的点云学习

本文是看了两个博主的内容，整理在这里是为了以后用时方便查找，更容易理解。引用的博文路径如下（本人也是刚开始看PCL的运用，本文是完全抄下面博主的内容，觉得这位博主写的很详细很清楚，并且自己运行了一遍有…
阅读更多...
Java17新特性详解含示例代码（值得珍藏）

Java17新特性详解含示例代码（值得珍藏）

1. 概述 Java 17 是 Java 开发工具包（JDK）的一个重要版本，它带来了一系列的新特性和改进，以进一步增强 Java 语言的功能和性能。以下是 Java 17 中的一些主要新特性及其详细说明。 2. 新特性详解 JEP 356: Enhanced Pseudo-Ran…
阅读更多...
uniapp uni.chooseLocation调用走失败那里,错误码：112

uniapp uni.chooseLocation调用走失败那里,错误码：112

问题：我配置了百度上所有能配置的，一直调用不成功，如下图配置的 1:第一个 配置 代码： "permission": {"scope.userLocation": {"desc": "你的位置信息将用于小程序位置接口的效果展示"}…
阅读更多...
openpose之使用摄像头检测并输出到json文件

openpose之使用摄像头检测并输出到json文件

编程如画，我是panda！ 前言 之前给大家分享了如何搭建openpose环境，并进行了测试案例，但是如果要使用摄像头的话，还需要修改一下运行文件，并且这次会教大家如何输出到json文件 。 如果环境还没有搭建好&am…
阅读更多...
Unix时间戳

Unix时间戳

时间戳，相信很多相关专业的人，计算机软件电子等等都会听过。由于最早是由Unix系统使用所以又叫Unix时间戳。 Unix 时间戳（Unix Timestamp）定义为从UTC（世界协调时）/GMT（格林尼治时）…
阅读更多...
iPhone解锁工具---AnyMP4 iPhone Unlocker 中文

iPhone解锁工具---AnyMP4 iPhone Unlocker 中文

AnyMP4 iPhone Unlocker是一款功能强大的iPhone解锁软件，旨在帮助用户轻松解锁iPhone，从而在电脑上进行数据备份、传输和编辑。该软件支持多种iPhone型号，包括最新的iPhone 14系列，并支持多种解锁模式，如屏幕密码解锁、…
阅读更多...
【Docker】安装 Nacos容器并根据Nginx实现负载均衡

【Docker】安装 Nacos容器并根据Nginx实现负载均衡

🎉🎉欢迎来到我的CSDN主页！🎉🎉 🏅我是Java方文山，一个在CSDN分享笔记的博主。📚📚 🌟推荐给大家我的专栏《Docker实战》。🎯🎯 &…
阅读更多...
Go后端开发 -- 反射reflect 结构体标签

Go后端开发 -- 反射reflect 结构体标签

Go后端开发 – 反射reflect && 结构体标签 文章目录 Go后端开发 -- 反射reflect && 结构体标签一、反射reflect1.编程语言中反射的概念2.interface 和反射3.变量内置的pair结构4.reflect的基本功能TypeOf和ValueOf5.从relfect.Value中获取接口interface的信息6…
阅读更多...
2018年认证杯SPSSPRO杯数学建模D题(第二阶段)投篮的最佳出手点全过程文档及程序

2018年认证杯SPSSPRO杯数学建模D题(第二阶段)投篮的最佳出手点全过程文档及程序

2018年认证杯SPSSPRO杯数学建模 D题 投篮的最佳出手点 原题再现： 影响投篮命中率的因素不仅仅有出手角度、球感、出手速度，还有出手点的选择。规范的投篮动作包含两膝微屈、重心落在两脚掌上、下肢蹬地发力、身体随之向前上方伸展、同时抬肘向投篮方向…
阅读更多...
SpringBoot：前端提交数据，服务端无法获取数据

SpringBoot：前端提交数据，服务端无法获取数据

http://www.xxx.com?phone111111111111&code1332 上述访问传值方式为键值对方式，服务端springmvc获取 >> // 在HttpServlet实现类的doGet、doPost方法中获取前端传来的值 doGet(ServerHttpRequest request){String phone request.getParameter("…
阅读更多...
Codeforces Round 767 (Div. 1) D2. Game on Sum (Hard Version)（博弈 期望 dp 贡献）

Codeforces Round 767 (Div. 1) D2. Game on Sum (Hard Version)（博弈 期望 dp 贡献）

题目 t(t<1e5)组样例&#xff0c;每次给定n,m,k(m<n<1e6&#xff0c;0<k<1e97) 有一个游戏&#xff0c;持续n轮&#xff0c;每轮Alice先选一个[0,k]的实数&#xff0c; Bob决定从总分里加上这个值还是减去这个值 特别地&#xff0c;n轮里&#xff0c;Bob选择…
阅读更多...
Unity Mirror VR联机开发 实战篇（二）

Unity Mirror VR联机开发 实战篇（二）

一、迁移示例中的联机物体 1、将MirrorExamplesVR工程中的部分文件夹复制到自己的工程中。 1、打开MirrorExamplesVR中的 SceneVR-Common场景。 2、将场景中没用的东西都删掉&#xff0c;只留下面这些&#xff0c;新建一个空物体XR Mirror&#xff0c;将所有剩下的物体拖成XR …
阅读更多...
Elastic 8.12：AI Assistant for Observability 正式发布，更新至 Apache Lucene 9.9

Elastic 8.12：AI Assistant for Observability 正式发布，更新至 Apache Lucene 9.9

作者&#xff1a;来自 Elastic Brian Bergholm 今天&#xff0c;我们很高兴地宣布 Elastic 8.12 全面上市。 有哪些新的功能&#xff1f; 8.12 版本的两个最重要的组成部分包括 Elastic AI Assistant for Observability 的 正式发布版 和 Apache Lucene 9.9 的更新&#xff08…
阅读更多...
网络安全B模块（笔记详解）- SQL注入

网络安全B模块（笔记详解）- SQL注入

简单sql注入 1.使用渗透机场景kali中工具扫描服务器场景，将apache的端口号和版本号作为Flag提交（格式：端口号_版本号） Flag：8081_7.5 2.使用渗透机场景windows7访问服务器场景SQL网站，并将网站中概述页面中的Flag提交； Flag：sql_is_good 3.使用渗透机场景windows7访问…
阅读更多...
AR与AI融合加速，医疗护理更便捷

AR与AI融合加速，医疗护理更便捷

根据Reports and Data的AR市场发展报告&#xff0c;到2026年&#xff0c;预计医疗保健市场中的AR/VR行业规模将达到70.5亿美元。这一趋势主要受到对创新诊断技术、神经系统疾病和疾病意识不断增长的需求驱动。信息技术领域的进步&#xff0c;包括笔记本电脑、计算机、互联网连接…
阅读更多...
最新文章

Copyright @ 2022~2023