文章目录
- 部署
- Docker
- 测试
- 启动
- 测试
- 创建Topic
- 生产者
- 消费者
- 集成Manager的部署
- dockerfile
- 效果
- 参考资料
在我们做和kafka开发相关的工作时，往往希望独立部署一套kafka测试环境。而kafka部署时，不能只是简单安装kafka自身组件，还要安装zookeeper、JDK之类的辅助软件。这让其部署变得不是那么方便。本文将使用kafka和zookeeper的官方编译包来做部署。
部署
Docker
# dockerfile
FROM ubuntu:22.04
WORKDIR /home/kafka-alone/
RUN apt-get update && apt-get upgrade -y
RUN apt-get install default-jdk -y
RUN apt-get install wget -y
RUN wget https://downloads.apache.org/kafka/3.7.0/kafka_2.13-3.7.0.tgz
RUN tar -xvzf kafka_2.13-3.7.0.tgz
RUN wget https://archive.apache.org/dist/zookeeper/zookeeper-3.9.2/apache-zookeeper-3.9.2-bin.tar.gz
RUN tar -xvzf apache-zookeeper-3.9.2-bin.tar.gz
RUN mv apache-zookeeper-3.9.2-bin zookeeper
RUN mv kafka_2.13-3.7.0 kafka
RUN rm -rf kafka_2.13-3.7.0.tgz
RUN rm -rf apache-zookeeper-3.9.2-bin.tar.gz
RUN mv zookeeper/conf/zoo_sample.cfg zookeeper/conf/zoo.cfg
EXPOSE 9092
RUN echo "./kafka/bin/zookeeper-server-start.sh ./kafka/config/zookeeper.properties &" >> start.sh
RUN echo "./kafka/bin/kafka-server-start.sh ./kafka/config/server.properties" >> start.sh
CMD ["/bin/bash", "-ce", "sh start.sh"]
然后使用下面指令编译成镜像
docker build --pull --rm -f "dockerfile" -t kafka-all-in-one:latest "."
测试
启动
docker container run -d -p 9092:9092 --name kafka-all-in-one-container kafka-all-in-one:latest
容器内的执行日志如下
测试
创建Topic
docker exec -it kafka-all-in-one-container /bin/bash kafka/bin/kafka-topics.sh --create --topic test-topic --bootstrap-server localhost:9092
生产者
docker exec -it kafka-all-in-one-container /bin/bash kafka/bin/kafka-console-producer.sh --topic test-topic --bootstrap-server localhost:9092
然后在终端中输入 hello world两个单词。
消费者
docker exec -it kafka-all-in-one-container /bin/bash kafka/bin/kafka-console-consumer.sh --topic test-topic --from-beginning --bootstrap-server localhost:9092
消费者就会收到生产者发布的两个消息。
集成Manager的部署
我们集成Kafka Manager到docker中，以可视化的方式管理Kafka。
dockerfile
FROM ubuntu:22.04
WORKDIR /home/kafka-alone/
RUN apt-get update && apt-get upgrade -y
RUN apt-get install default-jdk -y
RUN apt-get install wget -y
RUN wget https://downloads.apache.org/kafka/3.7.0/kafka_2.13-3.7.0.tgz
RUN tar -xvzf kafka_2.13-3.7.0.tgz
RUN wget https://archive.apache.org/dist/zookeeper/zookeeper-3.9.2/apache-zookeeper-3.9.2-bin.tar.gz
RUN tar -xvzf apache-zookeeper-3.9.2-bin.tar.gz
RUN mv apache-zookeeper-3.9.2-bin zookeeper
RUN mv kafka_2.13-3.7.0 kafka
RUN rm -rf kafka_2.13-3.7.0.tgz
RUN rm -rf apache-zookeeper-3.9.2-bin.tar.gz
RUN mv zookeeper/conf/zoo_sample.cfg zookeeper/conf/zoo.cfg
EXPOSE 9092
RUN echo "./kafka/bin/zookeeper-server-start.sh ./kafka/config/zookeeper.properties &" >> start.sh
RUN echo "./kafka/bin/kafka-server-start.sh ./kafka/config/server.properties &" >> start.sh
RUN apt-get install unzip -y
RUN wget https://github.com/yahoo/CMAK/releases/download/3.0.0.6/cmak-3.0.0.6.zip
RUN unzip cmak-3.0.0.6.zip
RUN mv cmak-3.0.0.6 cmak
RUN rm -rf cmak-3.0.0.6.zip
RUN sed -i 's/kafka-manager-zookeeper:2181/localhost:2181/g' cmak/conf/application.conf
RUN sed -i 's/cmak.zkhosts/#cmak.zkhosts/g' cmak/conf/application.conf
RUN echo "./cmak/bin/cmak -Dconfig.file=./cmak/conf/application.conf -Dhttp.port=8080" >> start.sh
EXPOSE 8080
CMD ["/bin/bash", "-ce", "sh start.sh"]
效果
使用下面命令部署后
docker build --pull --rm -f "dockerfile" -t kafka-all-in-one:latest "."
docker container run -d -p 9092:9092 -p 8080:8080 --name kafka-all-in-one-container kafka-all-in-one:latest
参考资料
- https://www.cnblogs.com/szx666/p/14635910.html
- https://blog.csdn.net/qq_44519484/article/details/121908565
- https://medium.com/yavar/installation-of-apache-kafka-on-ubuntu-22-04-59e57f878979
- https://akhq.io/docs/installation.html#docker
- https://github.com/yahoo/CMAK
