使用docker 和 kubnernetes 部署单节点/多节点 kafka 环境

参考资料

• https://kafka.apachecn.org/documentation.html#configuration

kafka的broker有三个核心配置

• broker.id
• log.dirs
• zookeeper.connect

docker启动单节点kafka环境

启动zookeeper

可配置的环境变量，https://gallery.ecr.aws/bitnami/zookeeper

$ docker run -it --rm --name my-zookeeper -p 2181:2181 -e ALLOW_ANONYMOUS_LOGIN=yes zookeeper:3.7# 查看节点状态
$ /apache-zookeeper-3.7.0-bin/bin/zkServer.sh status

启动kafka

可配置的环境变量，https://gallery.ecr.aws/bitnami/kafka

$ docker run -it --rm --name my-kafka -p 9092:9092 --link my-zookeeper \-e KAFKA_BROKER_ID=0 \-e KAFKA_ZOOKEEPER_CONNECT=my-zookeeper:2181 \-e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092 \-e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 \-e ALLOW_PLAINTEXT_LISTENER=yes public.ecr.aws/bitnami/kafka:3.2.3

查看和创建topic

$ kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --list
$ kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --topic test --create --partitions 1 --replication-factor 1

测试本地环境

$ kafka-console-consumer.sh --bootstrap-server 127.0.0.1:9092 --topic test --from-beginning$ kafka-console-producer.sh --broker-list 127.0.0.1:9092 --topic test

docker-compose启动单节点kafka

使用docker-composer部署

version: '3.8'
services:dk-zookeeper:image: zookeeper:3.7.0ports:- 2182:2181environment:- ALLOW_ANONYMOUS_LOGIN=yesdk-kafka-01:image: public.ecr.aws/bitnami/kafka:3.2.3ports:- 9092:9092environment:KAFKA_BROKER_ID: 0KAFKA_CFG_LISTENERS: PLAINTEXT://:9092KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://127.0.0.1:9092# KAFKA_CREATE_TOPICS: "test:1:0" //创建test的topic，具备1分区0副本KAFKA_ZOOKEEPER_CONNECT: dk-zookeeper:2181KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092ALLOW_PLAINTEXT_LISTENER: yesdepends_on:- dk-zookeeper

docker-compose启动多节点kafka集群

https://blog.csdn.net/qq_39526294/article/details/124293954

创建自定义网络

$ docker network create zookeeper_network

部署zookeeper

创建zookeeper服务

version: '3.8'networks:default:external:name: zookeeper_networkservices:zoo1:image: zookeeper:3.7.0hostname: zoo1ports:- 2181:2181environment:ZOO_MY_ID: 1ZOO_SERVERS: server.1=0.0.0.0:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=zoo3:2888:3888;2181zoo2:image: zookeeper:3.7.0hostname: zoo2ports:- 2182:2181environment:ZOO_MY_ID: 2ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=0.0.0.0:2888:3888;2181 server.3=zoo3:2888:3888;2181zoo3:image: zookeeper:3.7.0hostname: zoo3ports:- 2183:2181environment:ZOO_MY_ID: 3ZOO_SERVERS: server.1=zoo1:2888:3888;2181 server.2=zoo2:2888:3888;2181 server.3=0.0.0.0:2888:3888;2181

启动zookeeper

$ docker-compose up

部署kafka

创建kafka服务

version: '3.8'networks:default:external:name: zookeeper_networkservices:kafka1:image: public.ecr.aws/bitnami/kafka:3.2.3restart: unless-stoppedcontainer_name: kafka1hostname: kafka1ports:- "9092:9092"external_links:- zoo1- zoo2- zoo3environment:KAFKA_BROKER_ID: 1KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://172.31.18.4:9092KAFKA_ADVERTISED_HOST_NAME: kafka1KAFKA_ADVERTISED_PORT: 9092KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181,zoo2:2181,zoo3:2181"ALLOW_PLAINTEXT_LISTENER: yeskafka2:image: public.ecr.aws/bitnami/kafka:3.2.3restart: unless-stoppedcontainer_name: kafka2hostname: kafka2ports:- "9093:9092"external_links:- zoo1- zoo2- zoo3environment:KAFKA_BROKER_ID: 2KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.0.225:9093    ## 宿主机IPKAFKA_ADVERTISED_HOST_NAME: kafka2KAFKA_ADVERTISED_PORT: 9093KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181,zoo2:2181,zoo3:2181"ALLOW_PLAINTEXT_LISTENER: yeskafka3:image: public.ecr.aws/bitnami/kafka:3.2.3restart: unless-stoppedcontainer_name: kafka3hostname: kafka3ports:- "9094:9092"external_links:- zoo1- zoo2- zoo3environment:KAFKA_BROKER_ID: 3KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://192.168.0.225:9094   ## 宿主机IPKAFKA_ADVERTISED_HOST_NAME: kafka3KAFKA_ADVERTISED_PORT: 9094KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181,zoo2:2181,zoo3:2181"ALLOW_PLAINTEXT_LISTENER: yeskafka-manager:image: sheepkiller/kafka-manager:latestrestart: unless-stoppedcontainer_name: kafka-managerhostname: kafka-managerports:- "9000:9000"links:- kafka1- kafka2- kafka3external_links:- zoo1- zoo2- zoo3environment:ZK_HOSTS: zoo1:2181,zoo2:2181,zoo3:2181KAFKA_BROKERS: kafka1:9092,kafka2:9093,kafka3:9094

启动kafka

$ docker-compose up

查看kafka ui界面

kubernetes启动kafka集群

注意，这里目的仅仅是临时测试，因而回收策略设置为Delete

手动部署

单节点部署

apiVersion: v1
kind: Service
metadata:labels:app: zookeeper-servicename: zookeeper-service
spec:ports:- name: zookeeper-portport: 2181targetPort: 2181selector:app: zookeeper
---
apiVersion: apps/v1
kind: Deployment
metadata:labels:app: zookeepername: zookeeper
spec:replicas: 1selector:matchLabels:app: zookeepertemplate:metadata:labels:app: zookeeperspec:containers:#- image: wurstmeister/zookeeper- image: xxxxxxxxx.dkr.ecr.cn-north-1.amazonaws.com.cn/zookeeper:3.7.0imagePullPolicy: IfNotPresentname: zookeeperports:- containerPort: 2181
---
apiVersion: v1
kind: Service
metadata:name: kafka-servicelabels:app: kafka
spec:type: NodePortports:- port: 9092name: kafka-porttargetPort: 9092protocol: TCPselector:app: kafka
---
kind: Deployment
apiVersion: apps/v1
metadata:name: kafka-deployment
spec:replicas: 1selector:matchLabels:app: kafkatemplate:metadata:labels:app: kafkaspec:containers:- name: kafkaimage: public.ecr.aws/bitnami/kafka:3.2.3imagePullPolicy: IfNotPresentports:- containerPort: 9092env:- name: KAFKA_ADVERTISED_PORTvalue: "9092"- name: KAFKA_ADVERTISED_HOST_NAMEvalue: "kafka-host"- name: KAFKA_ZOOKEEPER_CONNECTvalue: "zookeeper-service:2181"- name: KAFKA_BROKER_IDvalue: "1"- name: ALLOW_PLAINTEXT_LISTENERvalue: "yes"

多节点部署

https://blog.51cto.com/binghe001/5245687

相对来说比较复杂，网络上较多demo

使用operator部署

https://github.com/strimzi/strimzi-kafka-operator

https://github.com/strimzi/strimzi-kafka-operator/blob/0.34.0/examples/kafka/kafka-ephemeral.yaml

一键部署operator

kubectl create -f 'https://strimzi.io/install/latest?namespace=kafka' -n kafka

创建单节点的zk-kafka环境

kubectl apply -f https://raw.githubusercontent.com/strimzi/strimzi-kafka-operator/0.34.0/examples/kafka/kafka-ephemeral-single.yaml -n kafka 

创建多节点临时存储kafka集群

kubectl apply -f https://raw.githubusercontent.com/strimzi/strimzi-kafka-operator/0.34.0/examples/kafka/kafka-ephemeral.yaml -n kafka 

yaml文件如下

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:name: my-cluster
spec:kafka:version: 3.4.0replicas: 3listeners:- name: plainport: 9092type: internaltls: false- name: tlsport: 9093type: internaltls: trueconfig:offsets.topic.replication.factor: 3transaction.state.log.replication.factor: 3transaction.state.log.min.isr: 2default.replication.factor: 3min.insync.replicas: 2inter.broker.protocol.version: "3.4"storage:type: ephemeralzookeeper:replicas: 3storage:type: ephemeralentityOperator:topicOperator: {}userOperator: {}

operator的方式还是方便的，删除环境只需要删除crd即可

$ k get pod -n kafka
NAME                                          READY   STATUS    RESTARTS   AGE
my-cluster-entity-operator-578d74f8f9-p9w5c   1/3     Running   0          16s
my-cluster-kafka-0                            1/1     Running   0          60s
my-cluster-kafka-1                            1/1     Running   0          60s
my-cluster-kafka-2                            1/1     Running   0          60s
my-cluster-zookeeper-0                        1/1     Running   0          114s
my-cluster-zookeeper-1                        1/1     Running   0          114s
my-cluster-zookeeper-2                        1/1     Running   0          114s
strimzi-cluster-operator-58967f75c7-8gp4w     1/1     Running   0          7m2s

测试集群

$ kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --list$ kafka-topics.sh --bootstrap-server 127.0.0.1:9092 --topic test --create --partitions 1 --replication-factor 1$ kafka-topics --zookeeper zookeeper-service:2181  --list$ kafka-console-producer --topic test-topic --broker-list localhost:9092$ kafka-console-consumer --topic test-topic --from-beginning --bootstrap-server localhost:9092