杰霸霸博客

主题

Broker节点服役和退役 ​

1. 服役新节点 ​

1.1 新节点准备 ​

新增一台hadoop108机器。 参照Kafka单机版搭建准备环境的下载解压章节进行搭建。

1.2 配置文件调整 ​

修改配置文件$KAFKA_HOME/config/server.properties

sh
# 调整broker.id
broker.id=4
#kafka 运行日志(数据)存放的路径，路径不需要提前创建
log.dirs=/opt/module/kafka-3.6.1/data
#配置连接 Zookeeper集群地址
zookeeper.connect=hadoop105:2181,hadoop106:2181,hadoop107:2181/kafka

1.3 启动Hadoop108的Kafka ​

sh
[jack@hadoop108 kafka-3.6.1]$ bin/kafka-server-start.sh -daemon ./config/server.properties

2. 执行负载均衡服役新节点 ​

2.1 创建一个要均衡的主题文件 ​

sh
[jack@hadoop105 kafka-3.6.1]$ vim topics-to-move.json
 {
    "topics": [
        {"topic": "hadoop"}
    ],
    "version": 1
}

2.2 生成一个负载均衡的计划 ​

sh
[jack@hadoop105 kafka-3.6.1]$ ./bin/kafka-reassign-partitions.sh --bootstrap-server hadoop106:9092 --topics-to-move-json-file topics-to-move.json --broker-list "1,2,3,4" --generate
Current partition replica assignment
{"version":1,"partitions":[{"topic":"hadoop","partition":0,"replicas":[2,1,3],"log_dirs":["any","any","any"]},{"topic":"hadoop","partition":1,"replicas":[3,2,1],"log_dirs":["any","any","any"]},{"topic":"hadoop","partition":2,"replicas":[1,3,2],"log_dirs":["any","any","any"]},{"topic":"hadoop","partition":3,"replicas":[2,3,1],"log_dirs":["any","any","any"]}]}

Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"hadoop","partition":0,"replicas":[1,2,3],"log_dirs":["any","any","any"]},{"topic":"hadoop","partition":1,"replicas":[2,3,4],"log_dirs":["any","any","any"]},{"topic":"hadoop","partition":2,"replicas":[3,4,1],"log_dirs":["any","any","any"]},{"topic":"hadoop","partition":3,"replicas":[4,1,2],"log_dirs":["any","any","any"]}]}

可以看到执行命令返回两部分，第一份部分是当前分区副本的情况，第二个是将要负载均衡计划的情况，重复命令执行生产新的计划。

2.3 创建副本存储计划文件 ​

如果对上一步的计划确认后，将内容保存到文件中

sh
[jack@hadoop105 kafka-3.6.1]$  vi increase-replication-factor.json
{"version":1,"partitions":[{"topic":"hadoop","partition":0,"replicas":[1,2,3],"log_dirs":["any","any","any"]},{"topic":"hadoop","partition":1,"replicas":[2,3,4],"log_dirs":["any","any","any"]},{"topic":"hadoop","partition":2,"replicas":[3,4,1],"log_dirs":["any","any","any"]},{"topic":"hadoop","partition":3,"replicas":[4,1,2],"log_dirs":["any","any","any"]}]}

2.4 执行副本存储计划 ​

sh
[jack@hadoop105 kafka-3.6.1]$ ./bin/kafka-reassign-partitions.sh --bootstrap-server hadoop107:9092 --reassignment-json-file increase-replication-factor.json --execute
Current partition replica assignment

{"version":1,"partitions":[{"topic":"hadoop","partition":0,"replicas":[2,1,3],"log_dirs":["any","any","any"]},{"topic":"hadoop","partition":1,"replicas":[3,2,1],"log_dirs":["any","any","any"]},{"topic":"hadoop","partition":2,"replicas":[1,3,2],"log_dirs":["any","any","any"]},{"topic":"hadoop","partition":3,"replicas":[2,3,1],"log_dirs":["any","any","any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Successfully started partition reassignments for hadoop-0,hadoop-1,hadoop-2,hadoop-3

2.5 验证副本存储计划 ​

可以中途查看负载均衡机构执行进度

sh
[jack@hadoop105 kafka-3.6.1]$ ./bin/kafka-reassign-partitions.sh --bootstrap-server hadoop107:9092 --reassignment-json-file increase-replication-factor.json --verify
Status of partition reassignment:
Reassignment of partition hadoop-0 is completed.
Reassignment of partition hadoop-1 is still in progress.
Reassignment of partition hadoop-2 is still in progress.
Reassignment of partition hadoop-3 is still in progress.
jack@hadoop105 kafka-3.6.1]$ ./bin/kafka-reassign-partitions.sh --bootstrap-server hadoop107:9092 --reassignment-json-file increase-replication-factor.json --verify
Status of partition reassignment:
Reassignment of partition hadoop-0 is completed.
Reassignment of partition hadoop-1 is completed.
Reassignment of partition hadoop-2 is completed.
Reassignment of partition hadoop-3 is completed.

Clearing broker-level throttles on brokers 1,2,3,4
Clearing topic-level throttles on topic hadoop

3. 退役旧节点 ​

先按照退役一台节点，生成执行计划，然后按照服役时操作流程执行负载均衡。

3.1 创建一个要均衡的主题 ​

sh
[jack@hadoop105 kafka-3.6.1]$ vim topics-to-move.json
 {
    "topics": [
        {"topic": "hadoop"}
    ],
    "version": 1
}

3.2 创建执行计划 ​

sh
[jack@hadoop105 kafka-3.6.1]$ ./bin/kafka-reassign-partitions.sh --bootstrap-server hadoop106:9092 --topics-to-move-json-file topics-to-move.json --broker-list "1,2,3" --generate
Current partition replica assignment
{"version":1,"partitions":[{"topic":"hadoop","partition":0,"replicas":[1,2,3],"log_dirs":["any","any","any"]},{"topic":"hadoop","partition":1,"replicas":[2,3,4],"log_dirs":["any","any","any"]},{"topic":"hadoop","partition":2,"replicas":[3,4,1],"log_dirs":["any","any","any"]},{"topic":"hadoop","partition":3,"replicas":[4,1,2],"log_dirs":["any","any","any"]}]}

Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"hadoop","partition":0,"replicas":[1,2,3],"log_dirs":["any","any","any"]},{"topic":"hadoop","partition":1,"replicas":[2,3,1],"log_dirs":["any","any","any"]},{"topic":"hadoop","partition":2,"replicas":[3,1,2],"log_dirs":["any","any","any"]},{"topic":"hadoop","partition":3,"replicas":[1,3,2],"log_dirs":["any","any","any"]}]}

3.3 创建副本存储计划 ​

所有副本存储在 broker1、broker2、broker3中

sh
[jack@hadoop105 kafka-3.6.1]$ vi increase-replication-factor.json
{"version":1,"partitions":[{"topic":"hadoop","partition":0,"replicas":[1,2,3],"log_dirs":["any","any","any"]},{"topic":"hadoop","partition":1,"replicas":[2,3,1],"log_dirs":["any","any","any"]},{"topic":"hadoop","partition":2,"replicas":[3,1,2],"log_dirs":["any","any","any"]},{"topic":"hadoop","partition":3,"replicas":[1,3,2],"log_dirs":["any","any","any"]}]}

3.4 执行副本存储计划 ​

sh
[jack@hadoop105 kafka-3.6.1]$ ./bin/kafka-reassign-partitions.sh --bootstrap-server hadoop107:9092 --reassignment-json-file increase-replication-factor.json --execute
Current partition replica assignment

{"version":1,"partitions":[{"topic":"hadoop","partition":0,"replicas":[1,2,3],"log_dirs":["any","any","any"]},{"topic":"hadoop","partition":1,"replicas":[2,3,4],"log_dirs":["any","any","any"]},{"topic":"hadoop","partition":2,"replicas":[3,4,1],"log_dirs":["any","any","any"]},{"topic":"hadoop","partition":3,"replicas":[4,1,2],"log_dirs":["any","any","any"]}]}

Save this to use as the --reassignment-json-file option during rollback
Successfully started partition reassignments for hadoop-0,hadoop-1,hadoop-2,hadoop-3

3.5 验证副本存储计划 ​

sh
[jack@hadoop107 kafka-3.6.1]$ ./bin/kafka-reassign-partitions.sh --bootstrap-server hadoop107:9092 --reassignment-json-file increase-replication-factor.json --verify
Status of partition reassignment:
Reassignment of partition hadoop-0 is completed.
Reassignment of partition hadoop-1 is completed.
Reassignment of partition hadoop-2 is completed.
Reassignment of partition hadoop-3 is completed.

Clearing broker-level throttles on brokers 1,2,3,4
Clearing topic-level throttles on topic hadoop
[jack@hadoop107 kafka-3.6.1]$ ./bin/kafka-topics.sh --bootstrap-server hadoop106:9092 --describe --topic hadoop
Topic: hadoop	TopicId: 7K55cLp0RjSJZtZ8xIiM5Q	PartitionCount: 4	ReplicationFactor: 3	Configs: 
	Topic: hadoop	Partition: 0	Leader: 1	Replicas: 1,2,3	Isr: 1,3,2
	Topic: hadoop	Partition: 1	Leader: 2	Replicas: 2,3,1	Isr: 3,2,1
	Topic: hadoop	Partition: 2	Leader: 1	Replicas: 3,1,2	Isr: 1,3,2
	Topic: hadoop	Partition: 3	Leader: 1	Replicas: 1,3,2	Isr: 1,2,3

3.6 执行停止命令 ​

在hadoop108上执行停止命令即可。

sh
[jack@hadoop108 kafka]$ bin/kafka-server-stop.sh