杰霸霸博客

主题

Kafka集群压力测试 ​

使用Kafka官方自带的脚本，对Kafka进行压测：

  • 生产者压测脚本：kafka-producer-perf-test.sh
  • 消费者压测脚本：kafka-consumer-perf-test.sh

1. Kafka Producer压力测试 ​

1.1 创建一个Topic叫test ，设置为3个分区3个副本 ​

sh
[jack@hadoop105 kafka-3.6.1]$ ./bin/kafka-topics.sh --bootstrap-server 192.168.101.105:9092 --create --topic test --replication-factor 3 --partitions 3
Created topic test.

1.2 使用kafka-producer-perf-test.sh测试test主题 ​

sh
[jack@hadoop105 kafka-3.6.1]$ ./bin/kafka-producer-perf-test.sh --topic test --record-size 1024 --num-records 1000000 --throughput 10000 --producer-props bootstrap.servers=192.168.101.105:9092,192.168.101.106:9092,192.168.101.107:9092 batch.size=16384 linger.ms=0

参数说明：

  • record-size是一条信息有多大，单位是字节，本次测试设置为1k。
  • num-records是总共发送多少条信息，本次测试设置为100万条。
  • throughput是每秒多少条信息，设成-1，表示不限流，尽可能快的生产数据，可测出生产者最大吞吐量。本次实验设置为每秒钟1万条。
  • producer-props后面可以配置生产者相关参数，batch.size 配置为16k。
    输出结果：
sh
[jack@hadoop105 kafka-3.6.1]$ ./bin/kafka-producer-perf-test.sh --topic test --record-size 1024 --num-records 1000000 --throughput 10000 --producer-props bootstrap.servers=192.168.101.105:9092,192.168.101.106:9092,192.168.101.107:9092 batch.size=16384 linger.ms=0 
12556 records sent, 2508.2 records/sec (2.45 MB/sec), 1929.7 ms avg latency, 3268.0 ms max latency.
29475 records sent, 5895.0 records/sec (5.76 MB/sec), 4615.4 ms avg latency, 5970.0 ms max latency.
34020 records sent, 6801.3 records/sec (6.64 MB/sec), 4556.4 ms avg latency, 5762.0 ms max latency.
34575 records sent, 6909.5 records/sec (6.75 MB/sec), 4647.2 ms avg latency, 5155.0 ms max latency.
29835 records sent, 5938.5 records/sec (5.80 MB/sec), 4637.2 ms avg latency, 5518.0 ms max latency.
36300 records sent, 7257.1 records/sec (7.09 MB/sec), 4819.1 ms avg latency, 5639.0 ms max latency.
40350 records sent, 8068.4 records/sec (7.88 MB/sec), 3958.3 ms avg latency, 4445.0 ms max latency.
40755 records sent, 8149.4 records/sec (7.96 MB/sec), 3648.7 ms avg latency, 4055.0 ms max latency.
33390 records sent, 6676.7 records/sec (6.52 MB/sec), 4331.7 ms avg latency, 4962.0 ms max latency.
49950 records sent, 9986.0 records/sec (9.75 MB/sec), 3566.2 ms avg latency, 4936.0 ms max latency.
50385 records sent, 10064.9 records/sec (9.83 MB/sec), 3091.5 ms avg latency, 3399.0 ms max latency.
35490 records sent, 7096.6 records/sec (6.93 MB/sec), 4023.6 ms avg latency, 4882.0 ms max latency.
48495 records sent, 9695.1 records/sec (9.47 MB/sec), 3337.7 ms avg latency, 4653.0 ms max latency.
45960 records sent, 9177.3 records/sec (8.96 MB/sec), 3050.2 ms avg latency, 3705.0 ms max latency.
38985 records sent, 7793.9 records/sec (7.61 MB/sec), 4111.9 ms avg latency, 4906.0 ms max latency.
47670 records sent, 9530.2 records/sec (9.31 MB/sec), 3329.9 ms avg latency, 4078.0 ms max latency.
43995 records sent, 8795.5 records/sec (8.59 MB/sec), 3323.2 ms avg latency, 3706.0 ms max latency.
40605 records sent, 8088.6 records/sec (7.90 MB/sec), 4008.2 ms avg latency, 4745.0 ms max latency.
50370 records sent, 10065.9 records/sec (9.83 MB/sec), 3099.4 ms avg latency, 3794.0 ms max latency.
44160 records sent, 8828.5 records/sec (8.62 MB/sec), 3158.9 ms avg latency, 3889.0 ms max latency.
41985 records sent, 8397.0 records/sec (8.20 MB/sec), 4041.7 ms avg latency, 4769.0 ms max latency.
47370 records sent, 9474.0 records/sec (9.25 MB/sec), 3167.7 ms avg latency, 3637.0 ms max latency.
43230 records sent, 8640.8 records/sec (8.44 MB/sec), 3166.1 ms avg latency, 3907.0 ms max latency.
41760 records sent, 8352.0 records/sec (8.16 MB/sec), 4121.2 ms avg latency, 4928.0 ms max latency.
1000000 records sent, 8054.512944 records/sec (7.87 MB/sec), 3693.53 ms avg latency, 5970.00 ms max latency, 3519 ms 50th, 4902 ms 95th, 5355 ms 99th, 5852 ms 99.9th.

可以看到当前Kafka集群生产者默认吞吐量为7.87 MB/s

1.3 调整 batch.size大小 ​

  1. 清空之前test主题数据
sh
[jack@hadoop105 kafka-3.6.1]$ cat delete-topic-data.json 
{
  "partitions": [
    {
      "topic": "test",
      "partition": 1,
      "offset": -1
    }
  ],
  "version": 1
}
[jack@hadoop105 kafka-3.6.1]bin/kafka-delete-records.sh --bootstrap-server localhost:9092 --offset-json-file delete-topic-data.json 
Executing records delete operation
Records delete operation completed:
partition: test-1	low_watermark: 172800
  1. 提高batch.size设置为32k
sh
[jack@hadoop105 kafka-3.6.1]$./bin/kafka-producer-perf-test.sh --topic test --record-size 1024 --num-records 1000000 --throughput 10000 --producer-props bootstrap.servers=192.168.101.105:9092,192.168.101.106:9092,192.168.101.107:9092 batch.size=32768 linger.ms=0

输出结果：

sh
[jack@hadoop105 kafka-3.6.1]$./bin/kafka-producer-perf-test.sh --topic test --record-size 1024 --num-records 1000000 --throughput 10000 --producer-props bootstrap.servers=192.168.101.105:9092,192.168.101.106:9092,192.168.101.107:9092 batch.size=32768 linger.ms=0 
40704 records sent, 8118.1 records/sec (7.93 MB/sec), 1060.1 ms avg latency, 1640.0 ms max latency.
59299 records sent, 11826.7 records/sec (11.55 MB/sec), 319.7 ms avg latency, 1035.0 ms max latency.
50055 records sent, 9995.0 records/sec (9.76 MB/sec), 52.7 ms avg latency, 223.0 ms max latency.
50524 records sent, 10092.7 records/sec (9.86 MB/sec), 44.8 ms avg latency, 274.0 ms max latency.
49002 records sent, 9780.8 records/sec (9.55 MB/sec), 29.8 ms avg latency, 180.0 ms max latency.
49776 records sent, 9790.7 records/sec (9.56 MB/sec), 39.5 ms avg latency, 200.0 ms max latency.
50807 records sent, 10141.1 records/sec (9.90 MB/sec), 51.6 ms avg latency, 258.0 ms max latency.
51093 records sent, 10218.6 records/sec (9.98 MB/sec), 22.5 ms avg latency, 183.0 ms max latency.
50167 records sent, 10021.4 records/sec (9.79 MB/sec), 29.9 ms avg latency, 242.0 ms max latency.
50295 records sent, 10042.9 records/sec (9.81 MB/sec), 31.1 ms avg latency, 201.0 ms max latency.
50140 records sent, 10028.0 records/sec (9.79 MB/sec), 20.3 ms avg latency, 156.0 ms max latency.
49657 records sent, 9925.4 records/sec (9.69 MB/sec), 37.6 ms avg latency, 202.0 ms max latency.
50303 records sent, 10058.6 records/sec (9.82 MB/sec), 24.4 ms avg latency, 146.0 ms max latency.
50050 records sent, 10008.0 records/sec (9.77 MB/sec), 14.2 ms avg latency, 117.0 ms max latency.
49758 records sent, 9949.6 records/sec (9.72 MB/sec), 40.2 ms avg latency, 188.0 ms max latency.
50312 records sent, 10050.3 records/sec (9.81 MB/sec), 18.5 ms avg latency, 127.0 ms max latency.
50090 records sent, 10012.0 records/sec (9.78 MB/sec), 13.5 ms avg latency, 120.0 ms max latency.
50003 records sent, 9998.6 records/sec (9.76 MB/sec), 50.4 ms avg latency, 242.0 ms max latency.
50037 records sent, 10005.4 records/sec (9.77 MB/sec), 16.2 ms avg latency, 119.0 ms max latency.
1000000 records sent, 9989.111868 records/sec (9.75 MB/sec), 89.95 ms avg latency, 1640.00 ms max latency, 17 ms 50th, 624 ms 95th, 1274 ms 99th, 1507 ms 99.9th.

可以看到当前Kafka集群生产者吞吐量为9.75 MB/s

1.4 调整linger.ms时间 ​

  1. 清空之前test主题数据
sh
[jack@hadoop105 kafka-3.6.1]$ bin/kafka-delete-records.sh   --bootstrap-server localhost:9092   --offset-json-file delete-topic-data.json 
Executing records delete operation
Records delete operation completed:
partition: test-1	low_watermark: 1402208
  1. 增大linger.ms时间为50ms
sh
[jack@hadoop105 kafka-3.6.1]$ ./bin/kafka-producer-perf-test.sh --topic test --record-size 1024 --num-records 1000000 --throughput 10000 --producer-props bootstrap.servers=192.168.101.105:9092,192.168.101.106:9092,192.168.101.107:9092 batch.size=32768 linger.ms=50

输出结果：

sh
[jack@hadoop105 kafka-3.6.1]$ ./bin/kafka-producer-perf-test.sh --topic test --record-size 1024 --num-records 1000000 --throughput 10000 --producer-props bootstrap.servers=192.168.101.105:9092,192.168.101.106:9092,192.168.101.107:9092 batch.size=32768 linger.ms=50 
39123 records sent, 7804.3 records/sec (7.62 MB/sec), 1065.5 ms avg latency, 1508.0 ms max latency.
57567 records sent, 11513.4 records/sec (11.24 MB/sec), 857.5 ms avg latency, 1398.0 ms max latency.
52545 records sent, 10502.7 records/sec (10.26 MB/sec), 125.0 ms avg latency, 562.0 ms max latency.
50995 records sent, 10199.0 records/sec (9.96 MB/sec), 33.7 ms avg latency, 197.0 ms max latency.
49600 records sent, 9918.0 records/sec (9.69 MB/sec), 27.0 ms avg latency, 149.0 ms max latency.
49312 records sent, 9805.5 records/sec (9.58 MB/sec), 38.3 ms avg latency, 203.0 ms max latency.
51708 records sent, 10335.4 records/sec (10.09 MB/sec), 30.7 ms avg latency, 197.0 ms max latency.
49972 records sent, 9992.4 records/sec (9.76 MB/sec), 14.6 ms avg latency, 139.0 ms max latency.
49693 records sent, 9920.7 records/sec (9.69 MB/sec), 35.4 ms avg latency, 154.0 ms max latency.
50468 records sent, 10093.6 records/sec (9.86 MB/sec), 19.9 ms avg latency, 185.0 ms max latency.
49972 records sent, 9988.4 records/sec (9.75 MB/sec), 14.1 ms avg latency, 160.0 ms max latency.
50096 records sent, 10019.2 records/sec (9.78 MB/sec), 46.0 ms avg latency, 238.0 ms max latency.
50003 records sent, 10000.6 records/sec (9.77 MB/sec), 16.0 ms avg latency, 184.0 ms max latency.
49786 records sent, 9935.3 records/sec (9.70 MB/sec), 14.1 ms avg latency, 132.0 ms max latency.
50344 records sent, 10064.8 records/sec (9.83 MB/sec), 45.1 ms avg latency, 274.0 ms max latency.
49910 records sent, 9982.0 records/sec (9.75 MB/sec), 14.7 ms avg latency, 185.0 ms max latency.
50127 records sent, 10021.4 records/sec (9.79 MB/sec), 15.6 ms avg latency, 206.0 ms max latency.
50065 records sent, 10009.0 records/sec (9.77 MB/sec), 58.6 ms avg latency, 305.0 ms max latency.
50003 records sent, 9998.6 records/sec (9.76 MB/sec), 13.0 ms avg latency, 130.0 ms max latency.
1000000 records sent, 9977.152321 records/sec (9.74 MB/sec), 120.64 ms avg latency, 1508.00 ms max latency, 14 ms 50th, 969 ms 95th, 1320 ms 99th, 1462 ms 99.9th

可以看到当前Kafka集群生产者吞吐量为9.74 MB/s, 主要是因为压测消息累计大小都快速增大，还没到linger.ms的阈值，而率先已到达batch.size，导致linger.ms没有起作用。

1.5 调整压缩方式 ​

  1. 清空之前test主题数据
sh
[jack@hadoop105 kafka-3.6.1]$ bin/kafka-delete-records.sh   --bootstrap-server localhost:9092   --offset-json-file delete-topic-data.json 
Executing records delete operation
Records delete operation completed:
partition: test-1	low_watermark: 1402208
  1. 默认的压缩方式是none。本次compression.type设置为snappy。
sh
[jack@hadoop105 kafka-3.6.1]$ ./bin/kafka-producer-perf-test.sh --topic test --record-size 1024 --num-records 1000000 --throughput 10000 --producer-props bootstrap.servers=192.168.101.105:9092,192.168.101.106:9092,192.168.101.107:9092 batch.size=32768 linger.ms=50 compression.type=snappy
27387 records sent, 5474.1 records/sec (5.35 MB/sec), 1268.4 ms avg latency, 2544.0 ms max latency.
53885 records sent, 10725.5 records/sec (10.47 MB/sec), 2124.6 ms avg latency, 2644.0 ms max latency.
60083 records sent, 11978.3 records/sec (11.70 MB/sec), 1475.9 ms avg latency, 2048.0 ms max latency.
58725 records sent, 11728.6 records/sec (11.45 MB/sec), 245.6 ms avg latency, 956.0 ms max latency.
49412 records sent, 9872.5 records/sec (9.64 MB/sec), 33.7 ms avg latency, 187.0 ms max latency.
50636 records sent, 10115.1 records/sec (9.88 MB/sec), 48.2 ms avg latency, 215.0 ms max latency.
50463 records sent, 10090.6 records/sec (9.85 MB/sec), 33.3 ms avg latency, 187.0 ms max latency.
50069 records sent, 10001.8 records/sec (9.77 MB/sec), 20.9 ms avg latency, 149.0 ms max latency.
48849 records sent, 9763.9 records/sec (9.54 MB/sec), 30.2 ms avg latency, 175.0 ms max latency.
51248 records sent, 10247.6 records/sec (10.01 MB/sec), 28.2 ms avg latency, 163.0 ms max latency.
49989 records sent, 9995.8 records/sec (9.76 MB/sec), 15.8 ms avg latency, 128.0 ms max latency.
49765 records sent, 9949.0 records/sec (9.72 MB/sec), 33.6 ms avg latency, 160.0 ms max latency.
50265 records sent, 10053.0 records/sec (9.82 MB/sec), 27.4 ms avg latency, 144.0 ms max latency.
50035 records sent, 10003.0 records/sec (9.77 MB/sec), 17.1 ms avg latency, 142.0 ms max latency.
49852 records sent, 9966.4 records/sec (9.73 MB/sec), 32.9 ms avg latency, 164.0 ms max latency.
49625 records sent, 9925.0 records/sec (9.69 MB/sec), 24.4 ms avg latency, 187.0 ms max latency.
50555 records sent, 10109.0 records/sec (9.87 MB/sec), 21.8 ms avg latency, 173.0 ms max latency.
49800 records sent, 9956.0 records/sec (9.72 MB/sec), 41.5 ms avg latency, 204.0 ms max latency.
50282 records sent, 10052.4 records/sec (9.82 MB/sec), 19.2 ms avg latency, 130.0 ms max latency.
1000000 records sent, 9985.820135 records/sec (9.75 MB/sec), 274.76 ms avg latency, 2644.00 ms max latency, 21 ms 50th, 1977 ms 95th, 2266 ms 99th, 2381 ms 99.9th

可以看到当前Kafka集群生产者吞吐量为9.75 MB/s 3. 清除数据，compression.type设置为gzip

sh
[jack@hadoop105 kafka-3.6.1]$ ./bin/kafka-producer-perf-test.sh --topic test --record-size 1024 --num-records 1000000 --throughput 10000 --producer-props bootstrap.servers=192.168.101.105:9092,192.168.101.106:9092,192.168.101.107:9092 batch.size=32768 linger.ms=50 compression.type=gzip
15500 records sent, 3086.4 records/sec (3.01 MB/sec), 308.6 ms avg latency, 1497.0 ms max latency.
39177 records sent, 7678.8 records/sec (7.50 MB/sec), 584.2 ms avg latency, 1527.0 ms max latency.
36382 records sent, 7270.6 records/sec (7.10 MB/sec), 2471.7 ms avg latency, 3782.0 ms max latency.
36467 records sent, 7252.8 records/sec (7.08 MB/sec), 4807.1 ms avg latency, 6283.0 ms max latency.
54922 records sent, 10942.8 records/sec (10.69 MB/sec), 4855.2 ms avg latency, 6446.0 ms max latency.
52196 records sent, 10180.6 records/sec (9.94 MB/sec), 4093.2 ms avg latency, 4787.0 ms max latency.
38104 records sent, 7611.7 records/sec (7.43 MB/sec), 5293.5 ms avg latency, 6623.0 ms max latency.
59626 records sent, 11749.0 records/sec (11.47 MB/sec), 4541.8 ms avg latency, 6383.0 ms max latency.
48933 records sent, 9664.8 records/sec (9.44 MB/sec), 4154.4 ms avg latency, 4993.0 ms max latency.
39420 records sent, 7762.9 records/sec (7.58 MB/sec), 5220.4 ms avg latency, 6100.0 ms max latency.
43868 records sent, 8756.1 records/sec (8.55 MB/sec), 5485.2 ms avg latency, 6248.0 ms max latency.
57286 records sent, 11432.0 records/sec (11.16 MB/sec), 4395.7 ms avg latency, 5618.0 ms max latency.
45290 records sent, 9054.4 records/sec (8.84 MB/sec), 4100.3 ms avg latency, 5252.0 ms max latency.
46259 records sent, 9180.2 records/sec (8.97 MB/sec), 4594.9 ms avg latency, 5592.0 ms max latency.
55161 records sent, 11016.8 records/sec (10.76 MB/sec), 4005.1 ms avg latency, 4531.0 ms max latency.
50216 records sent, 9902.6 records/sec (9.67 MB/sec), 4012.7 ms avg latency, 4933.0 ms max latency.
50887 records sent, 10151.0 records/sec (9.91 MB/sec), 5048.3 ms avg latency, 5961.0 ms max latency.
60970 records sent, 12191.6 records/sec (11.91 MB/sec), 3854.6 ms avg latency, 4765.0 ms max latency.
53191 records sent, 10638.2 records/sec (10.39 MB/sec), 3787.2 ms avg latency, 4661.0 ms max latency.
47442 records sent, 9445.0 records/sec (9.22 MB/sec), 4976.7 ms avg latency, 5614.0 ms max latency.
1000000 records sent, 9545.810344 records/sec (9.32 MB/sec), 4127.81 ms avg latency, 6623.00 ms max latency, 4204 ms 50th, 5758 ms 95th, 6155 ms 99th, 6555 ms 99.9th.

可以看到当前Kafka集群生产者吞吐量为9.75 MB/s

1.5 调整缓存大小 ​

  1. 清空之前test主题数据
sh
[jack@hadoop105 kafka-3.6.1]$ bin/kafka-delete-records.sh   --bootstrap-server localhost:9092   --offset-json-file delete-topic-data.json 
Executing records delete operation
Records delete operation completed:
partition: test-1	low_watermark: 2413655
  1. 默认生产者端缓存大小32m。本次buffer.memory设置为 64m。
sh
[jack@hadoop105 kafka-3.6.1]$ ./bin/kafka-producer-perf-test.sh --topic test --record-size 1024 --num-records 1000000 --throughput 10000 --producer-props bootstrap.servers=192.168.101.105:9092,192.168.101.106:9092,192.168.101.107:9092 batch.size=32768 linger.ms=50 compression.type=gzip  buffer.memory=67108864
11471 records sent, 2284.6 records/sec (2.23 MB/sec), 1470.1 ms avg latency, 2092.0 ms max latency.
32474 records sent, 6439.4 records/sec (6.29 MB/sec), 2307.0 ms avg latency, 3462.0 ms max latency.
42695 records sent, 8516.9 records/sec (8.32 MB/sec), 3607.2 ms avg latency, 4479.0 ms max latency.
30857 records sent, 6097.0 records/sec (5.95 MB/sec), 6358.0 ms avg latency, 7172.0 ms max latency.
47289 records sent, 9446.5 records/sec (9.23 MB/sec), 7806.6 ms avg latency, 8559.0 ms max latency.
50977 records sent, 10148.7 records/sec (9.91 MB/sec), 7706.4 ms avg latency, 8320.0 ms max latency.
32807 records sent, 6461.9 records/sec (6.31 MB/sec), 9843.5 ms avg latency, 10698.0 ms max latency.
46897 records sent, 9349.5 records/sec (9.13 MB/sec), 10762.1 ms avg latency, 11312.0 ms max latency.
53192 records sent, 10591.8 records/sec (10.34 MB/sec), 9421.6 ms avg latency, 11133.0 ms max latency.
49232 records sent, 9842.5 records/sec (9.61 MB/sec), 8929.6 ms avg latency, 9450.0 ms max latency.
40470 records sent, 8085.9 records/sec (7.90 MB/sec), 9137.3 ms avg latency, 10618.0 ms max latency.
40121 records sent, 8021.0 records/sec (7.83 MB/sec), 10019.6 ms avg latency, 11282.0 ms max latency.
46896 records sent, 9373.6 records/sec (9.15 MB/sec), 7722.9 ms avg latency, 9015.0 ms max latency.
44104 records sent, 8815.5 records/sec (8.61 MB/sec), 9538.0 ms avg latency, 10630.0 ms max latency.
55807 records sent, 11154.7 records/sec (10.89 MB/sec), 9707.8 ms avg latency, 10476.0 ms max latency.
56036 records sent, 11200.5 records/sec (10.94 MB/sec), 8037.3 ms avg latency, 9149.0 ms max latency.
40891 records sent, 8165.1 records/sec (7.97 MB/sec), 8953.6 ms avg latency, 9862.0 ms max latency.
59767 records sent, 11886.8 records/sec (11.61 MB/sec), 9418.6 ms avg latency, 10134.0 ms max latency.
59273 records sent, 11772.2 records/sec (11.50 MB/sec), 7837.3 ms avg latency, 9269.0 ms max latency.
40033 records sent, 7998.6 records/sec (7.81 MB/sec), 8471.7 ms avg latency, 9791.0 ms max latency.
88415 records sent, 17683.0 records/sec (17.27 MB/sec), 8232.3 ms avg latency, 9961.0 ms max latency.
1000000 records sent, 9360.578858 records/sec (9.14 MB/sec), 8117.87 ms avg latency, 11312.00 ms max latency, 8593 ms 50th, 10575 ms 95th, 11054 ms 99th, 11264 ms 99.9th.

可以看到当前Kafka集群生产者吞吐量为9.14 MB/s

2. Kafka Consumer压力测试 ​

2.1 修改config/consumer.properties 文件中的一次拉取条数为500 ​

sh
## 默认就是500
max.poll.records=500

2.2 消费100万条日志进行压测 ​

sh
[jack@hadoop105 kafka-3.6.1]$ ./bin/kafka-consumer-perf-test.sh --bootstrap-server hadoop105:9092,hadoop106:9092,hadoop107:9092 --topic test --messages 1000000 --consumer.config config/consumer.properties
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2024-03-19 14:42:54:449, 2024-03-19 14:43:12:153, 976.5811, 55.1616, 1000019, 56485.4835, 3877, 13827, 70.6286, 72323.6421

参数说明：

  • bootstrap-server 指定 Kafka 集群地址
  • topic 指定 topic 的名称
  • messages 总共要消费的消息个数。本次实验 100 万条。 可以看到当前消费速度(MB.sec这一列)为55.1616MB/s

2.3 修改拉取条数参数为2000 ​

sh
## 修改拉取条数参数为2000
[jack@hadoop105 kafka-3.6.1]$ vi config/consumer.properties
[jack@hadoop105 kafka-3.6.1]$ ./bin/kafka-consumer-perf-test.sh --bootstrap-server hadoop105:9092,hadoop106:9092,hadoop107:9092 --topic test --messages 1000000 --consumer.config config/consumer.properties
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2024-03-19 14:48:40:996, 2024-03-19 14:48:57:221, 976.5859, 60.1902, 1000024, 61634.7612, 3852, 12373, 78.9288, 80823.0825

可以看到当前消费速度(MB.sec这一列)为60.1902MB/s

2.4 调整fetch.max.bytes大小 ​

  1. 修改consumer.properties文件中的拉取一批数据大小100m,默认为50m
sh
[jack@hadoop105 kafka-3.6.1]$ vi config/consumer.properties
fetch.max.bytes=104857600
[jack@hadoop105 kafka-3.6.1]$ ./bin/kafka-consumer-perf-test.sh --bootstrap-server hadoop105:9092,hadoop106:9092,hadoop107:9092 --topic test --messages 1000000 --consumer.config config/consumer.properties
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2024-03-19 14:55:34:244, 2024-03-19 14:55:50:296, 976.6934, 60.8456, 1000134, 62305.8809, 4054, 11998, 81.4047, 83358.3931

可以看到当前消费速度(MB.sec这一列)为60.8456MB/s