13518219792
提示：一直在用kafka但是对它的性能一直没有啥认识，趁着这次优化算法中台的算法效率问题，梳理一下这块。
创新互联建站长期为上千客户提供的网站建设服务，团队从业经验10年，关注不同地域、不同群体，并针对不同对象提供差异化的产品和服务；打造开放共赢平台，与合作伙伴共同营造健康的互联网生态环境。为崇川企业提供专业的成都做网站、网站制作、成都外贸网站建设，崇川网站改版等技术服务。拥有十余年丰富建站经验和众多成功案例,为您定制开发。 文章目录
- 1.硬件资源
- 2.测试
- 3.总结
kafka是个消息中间件，现在我们要搞清楚它的性能，离开硬件资源配置说性能都是xjb整。
cpu:8h memery:16G disk:1.5T
cpu 查看资源
cat /proc/cpuinfo | grep name | cut -f2 -d: | uniq -c #查看cpu型号8 Intel® Xeon® Gold 5218 CPU @ 2.30GHz
硬盘信息
安装工具smartmontoolsyum -y install smartmontools
查看磁盘型号：smartctl --all /dev/sda[root@localhost bin]# smartctl --all /dev/sda
smartctl 7.0 2018-12-30 r4883 [x86_64-linux-3.10.0-957.el7.x86_64] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Vendor: VMware
Product: Virtual disk
Revision: 2.0
Compliance: SPC-4
User Capacity: 1,649,267,441,664 bytes [1.64 TB]
Logical block size: 512 bytes
LU is fully provisioned
Rotation Rate: 15000 rpm
Device type: disk
Local Time is: Wed Dec 28 19:27:45 2022 CST
SMART support is: Unavailable - device lacks SMART capability.
=== START OF READ SMART DATA SECTION ===
Current Drive Temperature: 0 C
Drive Trip Temperature: 0 C
Error Counter logging not supported
Device does not support Self Test logging命令指引看这里:https://blog.51cto.com/u_15127621/4055708
2.测试测试脚本：
kafka-producer-perf-test --topic test --num-records 100 --record-size 1 --throughput 100 --producer-props bootstrap.servers=localhost:9092
localhost 不好使的话，换成127.0.0.1 或者 实际ip， 和config的配置有关。
kafka 被运维装到了这里，里面的shell脚本如下面：
[root@localhost bin]# pwd
/usr/local/kafka/bin
[root@localhost bin]# ls -alh
总用量 464K
drwxr-xr-x. 3 root root 4.0K 10月 27 08:53 .
drwxr-xr-x. 8 root root 117 12月 19 11:04 ..
-rwxr-xr-x. 1 root root 1.4K 9月 14 2021 connect-distributed.sh
-rwxr-xr-x. 1 root root 1.4K 9月 14 2021 connect-mirror-maker.sh
-rwxr-xr-x. 1 root root 1.4K 9月 14 2021 connect-standalone.sh
-rwxr-xr-x. 1 root root 861 9月 14 2021 kafka-acls.sh
-rwxr-xr-x. 1 root root 873 9月 14 2021 kafka-broker-api-versions.sh
-rwxr-xr-x. 1 root root 860 9月 14 2021 kafka-cluster.sh
-rwxr-xr-x. 1 root root 864 9月 14 2021 kafka-configs.sh
-rwxr-xr-x. 1 root root 945 9月 14 2021 kafka-console-consumer.sh
-rwxr-xr-x. 1 root root 944 9月 14 2021 kafka-console-producer.sh
-rwxr-xr-x. 1 root root 871 9月 14 2021 kafka-consumer-groups.sh
-rwxr-xr-x. 1 root root 948 9月 14 2021 kafka-consumer-perf-test.sh
-rwxr-xr-x. 1 root root 871 9月 14 2021 kafka-delegation-tokens.sh
-rwxr-xr-x. 1 root root 869 9月 14 2021 kafka-delete-records.sh
-rwxr-xr-x. 1 root root 866 9月 14 2021 kafka-dump-log.sh
-rwxr-xr-x. 1 root root 863 9月 14 2021 kafka-features.sh
-rwxr-xr-x. 1 root root 870 9月 14 2021 kafka-leader-election.sh
-rwxr-xr-x. 1 root root 863 9月 14 2021 kafka-log-dirs.sh
-rwxr-xr-x. 1 root root 873 9月 14 2021 kafka-metadata-shell.sh
-rwxr-xr-x. 1 root root 862 9月 14 2021 kafka-mirror-maker.sh
-rwxr-xr-x. 1 root root 886 9月 14 2021 kafka-preferred-replica-election.sh
-rwxr-xr-x. 1 root root 959 9月 14 2021 kafka-producer-perf-test.sh
-rwxr-xr-x. 1 root root 874 9月 14 2021 kafka-reassign-partitions.sh
-rwxr-xr-x. 1 root root 874 9月 14 2021 kafka-replica-verification.sh
-rwxr-xr-x. 1 root root 11K 9月 14 2021 kafka-run-class.sh
-rwxr-xr-x. 1 root root 1.4K 9月 14 2021 kafka-server-start.sh
-rwxr-xr-x. 1 root root 1.4K 9月 14 2021 kafka-server-stop.sh
-rwxr-xr-x. 1 root root 860 9月 14 2021 kafka-storage.sh
-rwxr-xr-x. 1 root root 945 9月 14 2021 kafka-streams-application-reset.sh
-rwxr-xr-x. 1 root root 863 9月 14 2021 kafka-topics.sh
-rwxr-xr-x. 1 root root 958 9月 14 2021 kafka-verifiable-consumer.sh
-rwxr-xr-x. 1 root root 958 9月 14 2021 kafka-verifiable-producer.sh
-rw-------. 1 root root 304K 12月 28 18:56 nohup.out
-rwxr-xr-x. 1 root root 1.7K 9月 14 2021 trogdor.sh
drwxr-xr-x. 2 root root 4.0K 9月 14 2021 windows
-rwxr-xr-x. 1 root root 867 9月 14 2021 zookeeper-security-migration.sh
-rwxr-xr-x. 1 root root 1.4K 9月 14 2021 zookeeper-server-start.sh
-rwxr-xr-x. 1 root root 1.4K 9月 14 2021 zookeeper-server-stop.sh
-rwxr-xr-x. 1 root root 1019 9月 14 2021 zookeeper-shell.sh测试命令 sh kafka-producer-perf-test.sh --topic test1 --num-records 100 --record-size 1 --throughput 100 --producer-props bootstrap.servers=10.10.90.210:9092
[root@localhost bin]# sh kafka-producer-perf-test.sh --topic test1 --num-records 100 --record-size 1 --throughput 100 --producer-props bootstrap.servers=10.10.90.210:9092
100 records sent, 99.601594 records/sec (0.00 MB/sec), 8.77 ms avg latency, 341.00 ms max latency, 1 ms 50th, 14 ms 95th, 341 ms 99th, 341 ms 99.9th.num-records，消息数量
record-size，单条消息的大小，单位为字节
throughput，多少秒一条消息，-1为没有间歇
[root@localhost bin]# sh kafka-producer-perf-test.sh --topic test1 --num-records 100 --record-size 1 --throughput -1 --producer-props bootstrap.servers=10.10.90.210:9092
100 records sent, 267.379679 records/sec (0.00 MB/sec), 18.74 ms avg latency, 367.00 ms max latency, 15 ms 50th, 18 ms 95th, 367 ms 99th, 367 ms 99.9th.可以看到单点kafka， 消息没有间歇的前提下 1kb 一条数据，可以发 267条/秒
[root@localhost bin]# sh kafka-producer-perf-test.sh --topic test1 --num-records 100 --record-size 1 --throughput 100 --producer-props bootstrap.servers=10.10.90.210:9092
100 records sent, 99.601594 records/sec (0.00 MB/sec), 8.77 ms avg latency, 341.00 ms max latency, 1 ms 50th, 14 ms 95th, 341 ms 99th, 341 ms 99.9th.
[root@localhost bin]# sh kafka-producer-perf-test.sh --topic test1 --num-records 100 --record-size 1 --throughput -1 --producer-props bootstrap.servers=10.10.90.210:9092
100 records sent, 267.379679 records/sec (0.00 MB/sec), 18.74 ms avg latency, 367.00 ms max latency, 15 ms 50th, 18 ms 95th, 367 ms 99th, 367 ms 99.9th.
[root@localhost bin]# sh kafka-producer-perf-test.sh --topic test1 --num-records 100 --record-size 10 --throughput -1 --producer-props bootstrap.servers=10.10.90.210:9092
100 records sent, 280.112045 records/sec (0.00 MB/sec), 15.42 ms avg latency, 349.00 ms max latency, 12 ms 50th, 14 ms 95th, 349 ms 99th, 349 ms 99.9th.
[root@localhost bin]# sh kafka-producer-perf-test.sh --topic test1 --num-records 100 --record-size 10 --throughput -1 --producer-props bootstrap.servers=10.10.90.210:9092
100 records sent, 267.379679 records/sec (0.00 MB/sec), 16.51 ms avg latency, 367.00 ms max latency, 13 ms 50th, 15 ms 95th, 367 ms 99th, 367 ms 99.9th.
[root@localhost bin]# sh kafka-producer-perf-test.sh --topic test1 --num-records 100 --record-size 10 --throughput -1 --producer-props bootstrap.servers=10.10.90.210:9092
100 records sent, 285.714286 records/sec (0.00 MB/sec), 17.30 ms avg latency, 344.00 ms max latency, 14 ms 50th, 16 ms 95th, 344 ms 99th, 344 ms 99.9th.
[root@localhost bin]# sh kafka-producer-perf-test.sh --topic test1 --num-records 100 --record-size 100 --throughput -1 --producer-props bootstrap.servers=10.10.90.210:9092
100 records sent, 275.482094 records/sec (0.03 MB/sec), 16.33 ms avg latency, 356.00 ms max latency, 13 ms 50th, 15 ms 95th, 356 ms 99th, 356 ms 99.9th.
[root@localhost bin]# sh kafka-producer-perf-test.sh --topic test1 --num-records 100 --record-size 100 --throughput -1 --producer-props bootstrap.servers=10.10.90.210:9092
100 records sent, 264.550265 records/sec (0.03 MB/sec), 15.85 ms avg latency, 371.00 ms max latency, 12 ms 50th, 14 ms 95th, 371 ms 99th, 371 ms 99.9th.
[root@localhost bin]# sh kafka-producer-perf-test.sh --topic test1 --num-records 100 --record-size 1000 --throughput -1 --producer-props bootstrap.servers=10.10.90.210:9092
100 records sent, 268.096515 records/sec (0.26 MB/sec), 20.13 ms avg latency, 360.00 ms max latency, 17 ms 50th, 18 ms 95th, 360 ms 99th, 360 ms 99.9th.
[root@localhost bin]# sh kafka-producer-perf-test.sh --topic test1 --num-records 100 --record-size 10000 --throughput -1 --producer-props bootstrap.servers=10.10.90.210:9092
100 records sent, 246.305419 records/sec (2.35 MB/sec), 45.08 ms avg latency, 344.00 ms max latency, 44 ms 50th, 60 ms 95th, 344 ms 99th, 344 ms 99.9th.
[root@localhost bin]# sh kafka-producer-perf-test.sh --topic test1 --num-records 100 --record-size 100000 --throughput -1 --producer-props bootstrap.servers=10.10.90.210:9092
100 records sent, 229.885057 records/sec (21.92 MB/sec), 60.76 ms avg latency, 353.00 ms max latency, 58 ms 50th, 79 ms 95th, 353 ms 99th, 353 ms 99.9th.
[root@localhost bin]# sh kafka-producer-perf-test.sh --topic test1 --num-records 100 --record-size 1000000 --throughput -1 --producer-props bootstrap.servers=10.10.90.210:9092
100 records sent, 138.121547 records/sec (131.72 MB/sec), 115.93 ms avg latency, 353.00 ms max latency, 122 ms 50th, 137 ms 95th, 353 ms 99th, 353 ms 99.9th.
[root@localhost bin]# sh kafka-producer-perf-test.sh --topic test1 --num-records 100 --record-size 1000000 --throughput -1 --producer-props bootstrap.servers=10.10.90.210:9092
100 records sent, 142.857143 records/sec (136.24 MB/sec), 105.84 ms avg latency, 339.00 ms max latency, 121 ms 50th, 141 ms 95th, 339 ms 99th, 339 ms 99.9th.从以上脚本可以看出，仅仅是修改了 record-size，也就是消息体的大小，当1,10,100,1000,10000kb的时候，都是250records/s 以上，max latency延时则从10毫秒到了几十毫秒。
当增加到100000kb （100MB），1000000kb（1G）的时候，显著下降为100+records/s 延时也增加到100毫秒以上，但正常情况下也没有人会发100MB，或者 1G每条消息的数据吧。
2018年我在搞直播系统的时候，当时的答题回传答案用的kafka，那时候看的二手技术博客说单条消息大于10kb的时候，kafka性能会显著下降，现在看不是那么回事。眼见为实，到底咋样，拉出来溜溜！
3.总结单体kafka， 8h 16G 的服务器，1kb每条消息， kafka的承载量是接近300条消息/s，延时在10毫秒，
随之，单体消息的数据量增加，延时在增加，但1kb----10MB的数据，消息承载量没有显著下降从280–240，平均延时延时从10毫秒到50毫秒
你是否还在寻找稳定的海外服务器提供商？创新互联www.cdcxhl.cn海外机房具备T级流量清洗系统配攻击溯源，准确流量调度确保服务器高可用性，企业级服务器适合批量采购，新人活动首月15元起，快前往官网查看详情吧
