ebpf的主从同步实现，QPS测试与内存池QPS测试。

2026-01-30 16:00:06 +00:00
parent 2bdb48d63d
commit fbdcff6878
23 changed files with 599 additions and 383 deletions
--- a/README.md
+++ b/README.md
@@ -3,14 +3,14 @@
 ## 需求
 1. ntyco需要作为kvstore的submodule,通过git clone一次下载。 **完成**。
 2. README需要包含编译步骤，测试方案与可行性，性能数据。  **完成**。
-3. 全量持久化保存数据集。 **BUG FIX，完成**。
+3. 全量持久化保存数据集。 **BUG FIX**。
 4. 持久化的性能数据。
 5. 特殊字符，可以解决redis的resp协议。 **完成**。
 6. 实现配置文件，把日志级别，端口ip，主从模式，持久化方案。 **完成**。
 7. 持久化落盘用io_uring，加载配置文件用mmap。   **完成**。
-8. 主从同步的性能，开启与关闭性能做到。
-9. 主从同步600w条,出现的coredump。
-10. 主从同步用ebpf实现。
+8. 主从同步的性能，开启与关闭性能做到5%?。
+9. 主从同步600w条,出现的coredump。     **完成**。
+10. 主从同步用ebpf实现。               **BUG FIX**。
 11. 内存池测试qps与虚拟内存，物理内存。
 12. 实现一个内存泄露检测组件。          **完成**。

@@ -34,6 +34,191 @@ make
 ```

 ## 测试
+### 测试1：性能测试
+测试条件：
+1. 不启用持久化。
+2. 不启用主从同步。
+2. pipline：
+    1. RSET 100w 条， p:i v:i -> +OK 
+    2. RGET 100w 条， p:i -> +v:i
+    3. RDEL 100w 条。 p:i -> +OK
+3. 重复 15 次.
+4. 本机发送请求。
+
+内存分配: malloc
+```bash
+lian@ubuntu:~/share/9.1-kvstore$ ./test-redis/testcase 192.168.10.129 8888 3
+Connected to 192.168.10.129:8888
+BATCH (N=3000000) --> time_used=3294 ms, qps=910746
+BATCH (N=3000000) --> time_used=3501 ms, qps=856898
+BATCH (N=3000000) --> time_used=3457 ms, qps=867804
+BATCH (N=3000000) --> time_used=3351 ms, qps=895255
+BATCH (N=3000000) --> time_used=3320 ms, qps=903614
+BATCH (N=3000000) --> time_used=3551 ms, qps=844832
+BATCH (N=3000000) --> time_used=3354 ms, qps=894454
+BATCH (N=3000000) --> time_used=3475 ms, qps=863309
+BATCH (N=3000000) --> time_used=3404 ms, qps=881316
+BATCH (N=3000000) --> time_used=3460 ms, qps=867052
+BATCH (N=3000000) --> time_used=3392 ms, qps=884433
+BATCH (N=3000000) --> time_used=3427 ms, qps=875401
+BATCH (N=3000000) --> time_used=3441 ms, qps=871839
+BATCH (N=3000000) --> time_used=3471 ms, qps=864304
+BATCH (N=3000000) --> time_used=3354 ms, qps=894454
+BATCH (N=3000000) --> time_used=3447 ms, qps=870322
+BATCH (N=3000000) --> time_used=3364 ms, qps=891795
+BATCH (N=3000000) --> time_used=3200 ms, qps=937500
+BATCH (N=3000000) --> time_used=3159 ms, qps=949667
+BATCH (N=3000000) --> time_used=3482 ms, qps=861573
+BATCH (N=3000000) --> time_used=3474 ms, qps=863557
+BATCH (N=3000000) --> time_used=3591 ms, qps=835421
+BATCH (N=3000000) --> time_used=3466 ms, qps=865551
+BATCH (N=3000000) --> time_used=3425 ms, qps=875912
+BATCH (N=3000000) --> time_used=3346 ms, qps=896592
+BATCH (N=3000000) --> time_used=3532 ms, qps=849377
+BATCH (N=3000000) --> time_used=3471 ms, qps=864304
+BATCH (N=3000000) --> time_used=3616 ms, qps=829646
+BATCH (N=3000000) --> time_used=3403 ms, qps=881575
+BATCH (N=3000000) --> time_used=3426 ms, qps=875656
+BATCH (N=3000000) --> time_used=3493 ms, qps=858860
+BATCH (N=3000000) --> time_used=3411 ms, qps=879507
+BATCH (N=3000000) --> time_used=3422 ms, qps=876680
+BATCH (N=3000000) --> time_used=3556 ms, qps=843644
+BATCH (N=3000000) --> time_used=3285 ms, qps=913242
+BATCH (N=3000000) --> time_used=3486 ms, qps=860585
+BATCH (N=3000000) --> time_used=3427 ms, qps=875401
+BATCH (N=3000000) --> time_used=3563 ms, qps=841987
+BATCH (N=3000000) --> time_used=3304 ms, qps=907990
+BATCH (N=3000000) --> time_used=3582 ms, qps=837520
+BATCH (N=3000000) --> time_used=3468 ms, qps=865051
+BATCH (N=3000000) --> time_used=3360 ms, qps=892857
+BATCH (N=3000000) --> time_used=3426 ms, qps=875656
+BATCH (N=3000000) --> time_used=3186 ms, qps=941619
+BATCH (N=3000000) --> time_used=3251 ms, qps=922792
+BATCH (N=3000000) --> time_used=3400 ms, qps=882352
+BATCH (N=3000000) --> time_used=3446 ms, qps=870574
+BATCH (N=3000000) --> time_used=3302 ms, qps=908540
+BATCH (N=3000000) --> time_used=3072 ms, qps=976562
+BATCH (N=3000000) --> time_used=3458 ms, qps=867553
+average qps:880462
+ALL TESTS PASSED.
+```
+
+内存分配: 自实现内存池
+```bash
+lian@ubuntu:~/share/9.1-kvstore$ ./test-redis/testcase 192.168.10.129 8888 3
+Connected to 192.168.10.129:8888
+BATCH (N=3000000) --> time_used=3552 ms, qps=844594
+BATCH (N=3000000) --> time_used=3404 ms, qps=881316
+BATCH (N=3000000) --> time_used=3083 ms, qps=973078
+BATCH (N=3000000) --> time_used=3315 ms, qps=904977
+BATCH (N=3000000) --> time_used=3462 ms, qps=866551
+BATCH (N=3000000) --> time_used=3334 ms, qps=899820
+BATCH (N=3000000) --> time_used=3358 ms, qps=893388
+BATCH (N=3000000) --> time_used=3423 ms, qps=876424
+BATCH (N=3000000) --> time_used=3319 ms, qps=903886
+BATCH (N=3000000) --> time_used=3327 ms, qps=901713
+BATCH (N=3000000) --> time_used=3157 ms, qps=950269
+BATCH (N=3000000) --> time_used=3241 ms, qps=925640
+BATCH (N=3000000) --> time_used=3301 ms, qps=908815
+BATCH (N=3000000) --> time_used=3345 ms, qps=896860
+BATCH (N=3000000) --> time_used=3319 ms, qps=903886
+BATCH (N=3000000) --> time_used=3312 ms, qps=905797
+BATCH (N=3000000) --> time_used=3337 ms, qps=899011
+BATCH (N=3000000) --> time_used=3309 ms, qps=906618
+BATCH (N=3000000) --> time_used=3385 ms, qps=886262
+BATCH (N=3000000) --> time_used=3328 ms, qps=901442
+BATCH (N=3000000) --> time_used=3194 ms, qps=939261
+BATCH (N=3000000) --> time_used=3309 ms, qps=906618
+BATCH (N=3000000) --> time_used=3262 ms, qps=919681
+BATCH (N=3000000) --> time_used=3314 ms, qps=905250
+BATCH (N=3000000) --> time_used=3382 ms, qps=887049
+BATCH (N=3000000) --> time_used=3296 ms, qps=910194
+BATCH (N=3000000) --> time_used=3331 ms, qps=900630
+BATCH (N=3000000) --> time_used=3279 ms, qps=914913
+BATCH (N=3000000) --> time_used=2996 ms, qps=1001335
+BATCH (N=3000000) --> time_used=3387 ms, qps=885739
+BATCH (N=3000000) --> time_used=3346 ms, qps=896592
+BATCH (N=3000000) --> time_used=3241 ms, qps=925640
+BATCH (N=3000000) --> time_used=3353 ms, qps=894721
+BATCH (N=3000000) --> time_used=3366 ms, qps=891265
+BATCH (N=3000000) --> time_used=3387 ms, qps=885739
+BATCH (N=3000000) --> time_used=3382 ms, qps=887049
+BATCH (N=3000000) --> time_used=3358 ms, qps=893388
+BATCH (N=3000000) --> time_used=3372 ms, qps=889679
+BATCH (N=3000000) --> time_used=3467 ms, qps=865301
+BATCH (N=3000000) --> time_used=3243 ms, qps=925069
+BATCH (N=3000000) --> time_used=3191 ms, qps=940144
+BATCH (N=3000000) --> time_used=3365 ms, qps=891530
+BATCH (N=3000000) --> time_used=3218 ms, qps=932256
+BATCH (N=3000000) --> time_used=3332 ms, qps=900360
+BATCH (N=3000000) --> time_used=3172 ms, qps=945775
+BATCH (N=3000000) --> time_used=3211 ms, qps=934288
+BATCH (N=3000000) --> time_used=3124 ms, qps=960307
+BATCH (N=3000000) --> time_used=3043 ms, qps=985869
+BATCH (N=3000000) --> time_used=3086 ms, qps=972132
+BATCH (N=3000000) --> time_used=3201 ms, qps=937207
+average qps:911106
+ALL TESTS PASSED.
+```
+
+内存分配：jemalloc
+```shell
+lian@ubuntu:~/share/9.1-kvstore$ ./test-redis/testcase 192.168.10.129 8888 3
+Connected to 192.168.10.129:8888
+BATCH (N=3000000) --> time_used=3197 ms, qps=938379
+BATCH (N=3000000) --> time_used=3221 ms, qps=931387
+BATCH (N=3000000) --> time_used=3360 ms, qps=892857
+BATCH (N=3000000) --> time_used=3292 ms, qps=911300
+BATCH (N=3000000) --> time_used=3407 ms, qps=880540
+BATCH (N=3000000) --> time_used=3317 ms, qps=904431
+BATCH (N=3000000) --> time_used=3337 ms, qps=899011
+BATCH (N=3000000) --> time_used=3384 ms, qps=886524
+BATCH (N=3000000) --> time_used=3355 ms, qps=894187
+BATCH (N=3000000) --> time_used=3379 ms, qps=887836
+BATCH (N=3000000) --> time_used=3243 ms, qps=925069
+BATCH (N=3000000) --> time_used=3377 ms, qps=888362
+BATCH (N=3000000) --> time_used=3212 ms, qps=933997
+BATCH (N=3000000) --> time_used=3248 ms, qps=923645
+BATCH (N=3000000) --> time_used=3234 ms, qps=927643
+BATCH (N=3000000) --> time_used=3152 ms, qps=951776
+BATCH (N=3000000) --> time_used=3089 ms, qps=971188
+BATCH (N=3000000) --> time_used=3287 ms, qps=912686
+BATCH (N=3000000) --> time_used=3079 ms, qps=974342
+BATCH (N=3000000) --> time_used=3261 ms, qps=919963
+BATCH (N=3000000) --> time_used=3123 ms, qps=960614
+BATCH (N=3000000) --> time_used=3234 ms, qps=927643
+BATCH (N=3000000) --> time_used=3056 ms, qps=981675
+BATCH (N=3000000) --> time_used=3040 ms, qps=986842
+BATCH (N=3000000) --> time_used=3187 ms, qps=941324
+BATCH (N=3000000) --> time_used=3311 ms, qps=906070
+BATCH (N=3000000) --> time_used=3155 ms, qps=950871
+BATCH (N=3000000) --> time_used=3318 ms, qps=904159
+BATCH (N=3000000) --> time_used=3372 ms, qps=889679
+BATCH (N=3000000) --> time_used=3254 ms, qps=921942
+BATCH (N=3000000) --> time_used=3386 ms, qps=886001
+BATCH (N=3000000) --> time_used=3413 ms, qps=878992
+BATCH (N=3000000) --> time_used=3474 ms, qps=863557
+BATCH (N=3000000) --> time_used=3412 ms, qps=879249
+BATCH (N=3000000) --> time_used=3414 ms, qps=878734
+BATCH (N=3000000) --> time_used=3325 ms, qps=902255
+BATCH (N=3000000) --> time_used=3346 ms, qps=896592
+BATCH (N=3000000) --> time_used=3345 ms, qps=896860
+BATCH (N=3000000) --> time_used=3582 ms, qps=837520
+BATCH (N=3000000) --> time_used=3412 ms, qps=879249
+BATCH (N=3000000) --> time_used=3370 ms, qps=890207
+BATCH (N=3000000) --> time_used=3375 ms, qps=888888
+BATCH (N=3000000) --> time_used=3190 ms, qps=940438
+BATCH (N=3000000) --> time_used=3324 ms, qps=902527
+BATCH (N=3000000) --> time_used=3253 ms, qps=922225
+BATCH (N=3000000) --> time_used=3230 ms, qps=928792
+BATCH (N=3000000) --> time_used=3294 ms, qps=910746
+BATCH (N=3000000) --> time_used=3295 ms, qps=910470
+BATCH (N=3000000) --> time_used=3148 ms, qps=952986
+BATCH (N=3000000) --> time_used=3228 ms, qps=929368
+average qps:914031
+ALL TESTS PASSED.
+```
+

 ### 面试题
 1. 为什么会实现kvstore，使用场景在哪里？