367 lines
14 KiB
Markdown
367 lines
14 KiB
Markdown
# 9.1 Kvstore
|
||
|
||
## 需求
|
||
1. ntyco需要作为kvstore的submodule,通过git clone一次下载。 **完成**。
|
||
2. README需要包含编译步骤，测试方案与可行性，性能数据。 **完成**。
|
||
3. 全量持久化保存数据集。 **BUG FIX**。
|
||
4. 持久化的性能数据。 **完成**。
|
||
5. 特殊字符，可以解决redis的resp协议。 **完成**。
|
||
6. 实现配置文件，把日志级别，端口ip，主从模式，持久化方案。 **完成**。
|
||
7. 持久化落盘用io_uring，加载配置文件用mmap。 **完成**。
|
||
8. 主从同步的性能，开启与关闭性能做到5%?。
|
||
9. 主从同步600w条,出现的coredump。 **BUG FIX**。
|
||
10. 主从同步用ebpf实现。 **完成**。
|
||
11. 内存池测试qps与虚拟内存，物理内存。 **完成**。
|
||
12. 实现一个内存泄露检测组件。 **完成**。
|
||
|
||
|
||
## 环境安装与编译
|
||
```shell
|
||
# xml
|
||
sudo apt install libxml2 libxml2-dev
|
||
# hiredis client
|
||
sudo apt install -y libhiredis-dev
|
||
# bpftrace
|
||
sudo apt install -y bpftrace libelf libelf-dev clang
|
||
|
||
git clone git@gitlab.0voice.com:lianyiheng/9.1-kvstore.git
|
||
cd 9.1-kvstore/
|
||
git submodule update --init --recursive
|
||
./init.sh
|
||
make
|
||
|
||
|
||
```
|
||
|
||
## 测试
|
||
### 测试1：性能测试
|
||
测试条件：
|
||
1. 不启用持久化。
|
||
2. 不启用主从同步。
|
||
3. pipline：
|
||
1. RSET 100w 条， p:i v:i -> +OK
|
||
2. RGET 100w 条， p:i -> +v:i
|
||
3. RDEL 100w 条。 p:i -> +OK
|
||
4. 本机发送请求。
|
||
|
||
#### 内存分配: malloc
|
||
```bash
|
||
lian@ubuntu:~/share/9.1-kvstore$ ./test-redis/testcase 192.168.10.129 8888 3
|
||
Connected to 192.168.10.129:8888
|
||
BATCH (N=3000000) --> time_used=3294 ms, qps=910746
|
||
BATCH (N=3000000) --> time_used=3501 ms, qps=856898
|
||
BATCH (N=3000000) --> time_used=3457 ms, qps=867804
|
||
BATCH (N=3000000) --> time_used=3351 ms, qps=895255
|
||
BATCH (N=3000000) --> time_used=3320 ms, qps=903614
|
||
BATCH (N=3000000) --> time_used=3551 ms, qps=844832
|
||
BATCH (N=3000000) --> time_used=3354 ms, qps=894454
|
||
BATCH (N=3000000) --> time_used=3475 ms, qps=863309
|
||
BATCH (N=3000000) --> time_used=3404 ms, qps=881316
|
||
BATCH (N=3000000) --> time_used=3460 ms, qps=867052
|
||
BATCH (N=3000000) --> time_used=3392 ms, qps=884433
|
||
BATCH (N=3000000) --> time_used=3427 ms, qps=875401
|
||
BATCH (N=3000000) --> time_used=3441 ms, qps=871839
|
||
BATCH (N=3000000) --> time_used=3471 ms, qps=864304
|
||
BATCH (N=3000000) --> time_used=3354 ms, qps=894454
|
||
BATCH (N=3000000) --> time_used=3447 ms, qps=870322
|
||
BATCH (N=3000000) --> time_used=3364 ms, qps=891795
|
||
BATCH (N=3000000) --> time_used=3200 ms, qps=937500
|
||
BATCH (N=3000000) --> time_used=3159 ms, qps=949667
|
||
BATCH (N=3000000) --> time_used=3482 ms, qps=861573
|
||
BATCH (N=3000000) --> time_used=3474 ms, qps=863557
|
||
BATCH (N=3000000) --> time_used=3591 ms, qps=835421
|
||
BATCH (N=3000000) --> time_used=3466 ms, qps=865551
|
||
BATCH (N=3000000) --> time_used=3425 ms, qps=875912
|
||
BATCH (N=3000000) --> time_used=3346 ms, qps=896592
|
||
BATCH (N=3000000) --> time_used=3532 ms, qps=849377
|
||
BATCH (N=3000000) --> time_used=3471 ms, qps=864304
|
||
BATCH (N=3000000) --> time_used=3616 ms, qps=829646
|
||
BATCH (N=3000000) --> time_used=3403 ms, qps=881575
|
||
BATCH (N=3000000) --> time_used=3426 ms, qps=875656
|
||
BATCH (N=3000000) --> time_used=3493 ms, qps=858860
|
||
BATCH (N=3000000) --> time_used=3411 ms, qps=879507
|
||
BATCH (N=3000000) --> time_used=3422 ms, qps=876680
|
||
BATCH (N=3000000) --> time_used=3556 ms, qps=843644
|
||
BATCH (N=3000000) --> time_used=3285 ms, qps=913242
|
||
BATCH (N=3000000) --> time_used=3486 ms, qps=860585
|
||
BATCH (N=3000000) --> time_used=3427 ms, qps=875401
|
||
BATCH (N=3000000) --> time_used=3563 ms, qps=841987
|
||
BATCH (N=3000000) --> time_used=3304 ms, qps=907990
|
||
BATCH (N=3000000) --> time_used=3582 ms, qps=837520
|
||
BATCH (N=3000000) --> time_used=3468 ms, qps=865051
|
||
BATCH (N=3000000) --> time_used=3360 ms, qps=892857
|
||
BATCH (N=3000000) --> time_used=3426 ms, qps=875656
|
||
BATCH (N=3000000) --> time_used=3186 ms, qps=941619
|
||
BATCH (N=3000000) --> time_used=3251 ms, qps=922792
|
||
BATCH (N=3000000) --> time_used=3400 ms, qps=882352
|
||
BATCH (N=3000000) --> time_used=3446 ms, qps=870574
|
||
BATCH (N=3000000) --> time_used=3302 ms, qps=908540
|
||
BATCH (N=3000000) --> time_used=3072 ms, qps=976562
|
||
BATCH (N=3000000) --> time_used=3458 ms, qps=867553
|
||
average qps:880462
|
||
ALL TESTS PASSED.
|
||
```
|
||
|
||
#### 内存分配: 自实现内存池
|
||
```bash
|
||
lian@ubuntu:~/share/9.1-kvstore$ ./test-redis/testcase 192.168.10.129 8888 3
|
||
Connected to 192.168.10.129:8888
|
||
BATCH (N=3000000) --> time_used=3241 ms, qps=925640
|
||
BATCH (N=3000000) --> time_used=3047 ms, qps=984574
|
||
BATCH (N=3000000) --> time_used=3085 ms, qps=972447
|
||
BATCH (N=3000000) --> time_used=3119 ms, qps=961846
|
||
BATCH (N=3000000) --> time_used=3104 ms, qps=966494
|
||
BATCH (N=3000000) --> time_used=3163 ms, qps=948466
|
||
BATCH (N=3000000) --> time_used=3033 ms, qps=989119
|
||
BATCH (N=3000000) --> time_used=3170 ms, qps=946372
|
||
BATCH (N=3000000) --> time_used=3299 ms, qps=909366
|
||
BATCH (N=3000000) --> time_used=3272 ms, qps=916870
|
||
BATCH (N=3000000) --> time_used=3294 ms, qps=910746
|
||
BATCH (N=3000000) --> time_used=3182 ms, qps=942803
|
||
BATCH (N=3000000) --> time_used=3190 ms, qps=940438
|
||
BATCH (N=3000000) --> time_used=3493 ms, qps=858860
|
||
BATCH (N=3000000) --> time_used=3111 ms, qps=964320
|
||
BATCH (N=3000000) --> time_used=3220 ms, qps=931677
|
||
BATCH (N=3000000) --> time_used=3067 ms, qps=978154
|
||
BATCH (N=3000000) --> time_used=3345 ms, qps=896860
|
||
BATCH (N=3000000) --> time_used=3381 ms, qps=887311
|
||
BATCH (N=3000000) --> time_used=3416 ms, qps=878220
|
||
BATCH (N=3000000) --> time_used=3192 ms, qps=939849
|
||
BATCH (N=3000000) --> time_used=3085 ms, qps=972447
|
||
BATCH (N=3000000) --> time_used=3150 ms, qps=952380
|
||
BATCH (N=3000000) --> time_used=3296 ms, qps=910194
|
||
BATCH (N=3000000) --> time_used=3001 ms, qps=999666
|
||
BATCH (N=3000000) --> time_used=3143 ms, qps=954502
|
||
BATCH (N=3000000) --> time_used=3111 ms, qps=964320
|
||
BATCH (N=3000000) --> time_used=3123 ms, qps=960614
|
||
BATCH (N=3000000) --> time_used=3257 ms, qps=921093
|
||
BATCH (N=3000000) --> time_used=3037 ms, qps=987816
|
||
BATCH (N=3000000) --> time_used=3135 ms, qps=956937
|
||
BATCH (N=3000000) --> time_used=3124 ms, qps=960307
|
||
BATCH (N=3000000) --> time_used=3276 ms, qps=915750
|
||
BATCH (N=3000000) --> time_used=3058 ms, qps=981033
|
||
BATCH (N=3000000) --> time_used=3024 ms, qps=992063
|
||
BATCH (N=3000000) --> time_used=3224 ms, qps=930521
|
||
BATCH (N=3000000) --> time_used=3235 ms, qps=927357
|
||
BATCH (N=3000000) --> time_used=3334 ms, qps=899820
|
||
BATCH (N=3000000) --> time_used=3427 ms, qps=875401
|
||
BATCH (N=3000000) --> time_used=3218 ms, qps=932256
|
||
BATCH (N=3000000) --> time_used=3191 ms, qps=940144
|
||
BATCH (N=3000000) --> time_used=3179 ms, qps=943692
|
||
BATCH (N=3000000) --> time_used=3104 ms, qps=966494
|
||
BATCH (N=3000000) --> time_used=3202 ms, qps=936914
|
||
BATCH (N=3000000) --> time_used=3184 ms, qps=942211
|
||
BATCH (N=3000000) --> time_used=3000 ms, qps=1000000
|
||
BATCH (N=3000000) --> time_used=3280 ms, qps=914634
|
||
BATCH (N=3000000) --> time_used=3141 ms, qps=955109
|
||
BATCH (N=3000000) --> time_used=3198 ms, qps=938086
|
||
BATCH (N=3000000) --> time_used=3126 ms, qps=959692
|
||
average qps:942837
|
||
ALL TESTS PASSED.
|
||
```
|
||
|
||
#### 内存分配：jemalloc
|
||
```shell
|
||
lian@ubuntu:~/share/9.1-kvstore$ ./test-redis/testcase 192.168.10.129 8888 3
|
||
Connected to 192.168.10.129:8888
|
||
BATCH (N=3000000) --> time_used=3511 ms, qps=854457
|
||
BATCH (N=3000000) --> time_used=3280 ms, qps=914634
|
||
BATCH (N=3000000) --> time_used=3603 ms, qps=832639
|
||
BATCH (N=3000000) --> time_used=3418 ms, qps=877706
|
||
BATCH (N=3000000) --> time_used=3353 ms, qps=894721
|
||
BATCH (N=3000000) --> time_used=3435 ms, qps=873362
|
||
BATCH (N=3000000) --> time_used=3250 ms, qps=923076
|
||
BATCH (N=3000000) --> time_used=3550 ms, qps=845070
|
||
BATCH (N=3000000) --> time_used=3536 ms, qps=848416
|
||
BATCH (N=3000000) --> time_used=3273 ms, qps=916590
|
||
BATCH (N=3000000) --> time_used=3224 ms, qps=930521
|
||
BATCH (N=3000000) --> time_used=3161 ms, qps=949066
|
||
BATCH (N=3000000) --> time_used=3143 ms, qps=954502
|
||
BATCH (N=3000000) --> time_used=3342 ms, qps=897666
|
||
BATCH (N=3000000) --> time_used=3410 ms, qps=879765
|
||
BATCH (N=3000000) --> time_used=3522 ms, qps=851788
|
||
BATCH (N=3000000) --> time_used=3035 ms, qps=988467
|
||
BATCH (N=3000000) --> time_used=3352 ms, qps=894988
|
||
BATCH (N=3000000) --> time_used=3226 ms, qps=929944
|
||
BATCH (N=3000000) --> time_used=3406 ms, qps=880798
|
||
BATCH (N=3000000) --> time_used=3336 ms, qps=899280
|
||
BATCH (N=3000000) --> time_used=3307 ms, qps=907166
|
||
BATCH (N=3000000) --> time_used=3171 ms, qps=946073
|
||
BATCH (N=3000000) --> time_used=3252 ms, qps=922509
|
||
BATCH (N=3000000) --> time_used=3296 ms, qps=910194
|
||
BATCH (N=3000000) --> time_used=3301 ms, qps=908815
|
||
BATCH (N=3000000) --> time_used=3403 ms, qps=881575
|
||
BATCH (N=3000000) --> time_used=3234 ms, qps=927643
|
||
BATCH (N=3000000) --> time_used=3348 ms, qps=896057
|
||
BATCH (N=3000000) --> time_used=3517 ms, qps=852999
|
||
BATCH (N=3000000) --> time_used=3354 ms, qps=894454
|
||
BATCH (N=3000000) --> time_used=3529 ms, qps=850099
|
||
BATCH (N=3000000) --> time_used=3473 ms, qps=863806
|
||
BATCH (N=3000000) --> time_used=3521 ms, qps=852030
|
||
BATCH (N=3000000) --> time_used=3370 ms, qps=890207
|
||
BATCH (N=3000000) --> time_used=3267 ms, qps=918273
|
||
BATCH (N=3000000) --> time_used=3352 ms, qps=894988
|
||
BATCH (N=3000000) --> time_used=3433 ms, qps=873871
|
||
BATCH (N=3000000) --> time_used=3374 ms, qps=889152
|
||
BATCH (N=3000000) --> time_used=3360 ms, qps=892857
|
||
BATCH (N=3000000) --> time_used=3463 ms, qps=866300
|
||
BATCH (N=3000000) --> time_used=3499 ms, qps=857387
|
||
BATCH (N=3000000) --> time_used=3294 ms, qps=910746
|
||
BATCH (N=3000000) --> time_used=3311 ms, qps=906070
|
||
BATCH (N=3000000) --> time_used=3443 ms, qps=871333
|
||
BATCH (N=3000000) --> time_used=3381 ms, qps=887311
|
||
BATCH (N=3000000) --> time_used=3422 ms, qps=876680
|
||
BATCH (N=3000000) --> time_used=3421 ms, qps=876936
|
||
BATCH (N=3000000) --> time_used=3322 ms, qps=903070
|
||
BATCH (N=3000000) --> time_used=3494 ms, qps=858614
|
||
average qps:892493
|
||
ALL TESTS PASSED.
|
||
```
|
||
|
||
### 测试2：持久化
|
||
测试条件：
|
||
1. 启用持久化。
|
||
2. 不启用主从同步。
|
||
3. pipline：
|
||
1. RSET 100w 条， p:i v:i -> +OK
|
||
2. RGET 100w 条， p:i -> +v:i
|
||
3. RDEL 100w 条。 p:i -> +OK
|
||
5. 本机发送请求。
|
||
|
||
```shell
|
||
lian@ubuntu:~/share/9.1-kvstore$ ./test-redis/testcase 192.168.10.129 8888 4
|
||
Connected to 192.168.10.129:8888
|
||
BATCH (N=3000000) --> time_used=3500 ms, qps=857142
|
||
BATCH (N=3000000) --> time_used=3322 ms, qps=903070
|
||
BATCH (N=3000000) --> time_used=3424 ms, qps=876168
|
||
BATCH (N=3000000) --> time_used=3483 ms, qps=861326
|
||
BATCH (N=3000000) --> time_used=3421 ms, qps=876936
|
||
BATCH (N=3000000) --> time_used=3519 ms, qps=852514
|
||
BATCH (N=3000000) --> time_used=3597 ms, qps=834028
|
||
BATCH (N=3000000) --> time_used=3504 ms, qps=856164
|
||
BATCH (N=3000000) --> time_used=3281 ms, qps=914355
|
||
BATCH (N=3000000) --> time_used=3446 ms, qps=870574
|
||
average qps:870227
|
||
ALL TESTS PASSED.
|
||
```
|
||
|
||
### 测试3：内存
|
||
|
||
#### malloc
|
||
```shell
|
||
VIRT 58504
|
||
RES 4604
|
||
|
||
插入 20w 删除 10w，重复 10 次，共计插入 200w 删除 100w。
|
||
BATCH (N=9000000) --> time_used=12897 ms, qps=1395673
|
||
|
||
VIRT 489M
|
||
RES 430M
|
||
|
||
插入 10w 删除 20w，重复 10 次，共计插入 100w 删除 200w。
|
||
BATCH (N=9000000) --> time_used=10033 ms, qps=1794079
|
||
|
||
VIRT 208M
|
||
RES 155M
|
||
```
|
||
|
||
|
||
|
||
|
||
#### jemalloc
|
||
```shell
|
||
VIRT 69376
|
||
RES 5408
|
||
|
||
插入 20w 删除 10w，重复 30 次，共计插入 600w 删除 300w。
|
||
BATCH (N=9000000) --> time_used=9436 ms, qps=1907587
|
||
|
||
VIRT 356M
|
||
RES 294M
|
||
|
||
插入 10w 删除 20w，重复 30 次，共计插入 300w 删除 600w。
|
||
BATCH (N=9000000) --> time_used=9353 ms, qps=1924516
|
||
|
||
VIRT 356M
|
||
RES 119M
|
||
```
|
||
|
||
|
||
|
||
|
||
#### mypool
|
||
```shell
|
||
VIRT 58504
|
||
RES 4636
|
||
|
||
插入 20w 删除 10w，重复 30 次，共计插入 600w 删除 300w。
|
||
BATCH (N=3000000) --> time_used=3184 ms, qps=1884422
|
||
|
||
VIRT 625M
|
||
RES 572M
|
||
|
||
插入 10w 删除 20w，重复 10 次，共计插入 100w 删除 200w。
|
||
BATCH (N=3000000) --> time_used=3022 ms, qps=1985440
|
||
|
||
VIRT 122M
|
||
RES 71492
|
||
```
|
||
|
||
|
||
|
||
|
||
### 测试4：主从同步
|
||
测试条件：
|
||
1. 不启用持久化。
|
||
2. 启用主从同步。
|
||
3. pipline：
|
||
1. RSET 100w 条， p:i v:i -> +OK
|
||
2. RGET 100w 条， p:i -> +v:i
|
||
3. RDEL 100w 条。 p:i -> +OK
|
||
5. 本机发送请求。
|
||
```shell
|
||
lian@ubuntu:~/share/9.1-kvstore$ ./test-redis/testcase 192.168.10.129 8888 4
|
||
Connected to 192.168.10.129:8888
|
||
BATCH (N=3000000) --> time_used=3702 ms, qps=810372
|
||
BATCH (N=3000000) --> time_used=3804 ms, qps=788643
|
||
BATCH (N=3000000) --> time_used=4076 ms, qps=736015
|
||
BATCH (N=3000000) --> time_used=3840 ms, qps=781250
|
||
BATCH (N=3000000) --> time_used=3824 ms, qps=784518
|
||
average qps:780159
|
||
ALL TESTS PASSED.
|
||
lian@ubuntu:~/share/9.1-kvstore$ ./test-redis/testcase 192.168.10.129 8888 4
|
||
Connected to 192.168.10.129:8888
|
||
BATCH (N=3000000) --> time_used=3958 ms, qps=757958
|
||
BATCH (N=3000000) --> time_used=4043 ms, qps=742023
|
||
BATCH (N=3000000) --> time_used=3729 ms, qps=804505
|
||
BATCH (N=3000000) --> time_used=3989 ms, qps=752068
|
||
BATCH (N=3000000) --> time_used=3603 ms, qps=832639
|
||
average qps:777838
|
||
ALL TESTS PASSED.
|
||
```
|
||
|
||
### 面试题
|
||
1. 为什么会实现kvstore，使用场景在哪里？
|
||
2. reactor, ntyco, io_uring的三种网络模型的性能差异？
|
||
3. 多线程的kvstore该如何改进？
|
||
4. 私有协议如何设计会更加安全可靠？
|
||
5. 协议改进以后，对已有的代码有哪些改变？
|
||
6. kv引擎实现了哪些？
|
||
7. 每个kv引擎的使用场景，以及性能差异？
|
||
8. 测试用例如何实现？并且保证代码覆盖率超过90%
|
||
9. 网络并发量如何？qps如何？
|
||
10. 能够跟哪些系统交互使用？
|
||
|
||
|
||
## 项目收获
|
||
reactor网络模型，用户态网络缓冲区的写法。\
|
||
特殊字符串支持的引擎层数据结构设计，支持\0作为键值存储。\
|
||
实现RESP协议的服务端协议解析。\
|
||
使用fork的Copy On Write机制，实现的异步快照创建，不会受到原字符串的影响。\
|
||
基于BinLog上OffSet的主从同步设计。\
|
||
基于bpf的实时数据同步设计。\
|
||
基于共享缓冲区+额外进程的实时数据同步设计。\
|
||
基于bpf的内存泄露探测功能，实现热插拔。\
|
||
实现支持分配可变长度内存块的内存池。\
|
||
实现专门uring线程实现异步的增量、全量落盘操作。\
|
||
使用配置文件规定端口、保存文件路径等，使用mmap加载到内存，使用libxml解析。\ |