ubuntu20.04 调试bcache源码

搭建单步调试bcache的环境，/dev/sdb作为backing dev， /dev/sdc作为cache dev。

一、宿主机环境

1）安装ubuntu 20.04 ：

参考ubuntu20.04 搭建kernel调试环境第一篇--安装系统_ubuntu kernel-CSDN博客安装，其中的第六节（安装qemu）、第七节（安装apache服务）安装完成即可。

2）配置网络

host（ubuntu系统）和guest（qemu运行的系统，待调试）之间需共享文件，所以需要配置网络。

参考ubuntu20.04 搭建kernel调试环境第六篇（上）--网络配置_ubuntu20 menuconfig-CSDN博客，按其中的第三节（ubuntu宿主机配置步骤）、第4节（buildroot配置）、第五节（启动虚拟机）配置，最终能ping通外网就行。

二、静态编译bcache-tools工具

guest运行的kernel是我们自己编译的，busybox只支持一些简单的命令，动态库也很少，所以无法使用apt-get install bcache-tools方式安装bcache的用户态工具。需要自己下载源码，然后用静态方式编译出可执行文件。

1）git下载bcache-tools源码：
git clone https://github.com/g2p/bcache-tools.git

2) Makefile改成静态编译

root@linux:/home/gsf/code/bcache-tool-git/bcache-tools# git diff Makefile
diff --git a/Makefile b/Makefile
index c824ae3..ccea7ee 100644
--- a/Makefile
+++ b/Makefile
@@ -3,7 +3,8 @@ PREFIX=/usr
UDEVLIBDIR=/lib/udev
DRACUTLIBDIR=/lib/dracut
INSTALL=install
-CFLAGS+=-O2 -Wall -g
+CFLAGS+=-O2 -Wall -g -static
+

3）编译bcache-tools

root@linux:/home/gsf/code/bcache-tool-git/bcache-tools# make

root@linux:/home/gsf/code/bcache-tool-git/bcache-tools# make install
install -m0755 make-bcache bcache-super-show    /usr/sbin/ （关注这一行中的两个文件）
install -m0755 probe-bcache bcache-register        /lib/udev/
install -m0644 69-bcache.rules    /lib/udev/rules.d/
install -m0644 -- *.8 /usr/share/man/man8/
install -D -m0755 initramfs/hook    /usr/share/initramfs-tools/hooks/bcache
install -D -m0755 initcpio/install    /usr/lib/initcpio/install/bcache
install -D -m0755 dracut/module-setup.sh /lib/dracut/modules.d/90bcache/module-setup.sh

可能遇到的错误1：

Perhaps you should add the directory containing `uuid.pc'
to the PKG_CONFIG_PATH environment variable
No package 'uuid' found
Package blkid was not found in the pkg-config search path.
Perhaps you should add the directory containing `blkid.pc'
to the PKG_CONFIG_PATH environment variable
No package 'blkid' found
make-bcache.c:11:10: fatal error: blkid.h: No such file or directory
11 | #include <blkid.h>
| ^~~~~~~~~

解决：apt-get install libblkid-dev

可能遇到的错误2：

root@linux:/home/gsf/code/bcache-tools/bcache-tools-master# make
cc -O2 -Wall -g `pkg-config --cflags uuid blkid` make-bcache.c bcache.o `pkg-config --libs uuid blkid` -o make-bcache
/usr/bin/ld: /tmp/ccsteIC1.o: in function `write_sb':
/home/gsf/code/bcache-tools/bcache-tools-master/make-bcache.c:277: undefined reference to `crc64'
collect2: error: ld returned 1 exit status
make: *** [<builtin>: make-bcache] Error 1

解决：源码bcache.c中的crc_table数组、crc64函数复制一份，放在bcache.h前面。

三、kernel编译

在编译kernel前，需通过make menuconfig打开以下选项：

<*> Block device as cache

[*] Bcache debugging

kernel编译参考：

ubuntu20.04 搭建kernel调试环境第三篇--kernel编译及运行_在qemu中运行f2fs-CSDN博客

四、qemu启动guest系统

以下命令在host即ubuntu中执行！

1）生成cache盘img，大小3G

root@linux:/home/gsf/linux-5.18.11# qemu-img create -f raw disk_raw_3G_C.qcow 3G

2）生成backing盘img，大小5G

qemu-img create -f raw disk_raw_5G_B.qcow 5G

3）生成用于测试nvme的img

qemu-img create -f raw disk10G.qcow 5G

3）启动guest系统

root@linux:/home/gsf/linux-5.18.11# qemu-system-x86_64 -kernel arch/x86_64/boot/bzImage -drive file=rootfs.f2fs,if=ide,format=raw,id=myid0 --nographic -append "root=/dev/sda console=ttyS0" -hdb disk_raw_5G_B.qcow -hdc disk_raw_3G_C.qcow -device nvme,drive=nvme1,serial=deadbeaf,num_queues=8 -drive file=disk10G.qcow,if=none,id=nvme1 -smp 4 -net nic,macaddr=52:54:00:12:34:56,model=e1000 -net bridge,id=net0,helper=/usr/lib/qemu/qemu-bridge-helper,br=virbr0

红色字段，创建/dev/sdb，在guest起来后，该设备会格式化成backing dev

蓝色字段，创建/dev/sdc，在guest起来后，该设备会格式化成cache dev

黄色字段，创建/dev/nvme0，作为nvme测试使用，bcache用不到。

guest系统启动后，输入root登录，dev文件如下：

# ls /dev （guest中执行的命令）

cpu_dma_latency sda tty26 tty56

fd sda1 tty27 tty57

full sdb tty28 tty58 ---sdb

fuse sdc tty29 tty59 ---sdc

nvme0 tty17 tty47 vcsu1

nvme0n1 tty18 tty48 vga_arbiter

nvme0n1p1 tty19 tty49 zero

五、拷贝bcache-tools到guset系统

1）将make-bcache拷贝到guest中

host系统中执行：

root@linux:/var/www/html# cp /usr/sbin/make-bcache ./

root@linux:/var/www/html# cp /usr/sbin/bcache-super-show ./

host系统中执行：

root@linux:/home/gsf# ifconfig

找到virbr0的ip地址，比如

virbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
        inet 192.168.122.1 netmask 255.255.255.0 broadcast 192.168.122.255
        ether 52:54:00:5e:c9:bb txqueuelen 1000 (Ethernet)
        RX packets 299 bytes 25524 (25.5 KB)
        RX errors 0 dropped 0 overruns 0 frame 0
        TX packets 431 bytes 1335005 (1.3 MB)
        TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

guest系统中执行：

# wget http://192.168.122.1/make-bcache （上面ifconfig看到的ip地址）
Connecting to 192.168.122.1 (192.168.122.1:80)
saving to 'make-bcache'
make-bcache 100% |********************************| 1273k 0:00:00 ETA
'make-bcache' saved
# wget http://192.168.122.1/bcache-super-show
Connecting to 192.168.122.1 (192.168.122.1:80)
saving to 'bcache-super-show'
bcache-super-show 100% |********************************| 908k 0:00:00 ETA
'bcache-super-show' saved

# chmod 777 make-bcache
# chmod 777 bcache-super-show
# ls
bcache-super-show make-bcache

六、配置bcache磁盘

以下命令在guest中执行！

1）格式化bcache设备

# /root/make-bcache -B /dev/sdb
UUID:            3a5fa145-1cc9-44e0-b91c-1beaf245355f
Set UUID:        25fceacc-0546-4c32-80d0-5c9881a1d67f
version:        1
block_size:        1
data_offset:        16
#
# /root/make-bcache -C /dev/sdc
UUID:            c66b56a8-bb3e-4d8e-9b9f-3b402cd6b258
Set UUID:        fe0af6ee-67e8-49c3-bfec-5633ba05f782
version:        0
nbuckets:        6144
block_size:        1
bucket_size:        1024
nr_in_set:        1
nr_this_dev:        0
first_bucket:        1

2）注册backing device

# echo /dev/sdb > /sys/fs/bcache/register
[ 2157.866302] bcache: register_bdev() registered backing device sdb

# 命令完成后，dev目录中会生成bcache0设备
# ls /dev/
autofs           ptmx             tty21            tty51
bcache0          pts              tty22            tty52
bsg              random           tty23            tty53

3）注册cache device

# echo /dev/sdc > /sys/fs/bcache/register
[ 146.450348] bcache: bch_journal_replay() journal replay done, 0 keys in 1 entries, seq 4
[ 146.453397] bcache: register_cache() registered cache device sdc

如果不注册cache device，就执行attach命令，会报下面错误：

# echo "fe0af6ee-67e8-49c3-bfec-5633ba05f782" > /sys/block/bcache0/bcache/attach
[ 114.804472] bcache: __cached_dev_store() Can't attach fe0af6ee-67e8-49c3-bfec-5633ba05f782
[ 114.804472] : cache set not found
sh: write error: No such file or directory

4）绑定backing dev到cache dev上

格式化的时候，/dev/sdb格式化成了backing dev， /dev/sdc格式化成了cache dev，通过bcache-super-show找到cache dev的cset.uuid值。

# /root/bcache-super-show /dev/sdc
sb.magic        ok
sb.first_sector        8 [match]
sb.csum            5E71456A3E5F13A8 [match]
sb.version        3 [cache device]

dev.label        (empty)
dev.uuid        c66b56a8-bb3e-4d8e-9b9f-3b402cd6b258
dev.sectors_per_block    1
dev.sectors_per_bucket    1024
dev.cache.first_sector    1024
dev.cache.cache_sectors    6290432
dev.cache.total_sectors    6291456
dev.cache.ordered    yes
dev.cache.discard    no
dev.cache.pos        0
dev.cache.replacement    0 [lru]

cset.uuid fe0af6ee-67e8-49c3-bfec-5633ba05f782 （该值用于写入backing dev的attach文件中）

执行绑定指令：

# echo "fe0af6ee-67e8-49c3-bfec-5633ba05f782" > /sys/block/bcache0/bcache/attach
[ 151.527075] bcache: bch_cached_dev_run() cached dev sdb is running already
[ 151.528160] bcache: bch_cached_dev_attach() Caching sdb as bcache0 on set fe0af6ee-67e8-49c3-bfec-5633ba05f782

5）backing dev格式化成具体的文件系统（比如f2fs）

# mkfs.f2fs /dev/bcache0

F2FS-tools: mkfs.f2fs Ver: 1.13.0 (2019-09-24)

Info: Disable heap-based policy
Info: Debug level = 0
Info: Trim is enabled
Info: [/dev/bcache0] Disk Model: QEMU HARDDISK
Info: Segments per section = 1
Info: Sections per zone = 1
Info: sector size = 512
Info: total sectors = 10485744 (5119 MB)
Info: zone aligned segment0 blkaddr: 512
Info: format version with
"Linux version 5.18.11-g50a65f78b667 (root@linux) (gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #20 SM"
Info: [/dev/bcache0] Discarding device
Info: This device doesn't support BLKSECDISCARD
Info: Discarded 5119 MB
Info: Overprovision ratio = 2.810%
Info: Overprovision segments = 148 (GC reserved = 79)
Info: format successful
[ 1491.536112] mkfs.f2fs (194) used greatest stack depth: 13264 bytes left

6）挂载/dev/bcache0

# mkdir /mnt/bcachemp 创建一个目录用于挂载/dev/bcache0

# mount /dev/bcache0 /mnt/bcachemp
[ 1672.244480] F2FS-fs (bcache0): Found nat_bits in checkpoint
[ 1672.366811] F2FS-fs (bcache0): Mounted with checkpoint version = 19642f45
[ 1672.371205] mount (196) used greatest stack depth: 13144 bytes left

# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root                98.0M     74.5M     23.5M 76% /
devtmpfs                 44.9M         0     44.9M   0% /dev
tmpfs                    46.8M         0     46.8M   0% /dev/shm
tmpfs                    46.8M     52.0K     46.7M   0% /tmp
tmpfs                    46.8M     20.0K     46.8M   0% /run
/dev/nvme0n1p1           10.0G      3.4G      6.6G 34% /mnt/nvmemp
/dev/bcache0              5.0G    340.0M      4.7G   7% /mnt/bcachemp (bcache 挂载成功

bcache设备挂载成功，可通过gdb调试代码（gdb调试可参考https://blog.csdn.net/geshifei/article/details/128235389）。