一、目标
在没有arm硬件的情况下，使用QEMU模拟器，在PC上模拟一块ARM开发板，对ARM Linux进行学习。
二、搭建步骤
首先先有一个Linux 开发环境，我目前使用的是Ubuntu20.
首先安装qemu，qemu的官网：https://www.qemu.org/
sudo apt install qemu-system
使用qemu 启动虚拟开发板运行：
qemu-system-arm -M vexpress-a9 -m 512M -kernel /home/prx/project/linux-5.10/arch/arm/boot/zImage -dtb /home/prx/project/linux-5.10/arch/arm/boot/dts/vexpress-v2p-ca9.dtb -nographic -append "init=/linuxrc root=/dev/mmcblk0 console=ttyAMA0" -sd /home/prx/project/busybox-1.36.0/disk.img
# -M：表示模拟的硬件开发板
# -m 512M：表示内存大小
# -kernel：kernel image地址
# -dtb：设备树地址
# -nographic：不需要图像界面
# -append：带一个字符串，作为bootcmd
# -sd：指定镜像
退出虚拟开发板的命令为：
ctrl+a松手之后再按x
好的，硬件虚拟环境搭建之后，就可以开始搭建软件环境了。
# 安装交叉编译工具，我们的目标是arm v7
sudo apt-get install gcc-arm-linux-gnueabi
# 设置环境变量
export CROSS_COMPILE=arm-linux-gnueabi-
export ARCH=arm
# 下载kernel 5.10 代码
wget https://mirror.bjtu.edu.cn/kernel/linux/kernel/v5.x/linux-5.10.tar.xz
tar -xf linux-5.10.tar.xz
# 编译内核
cd linux-5.10
make vexpress_defconfig
make -j8
# 我们所需的文件为
./arch/arm/boot/zImage
./arch/arm/boot/dts/vexpress-v2p-ca9.dtb
# 下载编译busybox
wget http://www.busybox.net/downloads/busybox-1.36.0.tar.bz2
make
make install
# 制作根文件系统
mkdir -p rootfs/{dev,etc/init.d,lib,proc,sys,tmp}
# 拷贝busybox的输出到根文件系统
sudo cp busybox-1.20.2/_install/* -r rootfs/
#拷贝libc
sudo cp -P /usr/arm-linux-gnueabi/lib/* rootfs/lib/
# 新增 rootfs/etc/fstab，让linux启动后自动mount文件系统
vim rootfs/etc/fstab
proc /proc proc defaults 0 0
tmpfs /tmp tmpfs defaults 0 0
sysfs /sys sysfs defaults 0 0
# 新增 rootfs/etc/init.d/rcS,linux启动之后的第一个shell脚本，在这里我们要做一些初始化
vim rootfs/etc/init.d/rcS
PATH=/sbin:/bin:/usr/sbin:/usr/bin:$PATH
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/lib:/usr/lib
export PATH LD_LIBRARY_PATH
mount -a
mdev -s
# 制作镜像文件
qemu-img create -f raw disk.img 512M
mkfs -t ext4 ./disk.img
接下来我们编写一个脚本start_qemu.sh，每次运行都会将rootfs打包到disk.img中，并启动qemu：
（显然这个脚本应该会随着我的使用慢慢更新）
mkdir /home/prx/project/tmpfs
sudo mount -o loop /home/prx/project/disk.img /home/prx/project/tmpfs
sudo cp -r /home/prx/project/rootfs/* /home/prx/project/tmpfs/
sudo umount /home/prx/project/tmpfs
rm -rf /home/prx/project/tmpfs
qemu-system-arm -M vexpress-a9 -m 512M -kernel /home/prx/project/linux-5.10/arch/arm/boot/zImage -dtb /home/prx/project/linux-5.10/arch/arm/boot/dts/vexpress-v2p-ca9.dtb -nographic -append "init=/linuxrc rw rootfstype=ext4 root=/dev/mmcblk0 console=ttyAMA0" -sd /home/prx/project/disk.img --fsdev local,id=kmod_dev,path=/home/prx/share,security_model=none -device virtio-9p-device,fsdev=kmod_dev,mount_tag=kmod_mount -net nic -net tap
启动qemu之后的日志：
ARNING: Image format was not specified for '/home/prx/project/busybox-1.36.0/disk.img' and probing guessed raw.
Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
Specify the 'raw' format explicitly to remove the restrictions.
Booting Linux on physical CPU 0x0
Linux version 5.10.0 (prx@mybuildserver01) (arm-linux-gnueabi-gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0, GNU ld (GNU Binutils for Ubuntu) 2.34) #2 SMP Thu Dec 5 16:24:18 UTC 2024
CPU: ARMv7 Processor [410fc090] revision 0 (ARMv7), cr=10c5387d
CPU: PIPT / VIPT nonaliasing data cache, VIPT nonaliasing instruction cache
OF: fdt: Machine model: V2P-CA9
Memory policy: Data cache writeback
Reserved memory: created DMA memory pool at 0x4c000000, size 8 MiB
OF: reserved mem: initialized node vram@4c000000, compatible id shared-dma-pool
cma: Reserved 16 MiB at 0x7f000000
Zone ranges:
Normal [mem 0x0000000060000000-0x000000007fffffff]
Movable zone start for each node
Early memory node ranges
node 0: [mem 0x0000000060000000-0x000000007fffffff]
Initmem setup node 0 [mem 0x0000000060000000-0x000000007fffffff]
CPU: All CPU(s) started in SVC mode.
percpu: Embedded 19 pages/cpu s46028 r8192 d23604 u77824
Built 1 zonelists, mobility grouping on. Total pages: 130048
Kernel command line: init=/linuxrc root=/dev/mmcblk0 console=ttyAMA0
printk: log_buf_len individual max cpu contribution: 4096 bytes
printk: log_buf_len total cpu_extra contributions: 12288 bytes
printk: log_buf_len min size: 16384 bytes
printk: log_buf_len: 32768 bytes
printk: early log buf free: 14784(90%)
Dentry cache hash table entries: 65536 (order: 6, 262144 bytes, linear)
Inode-cache hash table entries: 32768 (order: 5, 131072 bytes, linear)
mem auto-init: stack:off, heap alloc:off, heap free:off
Memory: 491824K/524288K available (7168K kernel code, 583K rwdata, 1748K rodata, 1024K init, 181K bss, 16080K reserved, 16384K cma-reserved)
SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
rcu: Hierarchical RCU implementation.
rcu: RCU event tracing is enabled.
rcu: RCU restricting CPUs from NR_CPUS=8 to nr_cpu_ids=4.
rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4
NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
GIC CPU mask not found - kernel will fail to boot.
GIC CPU mask not found - kernel will fail to boot.
L2C: platform modifies aux control register: 0x02020000 -> 0x02420000
L2C: DT/platform modifies aux control register: 0x02020000 -> 0x02420000
L2C-310 enabling early BRESP for Cortex-A9
L2C-310 full line of zeros enabled for Cortex-A9
L2C-310 dynamic clock gating disabled, standby mode disabled
L2C-310 cache controller enabled, 8 ways, 128 kB
L2C-310: CACHE_ID 0x410000c8, AUX_CTRL 0x46420001
random: get_random_bytes called from start_kernel+0x384/0x538 with crng_init=0
sched_clock: 32 bits at 24MHz, resolution 41ns, wraps every 89478484971ns
clocksource: arm,sp804: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275 ns
Failed to initialize '/bus@4000000/motherboard/iofpga@7,00000000/timer@12000': -22
smp_twd: clock not found -2
Console: colour dummy device 80x30
Calibrating local timer... 95.92MHz.
Calibrating delay loop... 1869.41 BogoMIPS (lpj=9347072)
pid_max: default: 32768 minimum: 301
Mount-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
Mountpoint-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
CPU: Testing write buffer coherency: ok
CPU0: Spectre v2: using BPIALL workaround
CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
Setting up static identity map for 0x60100000 - 0x60100060
rcu: Hierarchical SRCU implementation.
smp: Bringing up secondary CPUs ...
smp: Brought up 1 node, 1 CPU
SMP: Total of 1 processors activated (1869.41 BogoMIPS).
CPU: All CPU(s) started in SVC mode.
devtmpfs: initialized
VFP support v0.3: implementor 41 architecture 3 part 30 variant 9 rev 0
clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
futex hash table entries: 1024 (order: 4, 65536 bytes, linear)
NET: Registered protocol family 16
DMA: preallocated 256 KiB pool for atomic coherent allocations
cpuidle: using governor ladder
hw-breakpoint: debug architecture 0x4 unsupported.
Serial: AMBA PL011 UART driver
irq: type mismatch, failed to map hwirq-75 for interrupt-controller@1e001000!
SCSI subsystem initialized
usbcore: registered new interface driver usbfs
usbcore: registered new interface driver hub
usbcore: registered new device driver usb
Advanced Linux Sound Architecture Driver Initialized.
clocksource: Switched to clocksource arm,sp804
NET: Registered protocol family 2
tcp_listen_portaddr_hash hash table entries: 512 (order: 0, 6144 bytes, linear)
TCP established hash table entries: 4096 (order: 2, 16384 bytes, linear)
TCP bind hash table entries: 4096 (order: 3, 32768 bytes, linear)
TCP: Hash tables configured (established 4096 bind 4096)
UDP hash table entries: 256 (order: 1, 8192 bytes, linear)
UDP-Lite hash table entries: 256 (order: 1, 8192 bytes, linear)
NET: Registered protocol family 1
RPC: Registered named UNIX socket transport module.
RPC: Registered udp transport module.
RPC: Registered tcp transport module.
RPC: Registered tcp NFSv4.1 backchannel transport module.
hw perfevents: enabled with armv7_cortex_a9 PMU driver, 5 counters available
workingset: timestamp_bits=30 max_order=17 bucket_order=0
squashfs: version 4.0 (2009/01/31) Phillip Lougher
jffs2: version 2.2. (NAND) © 2001-2006 Red Hat, Inc.
9p: Installing v9fs 9p2000 file system support
io scheduler mq-deadline registered
io scheduler kyber registered
i2c i2c-0: Added multiplexed i2c bus 2
physmap-flash 40000000.flash: physmap platform flash device: [mem 0x40000000-0x43ffffff]
40000000.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000
Intel/Sharp Extended Query Table at 0x0031
Using buffer write method
physmap-flash 40000000.flash: physmap platform flash device: [mem 0x44000000-0x47ffffff]
40000000.flash: Found 2 x16 devices at 0x0 in 32-bit bank. Manufacturer ID 0x000000 Chip ID 0x000000
Intel/Sharp Extended Query Table at 0x0031
Using buffer write method
Concatenating MTD devices:
(0): "40000000.flash"
(1): "40000000.flash"
into device "40000000.flash"
physmap-flash 48000000.psram: physmap platform flash device: [mem 0x48000000-0x49ffffff]
libphy: Fixed MDIO Bus: probed
libphy: smsc911x-mdio: probed
smsc911x 4e000000.ethernet eth0: MAC Address: 52:54:00:12:34:56
isp1760 4f000000.usb: bus width: 32, oc: digital
isp1760 4f000000.usb: NXP ISP1760 USB Host Controller
isp1760 4f000000.usb: new USB bus registered, assigned bus number 1
isp1760 4f000000.usb: Scratch test failed.
isp1760 4f000000.usb: can't setup: -19
isp1760 4f000000.usb: USB bus 1 deregistered
usbcore: registered new interface driver usb-storage
ledtrig-cpu: registered to indicate activity on CPUs
usbcore: registered new interface driver usbhid
usbhid: USB HID core driver
NET: Registered protocol family 17
9pnet: Installing 9P2000 support
oprofile: using arm/armv7-ca9
Registering SWP/SWPB emulation handler
aaci-pl041 10004000.aaci: ARM AC'97 Interface PL041 rev0 at 0x10004000, irq 32
aaci-pl041 10004000.aaci: FIFO 512 entries
mmci-pl18x 10005000.mmci: Got CD GPIO
mmci-pl18x 10005000.mmci: Got WP GPIO
mmci-pl18x 10005000.mmci: mmc0: PL181 manf 41 rev0 at 0x10005000 irq 33,34 (pio)
10009000.uart: ttyAMA0 at MMIO 0x10009000 (irq = 37, base_baud = 0) is a PL011 rev1
printk: console [ttyAMA0] enabled
1000a000.uart: ttyAMA1 at MMIO 0x1000a000 (irq = 38, base_baud = 0) is a PL011 rev1
1000b000.uart: ttyAMA2 at MMIO 0x1000b000 (irq = 39, base_baud = 0) is a PL011 rev1
1000c000.uart: ttyAMA3 at MMIO 0x1000c000 (irq = 40, base_baud = 0) is a PL011 rev1
rtc-pl031 10017000.rtc: registered as rtc0
rtc-pl031 10017000.rtc: setting system clock to 2024-12-14T07:36:02 UTC (1734161762)
drm-clcd-pl111 1001f000.clcd: assigned reserved memory node vram@4c000000
drm-clcd-pl111 1001f000.clcd: using device-specific reserved memory
drm-clcd-pl111 1001f000.clcd: core tile graphics present
drm-clcd-pl111 1001f000.clcd: this device will be deactivated
drm-clcd-pl111 1001f000.clcd: Versatile Express init failed - -19
drm-clcd-pl111 10020000.clcd: DVI muxed to daughterboard 1 (core tile) CLCD
drm-clcd-pl111 10020000.clcd: initializing Versatile Express PL111
drm-clcd-pl111 10020000.clcd: found bridge on endpoint 0
drm-clcd-pl111 10020000.clcd: Using non-panel bridge
[drm] Initialized pl111 1.0.0 20170317 for 10020000.clcd on minor 0
mmc0: new SD card at address 4567
input: AT Raw Set 2 keyboard as /devices/platform/bus@4000000/bus@4000000:motherboard/bus@4000000:motherboard:iofpga@7,00000000/10006000.kmi/serio0/input/input0
mmcblk0: mmc0:4567 QEMU! 512 MiB
Console: switching to colour frame buffer device 128x48
drm-clcd-pl111 10020000.clcd: [drm] fb0: pl111drmfb frame buffer device
ALSA device list:
#0: ARM AC'97 Interface PL041 rev0 at 0x10004000, irq 32
input: ImExPS/2 Generic Explorer Mouse as /devices/platform/bus@4000000/bus@4000000:motherboard/bus@4000000:motherboard:iofpga@7,00000000/10007000.kmi/serio1/input/input2
random: fast init done
EXT4-fs (mmcblk0): mounted filesystem with ordered data mode. Opts: (null)
VFS: Mounted root (ext4 filesystem) readonly on device 179:0.
Freeing unused kernel memory: 1024K
Run /linuxrc as init process
random: crng init done
can't run '/etc/init.d/rcS': No such file or directory
Please press Enter to activate this console.
~ #
~ #
~ # ls
bin etc lib lost+found rootfs sys
dev init linuxrc proc sbin usr
~ #
三、共享文件
这一步会搭建qemu 虚拟机和服务器之间的文件共享，方便我们进行软件调试。
我们将会在宿主机创建一个~/share的共享目录，在虚拟机中使用这个目录来跑app
# 服务器（宿主机）安装nfs server
sudo apt install nfs-kernel-server
# 创建共享目录
sudo mkdir ~/share
# 我们希望所有客户端都可以访问该共享文件夹里面的内容，因此分配最高权限：
sudo chown nobody:nogroup ~/share
sudo chmod -R 777 ~/share
# 配置nfs server
sudo vim /etc/exports
/home/prx/share *(rw,sync,no_subtree_check,no_root_squash,insecure)
# 使能新配置
sudo exportfs -arv
# 在fstab中加入下面一行代码，表示/mnt 作为共享目录
kmod_mount /mnt 9p trans=virtio 0 0
为了配置共享目录，需要将虚拟机的启动命令后添加新的选项：
qemu-system-arm -M vexpress-a9 -m 512M -kernel /home/prx/project/linux-5.10/arch/arm/boot/zImage -dtb /home/prx/project/linux-5.10/arch/arm/boot/dts/vexpress-v2p-ca9.dtb -nographic -append "init=/linuxrc rw rootfstype=ext4 root=/dev/mmcblk0 console=ttyAMA0" -sd /home/prx/project/disk.img --fsdev local,id=kmod_dev,path=/home/prx/share,security_model=none -device virtio-9p-device,fsdev=kmod_dev,mount_tag=kmod_mount
四、配置arm虚拟机网络
为arm 虚拟配置网络，可以很方便我们进行开发，我是参考了 https://blog.csdn.net/jasonactions/article/details/118931633，其中不需要e1000网卡也是可以的。
4.1、配置网桥
sudo brctl addbr br0 # 添加一座名为 br0 的网桥
sudo brctl addif br0 eth0 # 在 br0 中添加一个接口
sudo brctl stp br0 off # 如果只有一个网桥，则关闭生成树协议
sudo brctl setfd br0 1 # 设置 br0 的转发延迟
sudo brctl sethello br0 1 # 设置 br0 的 hello 时间
sudo ifconfig br0 0.0.0.0 promisc up # 启用 br0 接口
sudo ifconfig eth0 0.0.0.0 promisc up # 启用网卡接口
sudo dhclient br0 # 从 dhcp 服务器获得 br0 的 IP 地址
sudo brctl show br0 # 查看虚拟网桥列表
sudo brctl showstp br0 # 查看 br0 的各接口信息
4.2、创建TAP 设备
创建一个 TAP 设备，给 QEMU 使用
sudo tunctl -t tap0 -u root # 创建一个 tap0 接口，只允许 root 用户访问
sudo brctl addif br0 tap0 # 在虚拟网桥中增加一个 tap0 接口
sudo ifconfig tap0 0.0.0.0 promisc up # 启用 tap0 接口
sudo brctl showstp br0
配置完成后的网络情况为：
prx@mybuildserver01:~/tmp/e1000e-3.8.4/src$ ifconfig
br0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 1500
inet 192.168.1.83 netmask 255.255.255.0 broadcast 192.168.1.255
inet6 2409:8a55:3115:1230:20c:29ff:fe4d:4f0e prefixlen 64 scopeid 0x0<global>
inet6 2409:8a55:3115:1230:6d78:c0f:2ad:4466 prefixlen 64 scopeid 0x0<global>
inet6 fe80::20c:29ff:fe4d:4f0e prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:4d:4f:0e txqueuelen 1000 (Ethernet)
RX packets 14280 bytes 9797641 (9.7 MB)
RX errors 0 dropped 11 overruns 0 frame 0
TX packets 8656 bytes 2909545 (2.9 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
ens33: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST> mtu 1500
inet6 2409:8a55:3115:1230:20c:29ff:fe4d:4f0e prefixlen 64 scopeid 0x0<global>
inet6 fe80::20c:29ff:fe4d:4f0e prefixlen 64 scopeid 0x20<link>
ether 00:0c:29:4d:4f:0e txqueuelen 1000 (Ethernet)
RX packets 26195 bytes 11815835 (11.8 MB)
RX errors 0 dropped 500 overruns 0 frame 0
TX packets 15385 bytes 6043108 (6.0 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 1000 (Local Loopback)
RX packets 23625 bytes 7430833 (7.4 MB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 23625 bytes 7430833 (7.4 MB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
tap0: flags=4355<UP,BROADCAST,PROMISC,MULTICAST> mtu 1500
ether 26:1e:c7:3e:ff:05 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
4.3、更新qemu启动参数
新增启动参数，指定使用tap
-net nic -net tap
更新rootfs/etc/init.d/rcS
PATH=/sbin:/bin:/usr/sbin:/usr/bin:$PATH
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/lib:/usr/lib
export PATH LD_LIBRARY_PATH
mount -a
mdev -s
# eth0 的网段和br0 的网段在同一个就可以
ifconfig eth0 192.168.1.183 netmask 255.255.255.0
五、trace内核函数
使用ftrace来跟踪内核函数的调用过程，可以帮助我们更好的学习源码。这里以trace kernel_clone() 这个函数为例：
首先需要打开内核ftrace功能：
如图所示开启 kernel function tracer，kernel function graph tracer（跟踪函数调用）
更新内核之后，修改fstab，使其能将tracefs挂载到根目录tracing下。
proc /proc proc defaults 0 0
tmpfs /tmp tmpfs defaults 0 0
sysfs /sys sysfs defaults 0 0
tracefs /tracing tracefs defaults 0 0
kmod_mount /mnt 9p trans=virtio 0 0
在虚拟机端运行如下命令抓取kernel_clone()函数的调用栈：
# 设置tracer，跟踪函数
echo function > /tracing/current_tracer
# 跟踪kernel_clone的函数调用
echo kernel_clone > /tracing/set_graph_function
# 开始跟踪
echo 1 > /tracing/tracing_on
# 运行app，会创建线程
/app/pthread
echo 0 > /tracing/tracing_on
# 查看trace log
cat /tracing/trace
