缘起
使用cbsd创建riscv jail的时候提示：
you have no qemu-user, please install qemu-devle with BSD_USER and STATIC ops (emulators/qemu-devel)
使用pkg安装之后，创建的riscv jail启动报错：
Starting jail: fbriscv, parallel timeout=5
chroot: /bin/sh: Exec format error
* Copying emulator to /bin
emulator error: /usr/sbin/chroot /usr/jails/basejail/base_riscv_riscv64_14.1 /bin/qemu-riscv64-static /bin/sh -c "echo ping": Assertion failed: (p_rcu_reader->depth != 0), function rcu_read_unlock, file /wrkdirs/usr/ports/emulators/qemu-user-static-devel/work/qemu-bsd-user-64ff0f053df/include/qemu/rcu.h, line 101.
于是准备重新编译安装qemu-devel
开工
首先安装prots系统：
git clone --depth 1 https://git.FreeBSD.org/ports.git /usr/ports
编译安装qemu-devel
进入/usr/ports/emulators/qemu-devel
执行make config
make config选中STATIC_LINK 回车
注意选了STATIC_LINK就不能选X11 support 和GTK3 GTK 3 GUI toolkit support
经实验，选中STATIC_LINK无法编译成功，要去掉才行。
另外编译的时候需要用
make而不能用make -j 8 ，带了-j参数会编译失败。
编译安装后提示：
FreeBSD host notes
==================
- Needs to set net.link.tap.user_open sysctl in order to use /dev/tap*
networking as non-root. Don't forget to adjust device node
permissions in /etc/devfs.rules.
- slirp (usermode networking) is fixed now, you still have to manually
do: echo nameserver 10.0.2.3 >/etc/resolv.conf but that is normal. And
you have to wait a bit for dhclient to do its thing; traffic to
address 10.0.2.2 is routed to 127.1 on the host.
- The -smb option (smb-export local dir to guest using the default slirp
networking) needs the samba port/package installed in addition to
qemu. (SAMBA knob.)
- If you want to use usb devices connected to the host in the guest
you can use usbredir over the network (see below); also unless you are
running qemu as root you then need to fix permissions for /dev/ugen*
device nodes: put a rule in /etc/devfs.rules, activate it in
/etc/rc.conf and run /etc/rc.d/devfs restart. Example devfs.rules:
[ugen_ruleset=20]
add path 'ugen*' mode 660 group operator
corresponding rc.conf line:
devfs_system_ruleset="ugen_ruleset"
- If you want to test the new usb network redirection (USBREDIR option)
see this thread by Hans de Goede <hdegoede <at> redhat.com>:
http://thread.gmane.org/gmane.comp.emulators.qemu/110176/focus=110183
Quote:
Example usage:
1) Start usbredirserver for a usb device:
sudo usbredirserver 045e:0772
2) Start qemu with usb2 support + a chardev talking to usbredirserver +
a usb-redir device using this chardev:
qemu -usb \
-readconfig docs/ich9-ehci-uhci.cfg \
-chardev socket,id=usbredirchardev,host=localhost,port=4000 \
-device usb-redir,chardev=usbredirchardev,id=usbredirdev ...
[you would replace docs/ich9-ehci-uhci.cfg with e.g.
/usr/local/share/doc/qemu/docs/ich9-ehci-uhci.cfg, but turns out
ehci was broken for me here with FreeBSD guests and the previous
qemu version at least, I got:
FETCHENTRY: entry at 22C5484 is of type 2 which is not supported yet
processing error - resetting ehci HC
Assertion failed: (0), function ehci_advance_state, file /data/ports/emulators/qemu-devel/work/qemu-0.15.0/hw/usb-ehci.c, line 2045.
The new qemu version works better tho.]
- Still usb: since the hub is no longer attached to the uchi controller and
the wakeup mechanism, resume interrupt is not implemented yet linux guests
will suspend the bus, i.e. they wont see devices usb_add'ed after its
(linux') uhci module got loaded. Workaround: either add devices before
linux loads the module or rmmod and modprobe it afterwards. [Not sure
if this still applies to the new libusb host code used on recent
10-current.]
- If you get repeated `atapi_poll called!' console messages with FreeBSD
guests or other weird cdrom problems then thats probably because the guest
has atapicam loaded, which for reasons still to be determined has problems
with qemu's now by default enabled cdrom dma. You can build the port with
CDROM_DMA disabled to disable it. [Looks like this is fixed in recent
FreeBSD guest versions.]
- If you build qemu wihout SDL and then get crashes running it try passing it
-nographic.
- qemu's network boot roms (-boot n) have a bug when bootfiles sizes are a
multiple of blksize, if this affects you (like with FreeBSD's /boot/pxeboot)
you can do like
cp /boot/pxeboot pxeboot-qemu && chmod +w pxeboot-qemu && echo >>pxeboot-qemu
and then use pxeboot-qemu. Actually you need recent btx code because
of the real mode boot problem, so use at least pxeboot from there.
- If you use slirp (usernet, the default) and want to mount nfs into the
guest and you are not running qemu as root, then mountd(8) on the
exporting box needs to be run with -n in order to accept requests from
ports >= 1024.
- (not FreeBSD-specific:) There have been reports of qcow2 corruption with (at
least) win2k guests on recent kvm (which uses similar qcow2 code than qemu
now, see this thread:
http://lists.gnu.org/archive/html/qemu-devel/2009-02/msg00713.html -
the consensus on that thread seems to be that qcow(2) code has always been
experimental and you should use raw images if you want reliability; raw is
also usually faster.) You should be able to migrate existing images to raw
using qemu-img(1)'s convert function; raw doesn't support advanced features
like snapshots tho. [a few important qcow2 bugfixed have been committed in
the meantime so this _might_ be less of an issue now; and meanwhile there
also is the new qed format - I don't know how stable that one is.]
- (also not FreeBSD-specific:) It is recommended to pass raw images using the
new -drive syntax, specifying format=raw explicitly in order to avoid
malicious guests being able to exploit the format autodetection thats
otherwise getting used. (Not that you should run malicious guests anyway,
but this eleminates at least a known attack vector.)
- qemu now has improved physical cdrom support, but still there is at
least one known problem: you need to have the guest eject the disc if you
want to change it/take it out, or otherwise the guest may continue using
state (like size) of the old disc. (You can also do like `change ide1-cd0
/dev/acd0' in the monitor after taking out the disc if a guest cannot eject
it itself.)
- The default configuration location (qemu-ifup script etc.) has been changed
from /etc to PREFIX/etc (usually /usr/local/etc). Move your files
accordingly.
- The pcap code (-net nic... -net pcap,ifname=...) should work properly now,
with only one exception: Advanced features like TSO used on the host
interface can cause oversize packets which now do get truncated to avoid
confusing/panicing guests but of course still will cause retransmissions.
So if you see slow throughput and `pcap_send: packet size > ..., truncating'
messages on qemu's tty try disabling TSO etc on the host interface at least
while using pcap.
- kqemu is no longer supported in qemu upstream after the 0.11 branch
was created, which means also not in this version. (Linux has moved
on to kvm now for qemu(-like) virtualization needs, so if you want qemu
to go faster and don't want to switch to virtualbox or stick to the older
emulators/qemu port which is at 0.11.1 atm and as such still supports
kqemu you should help getting the FreeBSD kvm port updated and
completed:
http://wiki.freebsd.org/FabioChecconi/PortingLinuxKVMToFreeBSD
)
===> SECURITY REPORT:
This port has installed the following files which may act as network
servers and may therefore pose a remote security risk to the system.
/usr/local/bin/qemu-img
/usr/local/bin/qemu-system-nios2
/usr/local/bin/qemu-system-riscv64
/usr/local/bin/qemu-system-mips64
/usr/local/bin/qemu-system-mips64el
/usr/local/bin/qemu-system-loongarch64
/usr/local/bin/qemu-system-microblaze
/usr/local/bin/qemu-system-sparc
/usr/local/bin/qemu-system-riscv32
/usr/local/bin/qemu-nbd
/usr/local/bin/qemu-system-ppc64
/usr/local/bin/qemu-system-avr
/usr/local/bin/qemu-system-sh4
/usr/local/bin/qemu-system-or1k
/usr/local/bin/qemu-system-cris
/usr/local/bin/qemu-x86_64
/usr/local/bin/qemu-system-mipsel
/usr/local/bin/qemu-system-alpha
/usr/local/bin/qemu-edid
/usr/local/bin/qemu-system-sparc64
/usr/local/bin/qemu-arm
/usr/local/bin/qemu-storage-daemon
/usr/local/bin/qemu-system-ppc
/usr/local/bin/qemu-system-microblazeel
/usr/local/bin/qemu-system-xtensa
/usr/local/bin/qemu-system-aarch64
/usr/local/bin/qemu-system-xtensaeb
/usr/local/bin/qemu-io
/usr/local/bin/qemu-i386
/usr/local/bin/qemu-system-sh4eb
/usr/local/bin/qemu-system-arm
/usr/local/bin/qemu-system-m68k
/usr/local/bin/qemu-system-i386
/usr/local/bin/qemu-system-x86_64
/usr/local/bin/qemu-system-s390x
/usr/local/bin/qemu-system-mips
/usr/local/bin/qemu-system-rx
/usr/local/bin/qemu-system-hppa
/usr/local/bin/qemu-system-tricore
If there are vulnerabilities in these programs there may be a security
risk to the system. FreeBSD makes no guarantee about the security of
ports included in the Ports Collection. Please type 'make deinstall'
to deinstall the port if this is a concern.
For more information, and contact details about the security
status of this software, see the following webpage:
https://www.qemu.org/
编译安装的时候有很多波折，见后面调试部分
pkg 安装qemu
本来已经pkg 安装qemu-devel，再把qemu装上，发现还是新装了几个软件
pkg install qemu装好后提示：
Message from qemu-8.2.2_1:
--
FreeBSD host notes
==================
- Needs to set net.link.tap.user_open sysctl in order to use /dev/tap*
networking as non-root. Don't forget to adjust device node permissions in
/etc/devfs.rules.
- Expect timer problems when guest kernel HZ is > hosts. The linux 2.6
kernel uses 1000 by default btw. (changed to 250 later, and recent linux
kernels now no longer have a fixed HZ, aka `tickless kernel'...)
Enabling /dev/rtc doesn't seem to help either (not included since it needs
a patch to emulators/rtc.)
- The -smb option (smb-export local dir to guest using the default
slirp networking) needs the samba port/package installed
in addition to qemu. (SAMBA knob.)
- If you want to use usb devices connected to the host in the guest
yot can use usbredir over the network (see below); also unless you are
running qemu as root you then need to fix permissions for /dev/ugen*
device nodes: if you are on 5.x or later (devfs) put a rule in
/etc/devfs.rules, activate it in /etc/rc.conf and run /etc/rc.d/devfs
restart. Example devfs.rules:
[ugen_ruleset=20]
add path 'ugen*' mode 660 group operator
corresponding rc.conf line:
devfs_system_ruleset="ugen_ruleset"
- Still usb: since the hub is no longer attached to the uchi controller and
the wakeup mechanism, resume interrupt is not implemented yet linux guests
will suspend the bus, i.e. they wont see devices usb_add'ed after its
(linux') uhci module got loaded. Workaround: either add devices before
linux loads the module or rmmod and modprobe it afterwards. [Not sure
if this still applies to the new libusb host code used on recent
10-current.]
- If you get repeated `atapi_poll called!' console messages with FreeBSD
guests or other weird cdrom problems then thats probably because the guest
has atapicam loaded, which for reasons still to be determined has problems
with qemu's now by default enabled cdrom dma. You can build the port with
CDROM_DMA disabled to disable it. [Looks like this is fixed in recent
FreeBSD guest versions.]
- If you build qemu wihout SDL and then get crashes running it try passing it
-nographic. This should probably be default in that case...
- qemu's network boot roms (-boot n) have a bug when bootfiles sizes are a
multiple of blksize, if this affects you (like with FreeBSD's /boot/pxeboot)
you can do like
cp /boot/pxeboot pxeboot-qemu && chmod +w pxeboot-qemu && echo >>pxeboot-qemu
and then use pxeboot-qemu. Actually you need recent btx code
(from after 7.0 was released) because of the real mode boot
problem, so use at least pxeboot from there. And I just did that
for the pxeboot extracted out of
ftp://ftp.freebsd.org/pub/FreeBSD/snapshots/200805/7.0-STABLE-200805-i386-bootonly.iso
and placed it here:
http://people.freebsd.org/~nox/qemu/pxeboot-qemu
- If you use slirp (usernet, the default) and want to mount nfs into the guest
and you are not running qemu as root, then mountd(8) on the exporting box
needs to be run with -n in order to accept requests from ports >= 1024.
root@fbhost:/usr/ports/emulators/qemu-devel #
后来又重新安装了qemu-devel
pkg install qemu-devel当时之所以编译安装，就是为了STATIC_LINK这个参数。但是编译安装的时候用了这个参数也编译不过去，所以不如直接pkg按钻个方便了。
调试
报错pkg-static: open(/usr/ports/textproc/py-sphinx/work-py311/.metadir.py311-sphinx): No such file or directory
--- security-check ---
--- fake-pkg.py311-sphinx ---
===> Registering installation for py311-sphinx-5.3.0_1,1 as automatic
pkg-static: open(/usr/ports/textproc/py-sphinx/work-py311/.metadir.py311-sphinx): No such file or directory
*** [fake-pkg.py311-sphinx] Error code 1
make[2]: stopped in /usr/ports/textproc/py-sphinx
尝试 pkg install textproc/py-sphinx
不行。尝试pkg install textproc/py-sphinx_rtd_theme
还是不行。尝试pkg install python311 ，还是不行
删除work-py311目录，不行
期间发现安装的python311无法执行，原来是老终端里没有更新path，重新登录即可执行python3.11
执行python3.11 -m ensurepip ,这样就安装好了Successfully installed pip-24.0 setuptools-65.5.0
还是不行。在python里面安装sphinx_rtd_theme
python3.11 -m pip install sphinx_rtd_theme还是不行，把Makefile修改一下，去掉sphinx的依赖
# BUILD_DEPENDS= sphinx-build:textproc/py-sphinx \
${PYTHON_PKGNAMEPREFIX}sphinx_rtd_theme>0:textproc/py-sphinx_rtd_theme@${PY_FLAV
OR} \
出现新的报错
如何提高pip安装软件的速度
python3.11 -m pip config set global.index-url https://mirror.baidu.com/pypi/simple报错pkg-static: open(/usr/ports/devel/bison/work/.metadir.bison): No such file or directory
===> Registering installation for bison-3.8.2_2,1 as automatic
pkg-static: open(/usr/ports/devel/bison/work/.metadir.bison): No such file or directory
*** [fake-pkg.bison] Error code 1
pkg install bison-3.8.2_1,1
搞定
报错没有cmake
pkg install gmake 搞定
报错没有ninja
发现Makefile还有几个需求，一起装了
pkg install ninja perl5 pkgconfig
结果不行，还是一个一个装吧：pkg install ninja
安装pkg install print/texinfo
报错../meson.build:2420:15: ERROR: C prefer_static library 'inotify' not found
Header "sys/inotify.h" has symbol "inotify_init" : YES
Header "sys/inotify.h" has symbol "inotify_init1" : YES
../meson.build:2420:15: ERROR: C prefer_static library 'inotify' not found
A full log can be found at /usr/ports/emulators/qemu-devel/work/qemu-ff202817dc2b0b3b42992fa7f1c
e503f081068fe/build/meson-logs/meson-log.txt
ERROR: meson setup failed
===> Script "configure" failed unexpectedly.
Please report the problem to bofh@FreeBSD.org [maintainer] and attach the
"/usr/ports/emulators/qemu-devel/work/qemu-ff202817dc2b0b3b42992fa7f1ce503f081068fe/config.log"
including the output of the failure of your make command. Also, it might be
a good idea to provide an overview of all packages installed on your system
怀疑是把Makefile里面的sphinx注释掉的原因。把注释去掉，还是报错
BUILD_DEPENDS= sphinx-build:textproc/py-sphinx到目录/usr/ports/textproc/py-sphinx_rtd_theme
然后make install
再回到目录/usr/ports/emulators/qemu-devel
make install ,依旧。
针对这个报错，重新编译meson
cd /usr/ports/devel/meson && make install
依旧。
但是把STATIC_LINK选项去掉，能编译下去了。
最后编译完了之后报错...
编译结束之后报错几个文件找不到
把make -j 8 改成make ，就又能继续下去了。
但是最终make install的时候报错：
===> Registering installation for qemu-devel-8.3.0.20240229
pkg-static: Unable to access file /usr/ports/emulators/qemu-devel/work/stage/usr/local/include/fdt.h:No such file or directory
pkg-static: Unable to access file /usr/ports/emulators/qemu-devel/work/stage/usr/local/include/libfdt.h:No such file or directory
pkg-static: Unable to access file /usr/ports/emulators/qemu-devel/work/stage/usr/local/include/libfdt_env.h:No such file or directory
pkg-static: Unable to access file /usr/ports/emulators/qemu-devel/work/stage/usr/local/lib/libfdt.a:No such file or directory
pkg-static: Unable to access file /usr/ports/emulators/qemu-devel/work/stage/usr/local/libdata/pkgconfig/libfdt.pc:No such file or directory
*** Error code 1
可能是前面pkg install qemu导致的，执行pkg install qemu-devel
然后再来make install，提示需要先uninstall reinstall ，结果还是报错文件找不到。
最后是重新删除build目录，创建build目录，重新make config ，make install，终于编译成功了。
