主板是精粤 b660i ,一个 8125 2.5G 网口和一个 8111 千兆网口。配置完直通后通过 web 添加 pci 将 8111 直通给 openwrt ,然后 openwrt 虚拟机启动时报错:
kvm: ../hw/pci/pci.c:1562: pci_irq_handler: Assertion `0 <= irq_num && irq_num < PCI_NUM_PINS' failed.
TASK ERROR: start failed: QEMU exited with code 1
pve syslog:
Dec 22 15:15:13 aio pvedaemon[1399]: <root@pam> starting task UPID:aio:00006CA9:0005885C:63A40401:qmstart:100:root@pam:
Dec 22 15:15:13 aio pvedaemon[27817]: start VM 100: UPID:aio:00006CA9:0005885C:63A40401:qmstart:100:root@pam:
Dec 22 15:15:13 aio kernel: vfio-pci 0000:04:00.0: Unable to change power state from D3cold to D0, device inaccessible
Dec 22 15:15:13 aio kernel: vfio-pci 0000:04:00.0: Unable to change power state from D3cold to D0, device inaccessible
Dec 22 15:15:14 aio systemd[1]: Started 100.scope.
Dec 22 15:15:14 aio systemd-udevd[27834]: Using default interface naming scheme 'v247'.
Dec 22 15:15:14 aio systemd-udevd[27834]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Dec 22 15:15:15 aio kernel: device tap100i0 entered promiscuous mode
Dec 22 15:15:15 aio kernel: fwbr100i0: port 1(fwln100i0) entered disabled state
Dec 22 15:15:15 aio kernel: vmbr0: port 2(fwpr100p0) entered disabled state
Dec 22 15:15:15 aio kernel: device fwln100i0 left promiscuous mode
Dec 22 15:15:15 aio kernel: fwbr100i0: port 1(fwln100i0) entered disabled state
Dec 22 15:15:15 aio kernel: device fwpr100p0 left promiscuous mode
Dec 22 15:15:15 aio kernel: vmbr0: port 2(fwpr100p0) entered disabled state
Dec 22 15:15:15 aio systemd-udevd[27834]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Dec 22 15:15:15 aio systemd-udevd[27834]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Dec 22 15:15:15 aio systemd-udevd[27837]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
Dec 22 15:15:15 aio systemd-udevd[27837]: Using default interface naming scheme 'v247'.
Dec 22 15:15:15 aio kernel: vmbr0: port 2(fwpr100p0) entered blocking state
Dec 22 15:15:15 aio kernel: vmbr0: port 2(fwpr100p0) entered disabled state
Dec 22 15:15:15 aio kernel: device fwpr100p0 entered promiscuous mode
Dec 22 15:15:15 aio kernel: vmbr0: port 2(fwpr100p0) entered blocking state
Dec 22 15:15:15 aio kernel: vmbr0: port 2(fwpr100p0) entered forwarding state
Dec 22 15:15:15 aio kernel: fwbr100i0: port 1(fwln100i0) entered blocking state
Dec 22 15:15:15 aio kernel: fwbr100i0: port 1(fwln100i0) entered disabled state
Dec 22 15:15:15 aio kernel: device fwln100i0 entered promiscuous mode
Dec 22 15:15:15 aio kernel: fwbr100i0: port 1(fwln100i0) entered blocking state
Dec 22 15:15:15 aio kernel: fwbr100i0: port 1(fwln100i0) entered forwarding state
Dec 22 15:15:15 aio kernel: fwbr100i0: port 2(tap100i0) entered blocking state
Dec 22 15:15:15 aio kernel: fwbr100i0: port 2(tap100i0) entered disabled state
Dec 22 15:15:15 aio kernel: fwbr100i0: port 2(tap100i0) entered blocking state
Dec 22 15:15:15 aio kernel: fwbr100i0: port 2(tap100i0) entered forwarding state
Dec 22 15:15:15 aio kernel: vfio-pci 0000:04:00.0: Unable to change power state from D3cold to D0, device inaccessible
Dec 22 15:15:15 aio kernel: vfio-pci 0000:04:00.0: Unable to change power state from D3cold to D0, device inaccessible
Dec 22 15:15:15 aio kernel: vfio-pci 0000:04:00.0: Unable to change power state from D3cold to D0, device inaccessible
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: vfio_cap_init: hiding cap 0xff@0xff
Dec 22 15:15:16 aio kernel: fwbr100i0: port 2(tap100i0) entered disabled state
Dec 22 15:15:16 aio kernel: fwbr100i0: port 2(tap100i0) entered disabled state
Dec 22 15:15:16 aio pvedaemon[1400]: VM 100 qmp command failed - VM 100 not running
Dec 22 15:15:16 aio pvestatd[1370]: VM 100 qmp command failed - VM 100 not running
Dec 22 15:15:16 aio kernel: vfio-pci 0000:04:00.0: Unable to change power state from D3cold to D0, device inaccessible
Dec 22 15:15:16 aio pvedaemon[27817]: start failed: QEMU exited with code 1
Dec 22 15:15:16 aio pvedaemon[1399]: <root@pam> end task UPID:aio:00006CA9:0005885C:63A40401:qmstart:100:root@pam: start failed: QEMU exited with code 1
0000:04:00
是千兆网口,0000:05:00
是无线网卡,删除掉千兆网口的 PCI 只保留无线网卡是可以正常启动的。
grub:
GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on pcie_acs_override=downstream,multifunction"
GRUB_CMDLINE_LINUX=""
vm config:
balloon: 512
boot: order=scsi0
cores: 4
hostpci0: 0000:04:00
hostpci1: 0000:05:00
memory: 2048
meta: creation-qemu=7.1.0,ctime=1671628207
name: openwrt
net0: virtio=6E:35:49:91:0D:46,bridge=vmbr0,firewall=1
numa: 0
onboot: 1
ostype: l26
scsi0: local-lvm:vm-100-disk-0,iothread=1,size=420M
scsihw: virtio-scsi-single
smbios1: uuid=a3c5ba9b-a5f7-4f32-b9e3-056f0901cc01
sockets: 1
vmgenid: 1aa8b1ee-37e5-4617-9f93-fd2c8fbf98ab
1
WinkeyLin 2022-12-22 15:47:19 +08:00 via Android
这个网口是不是被 pve 占用做管理网口了
|
3
onetown 2022-12-22 16:04:47 +08:00
试试 启动参数里加上 intel_iommu=on iommu=pt
|
4
onetown 2022-12-22 16:06:35 +08:00
另外, 如果不是板载的网卡的话, 可以试试换个 pci 插槽, 不过你这个估计是板载的。
这个报错其实是 kvm 预防你的 pci 设备直接访问主机内存, 主要是不在一个 iommo group 里的话就会 crash. |
5
CRUD OP @onetown 也是不行的老哥,一样的错误信息,网卡是板载的。看 syslog 里面相对比较明确的提示是`vfio-pci 0000:04:00.0: Unable to change power state from D3cold to D0, device inaccessible`,但是按这个提示没找到什么有用的线索。
|
6
onetown 2022-12-22 22:03:15 +08:00
@CRUD 嗯, 看日志是 vfio 驱动的问题, 如果你不是一定要上 dpdk 的话, 可以考虑把 vfio 屏蔽掉试试
内核在加载驱动的时候直接使用了 vfio 驱动, 你的网卡应该不是 100% vfio 兼容的. 在 /etc/modprobe.d/blacklist.conf 里追加一行 blacklist vfio-pci |
15
heider 351 天前
我是 770i 遇到了同样的问题
看文章 https://docs.opennebula.io/6.4/open_cluster_deployment/kvm_node/pci_passthrough.html VFIO 设备绑定章节好像可以解决直通问题,抽空试一下 |
16
s1e42NxZVE484pwH 157 天前
@heider #15 成功了吗?
|