IOMMU Configuration Issues¶

This section describes possible reasons why mapping an HBA device to the host system may fail and ways to resolve the problem.

Verify that the version of your Linux kernel and KVM support IOMMU functionality.
- To check the version of the Linux kernel, run uname -r.
- To check the version of KVM, run /usr/libexec/qemu-kvm --version.
Refer to the Supported Platforms and Hardware section for the list of supported versions.

Make sure that the IOMMU functionality is part of your host boot process by running the following command:

dmesg | grep -i "iommu"

Your output should look similar to the following:

[    0.000000] Command line: BOOT_IMAGE=(hd1,gpt2)/vmlinuz-5.14.0-611.16.1.el9_7.x86_64 root=/dev/mapper/rhel-root ro resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet crashkernel=1G-2G:192M,2G-64G:256M,64G-:512M intel_iommu=on iommu_pt
[    0.043575] Kernel command line: BOOT_IMAGE=(hd1,gpt2)/vmlinuz-5.14.0-611.16.1.el9_7.x86_64 root=/dev/mapper/rhel-root ro resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet crashkernel=1G-2G:192M,2G-64G:256M,64G-:512M intel_iommu=on iommu_pt
[    0.043714] DMAR: IOMMU enabled
[    0.163101] DMAR-IR: IOAPIC id 12 under DRHD base  0xfbffc000 IOMMU 2
[    0.163103] DMAR-IR: IOAPIC id 11 under DRHD base  0xf37fc000 IOMMU 1
[    0.163104] DMAR-IR: IOAPIC id 10 under DRHD base  0xeaffc000 IOMMU 0
[    0.163105] DMAR-IR: IOAPIC id 8 under DRHD base  0xe27fc000 IOMMU 3
[    0.163106] DMAR-IR: IOAPIC id 9 under DRHD base  0xe27fc000 IOMMU 3
[    0.441716] iommu: Default domain type: Translated
[    0.441716] iommu: DMA domain TLB invalidation policy: lazy mode
.
.
.

If the dmesg | grep -i "iommu" command does not produce any output, you can run the find /sys/kernel/iommu_groups/ -type d -maxdepth 1 command which should produce the output similar to the following:

/sys/kernel/iommu_groups/
/sys/kernel/iommu_groups/55
/sys/kernel/iommu_groups/83
.
.
.

If you do not get any output from either dmesg | grep -i "iommu" or find /sys/kernel/iommu_groups/ -maxdepth 1 -type d, go back and make sure you have performed the steps described in the Enabling IOMMU Support in BIOS/UEFI and Enabling IOMMU Support in Linux Kernel sections.

Make sure that the necessary drivers are installed and can be loaded by running the lsmod | grep "kvm" command.
- On AMD-based hosts, your output will look similar to the following if the drivers are installed:
```
kvm_amd               245760  4
kvm                  1404928  3 kvm_amd
ccp                   163840  1 kvm_amd
```
- On Intel-based hosts, your output will look similar to the following if the drivers are installed:
```
kvm_intel              356352  0
kvm                    970752  1 kvm_intel
irqbypass               12288  1 kvm
```

System Hang¶

After configuring PCI passthrough, it may be impossible to start OpenVMS on a KVM virtual machine because the system hangs on boot.

Open your CLI and perform the following steps to resolve this issue:

Reboot your OpenVMS system by sending the reset signal to your VM by executing virsh reset vm-name, and then, at the BOOTMGR> prompt, try booting OpenVMS.

If the system still hangs on boot, do the following:
1. Execute virsh destroy vm-name to shut down your VM.
2. Restart your VM by executing virsh start vm-name.
3. Boot OpenVMS.
  
  If the system still hangs on boot, proceed to the next step.
Deassign both ports of the Fibre Channel HBA device by running the virsh nodedev-detach pci_xxxx_xx_xx_x command, replacing the xxxx_xx_xx_x with the PCI addresses for both ports of the device that you found in the Assigning Fibre Channel HBA to Virtual Machine section.

Boot OpenVMS and check the system version by executing the PRODUCT SHOW PRODUCT VMS/FULL command:

$ PRODUCT SHOW PRODUCT VMS/FULL
---------------------- ----------- --------- ------------------------------- 
PRODUCT                KIT TYPE    STATE     MAINTENANCE                     
---------------------- ----------- --------- ------------------------------- 
VSI X86VMS VMS V9.2-3  Oper System Installed VSI X86VMS VMS923X_DCL V4.0
                                             VSI X86VMS VMS923X_PCSI V1.0
                                             VSI X86VMS VMS923X_PERF_UPD V1.0
                                             VSI X86VMS VMS923X_UPDATE V3.0

Make sure that you are running VSI OpenVMS for x86-64 Version 9.2-3 + Update V3 or higher and have the PERF_UPD V1.0 ECO kit installed.

Try re-assigning the Fibre Channel HBA device to your VM. If the problem persists, contact VSI Support via support@vmssoftware.com.

Error Messages¶

When executing the sudo virsh start vm-name command, the system produces the errors specified below:

error: Failed to start domain 'vm-name'
error: Requested operation is not valid: PCI device xxxx:xx:xx.x is in use by driver QEMU, domain vm-name

These errors signify that the Fibre Channel HBA device is in use by another VM.

To resolve this, shut down the virtual machine that uses the Fibre Channel HBA and run sudo virsh start vm-name again.

Alternatively, assign each VM their designated Fibre Channel HBA device.