-
Enable Intel VT-d in BIOS and in the kernel
Skip this step if Intel VT-d is already enabled and working.
-
Verify support
Verify if the PCI device with SR-IOV capabilities are detected. This example lists an Intel 82576 network interface card which supports SR-IOV. Use the lspci
command to verify if the device was detected.
# lspci
03:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
03:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
Note that the output has been modified to remove all other devices.
-
Start the SR-IOV kernel modules
If the device is supported the driver kernel module should be loaded automatically by the kernel. Optional parameters can be passed to the module using the modprobe
command. The Intel 82576 network interface card uses the igb
driver kernel module.
# modprobe igb [<option>=<VAL1>,<VAL2>,]
# lsmod |grep igb
igb 87592 0
dca 6708 1 igb
-
Activate Virtual Functions
The max_vfs
parameter of the igb
module allocates the maximum number of Virtual Functions. The max_vfs
parameter causes the driver to spawn, up to the value of the parameter in, Virtual Functions. For this particular card the valid range is 0
to 7
.
Remove the module to change the variable.
# modprobe -r igb
Restart the module with the max_vfs
set to 1
or any number of Virtual Functions up to the maximum supported by your device.
# modprobe igb max_vfs=7
-
Make the Virtual Functions persistent
The modprobe
command /etc/modprobe.d/igb.conf options igb max_vfs=7
-
Inspect the new Virtual Functions
Using the lspci
command, list the newly added Virtual Functions attached to the Intel 82576 network device.
# lspci | grep 82576
0b:00.0 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
0b:00.1 Ethernet controller: Intel Corporation 82576 Gigabit Network Connection (rev 01)
0b:10.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
0b:10.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
0b:10.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
0b:10.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
0b:10.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
0b:10.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
0b:10.6 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
0b:10.7 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
0b:11.0 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
0b:11.1 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
0b:11.2 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
0b:11.3 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
0b:11.4 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
0b:11.5 Ethernet controller: Intel Corporation 82576 Virtual Function (rev 01)
The identifier for the PCI device is found with the -n
parameter of the lspci
command. The Physical Functions corresponds to 0b:00.0
and 0b:00.1
. All the Virtual Functions have Virtual Function
in the description.
-
Verify devices exist with virsh
The libvirt
service must recognize the device before adding a device to a guest. libvirt
uses a similar notation to the lspci
output. All punctuation characters, ; and ., in lspci output are changed to underscores (_).
Use the virsh nodedev-list
command and the grep command to filter the Intel 82576 network device from the list of available host devices. 0b
is the filter for the Intel 82576 network devices in this example. This may vary for your system and may result in additional devices.
# virsh nodedev-list | grep 0b
pci_0000_0b_00_0
pci_0000_0b_00_1
pci_0000_0b_10_0
pci_0000_0b_10_1
pci_0000_0b_10_2
pci_0000_0b_10_3
pci_0000_0b_10_4
pci_0000_0b_10_5
pci_0000_0b_10_6
pci_0000_0b_11_7
pci_0000_0b_11_1
pci_0000_0b_11_2
pci_0000_0b_11_3
pci_0000_0b_11_4
pci_0000_0b_11_5
The serial numbers for the Virtual Functions and Physical Functions should be in the list.
-
Get device details with virsh
The pci_0000_0b_00_0
is one of the Physical Functions and pci_0000_0b_10_0
is the first corresponding Virtual Function for that Physical Function. Use the virsh nodedev-dumpxml
command to get advanced output for both devices.
<device>
<name>pci_0000_0b_00_0</name>
<parent>pci_0000_00_01_0</parent>
<driver>
<name>igb</name>
</driver>
<capability type='pci'>
<domain>0</domain>
<bus>11</bus>
<slot>0</slot>
<function>0</function>
<product id='0x10c9'>Intel Corporation</product>
<vendor id='0x8086'>82576 Gigabit Network Connection</vendor>
</capability>
</device>
# virsh nodedev-dumpxml pci_0000_0b_10_0
<device>
<name>pci_0000_0b_10_0</name>
<parent>pci_0000_00_01_0</parent>
<driver>
<name>igbvf</name>
</driver>
<capability type='pci'>
<domain>0</domain>
<bus>11</bus>
<slot>16</slot>
<function>0</function>
<product id='0x10ca'>Intel Corporation</product>
<vendor id='0x8086'>82576 Virtual Function</vendor>
</capability>
</device>
This example adds the Virtual Function
pci_0000_0b_10_0
to the guest in
Step 10. Note the
bus
,
slot
and
function
parameters of the Virtual Function, these are required for adding the device.
-
Detach the Virtual Functions
Devices attached to a host cannot be attached to guests. Red Hat Enterprise Linux automatically attaches new devices to the host. Detach the Virtual Function from the host so that the Virtual Function can be used by the guest. Detaching the Physical Function causes errors, only detach the required Virtual Functions.
# virsh nodedev-dettach pci_0000_0b_10_0
Device pci_0000_0b_10_0 dettached
-
Add the Virtual Function to the guest
-
Shut down the guest.
-
Use the output from the virsh nodedev-dumpxml pci_8086_10ca_0
command to calculate the values for the configuration file. Convert slot and function values to hexadecimal values (from decimal) to get the PCI bus addresses. Append "0x" to the beginning of the output to tell the computer that the value is a hexadecimal number.
The example device has the following values: bus = 3, slot = 16 and function = 1. Use the printf
utility to convert decimal values to hexadecimal values.
$ printf %x 3
3
$ printf %x 16
10
$ printf %x 1
1
This example would use the following values in the configuration file:
bus='0x03'
slot='0x10'
function='0x01'
-
Open the XML configuration file with the virsh edit
command. This example edits a guest named MyGuest
.
# virsh edit MyGuest
-
The default text editor will open the libvirt configuration file for the guest. Add the new device to the devices
section of the XML configuration file.
<hostdev mode='subsystem' type='pci'>
<source>
<address bus='0x03' slot='0x10' function='0x01'/>
</source>
</hostdev>
-
Save the configuration.
-
Restart
Restart the guest to complete the installation.
# virsh start MyGuest