xen-pciback

Assign a Dedicated Network Card or PCI Device to a Xen Virtual Machine on RHEL Host

Purpose: If you are running any network Intensive application on DomU and if you want to dedicate a NIC to the guest instead of shared by multiple guest then the below document is very usefull to you.

 I just recently spent more hours than I care to think about getting a NIC in Xen Dom0 hidden and passed to a DomU via pciback in RHEL5 .

  For a domain other than Domain0 to directly control a physical device, the device must first be hidden from Domain0 and the control must be explicitly granted to another domain.

  The first step in hiding a device is learning the identifier of the device in the system. A device is commonly referred numbers that specify the domain identifier, bus number, slot number, and function number by which the device is connected to the system. They are expressed in the format: (<domain>:<bus>:<slot>.<function>) . Here the term domain refers to a
domain and not a Xen domain.


   For example, (0000:01:02.3) refers to a device in the domain 0000, bus 01, slot 02, and function 3.

 On a Linux system, a good way to determine the correct identifier for a PCI device is using
the utility lspci. lspci lists or displays information on all the PCI buses in the system and the devices connected to them.

   lspci  |grep -i ether

06:07.0 Ethernet controller: Intel Corporation 82541GI Gigabit Ethernet Controller (rev 05)
07:08.0 Ethernet controller: Intel Corporation 82541GI Gigabit Ethernet Controller (rev 05)

Note: The leading 0000 is the default and will be assumed if not specified.
so the defualt device number looks some thing like 0000:06:07.0


  So the question here is finding out the proper device which one is eth0 and eth1.To find out proper device name you can use the below simple trick.


find /sys/bus/pci/drivers -name "0000:06:07.0"

/sys/bus/pci/drivers/e1000/0000:06:07.0

ll /sys/bus/pci/drivers/e1000/0000\:06\:07.0/

 net:peth0 -> ../../../../../class/net/peth0

net:peth0  This is your actual eth0 in xen environment passing all the traffic to/from xenbr0.

find /sys/bus/pci/drivers -name "0000:07:08.0"

/sys/bus/pci/drivers/e1000/0000:07:08.0

ll /sys/bus/pci/drivers/e1000/0000\:07\:08.0/

net:eth1 -> ../../../../../class/net/eth1

 so the second pci device number relates to eth1.



  After you have the identified the device, the next step is to hide it from Domain0. One way to accomplish this is to add a pciback.hide kernel parameter to the module entry in the GRUB configuration file, typically grub.conf  Domain0 kernel.It may also be necessary to add the pciback.permissive option for drivers that need to write the register the PCI device itself.

 pciback or pciback.hide is now a module. Putting it on GRUB’s kernel line is no longer sufficient.So dont put it in grub.conf.

 Put the following and only the following in your /etc/modprobe.conf

 options pciback hide=(07:08.0)


 Now you need to create a new initrd image. Before doing so make a backup copy of the one that you are currently using. Then create a new initrd so that the pciback module is loaded early on.

mkinitrd -f -v --preload=pciback   /boot/initrd-$(uname -r).img $(uname -r)


You can get output some thing of similar sort while creating  the new initrd image

Adding module pciback with options hide=(07:08.0)

 This will place the new initrd into your boot directory and overwrite the old one.

 Moving on to the /etc/xen directory we have a little work do here. In the domU config file make two entries like the following:

vi vguest2

##################
pci = [ "07:08.0"]
vif = [ '' ]

Now open /etc/xen/xend-pci-permissive.sxp and make an entry like the following:

(unconstrained_dev_ids
     #('0123:4567:89AB:CDEF')
     ('8086:1076') ## added to hides eth1 from Dom0
)

You get the 8086:1076 number from running lspci -n:

lspci -n |grep "07:08.0"

07:08.0 0200: 8086:1076 (rev 05)


 As you can see those number match up with the 07:08.0 number that we used in modprobe.conf and the domU config file.


  It’s time to reboot and check your work.

Once dom0 is up we need to look for certain entries in /var/log/messages:

grep pciback /var/log/messages
Apr 21 13:17:15 vhost1 kernel: pciback 0000:07:08.0: seizing device


 Once you see that the device is seized on Dom0  start the virtual guest  using the below command so that you can see all the boot messages on the terminal.

 xm create vguest2 -c

Then check the below two commands on the guest machine

  lspci |grep -i ether

00:00.0 Ethernet controller: Intel Corporation 82541GI Gigabit Ethernet Controller (rev 05)

 Since my Guest  machine is already running with 30 network interfaces,RHEL automatically  created the entry for eth31 without any input from my  part.

[root@node2 network-scripts]# ifconfig  eth31
eth31     Link encap:Ethernet  HWaddr 00:11:43:E5:03:D3 
          inet addr:10.65.70.250  Bcast:10.65.70.255  Mask:255.255.255.0
          inet6 addr: fe80::211:43ff:fee5:3d3/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:724 errors:0 dropped:0 overruns:0 frame:0
          TX packets:76 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:100
          RX bytes:144746 (141.3 KiB)  TX bytes:18900 (18.4 KiB)
          Base address:0xdcc0 Memory:df8e0000-df900000


  You can compare above HW addrress 00:11:43:E5:03:D3  with the one that already on host os.



  There you have the results of several hours of reading and trial and error. Don’t forget that if you install a new xen kernel you will again have to make a new initrd.

One way around having to create a new initrd is to create a file named pciback under /etc/sysconfig/mkinitrd. Put an entry into the pciback file that reads PREMODS=”$PREMODS pciback”. That should automatically install the pciback module when your new kernel creates it’s new mkinitrd.


Hope this helps.