xCAT - Stateless Image Customizations

The following procedure can be used to update/customize xCAT netboot images on the head node (xCAT master). Also, NIC drivers for the baremetal servers can be updated inside the netboot image.

1. On the Head Node (xCAT master) update the netboot image:

# chroot into the netboot image
chroot /install/netboot/centos/x86_64/bare-metal-worker/rootimg
mount -t proc proc /proc
cd ~
# update/customize the netboot image
yum clean all
yum update
# exit from chroot environment
umount  /proc
exit
# create stateless image from chroot netboot image
packimage centos-x86_64-netboot-bare-metal-worker
# if kernel RPMs are updated, then copy the new kernel and RAM disk image files to netboot directory
cp /install/netboot/centos/x86_64/bare-metal-worker/rootimg/boot/vmlinuz-2.6.32-642.13.1.el6.x86_64 /install/netboot/centos/x86_64/bare-metal-worker/kernel
cp /install/netboot/centos/x86_64/bare-metal-worker/rootimg/boot/initramfs-2.6.32-642.13.1.el6.x86_64.img /install/netboot/centos/x86_64/bare-metal-worker/initrd-stateless.gz

2. Create a slice with a baremetal node to boot up baremetal worker. Login to the worker node, download Mellanox Network Interface Card drivers and build RPMs.

Login to the baremetal worker:

cd ~
wget 'http://www.mellanox.com/downloads/Drivers/mlnx-en-3.3-1.0.0.0.tgz'
mkdir ~/tmp
tar xf mlnx-en-3.3-1.0.0.0.tgz
cd mlnx-en-3.3-1.0.0.0
./install.sh --build-only --tmpdir /root/tmp
# copy the RPMs to the head node
scp -r RPMS/rhel6u8 mcevik@10.101.0.1:~/mlnx-en-3.3-1.0.0.0-rhel6u8

3. On the Head Node (xCAT master) install NIC drivers to the chroot netboot image:

# first, copy the RPMs into the image directory
cp -r ~mcevik/mlnx-en-3.3-1.0.0.0-rhel6u8 /install/netboot/centos/x86_64/bare-metal-worker/rootimg/root/.
# chroot into the netboot image
chroot /install/netboot/centos/x86_64/bare-metal-worker/rootimg
mount -t proc proc /proc
# Uninstall existing RPMs
rpm -qa | egrep 'mlnx|mstflint'
rpm -e <RPMS>
 #rpm -e mlnx-en-doc-3.3-1.0.0.0.0.gf583963.rhel6u5.x86_64
 #rpm -e mlnx-en-debuginfo-3.3-1.0.0.0.0.gf583963.rhel6u5.x86_64
 #rpm -e mlnx-en-sources-3.3-1.0.0.0.0.gf583963.rhel6u5.x86_64
 #rpm -e kmod-mlnx-en-3.3-1.0.0.0.0.gf583963.rhel6u5.x86_64
 #rpm -e mlnx-en-utils-3.3-1.0.0.0.0.gf583963.rhel6u5.x86_64
# Install new RPMs
cd  /root/mlnx-en-3.3-1.0.0.0-rhel6u8/x86_64
rpm -ivh mlnx-en-*
rpm -ivh mstflint-*
rpm -ivh kmod-mlnx-*
# exit from chroot environment
umount  /proc
exit
# create stateless image from chroot netboot image
packimage centos-x86_64-netboot-bare-metal-worker

4. It is possible that the interfaces need to be renamed due to an update in kernel or a network interface card is changed. In this case, some modifications need to be done in netboot image files and an xCAT postscript.

Login to the baremetal server and check device information and udev renaming:

[root@Node1 ~]# dmesg | grep -i eth
usb 2-1.1.5: Product: RNDIS/Ethernet Gadget
usb 2-1.1.5: Product: RNDIS/Ethernet Gadget
cxgb4 0000:11:00.4: eth0: Chelsio T420-CR rev 2 1000/10GBASE-R SFP+ RNIC PCIe x8 5 GT/s MSI-X
cxgb4 0000:11:00.4: eth0: S/N: PT30122283, P/N: 110113440B0
cxgb4 0000:11:00.4: eth1: Chelsio T420-CR rev 2 1000/10GBASE-R SFP+ RNIC PCIe x8 5 GT/s MSI-X
cxgb4 0000:11:00.4: eth1: S/N: PT30122283, P/N: 110113440B0
. . .
udev: renamed network interface eth2 to em2
udev: renamed network interface eth3 to em3
. . .
cxgb4 0000:11:00.4: eth0: Chelsio T420-CR rev 2 1000/10GBASE-R SFP+ RNIC PCIe x8 5 GT/s MSI-X
cxgb4 0000:11:00.4: eth0: S/N: PT30122283, P/N: 110113440B0
cxgb4 0000:11:00.4: eth1: Chelsio T420-CR rev 2 1000/10GBASE-R SFP+ RNIC PCIe x8 5 GT/s MSI-X
cxgb4 0000:11:00.4: eth1: S/N: PT30122283, P/N: 110113440B0
udev: renamed network interface eth0 to p2p5
udev: renamed network interface eth1 to p2p6

On the head node, check the files in netboot image that will activate the interfaces on baremetal server when booting and modify these files with the correct names:

[root@osf-hn ~]# cd /install/netboot/centos/x86_64/bare-metal-worker/rootimg/etc/sysconfig/network-scripts
[root@osf-hn network-scripts]# ls -l ifcfg-*
-rw-r--r-- 1 root root  36 Apr 29  2014 ifcfg-em2
-rw-r--r-- 1 root root 254 Apr 12 09:14 ifcfg-lo
-rw-r--r-- 1 root root  69 Apr 30  2014 ifcfg-p2p1
-rw-r--r-- 1 root root  68 Oct  9  2014 ifcfg-p2p2
-rw-r--r-- 1 root root  68 Aug 29 20:44 ifcfg-p3p1
-rw-r--r-- 1 root root  69 Aug 29 20:44 ifcfg-p3p2

An xCAT postinstall script (fix_ifaces.sh) is used on some racks to adapt the interface configuration due to runtime factors.

Check which postscripts are used:

[root@osf-hn network-scripts]# tabdump postscripts
#node,postscripts,postbootscripts,comments,disable
"xcatdefaults","syslog,remoteshell,syncfiles","otherpkgs",,
"service","servicenode",,,
"openstack-workers",,"openvswitch",,
"baremetal-workers",,"fix_ifaces.sh",,

Modify fix_ifaces.sh if necessary:

[root@osf-hn network-scripts]# cat /install/postscripts/fix_ifaces.sh
#!/bin/bash
set -x

##
## Fix up Mellanox
## 
sed -i 's/eth0/p2p1/g' /etc/udev/rules.d/70-persistent-net.rules
sed -i 's/p3p6/p2p2/g' /etc/udev/rules.d/70-persistent-net.rules
udevadm control --reload

modprobe -r mlx4_en && modprobe mlx4_en

# Add VLAN support
modprobe 8021q

Create stateless netboot image:

[root@osf-hn network-scripts]# packimage centos-x86_64-netboot-bare-metal-worker
Navigation
Print/export