Zoetrope

Update: newer kernel & initrd 2.6.18-53.1.13.el5xen doesn't have this problem.

If you're running Centos5 and recently upgraded a Xen domU kernel to 2.6.18-53.1.6.el5xen you maybe got burned by the following problem:

Loading xenblk.ko module
XENBUS: Timeout connecting to device: device/vbd/51712 (state 3)
Scanning and configuring dmraid supported devices
Creating root device.
Mounting root filesystem.
mount: could not find filesystem '/dev/root'
Setting up other filesystems.
Setting up new root fs
setuproot: moving /dev failed: No such file or directory
no fstab.sys, mounting internal defaults
setuproot: error mounting /proc: No such file or directory
setuproot: error mounting /sys: No such file or directory
Switching to new root and running init.
unmounting old /dev
unmounting old /proc
unmounting old /sys
switchroot: mount failed: No such file or directory
Kernel panic - not syncing: Attempted to kill init!

Apparently the initrd is missing the xenblk module and any domU that's using tap:io in its Xen config can't connect to it's virtual disk image.

Here's the fix:

Change tap:io into file: in your xen configs and boot the domU.

disk = [ 'tap:aio:/var/lib/xen/images/domU.img,xvda,w', ]

into

disk = [ 'file:/var/lib/xen/images/basestar.img,xvda,w', ]

In your booted domU:

cd /boot
rm /boot/initrd-2.6.18-53.1.6.el5xen.img
mkinitrd -v --with=xenblk --omit-scsi-modules --omit-raid-modules /boot/initrd-2.6.18-53.1.6.el5xen.img 2.6.18-53.1.6.el5xen

Now you can change back to tap:io in your domU config and reboot the domU. You'll need to do this for every domU that has been upgraded to the 2.6.18-53.1.6.el5xen kernel.

See http://bugs.centos.org/view.php?id=2083 and http://bugs.centos.org/view.php?id=2543 for more info