Please see the article in the Linode Library on Running Custom Kernels with PV-GRUB for more information and additional resources.
PV-GRUB allows Xen guests to boot their own custom kernel that supports pv_ops (>2.6.18). This provides further flexibility, allowing the kernel of each guest to be customized for its own needs. It can also help in situations where resources are tight - no unnecessary code is in memory, and no useless processes are spawned. You should become familiar with this entire guide before attempting to deploy any new Linode using PV-GRUB.
Linode has PV-GRUB configured to search for menu.lst at (hd0)/boot/grub/menu.lst. This means that your first disk (ie. /dev/xvda) cannot be partitioned. It has to contain a filesystem with at least your /boot directory (including /boot/grub).
If this is a problem, it is suggested to create a small disk image for the sole purpose of housing the kernel and menu.lst. Note that a separate disk mounted as /boot is not sufficient, as the entire /boot/grub/menu.lst path must exist on the boot disk.
 Installing Stock Kernel
 Ubuntu 12.04 Precise
Install the virtual kernel metapackage and the bootloader configuration:
root@host:~# apt-get install linux-virtual grub-legacy-ec2
grub-legacy-ec2 is a package which provides the update-grub interface to generate /boot/grub/menu.lst, but does not actually contain a GRUB bootloader, and is suitable for any PV-GRUB installation (despite the "ec2" name).
Edit /boot/grub/menu.lst, and look for a line similar to:
Append "rootflags=nobarrier" to this line (do not uncomment the line):
# defoptions=console=hvc0 rootflags=nobarrier
"nobarrier" is needed as barriers are default mount options for ext3/4 as of Linux 3.2, yet they do not always play well with Xen.
Update your Linode profile, disable "Xenify Distro", and set the kernel to "pv-grub-x86_32" or "pv-grub-x86_64" depending on your installation, then reboot the profile.
These instructions will only work on the x86_64 (64-bit) build, the i686 (32-bit) kernel does not support the required xen modules.
Install the 'linux' package to get the default arch kernel
[root@host ~]# pacman -S linux
Modify /etc/mkinitcpio.d/linux.preset, and add this line as the first non-commented line
MODULES="xen-blkfront xen-fbfront xenfs xen-netfront xen-kbdfront"
Generate a new vmlinuz and initial ramdisk
[root@host ~]# mkinitcpio -p linux
Finally, we need a valid grub config that pv-grub can boot. This needs to go in a new file /boot/grub/menu.lst
# (0) Arch Linux title Arch Linux [/boot/vmlinuz-linux] root (hd0) kernel /boot/vmlinuz-linux root=/dev/xvda ro initrd /boot/initramfs-linux.img
Change to pv-grub in the Linode Manager, and you can boot your own kernel.
 Other Distros
On Ubuntu 10.04 Lucid - default grub is grub 2, which creates grub.cfg instead of the needed menu.lst
apt-get install linux-virtual grub
On Debian 5.0 Lenny a stock kernel may be used:
apt-get install linux-image-xen-686 initramfs-tools update-initramfs -c -k <kernel version e.g. 2.6.32-2-686>
On Ubuntu 8.04 Hardy and later releases, the server kernel may be used, but the default generic kernel is incompatible:
apt-get install linux-image-server
Edit by Jed Smith @ Linode
At least one customer has reported that Hardy requires this kernel instead:
apt-get install linux-image-xen
If you run into this error, check to see that your stock kernel is bootable as a domU; lack of Xen support, as well as a compression type unexpected by the Xen loader, will result in this error:
ERROR Invalid kernel: xc_dom_find_loader: no loader found
 Building Your Own Kernel
The host must either be running xen-sources or another kernel that supports pv_ops (as a general rule, versions above 2.6.18). As all hardware calls are handled by Xen, you can remove all hardware device drivers.
In order to work with Xen, a number of options that must be selected:
CONFIG_PARAVIRT_GUEST=y CONFIG_XEN=y CONFIG_PARAVIRT=y CONFIG_PARAVIRT_CLOCK=y CONFIG_XEN_BLKDEV_FRONTEND=y CONFIG_XEN_NETDEV_FRONTEND=y CONFIG_HVC_XEN=y CONFIG_XEN_BALLOON=y CONFIG_XEN_SCRUB_PAGES=y
Linode Xen can only boot kernels compressed with GZIP, make sure this matches exactly:
CONFIG_KERNEL_GZIP=y # CONFIG_KERNEL_BZIP2 is not set # CONFIG_KERNEL_LZMA is not set # CONFIG_KERNEL_XZ is not set # CONFIG_KERNEL_LZO is not set
If compiling your own kernel, you can use the Linode kernel you have booted as a starting point:
cd /usr/src/linux && gzcat /proc/config.gz > .config make oldconfig make menuconfig
In Gentoo try this to get a starting kernel config (bug #226173):
cd /usr/src/linux && gunzip -c /proc/config.gz > .config
 Gentoo Kernels 2.6.38-r2+
If your instance is not booting properly (messages regarding "blk devices" flash briefly, then return to Lish), try using an initramfs. The one that genkernel generates should work flawlessly. Tested working on sys-kernel/gentoo-sources-2.6.38-r4. Bsaner 10:08, 8 May 2011 (UTC)
A menu.lst is necessary for unattended boot. If pv-grub cannot find one, it will drop to a command line on the lish console, waiting for you to specify a kernel. The format is the same as that used by standard grub. The only noteworthy difference is the change of root from (hd0,0) to (hd0). Note that stock Linode images do not come with a menu.lst file - you will need to create one at /boot/grub/menu.lst.
Here is an example:
default 0 timeout 5 title Gentoo GNU/Linux 2.6.27-gentoo-r5 root (hd0) kernel /boot/kernel-2.6.27-gentoo-r5 root=/dev/xvda ro
The update-grub utility on Debian or Ubuntu systems can be used to create and manage the menu.lst file. If you created the system from a Linode template, then Install the grub package like this:
mkdir /boot/grub apt-get install grub update-grub
The groot cannot be a UUID in the Linode environment, so open the /etc/grub/menu.lst file in a text editor and set the groot option to (hd0) like this:
## default grub root device ## e.g. groot=(hd0,0) # groot=(hd0)
DomU kernel detection is currently broken in Ubuntu 9.04 Jaunty, so disable it like this:
## specify if running in Xen domU or have grub detect automatically ## update-grub will ignore non-xen kernels when running in domU and vice versa ## e.g. indomU=detect ## indomU=true ## indomU=false # indomU=false
Regenerate the menu.lst file after making these changes, and after installing or removing any local kernel:
 Console Missing
If the system console becomes unavailable after installing a custom kernel, or if lish doesn't issue a login prompt, then you need to tell init to point getty at hvc0 instead of tty1. This problem is likely to look like your boot process hanging if watching the console. This was originally done in /etc/inittab, then in Ubuntu 8.04 in /etc/event.d/tty1, and in Ubuntu 10.04 in /etc/init/tty1.conf. If you have multiple of these, you probably (only) need to modify the newest.
For /etc/inittab change:
0:2345:respawn:/sbin/getty 38400 tty1
0:2345:respawn:/sbin/getty 38400 hvc0
and disable the "Xenify Distro" option in the "Configuration Profile" at the Linode Manager page, and poweroff/poweron the virtual machine.
For /etc/event.d/tty1 or /etc/init/tty1.conf change:
exec /sbin/getty 38400 tty1
exec /sbin/getty 38400 hvc0
in a similar fashion. Note that the Xenify Distro option doesn't touch these (at this time), so you can leave Xenify Distro enabled.
You can safely run `/sbin/getty 38400 hvc0` through ssh to temporarily enable the console.
 Console Security
If you notice that you can't login as root anymore on the console, please be sure to add 'hvc0' to your /etc/securetty file.
 Boot Configuration
Enter Linode Configuration Profile in your Linode Manager. Change Kernel to pv-grub-x86_32 or pv-grub-x86_64, depending on installed kernel and userspace. Unless you developed a complex configuration set your root device to /dev/xvda.
That's it! You should now be booting using your custom kernel. If for some reason your menu.lst cannot be found or does not work, you will be dumped to a grub prompt which you can access via Lish. Try entering your menu.lst manually to track down the problem.