Trace: • netbsd_xen

This is an old revision of the document!

Table of Contents

NetBSD Xen

Installing NetBSD as Dom0

In Xen, Dom0 is the main OS that handles all the Xen virtual domains (called domU).

As of version 4.2, Xen can be built natively on NetBSD without patches, although I haven't had success getting it working – the libraries don't link correctly.

Initial Setup

First, do a NetBSD install as normal. This guide uses NetBSD amd64 6.0.1, and builds everything from source using pkgsrc and not the pre-built binaries.

Primary Kernel

There are two parts to the kernel for booting NetBSD with support for Xen as dom0. The first one is built by your system, and it is the kernel that NetBSD boots first. The second is a NetBSD GENERIC kernel provided by upstream, with Xen support. Both parts are needed to completely boot into a Xen environment where NetBSD can act as the primary domain.

Build and install the first Xen kernel: 

cd /usr/pkgsrc/sysutils/xenkernel42
make install

Once it's built, there will be a xen.gz and a xen-debug.gz file in /usr/pkgsrc/sysutils/xenkernel42.

Copy the xen.gz kernel to the root directory, and unzip it.

The NetBSD Xen guide keeps the kernel as a gzipped file, but when using GRUB as a bootloader, it wouldn't work for me. Unzipping it works fine. 

cp /usr/pkgsrc/sysutils/xenkernel42/work/xen.gz /
gunzip /xen.gz

Secondary Kernel

Next, download the Xen DOM0 kernel from NetBSD's FTP repository. This is a GENERIC kernel that ships with the OS, but rebuilt for Xen support. 

ftp ftp://ftp.netbsd.org/pub/NetBSD/`uname -r`/`uname -p`/binary/kernel/netbsd-XEN3_DOM0.gz

Move it to the root filesystem and unpack it as well: 

mv netbsd-XEN3_DOM0.gz /
gunzip /netbsd-XEN3_DOM0.gz

Xen Tools

As a master domain, you'll need some Xen tools installed to maintain installations. 

cd /usr/pkgsrc/sysutils/xentools42
make install

Setup GRUB Bootloader

There are two ways to boot into the NetBSD Xen kernel: using the original bootloader, or installing GRUB. I prefer using GRUB because I'm familiar with the syntax, and it can chainload into the NetBSD bootloader as well.

Install GRUB on the system: 

cd /usr/pkgsrc/sysutils/grub
make install

Create a configuration file for GRUB at /grub/menu.lst. This one boots into NetBSD Xen by default, using 512 MB of RAM. The bootloader is at the first partition of the first drive. 

# Digital Trike
default 0
timeout 10

# Load NetBSD Xen through GRUB
title NetBSD Xen
root (hd0,0,a)
kernel /xen dom0_mem=512M
module /netbsd-XEN3_DOM0 bootdev=sd0a ro console=tty0

# Jump to NetBSD bootloader
title NetBSD Bootloader
root (hd0,0,a)
chainloader +1

Install GRUB as your bootloader: 

grub-install '(hd0)'

You can pass the device name directly as well, if you know it: 

grub-install /dev/wd0d

An alternative way to install: 

grub --no-curses
root (hd0,0,a)
setup (hd0)

Setup NetBSD Bootloader

You can alternatively use the default NetBSD bootloader, although I haven't gotten it to display output to the VGA console yet.

Add this menu entry to /boot.cfg: 

menu=NetBSD Xen:load /netbsd.xen;multiboot /xen dom0_mem=512M

Copying Files

There are some example files installed with the xentools package that need to be copied over to the right locations so they can be used.

Init scripts
cp /usr/pkg/share/examples/rc.d/xen* /etc/rc.d/

Update /etc/rc.conf to start Xen on boot: 

xenwatchdog=YES
xencommons=YES
xend=YES
xendomains=YES
Xen Scripts
cp /usr/pkg/share/examples/xen/scripts/* /usr/pkg/etc/xen/
Device Files

Setup the new Xen device files: 

cd /dev && sh MAKEDEV xen

Installing NetBSD as DomU

Now to create a NetBSD virtual machine.

For the base directories in this guide, I'm going to use /var/xen as the parent directory of all files. For the specific vm, /var/xen/netbsd

Create Network Adapter

Bridge the main Ethernet device (bnx0 in this example) to create bridge0.

Create this file: /etc/ifconfig.bridge0 

create
!brconfig $int add bnx0 up

Reboot the dom0 to create the new Ethernet bridge.

Create Hard Drive

Create a hard drive with as many GBs as you want. In this example, I'm using 4 GB (4096): 

dd if=/dev/zero of=netbsd.img bs=1024k count=4096

Mount the drive, and give it an MBR: 

vnconfig vnd0 netbsd.img
fdisk -l /dev/vnd0d
vnconfig -u vnd0

Create Xen Configuration File

Create a new file, netbsd-xen: 

# use this kernel to boot and install
kernel = "/var/xen/netbsd-xen/netbsd-INSTALL_XEN3_DOMU"
# use this kernel once install is complete
# kernel = "/var/xen/netbsd-xen/netbsd-INSTALL_XEN3_DOMU"
memory = 1024
name = "netbsd-xen"
vcpus = 2
vif = [ 'mac=aa:00:00:50:02:f0, bridge=bridge0' ]
disk = [ 'file:/var/xen/netbsd-xen/netbsd.img,0x1,w' ]
root = "xbd0"

Install NetBSD

Install NetBSD as usual, using FTP to download the base install. 

xm create -c netbsd-xen

Once you're finished, shutdown the install, and close it out with xm as well: 

xm shutdown netbsd-xen
xm destroy netbsd-xen

Edit the netbsd-xen config file, and switch the kernel to the GENERIC kernel with Xen support instead of the install one. Then, boot into NetBSD as normal! 

xm create -c netbsd-xen

Installing Linux as DomU

I haven't been able to get this working, but this is the basic setup:

  • Create a disk and a swap image
  • Format the images
  • Install GRUB

They need GRUB installed on the root disk. Pygrub then runs from NetBSD and boots off the GRUB install of that box.

So, you need an image that is already created and formatted before doing anything. I haven't had much luck in trying to do this *from within NetBSD*. So for now, I'm going to go with using xen-create-image on another Linux box, and copying it over.

Here is a cobbled-together ubuntu-xen.cfg that probably won't work as-is: 

bootloader = '/usr/pkg/bin/pygrub'
vcpus       = '1'
memory      = '512'
root        = '/dev/vnd0 ro'
disk        = [
                'file:/var/xen/ubuntu-xen/swap.img,xvda1,w',
                'file:/var/xen/ubuntu-xen/disk.img,0x300,w'
              ]
name        = 'ubuntu-xen'
dhcp        = 'dhcp'
vif         = [ 'mac=00:16:3E:E8:0E:55, bridge=bridge0' ]
on_poweroff = 'destroy'
on_reboot   = 'restart'
on_crash    = 'restart'

Install Ubuntu as DomU

Download the Ubuntu Xen kernel, initrd, and Xen config files from the FTP repo.

Create a config file for Xen: 

kernel = "/var/xen/ubuntu/vmlinuz"
ramdisk = "/var/xen/ubuntu/initrd.gz"
name = "ubuntu"
memory = "2048"
disk = [ 'file:/var/xen/ubuntu/disk.img,xvda,w', ]
vcpus = 4
on_reboot = 'restart'
on_poweroff = 'destroy'
on_crash = 'destroy'
vif = ['mac=00:16:3e:00:00:11, bridge=bridge0']

Boot the new Xen image and install Ubuntu: 

xm create -c config