Jontow Homelab KVM

Ubuntu 15.04 Notes

Overview

This section should get a general build of a VM host running KVM with libvirt (virsh)
and 2TB of RAID1 (softraid) running on Ubuntu 15.04 server. The hardware platform
that this section targets/was tested on is a Sun Fire X2200 M2. There is a hardware
specific notes section below.

The disks will be setup with RAID and LVM, giving a maximum compromise of reliability
and flexibility. Swap partitions are setup outside of softraid to maximize performance.

Disk Config

(2) x 2.0 TB disks in chassis, built at install time using the following general procedure:
- From aux shell: fdisk /dev/sda and "g" to GPT the disk, repeat for /dev/sdb
- Create a new (bios_grub) partition on each disk: ~256MB is fine
- Create any swap partitions necessary
- Create a new identically sized partition on each disk, "Use as: physical volume for RAID"
- Configure Software RAID: RAID1, 2 devices, 0 spares, select the two identical partitions as members.
- Configure LVM:
  - Create VG with PV /dev/md0 (named "vm1-vg")
  - Create LV (20G) from VG "vm1-vg" (named "vm1-root")
  - Create LV (20G) from VG "vm1-vg" (named "vm1-iso-0")
  - Create LV (1T) from VG "vm1-vg" (named "vm1-store-0")
  - Setup "vm1-root" as / with ext4 FS
  - Setup "vm1-iso-0" as /vm1-iso with ext4 FS
  - Setup "vm1-store-0" as /vm1-store-0 with ext4 FS
- When asked, install grub to "/dev/sda" (it'll try to use "/dev/mapper", override this.)
- After firstboot of system, run "dpkg-reconfigure grub-pc" and choose to install on "/dev/sdb" as well.

Worthwhile external reading:

Packages

 $ sudo apt-get install zsh multitail screen sysstat openssh-server nfs-kernel-server
 $ sudo apt-get install qemu-kvm libvirt-bin openvswitch-switch openvswitch-common
 $ sudo apt-get install libpolkit-agent-1-0 libpolkit-backend-1-0 policykit-1
 $ sudo apt-get install virtinst virt-top

System Config

/etc/network/interfaces :

# The loopback network interface
auto lo
iface lo inet loopback

################################################################################
# eth1: management interface (non-VM traffic)
auto eth1
iface eth1 inet static
        address 10.49.249.20
        netmask 255.255.255.0
        network 10.49.249.0
        broadcast 10.49.249.255
        gateway 10.49.249.1
        # dns-* options are implemented by the resolvconf package, if installed
        dns-nameservers 10.49.249.1
        dns-search je

################################################################################
# eth0: vlan-tagged for VM traffic
auto eth0
iface eth0 inet manual
################################################################################
# eth2: storage/cross-connect (xc) net, used with a pair of VM hosts
auto eth2
iface eth2 inet static
        address 10.49.254.20
        netmask 255.255.255.0
        network 10.49.254.0
        broadcast 10.49.254.255

/etc/network/if-up.d/openvswitch :

#! /bin/sh
# Setup openvswitch

# Don't bother with loopback
if [ "$IFACE" = lo ]; then
        exit 0
fi


# Only run from ifup.
if [ "$MODE" != start ]; then
        exit 0
fi

if [ "$IFACE" = "eth0" ]; then
        /usr/bin/ovs-vsctl add-br br0
        /usr/bin/ovs-vsctl add-port br0 eth0
fi

exit 0

/etc/hosts :

127.0.0.1       localhost
127.0.1.1       vm1

10.49.249.20    vm1.je
10.49.249.21    vm2.je

10.49.254.20    vm1.xc  vm1
10.49.254.21    vm2.xc  vm2

# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

Setup storage repo:

 $ sudo ln -s /vm1-store-0 /vm-store-0

NFS

/etc/exports (on vm1):

 /vm1-store-0            vm2.xc(rw,sync,no_subtree_check,no_root_squash)

Enable/start NFS:

 $ sudo systemctl enable nfs-kernel-server
 $ sudo systemctl start nfs-kernel-server

Misc bugfixes / errors

"error: Diskfilter writes are not supported"

This occurs around bootup splash screen time, and does not affect system operation, but is ugly. Supposedly fixed in 15.10+.
Ugly quick fix:

 $ sudo vi /etc/grub.d/10_linux
 (Replace 'quick_boot="1"' with 'quick_boot="0"', then...)
 $ sudo update-grub

See bug: https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1274320

PCI-DMA: Out of IOMMU space for...

This SHOULD be platform-specific (Sun Fire X2200 M2). If you begin to see messages like this in your logs:

 Oct 16 18:04:55 vm1 kernel: [  113.785020] sata_nv 0000:00:05.0: PCI-DMA: Out of IOMMU space for 65536 bytes

You ought to consider the following fix:

 $ sudo vi /etc/default/grub

Add "iommu=memaper3" to this line:

 GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"

Result should be:

 GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=memaper=3"

Requires a reboot.

KVM Networking

KVM Networking is an odd beast, and I required more flexibility than was natively
included in most prebuilt versions. Ubuntu 15.04 and newer include good support for
binding guests into VLANs and not requiring a physical interface for each VLAN to get
into the machine. This is accomplished with openvswitch and some creativity. Here
are a couple XML files useful to "net-define" in virsh. A few examples should allow
understanding of creating others with unique 802.1Q vlan tags.

If you fancy having an untagged ("native") vlan on the physical port and actually
using it, well, I forget how I did that. I stopped using the method and instead
went for explicitly specifying tags.

ovs-vlan2.xml

<network>
  <name>ovs-vlan2</name>
  <forward mode='bridge'/>
  <bridge name='br0'/>
  <virtualport type='openvswitch'/>
  <portgroup name='ovs-vlan2' default='yes'>
    <vlan>
      <tag id='2'/>
    </vlan>
  </portgroup>
</network>

ovs-vlan900.xml

<network>
  <name>ovs-vlan900</name>
  <forward mode='bridge'/>
  <bridge name='br0'/>
  <virtualport type='openvswitch'/>
  <portgroup name='ovs-vlan900' default='yes'>
    <vlan>
      <tag id='900'/>
    </vlan>
  </portgroup>
</network>

Example VM

Here's an example VM XML file using the above, to be used with virsh ("define").

examplevm.xml

<domain type='kvm'>
  <name>examplevm</name>
  <memory unit='KiB'>1048576</memory>
  <currentMemory unit='KiB'>1048576</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <os>
    <type arch='x86_64' machine='pc-1.1'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>Opteron_G2</model>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none' io='native'/>
      <source file='/vm-store-0/img/examplevm.img'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>
    <disk type='block' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='network'>
      <source network='ovs-vlan2'/>
      <model type='e1000'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes'/>
    <video>
      <model type='cirrus' vram='16384' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </memballoon>
  </devices>
</domain>