Jontow Homelab KVM

Ubuntu 15.04 Notes

Overview

This section should get a general build of a VM host running KVM with libvirt (virsh)
and 2TB of RAID1 (softraid) running on Ubuntu 15.04 server. The hardware platform
that this section targets/was tested on is a Sun Fire X2200 M2. There is a hardware
specific notes section below.

The disks will be setup with RAID and LVM, giving a maximum compromise of reliability
and flexibility. Swap partitions are setup outside of softraid to maximize performance.

Disk Config

• (2) x 2.0 TB disks in chassis, built at install time using the following general procedure:
  • From aux shell: fdisk /dev/sda and "g" to GPT the disk, repeat for /dev/sdb
  • Create a new (bios_grub) partition on each disk: ~256MB is fine
  • Create any swap partitions necessary
  • Create a new identically sized partition on each disk, "Use as: physical volume for RAID"
  • Configure Software RAID: RAID1, 2 devices, 0 spares, select the two identical partitions as members.
  • Configure LVM:
    • Create VG with PV /dev/md0 (named "vm1-vg")
    • Create LV (20G) from VG "vm1-vg" (named "vm1-root")
    • Create LV (20G) from VG "vm1-vg" (named "vm1-iso-0")
    • Create LV (1T) from VG "vm1-vg" (named "vm1-store-0")
    • Setup "vm1-root" as / with ext4 FS
    • Setup "vm1-iso-0" as /vm1-iso with ext4 FS
    • Setup "vm1-store-0" as /vm1-store-0 with ext4 FS
  • When asked, install grub to "/dev/sda" (it'll try to use "/dev/mapper", override this.)
  • After firstboot of system, run "dpkg-reconfigure grub-pc" and choose to install on "/dev/sdb" as well.

Packages

 $ sudo apt-get install zsh multitail screen sysstat openssh-server nfs-kernel-server
 $ sudo apt-get install qemu-kvm libvirt-bin openvswitch-switch openvswitch-common
 $ sudo apt-get install libpolkit-agent-1-0 libpolkit-backend-1-0 policykit-1
 $ sudo apt-get install virtinst

System Config

• /etc/network/interfaces :

# The loopback network interface
auto lo
iface lo inet loopback

################################################################################
# eth1: management interface (non-VM traffic)
auto eth1
iface eth1 inet static
        address 10.49.249.20
        netmask 255.255.255.0
        network 10.49.249.0
        broadcast 10.49.249.255
        gateway 10.49.249.1
        # dns-* options are implemented by the resolvconf package, if installed
        dns-nameservers 10.49.249.1
        dns-search je

################################################################################
# eth0: vlan-tagged for VM traffic
auto eth0
iface eth0 inet manual
################################################################################
# eth2: storage/cross-connect (xc) net, used with a pair of VM hosts
auto eth2
iface eth2 inet static
        address 10.49.254.20
        netmask 255.255.255.0
        network 10.49.254.0
        broadcast 10.49.254.255

• /etc/network/if-up.d/openvswitch :

#! /bin/sh
# Setup openvswitch

# Don't bother with loopback
if [ "$IFACE" = lo ]; then
        exit 0
fi


# Only run from ifup.
if [ "$MODE" != start ]; then
        exit 0
fi

if [ "$IFACE" = "eth0" ]; then
        /usr/bin/ovs-vsctl add-br br0
        /usr/bin/ovs-vsctl add-port br0 eth0
fi

exit 0

• /etc/hosts :

127.0.0.1       localhost
127.0.1.1       vm1

10.49.249.20    vm1.je
10.49.249.21    vm2.je

10.49.254.20    vm1.xc  vm1
10.49.254.21    vm2.xc  vm2

# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

• Setup storage repo:

 $ sudo ln -s /vm1-store-0 /vm-store-0

NFS

• /etc/exports (on vm1):

 /vm1-store-0            vm2.xc(rw,sync,no_subtree_check,no_root_squash)

• Enable/start NFS:

 $ sudo systemctl enable nfs-kernel-server
 $ sudo systemctl start nfs-kernel-server

Misc bugfixes / errors

"error: Diskfilter writes are not supported"

This occurs around bootup splash screen time, and does not affect system operation, but is ugly. Supposedly fixed in 15.10+.
Ugly quick fix:

 $ sudo vi /etc/grub.d/10_linux
 (Replace 'quick_boot="1"' with 'quick_boot="0"', then...)
 $ sudo update-grub

PCI-DMA: Out of IOMMU space for...

This SHOULD be platform-specific (Sun Fire X2200 M2). If you begin to see messages like this in your logs:

 Oct 16 18:04:55 vm1 kernel: [  113.785020] sata_nv 0000:00:05.0: PCI-DMA: Out of IOMMU space for 65536 bytes

You ought to consider the following fix:

 $ sudo vi /etc/default/grub

Add "iommu=memaper3" to this line:

 GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"

Result should be:

 GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=memaper=3"

Requires a reboot.

KVM Networking

KVM Networking is an odd beast, and I required more flexibility than was natively
included in most prebuilt versions. Ubuntu 15.04 and newer include good support for
binding guests into VLANs and not requiring a physical interface for each VLAN to get
into the machine. This is accomplished with openvswitch and some creativity. Here
are a couple XML files useful to "net-define" in virsh. A few examples should allow
understanding of creating others with unique 802.1Q vlan tags.

ovs-vlan2.xml

<network>
  <name>ovs-vlan2</name>
  <forward mode='bridge'/>
  <bridge name='br0'/>
  <virtualport type='openvswitch'/>
  <portgroup name='ovs-vlan2' default='yes'>
    <vlan>
      <tag id='2'/>
    </vlan>
  </portgroup>
</network>

ovs-vlan900

<network>
  <name>ovs-vlan900</name>
  <forward mode='bridge'/>
  <bridge name='br0'/>
  <virtualport type='openvswitch'/>
  <portgroup name='ovs-vlan900' default='yes'>
    <vlan>
      <tag id='900'/>
    </vlan>
  </portgroup>
</network>