Difference between revisions of "Jontow Homelab KVM"

From WTFwiki
Jump to navigation Jump to search
(Created page with "== Ubuntu 15.04 Notes == === Packages == $ sudo apt-get install zsh multitail screen sysstat openssh-server nfs-kernel-server $ sudo apt-get install qemu-kvm libvirt-bin o...")
 
 
(11 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
== Ubuntu 15.04 Notes ==
 
== Ubuntu 15.04 Notes ==
−
=== Packages ==
+
=== Overview ===
 +
This section should get a general build of a VM host running KVM with libvirt (virsh)<br/>
 +
and 2TB of RAID1 (softraid) running on Ubuntu 15.04 server.  The hardware platform<br/>
 +
that this section targets/was tested on is a Sun Fire X2200 M2.  There is a hardware<br/>
 +
specific notes section below.<br/>
 +
<br/>
 +
The disks will be setup with RAID and LVM, giving a maximum compromise of reliability<br/>
 +
and flexibility.  Swap partitions are setup outside of softraid to maximize performance.<br/>
 +
 
 +
=== Disk Config ===
 +
* (2) x 2.0 TB disks in chassis, built at install time using the following general procedure:
 +
** From aux shell: fdisk /dev/sda and "g" to GPT the disk, repeat for /dev/sdb
 +
** Create a new (bios_grub) partition on each disk: ~256MB is fine
 +
** Create any swap partitions necessary
 +
** Create a new identically sized partition on each disk, "Use as: physical volume for RAID"
 +
** Configure Software RAID: RAID1, 2 devices, 0 spares, select the two identical partitions as members.
 +
** Configure LVM:
 +
*** Create VG with PV /dev/md0 (named "vm1-vg")
 +
*** Create LV (20G) from VG "vm1-vg" (named "vm1-root")
 +
*** Create LV (20G) from VG "vm1-vg" (named "vm1-iso-0")
 +
*** Create LV (1T) from VG "vm1-vg" (named "vm1-store-0")
 +
*** Setup "vm1-root" as / with ext4 FS
 +
*** Setup "vm1-iso-0" as /vm1-iso with ext4 FS
 +
*** Setup "vm1-store-0" as /vm1-store-0 with ext4 FS
 +
** When asked, install grub to "/dev/sda" (it'll try to use "/dev/mapper", override this.)
 +
** After firstboot of system, run "dpkg-reconfigure grub-pc" and choose to install on "/dev/sdb" as well.
 +
 
 +
Worthwhile external reading:
 +
* [https://wiki.debian.org/DebianInstaller/SoftwareRaidRoot https://wiki.debian.org/DebianInstaller/SoftwareRaidRoot]
 +
* [https://help.ubuntu.com/community/Installation/SoftwareRAID https://help.ubuntu.com/community/Installation/SoftwareRAID]
 +
 
 +
=== Packages ===
 
   $ sudo apt-get install zsh multitail screen sysstat openssh-server nfs-kernel-server
 
   $ sudo apt-get install zsh multitail screen sysstat openssh-server nfs-kernel-server
 
   $ sudo apt-get install qemu-kvm libvirt-bin openvswitch-switch openvswitch-common
 
   $ sudo apt-get install qemu-kvm libvirt-bin openvswitch-switch openvswitch-common
 
   $ sudo apt-get install libpolkit-agent-1-0 libpolkit-backend-1-0 policykit-1
 
   $ sudo apt-get install libpolkit-agent-1-0 libpolkit-backend-1-0 policykit-1
−
   $ sudo apt-get install virtinst
+
   $ sudo apt-get install virtinst virt-top iftop vlock smartmontools check-mk-agent xinetd
 +
 
 
=== System Config ===
 
=== System Config ===
 +
* /etc/network/interfaces :
 +
<pre>
 +
# The loopback network interface
 +
auto lo
 +
iface lo inet loopback
 +
 +
################################################################################
 +
# eth1: management interface (non-VM traffic)
 +
auto eth1
 +
iface eth1 inet static
 +
        address 10.49.249.20
 +
        netmask 255.255.255.0
 +
        network 10.49.249.0
 +
        broadcast 10.49.249.255
 +
        gateway 10.49.249.1
 +
        # dns-* options are implemented by the resolvconf package, if installed
 +
        dns-nameservers 10.49.249.1
 +
        dns-search je
 +
 +
################################################################################
 +
# eth0: vlan-tagged for VM traffic
 +
auto eth0
 +
iface eth0 inet manual
 +
################################################################################
 +
# eth2: storage/cross-connect (xc) net, used with a pair of VM hosts
 +
auto eth2
 +
iface eth2 inet static
 +
        address 10.49.254.20
 +
        netmask 255.255.255.0
 +
        network 10.49.254.0
 +
        broadcast 10.49.254.255
 +
</pre>
 +
* /etc/network/if-up.d/openvswitch :
 +
<pre>
 +
#! /bin/sh
 +
# Setup openvswitch
 +
 +
# Don't bother with loopback
 +
if [ "$IFACE" = lo ]; then
 +
        exit 0
 +
fi
 +
 +
 +
# Only run from ifup.
 +
if [ "$MODE" != start ]; then
 +
        exit 0
 +
fi
 +
 +
if [ "$IFACE" = "eth0" ]; then
 +
        /usr/bin/ovs-vsctl add-br br0
 +
        /usr/bin/ovs-vsctl add-port br0 eth0
 +
fi
 +
 +
exit 0
 +
</pre>
 +
 
* /etc/hosts :
 
* /etc/hosts :
 
<pre>
 
<pre>
Line 30: Line 118:
 
   $ sudo systemctl enable nfs-kernel-server
 
   $ sudo systemctl enable nfs-kernel-server
 
   $ sudo systemctl start nfs-kernel-server
 
   $ sudo systemctl start nfs-kernel-server
 +
=== Misc bugfixes / errors ===
 +
==== "error:  Diskfilter writes are not supported" ====
 +
This occurs around bootup splash screen time, and does not affect system operation, but is ugly.  Supposedly fixed in 15.10+.<br/>
 +
Ugly quick fix:<br/>
 +
  $ sudo vi /etc/grub.d/10_linux
 +
  (Replace 'quick_boot="1"' with 'quick_boot="0"', then...)
 +
  $ sudo update-grub
 +
 +
See bug: [https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1274320 https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1274320]
 +
 +
==== PCI-DMA: Out of IOMMU space for... ====
 +
This SHOULD be platform-specific (Sun Fire X2200 M2).  If you begin to see messages like this in your logs:<br/>
 +
  Oct 16 18:04:55 vm1 kernel: [  113.785020] sata_nv 0000:00:05.0: PCI-DMA: Out of IOMMU space for 65536 bytes
 +
You ought to consider the following fix:
 +
  $ sudo vi /etc/default/grub
 +
Add "iommu=memaper3" to this line:
 +
  GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
 +
Result should be:
 +
  GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=memaper=3"
 +
Requires a reboot.
 +
=== KVM Networking ===
 +
KVM Networking is an odd beast, and I required more flexibility than was natively<br/>
 +
included in most prebuilt versions.  Ubuntu 15.04 and newer include good support for<br/>
 +
binding guests into VLANs and not requiring a physical interface for each VLAN to get<br/>
 +
into the machine.  This is accomplished with openvswitch and some creativity.  Here<br/>
 +
are a couple XML files useful to "net-define" in virsh.  A few examples should allow<br/>
 +
understanding of creating others with unique 802.1Q vlan tags.<br/>
 +
<br/>
 +
If you fancy having an untagged ("native") vlan on the physical port and actually<br/>
 +
using it, well, I forget how I did that.  I stopped using the method and instead<br/>
 +
went for explicitly specifying tags.<br/>
 +
==== ovs-vlan2.xml ====
 +
<pre>
 +
<network>
 +
  <name>ovs-vlan2</name>
 +
  <forward mode='bridge'/>
 +
  <bridge name='br0'/>
 +
  <virtualport type='openvswitch'/>
 +
  <portgroup name='ovs-vlan2' default='yes'>
 +
    <vlan>
 +
      <tag id='2'/>
 +
    </vlan>
 +
  </portgroup>
 +
</network>
 +
</pre>
 +
==== ovs-vlan900.xml ====
 +
<pre>
 +
<network>
 +
  <name>ovs-vlan900</name>
 +
  <forward mode='bridge'/>
 +
  <bridge name='br0'/>
 +
  <virtualport type='openvswitch'/>
 +
  <portgroup name='ovs-vlan900' default='yes'>
 +
    <vlan>
 +
      <tag id='900'/>
 +
    </vlan>
 +
  </portgroup>
 +
</network>
 +
</pre>
 +
 +
=== Example VM ===
 +
Here's an example VM XML file using the above, to be used with virsh ("define").<br/>
 +
==== examplevm.xml ====
 +
<pre>
 +
<domain type='kvm'>
 +
  <name>examplevm</name>
 +
  <memory unit='KiB'>1048576</memory>
 +
  <currentMemory unit='KiB'>1048576</currentMemory>
 +
  <vcpu placement='static'>1</vcpu>
 +
  <os>
 +
    <type arch='x86_64' machine='pc-1.1'>hvm</type>
 +
    <boot dev='hd'/>
 +
  </os>
 +
  <features>
 +
    <acpi/>
 +
    <apic/>
 +
    <pae/>
 +
  </features>
 +
  <cpu mode='custom' match='exact'>
 +
    <model fallback='allow'>Opteron_G2</model>
 +
  </cpu>
 +
  <clock offset='utc'>
 +
    <timer name='rtc' tickpolicy='catchup'/>
 +
    <timer name='pit' tickpolicy='delay'/>
 +
    <timer name='hpet' present='no'/>
 +
  </clock>
 +
  <on_poweroff>destroy</on_poweroff>
 +
  <on_reboot>restart</on_reboot>
 +
  <on_crash>restart</on_crash>
 +
  <devices>
 +
    <emulator>/usr/bin/kvm</emulator>
 +
    <disk type='file' device='disk'>
 +
      <driver name='qemu' type='qcow2' cache='none' io='native'/>
 +
      <source file='/vm-store-0/img/examplevm.img'/>
 +
      <target dev='vda' bus='virtio'/>
 +
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
 +
    </disk>
 +
    <disk type='block' device='cdrom'>
 +
      <driver name='qemu' type='raw'/>
 +
      <target dev='hda' bus='ide'/>
 +
      <readonly/>
 +
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
 +
    </disk>
 +
    <controller type='usb' index='0' model='ich9-ehci1'>
 +
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x7'/>
 +
    </controller>
 +
    <controller type='usb' index='0' model='ich9-uhci1'>
 +
      <master startport='0'/>
 +
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0' multifunction='on'/>
 +
    </controller>
 +
    <controller type='usb' index='0' model='ich9-uhci2'>
 +
      <master startport='2'/>
 +
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x1'/>
 +
    </controller>
 +
    <controller type='usb' index='0' model='ich9-uhci3'>
 +
      <master startport='4'/>
 +
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x2'/>
 +
    </controller>
 +
    <controller type='pci' index='0' model='pci-root'/>
 +
    <controller type='ide' index='0'>
 +
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
 +
    </controller>
 +
    <interface type='network'>
 +
      <source network='ovs-vlan2'/>
 +
      <model type='e1000'/>
 +
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
 +
    </interface>
 +
    <serial type='pty'>
 +
      <target port='0'/>
 +
    </serial>
 +
    <console type='pty'>
 +
      <target type='serial' port='0'/>
 +
    </console>
 +
    <input type='mouse' bus='ps2'/>
 +
    <input type='keyboard' bus='ps2'/>
 +
    <graphics type='vnc' port='-1' autoport='yes'/>
 +
    <video>
 +
      <model type='cirrus' vram='16384' heads='1'/>
 +
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
 +
    </video>
 +
    <memballoon model='virtio'>
 +
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
 +
    </memballoon>
 +
  </devices>
 +
</domain>
 +
</pre>

Latest revision as of 17:38, 3 November 2015

Ubuntu 15.04 Notes

Overview

This section should get a general build of a VM host running KVM with libvirt (virsh)
and 2TB of RAID1 (softraid) running on Ubuntu 15.04 server. The hardware platform
that this section targets/was tested on is a Sun Fire X2200 M2. There is a hardware
specific notes section below.

The disks will be setup with RAID and LVM, giving a maximum compromise of reliability
and flexibility. Swap partitions are setup outside of softraid to maximize performance.

Disk Config

  • (2) x 2.0 TB disks in chassis, built at install time using the following general procedure:
    • From aux shell: fdisk /dev/sda and "g" to GPT the disk, repeat for /dev/sdb
    • Create a new (bios_grub) partition on each disk: ~256MB is fine
    • Create any swap partitions necessary
    • Create a new identically sized partition on each disk, "Use as: physical volume for RAID"
    • Configure Software RAID: RAID1, 2 devices, 0 spares, select the two identical partitions as members.
    • Configure LVM:
      • Create VG with PV /dev/md0 (named "vm1-vg")
      • Create LV (20G) from VG "vm1-vg" (named "vm1-root")
      • Create LV (20G) from VG "vm1-vg" (named "vm1-iso-0")
      • Create LV (1T) from VG "vm1-vg" (named "vm1-store-0")
      • Setup "vm1-root" as / with ext4 FS
      • Setup "vm1-iso-0" as /vm1-iso with ext4 FS
      • Setup "vm1-store-0" as /vm1-store-0 with ext4 FS
    • When asked, install grub to "/dev/sda" (it'll try to use "/dev/mapper", override this.)
    • After firstboot of system, run "dpkg-reconfigure grub-pc" and choose to install on "/dev/sdb" as well.

Worthwhile external reading:

Packages

 $ sudo apt-get install zsh multitail screen sysstat openssh-server nfs-kernel-server
 $ sudo apt-get install qemu-kvm libvirt-bin openvswitch-switch openvswitch-common
 $ sudo apt-get install libpolkit-agent-1-0 libpolkit-backend-1-0 policykit-1
 $ sudo apt-get install virtinst virt-top iftop vlock smartmontools check-mk-agent xinetd

System Config

  • /etc/network/interfaces :
# The loopback network interface
auto lo
iface lo inet loopback

################################################################################
# eth1: management interface (non-VM traffic)
auto eth1
iface eth1 inet static
        address 10.49.249.20
        netmask 255.255.255.0
        network 10.49.249.0
        broadcast 10.49.249.255
        gateway 10.49.249.1
        # dns-* options are implemented by the resolvconf package, if installed
        dns-nameservers 10.49.249.1
        dns-search je

################################################################################
# eth0: vlan-tagged for VM traffic
auto eth0
iface eth0 inet manual
################################################################################
# eth2: storage/cross-connect (xc) net, used with a pair of VM hosts
auto eth2
iface eth2 inet static
        address 10.49.254.20
        netmask 255.255.255.0
        network 10.49.254.0
        broadcast 10.49.254.255
  • /etc/network/if-up.d/openvswitch :
#! /bin/sh
# Setup openvswitch

# Don't bother with loopback
if [ "$IFACE" = lo ]; then
        exit 0
fi


# Only run from ifup.
if [ "$MODE" != start ]; then
        exit 0
fi

if [ "$IFACE" = "eth0" ]; then
        /usr/bin/ovs-vsctl add-br br0
        /usr/bin/ovs-vsctl add-port br0 eth0
fi

exit 0
  • /etc/hosts :
127.0.0.1       localhost
127.0.1.1       vm1

10.49.249.20    vm1.je
10.49.249.21    vm2.je

10.49.254.20    vm1.xc  vm1
10.49.254.21    vm2.xc  vm2

# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
  • Setup storage repo:
 $ sudo ln -s /vm1-store-0 /vm-store-0

NFS

  • /etc/exports (on vm1):
 /vm1-store-0            vm2.xc(rw,sync,no_subtree_check,no_root_squash)
  • Enable/start NFS:
 $ sudo systemctl enable nfs-kernel-server
 $ sudo systemctl start nfs-kernel-server

Misc bugfixes / errors

"error: Diskfilter writes are not supported"

This occurs around bootup splash screen time, and does not affect system operation, but is ugly. Supposedly fixed in 15.10+.
Ugly quick fix:

 $ sudo vi /etc/grub.d/10_linux
 (Replace 'quick_boot="1"' with 'quick_boot="0"', then...)
 $ sudo update-grub

See bug: https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/1274320

PCI-DMA: Out of IOMMU space for...

This SHOULD be platform-specific (Sun Fire X2200 M2). If you begin to see messages like this in your logs:

 Oct 16 18:04:55 vm1 kernel: [  113.785020] sata_nv 0000:00:05.0: PCI-DMA: Out of IOMMU space for 65536 bytes

You ought to consider the following fix: 

 $ sudo vi /etc/default/grub

Add "iommu=memaper3" to this line: 

 GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"

Result should be: 

 GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=memaper=3"

Requires a reboot.

KVM Networking

KVM Networking is an odd beast, and I required more flexibility than was natively
included in most prebuilt versions. Ubuntu 15.04 and newer include good support for
binding guests into VLANs and not requiring a physical interface for each VLAN to get
into the machine. This is accomplished with openvswitch and some creativity. Here
are a couple XML files useful to "net-define" in virsh. A few examples should allow
understanding of creating others with unique 802.1Q vlan tags.

If you fancy having an untagged ("native") vlan on the physical port and actually
using it, well, I forget how I did that. I stopped using the method and instead
went for explicitly specifying tags.

ovs-vlan2.xml

<network>
  <name>ovs-vlan2</name>
  <forward mode='bridge'/>
  <bridge name='br0'/>
  <virtualport type='openvswitch'/>
  <portgroup name='ovs-vlan2' default='yes'>
    <vlan>
      <tag id='2'/>
    </vlan>
  </portgroup>
</network>

ovs-vlan900.xml

<network>
  <name>ovs-vlan900</name>
  <forward mode='bridge'/>
  <bridge name='br0'/>
  <virtualport type='openvswitch'/>
  <portgroup name='ovs-vlan900' default='yes'>
    <vlan>
      <tag id='900'/>
    </vlan>
  </portgroup>
</network>

Example VM

Here's an example VM XML file using the above, to be used with virsh ("define").

examplevm.xml

<domain type='kvm'>
  <name>examplevm</name>
  <memory unit='KiB'>1048576</memory>
  <currentMemory unit='KiB'>1048576</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <os>
    <type arch='x86_64' machine='pc-1.1'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>Opteron_G2</model>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none' io='native'/>
      <source file='/vm-store-0/img/examplevm.img'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>
    <disk type='block' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='network'>
      <source network='ovs-vlan2'/>
      <model type='e1000'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes'/>
    <video>
      <model type='cirrus' vram='16384' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </memballoon>
  </devices>
</domain>
Retrieved from "http://wtf.hijacked.us/wiki/index.php?title=Jontow_Homelab_KVM&oldid=1250"