Running Multiple Linux Servers as KVM Virtual Machines

KVM is virtual machine software for Linux that allows you to run multiple guest operating systems on top of the real host system. Unlike, the basic versions of VMWare and VirtualBox, KVM is designed for running operating systems that do not require a GUI, e.g. servers. In the past at work we had two separate physical internal servers: one for students and one for staff. So when we did a hardware upgrade I setup KVM so that both servers could run as separate virtual guests on the single physical host. Most of the instructions I followed came from the Ubuntu KVM Guide, as well as the man pages for virt-inst.

Installing Ubuntu on the Host Machine

In this case I installed Ubuntu Server 10.04 LTS (Lucid) on a PC with: Intel i7-2600 3.4GHz; Asus P8H67-M motherboard; 4GB RAM; 2 x Western Digital Green 1.5TB hard drives. I followed a standard install using the following options:

Hostname: ictweb
Partitioning: /boot plus LVM including /, /tmp, /var and /home, as well as two logical volumes for the guest virtual machines.
Selected OpenSSH server and Virtual Machine Host as install package options.

I did a package update and upgrade with apt-get.

Preparing Host Machine for Virtual Machine Guests

By selecting the Virtual Machine Host during installation, most of the necessary packages (e.g. libvirt) were automatically installed. But some additional packages need to be manually installed:

$ sudo apt-get install qemu libcap2-bin virtinst

The X window system is not installed on the host, and so to initially access an installed VM guest I used VNC/SSH from my laptop. The access requires root password on the host, and so that must be set:

$ sudo su
root@ictweb:/home/sgordon# passwd

Optionally, change the SSH server port to 11111 (or any other unused other port number) by editing /etc/ssh/sshd_config and setting the Port field. Finally, I had to enable the Intel Virtualization Technology in the BIOS (and hence a reboot of the host machine).

Networking for Host and Guests

The intended network topology for the host and VM guests is shown below:

By selecting the Virtual Machine Host during installation, bridging/routing is correctly setup so that once the VM guests are installed, they can be accessed on their own LAN. By default the host obtained an IP for the Ethernet interface (eth0) from the network DHCP server, and a virtual bridge was created (virbr0) with IP 192.168.122.1. I edited /etc/network/interfaces to give the host a static IP, and also created IP aliases for the VM guests. The aliases will allow the guests to be accessed on publicly accessible addresses (public within our work LAN at least, not necessarily on the global Internet). The interfaces file looked like this:

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
        address 192.168.1.2
        network 192.168.1.0
        netmask 255.255.255.0
        broadcast 192.168.1.255
        gateway 192.168.1.1

auto eth0:0
iface eth0:0 inet static
        address 192.168.1.10
        network 192.168.1.0
        netmask 255.255.255.0
        broadcast 192.168.1.255
        gateway 192.168.1.1

auto eth0:1
iface eth0:1 inet static
        address 192.168.1.11
        network 192.168.1.0
        netmask 255.255.255.0
        broadcast 192.168.1.255
        gateway 192.168.1.1

eth0:0 is a virtual interface and will be used to make one of the VM guests accessible via 192.168.1.10 (similar for eth0:1). Restart networking for the changes to have effect:

$ sudo /etc/init.d/networking stop
$ sudo /etc/init.d/networking start

Here is the output of ifconfig on the host machine (with many unnecessary details removed):

sgordon@ictweb:~$ ifconfig
eth0      Link encap:Ethernet  HWaddr f4:6d:...
          inet addr:192.168.1.2  Bcast:192.168.1.255  Mask:255.255.255.0
          ...

eth0:0    Link encap:Ethernet  HWaddr f4:6d:...
          inet addr:192.168.1.10  Bcast:192.168.1.255  Mask:255.255.255.0
          ...

eth0:1    Link encap:Ethernet  HWaddr f4:6d:
          inet addr:192.168.1.11  Bcast:192.168.1.255  Mask:255.255.255.0
          ...

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          ...

virbr0    Link encap:Ethernet  HWaddr fe:54:...
          inet addr:192.168.122.1  Bcast:192.168.122.255  Mask:255.255.255.0
          ...

vnet0     Link encap:Ethernet  HWaddr fe:54:...
          ...

vnet1     Link encap:Ethernet  HWaddr fe:54:...
          ...

Configuring the Firewall/NAT on Host

The firewall/NAT must be configured so that traffic coming into the host can be forwarded to the appropriate VM guest. In particular, data to/from the IP aliases addresses (e.g. 192.168.1.10) must be forwarded to the real VM guest address (192.168.122.10). The NAT table of iptables is used for this. In addition, rules are needed to allow the router to forward packets to the SSH, Web and Email servers on the guests. The rules I applied were:

$ sudo iptables -t nat -I PREROUTING -d 192.168.1.10 -j DNAT --to-destination 192.168.122.10
$ sudo iptables -t nat -I POSTROUTING -s 192.168.122.10 -j SNAT --to-source 192.168.1.10
$ sudo iptables -t nat -I PREROUTING -d 192.168.1.11 -j DNAT --to-destination 192.168.122.11
$ sudo iptables -t nat -I POSTROUTING -s 192.168.122.11 -j SNAT --to-source 192.168.1.11
$ sudo iptables -I FORWARD -p tcp -d 192.168.122.10 --dport 22 -j ACCEPT
$ sudo iptables -I FORWARD -p tcp -d 192.168.122.10 --dport 25 -j ACCEPT
$ sudo iptables -I FORWARD -p tcp -d 192.168.122.10 --dport 80 -j ACCEPT
$ sudo iptables -I FORWARD -p tcp -d 192.168.122.11 --dport 22 -j ACCEPT
$ sudo iptables -I FORWARD -p tcp -d 192.168.122.11 --dport 25 -j ACCEPT
$ sudo iptables -I FORWARD -p tcp -d 192.168.122.11 --dport 80 -j ACCEPT

To automatically create the firewall table upon reboot I saved the current set of rules:

$ sudo iptables-save -c > ~/iptables.rules
$ sudo mv ~/iptables.rules /etc/
$ sudo chown root.root /etc/iptables.rules
$ sudo chmod go-rwx /etc/iptables.rules

And then added the following line to /etc/rc.local so that the rules will be restored on boot:

iptables-restore < /etc/iptables.rules

Other rules can be added to this file as needed for your firewall. (Update: I needed to add a sleep 60 command before the restoration of the rules in the rc.local above. The reason is to all all other processes to update the firewall before we restore the desired rules. There should be a better way to do this, probably using upstart).

Installing the Virtual Machine Guests

I had the ISO image for Ubuntu on a USB disk. The first task is to mount the USB disk (mine uses ext4 file system; the -t option below may need to be changed depending on the filesystem of your USB. Also, the device needs to be correct):

$ cd ~
$ mkdir usb
$ sudo mount -t ext4 /dev/sdg1 ~/usb

Set the capabilities/permissions for the QEMU executable, and also if necessary add your username to the cap_net_admin field in the file /etc/security/capabilities.conf.

$ sudo setcap cap_net_admin=ei /usr/bin/qemu-system-x86_64

Now use virt-install to start the install of the Ubuntu VM guest. I am creating a guest with name vmict, 1.5GB of RAM and installing on onoe of the available Logical Volumes setup previously. That is, the guest will run from a raw partition, not from a disk file.

$ sudo virt-install --connect qemu:///system \
--name vmict \
--ram 1500 \
--os-type=linux \
--os-variant=ubuntukarmic \
--accelerate \
--cdrom /home/sgordon/usb/lucid-server-amd64.iso \
--disk path=/dev/ictweb/vmict \
--vnc \
--noautoconsole \
--network network=default

Assuming no errors, the VM guest should have started. To check that it is running use virsh:

$ virsh -c qemu:///system list
 Id Name                 State
----------------------------------
  1 vmict                running

Now to see the graphical output of the guest (and proceed with the Ubuntu installation) we need to connect to it with VNC. On my laptop, which is running Ubuntu, is on the 192.168.1.0 network and has virt-viewer installed, I connected to the running guest:

$ virt-viewer --connect qemu+ssh://root@192.168.1.2/system vmict

The root password of the host machine will be asked for (twice). Then you should see the output of the VM in a window, allowing you to start the Ubuntu installation. In this case if you press F4 you can choose a Minimial virtual machine install. After installing Ubuntu on the guest, you should again connect and configure networking. In particular, edit the /etc/networks/interfaces file and set the static IP address, e.g.

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
        address 192.168.122.11
        network 192.168.122.0
        netmask 255.255.255.0
        broadcast 192.168.122.255
        gateway 192.168.122.1

Also make sure the SSH server is running (it should be if you installed openssh-server or selected the SSH Server option in the Ubuntu install). Now you should be able to access the VM guest at its IP alias via SSH. From the host or some other computer:

$ ssh 192.168.1.11

You should also test access the web server and email server, e.g. using telnet you should be able to connect:

$ telnet 192.168.1.11 80
Trying 192.168.1.11...
Connected to 192.168.1.11.
Escape character is '^]'.
^C
Connection closed by foreign host.

Console Access to Virtual Machine

Once logged into a guest via VNC or SSH, configure console access. This allows access to the guest from the host without using VNC or SSH. This is useful if you make a mistake on the guest and disable SSH or network access; you can obtain direct access to the guest via the host. Create and edit the file /etc/init/ttyS0.conf, add the following configuration that runs getty while the guest is running:

# ttyS0 - getty
#
# This service maintains a getty on ttyS0 from the point the system is
# started until it is shut down again.

start on stopped rc RUNLEVEL=[2345]
stop on runlevel [!2345]

respawn
exec /sbin/getty -L 115200 ttyS0 xterm

The service will start whenever the guest VM starts. However, you need to manually start it now (or reboot the guest):

$ sudo start ttyS0
ttyS0 start/running, process 1646

Now on the host machine you can use the console command in virsh to connect, although you must do it as sudo:

$ sudo virsh -c qemu:///session
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virsh # list
 Id Name                 State
----------------------------------
  2 vmict                running
  4 vmit                 running
virsh # console vmit
Connected to domain vmit
Escape character is ^]

Ubuntu 10.04.2 LTS it ttyS0

it login:

Note that after running the console command and the initial Escape character message is printed, you may have to press Enter to get the login. And once you log out, to exit the console use Ctrl-].

Starting VM Guests on Boot

To make sure the VM guests start automatically when the host boots, use virsh:

$ virsh -c qemu:///system 
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virsh # autostart vmict
Domain vmict marked as autostarted

virsh # autostart vmit
Domain vmit marked as autostarted

Created on Sun, 31 Jul 2011, 5:52pm

Last changed on Wed, 16 May 2012, 8:19pm