Troubleshooting with virtnet Virtual Networks
virtnet is a set of scripts for automatically creating a virtual network of Linux guests with VirtualBox. I developed them to make it easy for students to quickly create a network on their own computer to test different networking and security protocols and applications. However the steps for setting up virtnet are not trivial and the scripts do not include much error checking, and may have many bugs. Therefore here I will try to list common things that may go wrong, and possible approaches to fixing them.
How do I know if the topology was created correctly?
Unfortunately the error reporting in virtnet is not very good. So it is often hard to know if something goes wrong. Below is an example of the terminal output for a successful creation of a topology (with the output of the creation of nodes 3 to 7 omitted - it looks the same as node 2). There may be some slight differences on your output, but significant different messages suggest something when wrong.
iMac-34:host student$ bash vn-createtopology 26
vn: Starting to create new VM node1
vn: Cloning base ...
This machine does not have any snapshots
vn: VM base does not have any snapshots
vn: Taking snapshot of base (snapshot also called base)
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Machine has been successfully cloned as "node1"
vn: Virtual machine base cloned to node1
vn: Disconnecting cable 2 on node1
vn: Disconnecting cable 3 on node1
vn: Disconnecting cable 4 on node1
vn: Configuring NIC 2 on node1
vn: Setting port forwarding on node1
Waiting for VM "node1" to power on...
VM "node1" has been successfully started.
vn: Virtual machine node1 is now running
vn: Removing SSH keys
/Users/student/.ssh/known_hosts updated.
Original contents retained as /Users/student/.ssh/known_hosts.old
vn: Setting interfaces on guest using SSH ...
vn: New VM node1 has been created
vn: Starting to create new VM node2
vn: Cloning base ...
Name: base (UUID: 786e01ec-8234-4d2e-af7f-96f6b89cecf1) *
vn: VM base already has snapshot base
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Machine has been successfully cloned as "node2"
vn: Virtual machine base cloned to node2
vn: Disconnecting cable 2 on node2
vn: Disconnecting cable 3 on node2
vn: Disconnecting cable 4 on node2
vn: Configuring NIC 2 on node2
vn: Setting port forwarding on node2
vn: Configuring NIC 3 on node2
vn: Setting port forwarding on node2
Waiting for VM "node2" to power on...
VM "node2" has been successfully started.
vn: Virtual machine node2 is now running
vn: Removing SSH keys
/Users/student/.ssh/known_hosts updated.
Original contents retained as /Users/student/.ssh/known_hosts.old
vn: Setting interfaces on guest using SSH ...
vn: New VM node2 has been created
vn: Starting to create new VM node3
...
vn: New VM node7 has been created
vn: Starting to create new VM node8
vn: Cloning base ...
Name: base (UUID: 786e01ec-8234-4d2e-af7f-96f6b89cecf1) *
vn: VM base already has snapshot base
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Machine has been successfully cloned as "node8"
vn: Virtual machine base cloned to node8
vn: Disconnecting cable 2 on node8
vn: Disconnecting cable 3 on node8
vn: Disconnecting cable 4 on node8
vn: Configuring NIC 2 on node8
vn: Setting port forwarding on node8
Waiting for VM "node8" to power on...
VM "node8" has been successfully started.
vn: Virtual machine node8 is now running
vn: Removing SSH keys
/Users/student/.ssh/known_hosts updated.
Original contents retained as /Users/student/.ssh/known_hosts.old
vn: Setting interfaces on guest using SSH ...
Warning: Permanently added '[127.0.0.1]:2208' (RSA) to the list of known hosts.
vn: New VM node8 has been created
And to confirm that the topology works, I start all the nodes, log in to node 1 with SSH and then ping some of the other nodes.
iMac-34:host student$ VBoxManage startvm --type headless node1
Waiting for VM "node1" to power on...
VM "node1" has been successfully started.
iMac-34:host student$ VBoxManage startvm --type headless node2
Waiting for VM "node2" to power on...
VM "node2" has been successfully started.
iMac-34:host student$ VBoxManage startvm --type headless node3
Waiting for VM "node3" to power on...
VM "node3" has been successfully started.
iMac-34:host student$ VBoxManage startvm --type headless node4
Waiting for VM "node4" to power on...
VM "node4" has been successfully started.
iMac-34:host student$ VBoxManage startvm --type headless node5
Waiting for VM "node5" to power on...
VM "node5" has been successfully started.
iMac-34:host student$ VBoxManage startvm --type headless node6
Waiting for VM "node6" to power on...
VM "node6" has been successfully started.
iMac-34:host student$ VBoxManage startvm --type headless node7
Waiting for VM "node7" to power on...
VM "node7" has been successfully started.
iMac-34:host student$ VBoxManage startvm --type headless node8
Waiting for VM "node8" to power on...
VM "node8" has been successfully started.
iMac-34:host student$ ssh -p 2201 -l network localhost
The authenticity of host '[localhost]:2201 ([127.0.0.1]:2201)' can't be established.
RSA key fingerprint is 3d:cf:9e:d1:72:78:91:04:dd:8b:73:7b:93:fa:4d:80.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '[localhost]:2201' (RSA) to the list of known hosts.
network@localhost's password: network
Welcome to Ubuntu 12.04.3 LTS (GNU/Linux 3.8.0-29-generic i686)
* Documentation: https://help.ubuntu.com/
System information as of Thu Jan 30 08:34:24 ICT 2014
System load: 0.0 Processes: 68
Usage of /: 63.1% of 1.77GB Users logged in: 0
Memory usage: 22% IP address for eth0: 10.0.2.15
Swap usage: 0% IP address for eth1: 192.168.1.11
Graph this data and manage this system at https://landscape.canonical.com/
Last login: Thu Jan 30 08:28:42 2014
network@node1:~$ ping -c 3 192.168.2.1
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_req=1 ttl=64 time=0.668 ms
64 bytes from 192.168.2.1: icmp_req=2 ttl=64 time=0.841 ms
64 bytes from 192.168.2.1: icmp_req=3 ttl=64 time=0.836 ms
--- 192.168.2.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1998ms
rtt min/avg/max/mdev = 0.668/0.781/0.841/0.086 ms
network@node1:~$ ping -c 3 192.168.2.21
PING 192.168.2.21 (192.168.2.21) 56(84) bytes of data.
64 bytes from 192.168.2.21: icmp_req=1 ttl=63 time=2.39 ms
64 bytes from 192.168.2.21: icmp_req=2 ttl=63 time=1.69 ms
64 bytes from 192.168.2.21: icmp_req=3 ttl=63 time=1.72 ms
--- 192.168.2.21 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 1.694/1.937/2.393/0.326 ms
network@node1:~$ ping -c 3 192.168.3.2
PING 192.168.3.2 (192.168.3.2) 56(84) bytes of data.
64 bytes from 192.168.3.2: icmp_req=1 ttl=63 time=1.32 ms
64 bytes from 192.168.3.2: icmp_req=2 ttl=63 time=1.44 ms
64 bytes from 192.168.3.2: icmp_req=3 ttl=63 time=1.52 ms
--- 192.168.3.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 1.327/1.433/1.529/0.093 ms
network@node1:~$ ping -c 3 192.168.3.31
PING 192.168.3.31 (192.168.3.31) 56(84) bytes of data.
64 bytes from 192.168.3.31: icmp_req=1 ttl=62 time=1.98 ms
64 bytes from 192.168.3.31: icmp_req=2 ttl=62 time=2.22 ms
64 bytes from 192.168.3.31: icmp_req=3 ttl=62 time=2.28 ms
--- 192.168.3.31 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 1.982/2.162/2.286/0.141 ms
Error message while running vn-createtopology; SSH prompts for root password
Reason 1: The private key for the root user in the SVN directory does not have correct permissions. The file data/defaults/root/.ssh/id_rsa should be only readable by the user, not groups or others.
Solution 1: Change the permissions of data/defaults/root/.ssh/id_rsa by running the following on the host (assuming you are currently virtnet directory created when checking out from the SVN repository):
$ chmod go-rwx virtnet/data/defaults/root/.ssh/id_rsa
Error message while running vn-createtopology; node(s) are not created in VirtualBox
Reason 1: vn-createtopology automatically creates nodes named "node1", "node2", "node3", ... .If VirtualBox guests with these names already exist (e.g. from a previously created topology), then new nodes will not be created.
Solution 1: Rename the nodes with the conflicting names in VirtualBox, e.g. to oldnode1, before running vn-createtopology. Alternatively, if you no longer require the old nodes, delete them in VirtualBox, making sure to select the option to remove all files.
Topology number X does not exist
Reason 1: If you run vn-createtopology X where X is a number that is not currently available on the host and in the base VM, then you will get errors. If the topology doesn't exist and it is referred to elsewhere (e.g. in my examples), then most likely your SVN repository is not up to date.
Solution 1: Update your SVN repository on both the host and the base VM. To update on the host, enter the directory where the virtnet SVN repository was setup (e.g. /home/user/svn/virtnet) and run:
$ svn update
This should download the latest topologies (and other files) from the SVN server. You must also update the base VM. To do so, open base in VirtualBox and login and then:
network@base:~$ cd virtnet
network@base:~$ svn update
network@base:~$ sudo rm /etc/udev/rules.d/70-persistent-net.rules
network@base:~$ sudo poweroff
If the base VM has a snapshot called "base", then delete it. Now try to create the topology again, i.e. bash vn-createtopology X.
node is very slow to boot; presents error during boot about "Waiting for network configuration"
Reason 1: Most likely the base VM is not configured correctly so that when it is copied to a node, that node gets the base VMs MAC address in Ubuntu. However VirtualBox gives it a different MAC address, causing the network being unable to start upon boot. The most common cause for this is that the base VM has been started in VirtualBox (normally there should be no need to start the base VM unless you need to update SVN; instead you should always clone it with vn-createtopology).
Solution 1: Start the base VM and login and then:
network@base:~$ sudo rm /etc/udev/rules.d/70-persistent-net.rules
network@base:~$ sudo reboot
If the base VM has a snapshot called "base", then delete it. Now try to create the topology again.
Created on Mon, 27 Jan 2014, 10:44pm
Last changed on Thu, 30 Jan 2014, 11:56am