CentOS 5 Virtualization Guide - Chapter 16. Virtualization live migration

Chapter 16. Virtualization live migration

$8.99

16.1. A live migration example

Red Hat Virtualization includes the capabilities to support migration of para-virtualized guests between Red Hat Virtualization servers. Migration can either be performed in two ways:

Offline mode using the command xm migrate VirtualMachineName HostName. In this mode the virtual machine will be stopped on the original host and restarted on the new host.
Live mode using the --live option for the command xm migrate --live VirtualMachineName HostName.

Word usage note

Take note of the interchangeable use of relocation and migration throughout these section. The different terms are used to match the different naming conventions of certain configuration files. Both terms can be taken to mean the same thing, that is the relocation of one guest image from one server to another.

Itanium support note

Virtual machine migration is presently unsupported on the Itanium® architecture.

To enable the use of migration a few changes must be made to configuration file /etc/xen/xend-config.sxp. By default migration is disabled due to the potentially harmful affects on the host's security. Opening the relocation port carries the potential ability of unauthorized hosts and users to initiate migrate or connect to the relocation ports. As there is no specific authentication for relocation requests and the only control mechanism is based on hostnames and IP addresses special care should be taken to make sure the migration port and server is not accessible to unauthorized hosts.

A note on virtualization migration security

IP address and hostname filters offer only minimal security. Both of these attributes can be forged if the attacker knows the address or hostname of the migration client. The best method for securing migration is to isolate the network the host and client are on from external and unauthorized internal connections.

Enabling migration

Modify the following entries in /etc/xen/xend-config.sxp to enable migration, remove the comments preceding the parameters in the configuration file:

(xend-relocation-server yes)

The default value is no to keep the migration server deactivated. Unless you are using a trusted network, the domain virtual memory will be exchanged in raw form without encryption of the communication.

You modify the xend-relocation-hosts-allow option to restrict access to the migration server.

(xend-relocation-port 8002)

The parameter, (xend-relocation-port), specifies the port xend should use for the relocation interface, if xend-relocation-server is set to yes

The default value of this variable should work for most installations. If you change the value make sure you are using an unused port on the relocation server.

(xend-relocation-address '')

(xend-relocation-address)is the address the xend should listen on for relocation-socket connections, if xend-relocation-server is set.

The default is listen on all active interfaces, the parameter can be used to restrict the relocation server to only listen to a specific interface. The default value in /etc/xen/xend-config.sxp is an empty string( '' ). This value should be replaced with a valid list of addresses or regular expressions surrounded by single quotes.

(xend-relocation-hosts-allow '')

The (xend-relocation-hosts-allow ) parameter is used to control the hosts who are allowed to talk to the relocation port.

If the value is empty, as denoted in the example above by an empty string surrounded by single quotes, then all connections are allowed. This assumes the connection arrives on a port and interface which the relocation server listens on, see also xend-relocation-port and xend-relocation-address above).

Otherwise, the (xend-relocation-hosts-allow ) parameter should be a sequence of regular expressions separated by spaces. Any host with a fully-qualified domain name or an IP address which matches one of these regular expressions will be accepted.

An example of a (xend-relocation-hosts-allow ) attribute:

(xend-relocation-hosts-allow '^localhost$ ^localhost\\.localdomain$')

After you have configured the parameters in your configuration file you should reboot the host to restart your environment with the new parameters.

16.1. A live migration example

Below is an example of how to setup a simple environment for live migration. This configuration is using NFS for the shared storage. NFS is suitable for demonstration environments but for a production environment a more robust shared storage configuration using Fibre Channel or iSCSI and GFS is recommended.

The configuration below consists of two servers (et-virt07 and et-virt08), both of them are using eth1 as their default network interface hence they are using xenbr1 as their Red Hat Virtualization networking bridge. We are using a locally attached SCSI disk (/dev/sdb) on et-virt07 for shared storage using NFS .

Setup for live migration

Create and mount the directory used for the migration:

# mkdir /xentest
# mount /dev/sdb /xentest

Important

Ensure the directory is exported with the correct options. If you are exporting the default directory /var/lib/xen/images/ make sure you only export /var/lib/xen/images/ and not /var/lib/xen/ as this directory is used by the xend daemon and other tools. Sharing /var/lib/xen/ will cause unpredictable behavior.

# cat /etc/exports
/xentest  *(rw,async,no_root_squash)

Verify it is exported via NFS :

# showmount -e et-virt07
Export list for et-virt07:
/xentest *

Install the virtual machine

The install command in the example used for installing the virtual machine:

# virt-install -p -f /xentest/xentestTravelVM01.dsk -s 5 -n\
xentesttravelvm01 --vnc -r 1024 -l https://porkchop.devel.redhat.com/rel-eng/RHEL5-\
Server-20070105.0/4.92/x86_64/os/ -b xenbr1

Verify environment for migration

Make sure the virtualized network bridges are configured correctly and have the same name on both hosts:

[et-virt08 ~]# brctl show
bridge name     bridge id               STP enabled     interfaces
xenbr1          8000.feffffffffff       no              peth1
vif0.1

[et-virt07 ~]# brctl show
bridge name     bridge id               STP enabled     interfaces
xenbr1          8000.feffffffffff       no              peth1
vif0.1

Verify the relocation parameters are configured on both hosts:

[et-virt07 ~]# grep xend-relocation /etc/xen/xend-config.sxp |grep -v '#'
(xend-relocation-server yes)
(xend-relocation-port 8002)
(xend-relocation-address '')
(xend-relocation-hosts-allow '')

[et-virt08 ~]# grep xend-relocation /etc/xen/xend-config.sxp |grep -v '#'
(xend-relocation-server yes)
(xend-relocation-port 8002)
(xend-relocation-address '')
(xend-relocation-hosts-allow '')

Make sure the relocation server has started and is listening on the dedicated port for Xen migrations (8002):

[et-virt07 ~]# lsof -i :8002
COMMAND  PID  USER   FD   TYPE  DEVICE SIZE NODE NAME
python 3445 root 14u IPv4 10223 TCP *:teradataordbms (LISTEN)

[et-virt08 ~]# lsof -i :8002
COMMAND  PID USER   FD   TYPE DEVICE SIZE NODE NAME
python 3252 root 14u IPv4 10901 TCP *:teradataordbms (LISTEN)

Verify the NFS directory has been mounted on the other host and you can see and access the virtual machine image and file system:

[et-virt08 ~]# df /xentest
Filesystem           1K-blocks      Used Available Use% Mounted on
et-virt07:/xentest    70562400   2379712  64598336   4% /xentest

[et-virt08 ~]# file /xentest/xentestTravelVM01.dsk 
/xentest/xentestTravelVM01.dsk: x86 boot sector; partition 1: ID=0x83,
active, starthead 1, startsector 63, 208782 sectors; partition 2: ID=0x8e, 
starthead 0, startsector 208845, 10265535 sectors, code offset 0x48

[et-virt08 ~]# touch /xentest/foo
[et-virt08 ~]# rm -f /xentest/foo

Verification of save and restore on local host

Start up the virtual machine (if it has not yet):

[et-virt07 ~]# xm li
Name             ID Mem(MiB) VCPUs State   Time(s)
Domain-0         0     1880  8     r-----  50.7

[et-virt07 ~]# xm create xentesttravelvm01
Using config file "/etc/xen/xentesttravelvm01".
Going to boot Red Hat Enterprise Linux Server (2.6.18-1.2961.el5xen)
kernel: /vmlinuz-2.6.18-1.2961.el5xen
initrd: /initrd-2.6.18-1.2961.el5xen.img
Started domain xentesttravelvm01

Verify the virtual machine is running:

[et-virt07 ~]# xm li
Name                     ID Mem(MiB) VCPUs State   Time(s)
Domain-0                 0  983      8     r-----  58.2
xentesttravelvm01        1  1024     1     -b----  9.2

Save the virtual machine on the local host:

[et-virt07 xentest]# time xm save xentesttravelvm01 xentesttravelvm01.sav
real    0m15.744s
user    0m0.188s
sys     0m0.044s

[et-virt07 xentest]# ls -lrt xentesttravelvm01.sav
-rwxr-xr-x 1 root root 1075657716 Jan 12 06:46 xentesttravelvm01.sav

[et-virt07 xentest]# xm li
Name                  ID Mem(MiB) VCPUs State   Time(s)
Domain-0              0  975      8     r-----  110.7

Restore the virtual machine on the local host:

[et-virt07 xentest]# time xm restore xentesttravelvm01.sav
real    0m12.443s
user    0m0.072s
sys     0m0.044s

[et-virt07 xentest]# xm li
Name                   ID Mem(MiB) VCPUs State   Time(s)
Domain-0               0  975      8     r-----  118.5
xentesttravelvm01      3  1023     1     -b----  0.0

Live migration test

Run a simple loop inside the guest to print out time and hostname every 3 seconds.

Note

The local host's clock is set to a different time 4hrs ahead of the remote host's clock.

# while true
> do
> hostname ; date
> sleep 3
> done
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:50:16 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:50:19 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:50:22 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:50:25 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:22:24 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:22:27 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:22:30 EST 2007

Verify the virtual machine is running:

[et-virt08 xen]# xm li
Name                      ID Mem(MiB) VCPUs State   Time(s)
Domain-0                  0  975      4     r-----  45.9
xentesttravelvm01         1  1023     1     -b----  1.3

Initiate the live migration to et-virt08. in the example below et-virt07 is the hostname you are migrating to and <domain-id> must be replaced with a guest domain available to the host system.

[et-virt08 ~]# xm migrate --live <domain-id> et-virt07

Verify the virtual machine has been shut down on et-virt07

[et-virt07 xentest]# xm li
Name                        ID Mem(MiB) VCPUs State   Time(s)
Domain-0                    0  975      8     r-----  161.1

Verify the virtual machine has been migrated to et-virt08:

[et-virt08 ~]# xm li
Name                        ID Mem(MiB) VCPUs State   Time(s)
Domain-0                    0  975      4     r-----  46.3
xentesttravelvm01           1  1023     1     -b----  1.6

Testing the progress and initiating the live migration

Create the following script inside the virtual machine to log date and hostname during the migration. This script performs I/O tasks on the virtual machine's file system.

#!/bin/bash

while true
do
touch /var/tmp/$$.log
echo `hostname` >>  /var/tmp/$$.log
	echo `date`     >>  /var/tmp/$$.log
	cat  /var/tmp/$$.log
	df /var/tmp
	ls -l  /var/tmp/$$.log
	sleep 3
	done

Remember, that script is only for testing purposes and unnecessary for a live migration in a production environment.

Verify the virtual machine is running on et-virt08 before we try to migrate it to et-virt07:

[et-virt08 ~]# xm li
Name                      ID Mem(MiB) VCPUs State   Time(s)
Domain-0                  0  975      4     r-----  46.3
xentesttravelvm01         1  1023     1     -b----  1.6

Initiate a live migration to et-virt07. You can add the time command to see how long the migration takes:

[et-virt08 ~]# time xm migrate --live xentesttravelvm01 et-virt07
real    0m10.378s
user    0m0.068s
sys     0m0.052s

run the script inside the guest:

# ./doit
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:26:27 EST 2007
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
2983664   2043120    786536  73% /
-rw-r--r-- 1 root root 62 Jan 12 02:26 /var/tmp/2279.log
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:26:27 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:26:30 EST 2007
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
2983664   2043120    786536  73% /
-rw-r--r-- 1 root root 124 Jan 12 02:26 /var/tmp/2279.log
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:26:27 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:26:30 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:26:33 EST 2007
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
2983664   2043120    786536  73% /
-rw-r--r-- 1 root root 186 Jan 12 02:26 /var/tmp/2279.log
Fri Jan 12 02:26:45 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:26:48 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:26:51 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:54:57 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:55:00 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:55:03 EST 2007
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
2983664   2043120    786536  73% /
-rw-r--r-- 1 root root 744 Jan 12 06:55 /var/tmp/2279.log
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:26:27 EST 2007

Verify the virtual machine has been shut down on et-virt08:

[et-virt08 ~]# xm li
Name                    ID Mem(MiB) VCPUs State   Time(s)
Domain-0                0  975      4     r-----  56.3

Verify the virtual machine has started up on et-virt07:

[et-virt07 xentest]# xm li
Name                    ID Mem(MiB) VCPUs State   Time(s)
Domain-0                0  975      8     r-----  198.1
xentesttravelvm01       4  1023     1     -b----  1.0

Run through another cycle migrating from et-virt07 to et-virt08. Initiate a migration from et-virt07 to et-virt08:

[et-virt07 xentest]# time xm migrate --live xentesttravelvm01 et-virt08
real    0m11.490s
user    0m0.084s
sys     0m0.028s

Verify the virtual machine has been shut down:

[et-virt07 xentest]# xm li
Name                      ID Mem(MiB) VCPUs State   Time(s)
Domain-0                  0  975      8     r-----  221.7

Before initiating the migration start the simple script in the guest and note the change in time when migrating the guest:

# ./doit
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:57:53 EST 2007
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
2983664   2043120    786536  73% /
-rw-r--r-- 1 root root 62 Jan 12 06:57 /var/tmp/2418.log
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:57:53 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:57:56 EST 2007
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
2983664   2043120    786536  73% /
-rw-r--r-- 1 root root 124 Jan 12 06:57 /var/tmp/2418.log
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:57:53 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:57:56 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:58:00 EST 2007
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
2983664   2043120    786536  73% /
-rw-r--r-- 1 root root 186 Jan 12 06:57 /var/tmp/2418.log
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:57:53 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:57:56 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:58:00 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:30:00 EST 2007
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
2983664   2043120    786536  73% /
-rw-r--r-- 1 root root 248 Jan 12 02:30 /var/tmp/2418.log
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:57:53 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:57:56 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:58:00 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:30:00 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:30:03 EST 2007
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
2983664   2043120    786536  73% /
-rw-r--r-- 1 root root 310 Jan 12 02:30 /var/tmp/2418.log
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:57:53 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:57:56 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 06:58:00 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:30:00 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:30:03 EST 2007
dhcp78-218.lab.boston.redhat.com
Fri Jan 12 02:30:06 EST 2007
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
2983664   2043120    786536  73% /
-rw-r--r-- 1 root root 372 Jan 12 02:30 /var/tmp/2418.log

After the migration command completes on et-virt07 verify on et-virt08 that the virtual machine has started:

[et-virt08 ~]# xm li
Name                       ID Mem(MiB) VCPUs State   Time(s)
Domain-0                   0  975      4     r-----  67.3
xentesttravelvm01          2  1023     1     -b----  0.4

and run another cycle:

[et-virt08 ~]# time xm migrate --live xentesttravelvm01 et-virt07
real    0m10.378s
user    0m0.068s
sys     0m0.052s

At this point you have successfully performed an offline and a live migration test.