·:[ Jason On A Shtick ]:·
• • • • •

Recently, I have spent time with Xen on Fedora Core 5 and 6 setting up routine services like mail and proxies within their own virtual machines. Following some other Xen blogs, I got into the habit of creating a “base” Fedora image and then simply customizing copies of that image for each new VM. These are notes that I have compiled on managing that process.

First, some useful links:

• Xen
• Wikipedia’s virtual appliances site list
• A XenSource wiki post that gave me some ideas
• jailtime.org’s small prefab images
• FC6 Xen quickstart guide

You can create a new image using the FC6 DVD as the repository if you don’t have a local one, you just need FTP or HTTP access to the disc. The simplest hack is to put a symbolic link on the Domain-0 host in /var/www/html/ that refers to the disc’s mount point with ln -s /media/fedora_dvd /var/www/html/disk. Now, you can specify http://dom0_ip_addr/disk as the install location, assuming you have httpd running there.

Starting in FC6, there is a virt-install script that points to the old xenguest-install one – a sign of preparation for easily slipping in other VM solutions by using generic script names.

I go for about a 3GB image file. That gives me plenty of room for the base Fedora stuff plus some room to install things that don’t grow over time. If you need more space for some type of disk consuming application, you could resize your image file depending on how you laid out partitions, logical volumes, etc. I have also had luck in creating separate “data” image files to be mounted on /opt or some other useful location. I find these easier to deal with resizing because you will likely have only one partition there, not multiple ones plus a partition table to screw up. Steps to create a “data” image file and resize it later:

• Create a 1GB image file with dd if=/dev/zero of=data.img bs=1M count=1 seek=1024
• Format the image file as ext3 with mke2fs -F -j data.img

At this point, you can mount the data image file as any other and use it. Put an entry for the mount in /etc/fstab in the guest VM that will use it. You will also need to customize the Xen config file for the guest image to make the data image available as a particular device. To resize the data image, unmount it (shutdown guest that is using it) and then:

• Remove the journal (making it ext2) with tune2fs -O ^has_journal data.img
• Check things out with e2fsck -f data.img
• Resize the image file to 2.5GB with dd if=/dev/zero of=data.img bs=1M conv=notrunc count=1 seek=2500
• Grow the filesystem to match with resize2fs data.img
• Put a new journal back with tune2fs -j data.img

And you are done. You can also mount the data image file with the loop device to access it, but don’t do it while it is being accessed by a running guest:

• Mount using the loop device with mount -o loop data.img /mnt
• losetup will show you which loop device is used with losetup -a
• Access what you need at /mnt
• Unmount the image when done with umount /mnt

During the Fedora install, I choose a static IP address with my known DNS servers and network configuration. These choices simply save me some time later when I customize a copy of the image. When it comes time to select packages, I customize by deselecting everything except for vim-advanced under the “Editors” category.

Once the installer finishes, your new guest domain may or may not start up depending on how things are set up. I mount the guest image and make one change before starting it up to make some further changes.

With the exception of the first one, the following changes are made inside of the running guest image:

1. Before starting the guest, mount the image file and disable SELinux. Edit /etc/selinux/config and change SELINUX=enforcing to SELINUX=disabled, then unmount the image. Now you can start the guest without SELinux running at all. Alternatively, you could start the guest, make this change, then reboot the guest to get SELinux turned off before doing anything else.
2. Stop the yum-updatesd service with service yum-updatesd stop so you can run yum on your own.
3. Issue a yum update to get everything up to snuff. Been a while since you burned that FC6 DVD, huh? I am behind a larger firewall setup that prevents just any IP address from having outside access, so I have squid running on dom0, which does have access. In the guest, I edit /etc/yum.conf to include proxy=http://dom0_ip_addr:3128 so that yum uses the proxy. Note that you may need to deal with squid caching of packages, etc if you do this often.
4. Out of habit, I look for any rpm conflicts by running find / \( -name “*.rpmsave” -o -name “*.rpmnew” \) and resolving anything that needs it.
5. Install ntp. Xen guest domain clocks tend to stray a bit, so this takes care of any time problems. Run yum install ntp.
6. Edit /etc/init.d/ntpd and add echo 1 > /proc/sys/xen/independent_wallclock in the start function after the call to readconf. This marker must be reset at each boot to indicate that the guest will keep its own time.
7. Run ntsysv and enable/disable the services you want to run in this base guest. I turn off everything including hardware stuff, APM and RAID monitoring (do those matter in a guest?) except for anacron, atd, autofs, crond, gpm, iptables (and ip6tables if you have enabled IPv6 support), messagebus, network, ntpd, sendmail, sshd and syslog.
8. Edit /etc/sysconfig/iptables (and ip6tables as above) and make any firewall settings that you want. I generally remove the default Fedora rules that accept multicast and other traffic, leaving only the entries for lo, icmp, state in established/related and the final reject with icmp-host-prohibited. I filter ssh on port 22 a little further by only allowing access from certain subnets on the network with -A RH-Firewall-1-INPUT -s xx.xx.xx.0/255.255.255.0 -m state –state NEW -m tcp -p tcp –dport 22 -j ACCEPT.
9. Reboot the guest image. I have gotten into the habit of logging out and having Xen do it.

At this point, you have a stock Fedora image that can be customized however you like. When you are ready to create a copy, follow these steps.

1. Shutdown the base image if it is running.
2. Copy the image to a new file with cp basefc6.img customfc6.img.
3. Copy the Xen config file for the image in /etc/xen like cp basefc6 customfc6.
4. Edit the customfc6 config file and change the name of the image, the amount of memory you want, the disk path to the image file and add one to the fake MAC address to make it unique. I usually remove the uuid and let Xen generate one for each guest on the fly. If you are going to use a data image file, make sure it is made available in the disk entry.

Now, your custom image copy is bootable but you may want to go ahead and tweak the network settings before you start it. You can mount the filesystem within the image first to make changes. The mount -o loop used above may not get it done if you have logical volumes defined in the image. Without logical volumes, lomount -t ext3 -diskimage customfc6.img -partition 1 /mnt will mount partition 1. To see a list of partitions in the image, leave off the -partition 1 part. If you have volume groups, consult the FC6 Xen quickstart guide for how to use kpartx.

The following instructions assume you are working in the mounted filesystem of the image file, so prefix the paths with your mount point (/mnt in the example above).

1. Edit /etc/hosts and enter what you like. I personally don’t like for the loopback range to also have the actual host name and shortname. I separate them like:
   127.0.0.1 localhost.localdomain localhost
   xxx.xxx.xxx.xxx something.somewhere.org something
2. Edit /etc/sysconfig/network and change HOSTNAME to a full name like something.somewhere.org.
3. Edit /etc/sysconfig/network-scripts/ifcfg-eth0 and update as much as you can stand. Make sure the HWADDR entry matches the MAC address in the Xen config file for this image. Here is an example:
   DEVICE=eth0
   BROADCAST=172.31.255.255
   HWADDR=00:16:3E:26:6B:DD
   IPADDR=172.16.1.100
   IPV6ADDR=
   IPV6PREFIX=
   NETMASK=255.240.0.0
   NETWORK=172.16.0.0
   ONBOOT=yes
   
4. Edit /etc/resolv.conf and get your DNS on, such as:
   search somewhere.org
   nameserver xxx.xxx.xxx.xxx
   nameserver xxx.xxx.xxx.xxx
5. Edit /etc/sysconfig/iptables and make any firewall settings that you want for this custom VM.

Unmount the image filesystem with umount /mnt. Your custom image is now ready to boot! Depending on how long ago you updated your base image, you may want to yum update this new custom one.