Wednesday, June 28, 2006

ZFS Root on Solaris Part 2

The first post was just a quick post to show people the commands to use to create a ZFS root filesystem, with a small UFS grub boot partition. This post is an update using a fresh solaris install with comments of what is happening. Most of the steps follow Tabriz's blog on

First step is to do a clean Solaris install which I will leave out all the details except my setup used the following partitioning -

c0d0s0 /grub 150MB Location of the UFS grub boot code
c0d0s1 swap 1GB Standard Swap/Dump partition
c0d0s3 / 4GB Install/Upgrade UFS partition.
c0d0s7 ZFS all the rest later configured as "intdisk" pool, which contains
ZFS root & clones plus home directories etc

Once you have installed all of your standard software and configuration, now the fun begins.

Create a zfs pool, and turn off mounting filesystems by default

zpool create -f -m none intdisk c0d0s7

Create a zfs partition for the root filesystem. Being on a laptop, I am saving space by turning on compression. It may also give a performance boost at the same time. Note that I am using a symbolic name for the partition name. Later I am going to clone the filesystem and create a test environment.

zfs create intdisk/snv42_root
zfs set mountpoint=legacy intdisk/snv42_root
zfs set compression=on intdisk/snv42_root

Create a mountpoint for the zfs root and use ufsdump/ufsrestore to copy all of the UFS root filesystem. You could use cpio or tar, but you also want the data underneath /devices.

mkdir -m 0755 /zfsroot
echo "intdisk/snv42_root - /zfsroot zfs - yes -" >> /etc/vfstab
mount /zfsroot
cd /zfsroot
ufsdump 0f - / | ufsrestore -rf -

Configure /etc/system on zfs root to use the correct zfs filesystem. This configuation actually gets used by the kernel from within the boot archive, and gets copied each time you update the archive. Also make sure the zpool configuration gets added to the boot archive, and use a classic onliner to up the vfstab on the zfs root.

echo "rootfs:zfs" >> /zfsroot/etc/system
echo "zfsroot:intdisk/snv42_root" >> /zfsroot/etc/system
echo "etc/zfs/zpool.cache" >> /zfsroot/boot/solaris/filelist.ramdisk
grep -v 'intdisk/snv42_root' /etc/vfstab | awk '$3 == "/" { printf "intdisk/snv42_root\t-\t/\tzfs\t-\tno\t-\n" } ; $3 != "/" { print $0 }' > /zfsroot/etc/vfstab

Create a modified hack from Tabriz's blog to fix bootadm.

mv /zfsroot/sbin/bootadm /zfsroot/sbin/bootadm.real
cat - > /zfsroot/sbin/bootadm << EOM

/sbin/bootadm.real "\$@"
if [ "\$1" = "update-archive" -a -d /grub/boot/grub ]; then
/usr/bin/cp /platform/i86pc/boot_archive /boot/boot_archive
exit 0

chmod +x /zfsroot/sbin/bootadm

Now we are ready to update the boot archive and configure grub. The "root (hd0,0,a)" should point to the grub partition.

/usr/sbin/bootadm update-archive -R /zfsroot
cp -pr /zfsroot/boot /grub
cp /zfsroot/platform/i86pc/boot_archive /grub/boot/boot_archive
echo "title Solaris ZFS"
echo "root (hd0,0,a)"
echo "kernel /boot/multiboot"
echo "module /boot/boot_archive"
) >> /grub/boot/grub/menu.lst

cd /grub/boot/grub
installgrub stage1 stage2 /dev/rdsk/c0d0s0

Now you can reboot, and if all goes well you should be able to use the new entry in the grub menu to boot into the zfs partition. Ok, if it all looks good, lets try to configure another zfs root partition using a clone. To create copy we simply do a snapshot and a clone of the current zfs root partition. I am naming this partition "snv42_test". Note the changes...

zfs snapshot intdisk/snv42_root@initial
zfs clone intdisk/snv42_root@initial intdisk/snv42_test
zfs set mountpoint=legacy intdisk/snv42_test
zfs set compression=on intdisk/snv42_test
echo "intdisk/snv42_test - /zfsroot zfs - yes -" >> /etc/vfstab
mount /zfsroot

Follow similar steps you used last time. Again note the changes I have done to the boot archive name and filesystem name.

sed -e "s/snv42_root/snv42_test/" /etc/system > /zfsroot/etc/system
grep -v 'intdisk/snv42_test' /etc/vfstab | awk '$3 == "/" { printf "intdisk/snv42_test\t-\t/\tzfs\t-\tno\t-\n" } ; $3 != "/" { print $0 }' > /zfsroot/etc/vfstab

cat - > /zfsroot/sbin/bootadm << EOM

/sbin/bootadm.real "\$@"
if [ "\$1" = "update-archive" -a -d /grub/boot/grub ]; then
/usr/bin/cp /platform/i86pc/boot_archive /boot/boot_archive.test
exit 0

chmod +x /zfsroot/sbin/bootadm

/usr/sbin/bootadm update-archive -R /zfsroot
cp /zfsroot/platform/i86pc/boot_archive /grub/boot/boot_archive.test
echo "#"
echo "title Solaris ZFS test"
echo "root (hd0,0,a)"
echo "kernel /boot/multiboot"
echo "module /boot/boot_archive.test"
) >> /grub/boot/grub/menu.lst

Now you should be able to reboot, and you should now have a cloned test environment. Only changes from the original filesystem are added to the diskspace usage. The number of clones you can have is limited by the number of boot archives you can jam into the /grub partition, and to a lesser extent by the free space you have in the ZFS pool.

Have Fun!


Dick Davies said...

Doug, I'm probably being thick, but how does this differ from Tabriz' howto? Looks like you still waste 4gb on the UFS root partition in addition to the grub partition.

Doug Scott said...

You are correct that I do have a 4GB partition free. The difference is that I can use that space for whatever I want. It is not a duplicate of my zfs root partition.

This 4GB does not have to be on the same disk. I am looking at doing an install onto a external disk, an then transfering onto the internal laptop disk. This will allow me to use almost the whole disk for zfs.

Martin Englund said...

Nice work! I'll use your instruction now that I'm installing build 42a on my lab system:)

You could actually install Solaris on without a swap partition, and then use the ufs / as swap.

Dick Davies said...

Thanks Doug - as I said, I thought I'd missed something.

I'm looking at running a Compactflash root with n-way mirrored sata for a home NAS, this will help lots.

Anonymous said...

Doug, how would I go by converting /usr partition from ufs to zfs? --axisys

Doug Scott said...

axisys: My latest post has the instructions on how to get access to your zfs root filesystem from failsafe. Once you do this, from inside failsafe, "cd /a/usr". This should be the /usr from your zfs partition. You can check this with a "df -h .". I will assume that the partition you have /usr on is c0d0s4. You can copy /usr using the command -

ufsdump 0f - /dev/dsk/c0d0s4 | ufsrestore -rf -

Once this is finished, comment out the entry for /usr in /a/etc/vfstab, and reboot.

Leon Gregory said...

To quote you, "I am looking at doing an install onto a external disk, an then transfering onto the internal laptop disk. This will allow me to use almost the whole disk for zfs."

I am very interested in this possibility as what i would like to do is setup a pair of raid z arrays as my single storage pool. Temporarily attach a drive to install solaris to and then create the zfs root file system from that as per the guides.

However then, i then want to remove that extra drive, leaving just the zfs storage pool and use a usb flash drive to handle getting me into my zfs root file system.

What would that usb flash drive actually need to contain to get me into the zfs root file system?

An article on that would make me very happy as otherwise i am left with partitions on one disk making it a different size from all the others in its raid z array or a drive that doesnt really fit in my server lurking around.

I have a 1GB stick to throw in there so it doesn't have to be super compact. Simple is better here.

mark.horstman said...

Sorry if this is the wrong place to post, but maybe someone can shove in in the right direction.

I recently installed Solaris10 06/06 on a Sun SPARC V240 with a 74GB disk like this:

s0 / 512M
s1 swap 8192M
s2 overlap
s3 /usr 5120M
s4 /var 512M

/, /usr and /var are ufs. What I'd really like to do is put /usr and /var in a zfs pool. I've tried tried creating pool00 with usr and var filesystems, cpio'ing over everything, setting their mountpoints to legacy, updating my /etc/vfstab to:

pool00/usr - /usr zfs - yes -
pool00/var - /var zfs - yes -

but then the machine fails to boot (can't cut-n-paste the errors from the serial console).

Anyone know if this can be done? I've googled but can't find anything to help.

dick davies said...

martin: I just tried that, and it looks like the install hangs without a swap partition somewhere.

Admittedly I only have 512Mb of ram in this box, but thought I'd point it out.

Dick said...

If anyone had trouble with creating the boot-archive, try a recent SXCR image.

b44 failed at that point for me, due (I think to ). b46 is fine.

Excellent work!