ZFS Root on Solaris Part 3

Hopefully, you have had success and you now have root on ZFS. In this next post I will describe how to get access to the root partition from failsafe. If you tried like me, you will find that you cannot see the zfs filesystems from failsafe. This is because the zpool.cache configuration file in /etc/zfs has not been copied onto the failsafe miniroot. One thing that is not wise at this point is to use the command "zfs import -f [pool_name]", to import the pool. While it may work, it will probably make your pool inaccessible when you next boot. So what you need to do is to copy the configuration file from your running system into the failsafe miniroot archive. Let' get into it. The steps are -

1) Boot ZFS root partition and get access to root

2) gunzip your miniroot image into /tmp

    gzcat /grub/boot/x86.miniroot-safe > /tmp/miniroot.img

3) Using the loopback filesystem to mount the image on /mnt - replace the /dev/lofi/1 with appropriate returned device

    lofiadm -a /tmp/miniroot.img
    mount /dev/lofi/1 /mnt

4) Copy the file /etc/zfs/zpool.cache into /mnt/etc/zfs

    cp -p /etc/zfs/zpool.cache /mnt/etc/zfs

5) unmount the filesystem

    umount /mnt
    lofiadm -d /tmp/miniroot.img

6) gzip the image back to grub.

    gzip -c /tmp/miniroot.img > /grub/boot/x86.miniroot-safe

7) Edit the /grub/boot/grub/menu.lst and make sure that there is an identical "root" entry for failsafe and ZFS. Below are the entries for my machine.

    #---------- ADDED BY BOOTADM - DO NOT EDIT ----------
    title Solaris failsafe
    root (hd0,0,a)
    kernel /boot/multiboot kernel/unix -s
    module /boot/x86.miniroot-safe
    #---------------------END BOOTADM--------------------
    title Solaris ZFS
    root (hd0,0,a)
    kernel /boot/multiboot
    module /boot/boot_archive

8) You should now be able to reboot into failsafe and see your zfs pool with the "zpool list" command. Also you should be able to see your root filesystem with the "zfs list" command. When you booted, it would have asked you if you want to mount a filesystem on /a. If you answered yes, then unmount it with "umount /a". Now you can use the mount command to mount your ZFS root filesystem on to /a. Note, I previously named my root filesystem "intdisk/snv42_root". Change the command to suite your setup. Also don't forget to add the option "-F zfs". If you dont, mount will try NFS....

    mount -F zfs intdisk/snv42_root /a

Finally you can now cd into /a and fix your system. Enjoy!!!

ZFS Root on Solaris Part 2

The first post was just a quick post to show people the commands to use to create a ZFS root filesystem, with a small UFS grub boot partition. This post is an update using a fresh solaris install with comments of what is happening. Most of the steps follow Tabriz's blog on blogs.sun.com.


First step is to do a clean Solaris install which I will leave out all the details except my setup used the following partitioning -

    c0d0s0   /grub    150MB         Location of the UFS grub boot code
    c0d0s1   swap     1GB           Standard Swap/Dump partition
    c0d0s3   /        4GB           Install/Upgrade UFS partition.
    c0d0s7   ZFS      all the rest  later configured as "intdisk" pool, which contains
                                    ZFS root & clones plus home directories etc

Once you have installed all of your standard software and configuration, now the fun begins.

Create a zfs pool, and turn off mounting filesystems by default

    zpool create -f -m none intdisk c0d0s7

Create a zfs partition for the root filesystem. Being on a laptop, I am saving space by turning on compression. It may also give a performance boost at the same time. Note that I am using a symbolic name for the partition name. Later I am going to clone the filesystem and create a test environment.

    zfs create intdisk/snv42_root
    zfs set mountpoint=legacy intdisk/snv42_root
    zfs set compression=on intdisk/snv42_root

Create a mountpoint for the zfs root and use ufsdump/ufsrestore to copy all of the UFS root filesystem. You could use cpio or tar, but you also want the data underneath /devices.

    mkdir -m 0755 /zfsroot
    echo "intdisk/snv42_root - /zfsroot zfs - yes -" >> /etc/vfstab
    mount /zfsroot
    cd /zfsroot
    ufsdump 0f - / | ufsrestore -rf -

Configure /etc/system on zfs root to use the correct zfs filesystem. This configuation actually gets used by the kernel from within the boot archive, and gets copied each time you update the archive. Also make sure the zpool configuration gets added to the boot archive, and use a classic onliner to up the vfstab on the zfs root.

    echo "rootfs:zfs" >> /zfsroot/etc/system
    echo "zfsroot:intdisk/snv42_root" >> /zfsroot/etc/system
    echo "etc/zfs/zpool.cache" >> /zfsroot/boot/solaris/filelist.ramdisk
    grep -v 'intdisk/snv42_root' /etc/vfstab | awk '$3 == "/" { printf "intdisk/snv42_root\t-\t/\tzfs\t-\tno\t-\n" } ; $3 != "/" { print $0 }' > /zfsroot/etc/vfstab

Create a modified hack from Tabriz's blog to fix bootadm.

    mv /zfsroot/sbin/bootadm /zfsroot/sbin/bootadm.real
    cat - > /zfsroot/sbin/bootadm << EOM
    #!/usr/bin/sh

    /sbin/bootadm.real "\$@"
    if [ "\$1" = "update-archive" -a -d /grub/boot/grub ]; then
        /usr/bin/cp /platform/i86pc/boot_archive /boot/boot_archive
    fi
    exit 0
    EOM

    chmod +x /zfsroot/sbin/bootadm

Now we are ready to update the boot archive and configure grub. The "root (hd0,0,a)" should point to the grub partition.

    /usr/sbin/bootadm update-archive -R /zfsroot
    cp -pr /zfsroot/boot /grub
    cp /zfsroot/platform/i86pc/boot_archive /grub/boot/boot_archive
    (
        echo "title Solaris ZFS"
        echo "root (hd0,0,a)"
        echo "kernel /boot/multiboot"
        echo "module /boot/boot_archive"
    ) >> /grub/boot/grub/menu.lst

    cd /grub/boot/grub
    installgrub stage1 stage2 /dev/rdsk/c0d0s0

Now you can reboot, and if all goes well you should be able to use the new entry in the grub menu to boot into the zfs partition. Ok, if it all looks good, lets try to configure another zfs root partition using a clone. To create copy we simply do a snapshot and a clone of the current zfs root partition. I am naming this partition "snv42_test". Note the changes...

    zfs snapshot intdisk/snv42_root@initial
    zfs clone intdisk/snv42_root@initial intdisk/snv42_test
    zfs set mountpoint=legacy intdisk/snv42_test
    zfs set compression=on intdisk/snv42_test
    echo "intdisk/snv42_test - /zfsroot zfs - yes -" >> /etc/vfstab
    mount /zfsroot

Follow similar steps you used last time. Again note the changes I have done to the boot archive name and filesystem name.

    sed -e "s/snv42_root/snv42_test/" /etc/system > /zfsroot/etc/system
    grep -v 'intdisk/snv42_test' /etc/vfstab | awk '$3 == "/" { printf "intdisk/snv42_test\t-\t/\tzfs\t-\tno\t-\n" } ; $3 != "/" { print $0 }' > /zfsroot/etc/vfstab
    
    cat - > /zfsroot/sbin/bootadm << EOM
    #!/usr/bin/sh

    /sbin/bootadm.real "\$@"
    if [ "\$1" = "update-archive" -a -d /grub/boot/grub ]; then
        /usr/bin/cp /platform/i86pc/boot_archive /boot/boot_archive.test
    fi
    exit 0
    EOM

    chmod +x /zfsroot/sbin/bootadm
    
    /usr/sbin/bootadm update-archive -R /zfsroot
    cp /zfsroot/platform/i86pc/boot_archive /grub/boot/boot_archive.test
    (
        echo "#"
        echo "title Solaris ZFS test"
        echo "root (hd0,0,a)"
        echo "kernel /boot/multiboot"
        echo "module /boot/boot_archive.test"
    ) >> /grub/boot/grub/menu.lst

Now you should be able to reboot, and you should now have a cloned test environment. Only changes from the original filesystem are added to the diskspace usage. The number of clones you can have is limited by the number of boot archives you can jam into the /grub partition, and to a lesser extent by the free space you have in the ZFS pool.

Have Fun!

ZFS root on Solaris

Below is the log of commands I used to have a ZFS root partition with a small ufs for grub. Before I was swapping between 2 root partitions for upgrade, with everything I wanted to keep between upgrades on a ZFS partition. This time I divided one of the root partitions into a small UFS partition, and the rest I used as a ZFS root.

The next step is a total backup (done), and repartition the disk into a small UFS boot partition for grub boots, a ~4GB ufs install partition, and the rest including root will be all on one ufs partition. On a laptop, the extra diskspace is very handy. I will also spend some time on writing better notes :-)

newfs /dev/dsk/c0d0s0
if [ ! -d /altroot ]; then
        mkdir -m 0755 /altroot
fi
echo "/dev/dsk/c0d0s0 /dev/rdsk/c0d0s0 /altroot ufs 3 yes -" >> /etc/vfstab
mount /altroot
zpool create -m none rootdisk c0d0s4
zfs create rootdisk/root
zfs set mountpoint=legacy rootdisk/root
zfs set compression=on rootdisk/root

/etc/vfstab
rootdisk/root   -       /zfsroot        zfs     -       yes     -

cd /zfsroot
ufsdump 0f - / | ufsrestore -rf -

echo "rootfs:zfs" >> /zfsroot/etc/system
echo "zfsroot:rootdisk/root" >> /zfsroot/etc/system
echo "etc/zfs/zpool.cache" >> /zfsroot/boot/solaris/filelist.ramdisk
grep -v 'rootdisk/root' /etc/vfstab | awk '$3 == "/" { printf "rootdisk/root\t-\t/\tzfs\t-\tno\t-\n" } ; $3 != "/" { print $0 }' > /zfsroot/etc/vfstab
mv /zfsroot/sbin/bootadm /zfsroot/sbin/bootadm.real

cat - > /zfsroot/sbin/bootadm << EOM
#!/usr/bin/sh

/sbin/bootadm.real "\$@"
/usr/bin/cp /platform/i86pc/boot_archive /boot/boot_archive
exit 0
EOM

chmod +x /zfsroot/sbin/bootadm
/usr/sbin/bootadm update-archive -R /zfsroot
cp -pr /zfsroot/boot /altroot
cp /zfsroot/platform/i86pc/boot_archive /altroot/boot/boot_archive

(
        echo "title Solaris ZFS"
        echo "kernel /boot/multiboot"
        echo "module /boot/boot_archive"
) >> /altroot/boot/grub/menu.lst

cd /altroot/boot/grub
installgrub stage1 stage2 /dev/rdsk/c0d0s0

Happy Bithday OpenSolaris

Today, as some are aware is the first birthday of the OpenSolaris Community, therefore it is time to reflect on the substantial progress that has made and the future. To fill in the picture properly. I will digress a little to go back in time to the pre-Internet days when dinosaurs and BSD roamed the earth. Being a young system admin at a University the access to Solaris source code was obtainable, and easy to modify for our environment. Therefore to me Solaris was always open source. On many occasions, a quick flick through the files, you could quickly solve problems you were having. Insulated from the outside world, and connected to other University's all over the world (pre and post Internet) and having a regular update of source delivered on tapes, open source was always easily obtainable, and very useful.

Somewhere down the track Sun decided to jump camps from BSD to System V Unix. For what ever reasons they had, to me this is when Sun changed focus from a company supporting research by building cheap high performance workstations, to focus on the server business market. While, this change at the time prepared them for the first Internet age, it was a shift away from the largely open source BSD, to the very much proprietary System V.

Many years down the track, with the help of Wall Street growing the Internet much faster than it should, the Internet bubble went Bang! To add to Sun's problems of the times, Intel PC chips at the low end were becoming performance competitive with Sparc, and the IBM's Power PC was impressive at the top end. Like vultures surrounding the dying Sun corpse, Sun competitors, tried to deliver the knockout blow, by helping make the Open Source Linux kernel competitive with Sun's crown jewel Solaris. This is very ironic, as to this day these supporters of the Linux kernel are right up there next to Microsoft being the most closed source companies in the world. Sun could not win a trick. Behind the scenes, they were preparing to Open Source Solaris. They had brought software companies such as Staroffice, and turned them into Open Source, many parts of their software stack were being Open Source'd, but somehow the wider Open Source community label them as somehow “closed”.

Ok, one year ago to this day, Sun finally created the OpenSolaris community, and released large portions of Solaris as Open Source under a Mozilla type license. What does this do for Sun?

• Sun is now the largest contributer of Open Source software (and hardware), on the planet bar none (something some slash dotter-er's may never accept). It is now difficult (or biased) to label Sun as a “closed” company.


• Solaris is now certified to run on 100's of platforms. This is true remarkable as it is an order of magnitude greater, than supported Linux platforms. Now with Solaris on non Sun platforms, not being treated like a second class citizen, and with Sun having an extremely good x86 product line. It now gives Sun the potential to reach out to new customers, that they could have on dreamed about having a couple of years ago.


• Through the Open Solaris community, Sun is showing just how transparent and open the company really is. Many of Sun critics over the last year since the launch of Open Solaris and now Open Sparc, are either changing their opinions of Sun openness, focusing on other areas such as Java Open Sourcing, or are plainly showing that they have vested interests which are always will be opposed to Sun.


• Just looking through the Solaris source code shows that it is extremely well written and structured, which shows the reason why Solaris has the reputation for being reliable and secure. It also show, that their developers are extremely talented and well organized. This demonstrates to customers, that Sun are not only able to deliver a good product, but it also has the ability to support it.

Now, more importantly, what has this done for the wider community?

• It is now much easier for individuals and companies to write device drivers for their product. This was always one area (along with Sun dropping it for awhile) for Solaris on non Sparc systems, that always had major problems.


• Other Open Source community's such as BSD, can now implement advanced Solaris features such as dtrace, zfs etc into their codebase. Ironically since the Linux kernel license is more restrictive than that of BSD, they have less freedom to do this in Linux.


• From personal experience in the OpenSolaris over the year, I can testify that Sun is by a long way the most transparent company I have ever seen. While they are tight lipped over the release schedule and configuration for some of their hardware (which is understandable), you can quite easily go into their software forums, and not only see their internal debates, but you can also put your 2 cents worth in and it is taken quite seriously.


• I read the other day, that Dell is now very please with its ability to field 90% support calls for Linux without having to run off to Red Hat (which they then may pass onto a public mailing list). After laughing for 5 minutes I picked myself up off the floor and thought about it. For many years I would generally spend a fair amount of time looking into the problem (to make sure it wasn't something I did), before I would lodge a support call to Sun or others. Generally the amount of time and effort to go through the support channels for what was most likely a trivial/stupid problem or a known bug, was greater than solving the problem myself. Now days, many simple annoying problems are just a matter of just writing to the relevant Sun's forums. Not only do you get a quick response, you will generally get a response back one of the developers (or fellow external community member), who is only too pleased to help. If you are a customer, this is great for the 90% Dell type questions, for which you need a simple quick answer. For Sun it not only lets their developers know exactly the problems the customers are having, but it helps free up the support network to focus on more difficult support issues.


• When not at work, for many years at home I have survived with my primary desktop being a Solaris machine such as a Sun Blade 100 with a Sun PCI card for windows apps, and a linux development box somewhere on the floor. Now days, I survive of 1 laptop with 2 100GB discs (physically swapped). On one disc I have one partition with Windows which came as a Microsoft tax, and Solaris running in vmware. The other partition is Linux, which currently is the latest Ubuntu. Most people for the day to day work, as I did in the past would use this disk. Today, it is just spends its life sitting in a draw, and I now prefer to use the other disc which is 100% OpenSolaris. Many of the reasons I would use the other disk are now gone. The effort in swapping disks is greater than the value of change. From a pure user experience, there is little difference between running on JDS/Gnone Solaris than Gnome Linux. Of course two standout point here is the lack of 3D graphics support for ATI cards is non existent, so most good games (other than freeciv) are out. Also, the ability to pull packages down through apt-get is surely missed. Running one of the other OpenSolaris distro's would close this gap. The benefits for a developer on Open Solaris with the combination of features such as ZFS, dtrace, zones, Sun Studio, Netbeans, etc are very significant.

Now we have made it to the first birthday of the Open Solaris community, what are some of the more interesting things planned in Open Solaris for the future? and what I would like to see. I will do my top ten.

1) Xen – Solaris on Dom0, would be very nice. Having relatively limited resources on my laptop, products such as VMware are still a bit heavy to emulate a datacenter. The combination of Xen, and Zones, go a long way to solving this problem.

2) Trusted Solaris Extentions – Anybody who works with me knows that I do put great importance on security, and control on who can do what on a system. While Solaris 10 has many of the features migrated from Trusted Solaris, it lacks the whole security from day one approach that Trusted Solaris has. Today Solaris 10 has many things turned on by default to make life easy for a Sysadmin. Trusted Solaris takes the hardline approach where the default profile, makes a Sysadmin think about what should be turned on, who should have access to what, and eithen who should talk to who.

3) ZFS boot – While I have a great fondness of ufs, especially its ability in Solaris 10 to mirror disks from jumpstart, zfs is just so much more efficient. The only fix barrier is the number and size of disks I have attached. Live upgrade into a snapshot would be a major win, and very cool.

4) Sparks etc projects – Coming from a background of a large University environment, LDAP directories, authentication, authorization, provisioning a user, can be difficult. While Sun Identity and Directory products are technically superior to other products on the market, they do no easily transform to the desktop. The Active Directory and Windows desktop combination when setup by a good sysadmin, is superior to anything Solaris and Linux has. The level, of control direct control of the desktop from the directory is impressive, when implemented properly. An advancement is this area is greatly needed for the Unix/Linux world.

5) JDS – Vermillion is coming on very nicely, and is catching quickly up to the commercial Linux Gnome implementations. The sooner CDE is shown the door the better.

6) Device Drivers – Great inroads have been made into the one big advantage that Windows and Linux have had over Solaris x86 (Sparc always had good support for its hardware). While you would class it as a small subset of drivers, it mostly covers all of the important ones. Solaris actually has an advantage in the future here, as it has much less baggage of old drivers). On my laptop, the only thing I am really missing that I want is a good 3D driver for the graphics card. The others just don't affect me working day to day, and I rarely ever used them under the other OS's. For mobile phone junkies bluetooth would be nice.

7) Porting Solaris to other devices – I would feel safer if my mobile phone (or my fridge) was running Solaris rather than something from Microsoft, or eithen Linux. While it is not a big thing, as long as the phone (or fridge) does its function, I am generally pleased....

8) GNU – While GNU/Solaris pre dates GNU/Linux, the creation of such a beast with the latest versions has been up to the owner. Having the GNU binaries and libraries in known configuration, makes it easier port other applications across. Also having up to date versions would negate the need to uninstall the delivered version or compile the newer version for another directory and have dynamic library madness.

9) Patching and Packaging – This is can be a painful area for sysadmins. Adding zones while in theory makes life easier, currently in practice, it can be troublesome. Now that the package tool source code has been released, it may spur the community to come up with some alternatives and tools.

10) Games – How could I leave this to last :-) Solaris has been lacking in this area for a long time. This is due to the fact that before, buying a Sparc machine with a 3D graphics card, would cost you a lot of money, as you were really buying something designed as a workstation. Only that Nvidia and ATI, now offer the closed source 3D driver that Linux is a viable platform for games. Having a ATI graphics card in my laptop, I cannot eithen run Java games which need 3D drivers. For the moment it is freeciv (the latest beta is really nice). I hope ATI will release a driver before the next birthday.

To summarize, all in all to extend an Australian term “Its been a bloody good year for OpenSolaris”, lets make 'every' next year better!!!

P.S. A special thanks to Scott, Jonathon, and the team for the leadership to create OpenSolaris.