Whats happening

First the bad news - After the OpenSolaris.org developer summit I have not had much time to devote to OpenSolaris :(

The good news is the reason why this has happened is that I have been getting ready to start work in Scotland as a contractor for Fidelity National Information Services. Due to NDA's I can not reveal much about the work, but if you look at their website you can see that it involves software for banks. At the moment I am just waiting for some paperwork needed for my visa. Since I am coming from Bangkok I had to do a Tuberculosis test. This involved an X-Ray from a hospital in Bangkok, which the process was relatively simple considering a language barrier. English is not as widely spoken in Thailand as people may think, and my Thai sux. BTW the TB results was negative....

Over the next year at least, I will do a lot of flying between Glasgow and Bangkok. This will be interesting. Not only a big change in timezone, but a huge change in climate. The temperature in Bangkok is around 32C (max) and 24C (min), while in Glasgow it is warm 12C (max) and 7C (min) and getting colder.

Being the new kid on the block, I drew the short straw and will be working over Christmas and New Years. My wife is a little upset about this as she wont see me for about 8 weeks. On the plus side is it will be yet another country I will be able to experience Christmas and New Year. Last year I was in Phayao, Thailand. The year before that I was in Kobe and Tokyo, Japan.

As the software is developed on Windows, I have had to change my laptop back to Windows, with Solaris Express using vmware. It reminds me just how much Windows is a horrible development platform. It really does sux. Luckily it is not what the customer chose to use. I can only say that they have made an excellent choice of both operating system and hardware platform :)

Another plus is that hopefully I will have time to visit the OpenSolaris user groups within the region. I will try to at least to go to a meeting of the world famous Ireland OpenSolaris User Group. This should be some good craic.

The next post I will try to find a pub with wi-fi, open fire and fine ale.

Pirate Busters

A Bangkok Post story today "Fighting the pirates" says there was a US-led anti-piracy conference in Bangkok this week. The was 60 police, customs officials and prosecutors from 13 countries attending. I think the key words from this is "US-led", as the US are very keen to pressure countries such as Thailand to clamp down on counterfeit products.

In the article, A US Cyber spook said "counterfeit products not only hurt the economy, but are also a serious threat to the health and safety of everyone". For Thailand, I can see the safety aspect from what he is saying in relation to fake aircraft parts and pharmaceuticals. As far as the economy side goes I think a broad crackdown on every thing pirated would only be beneficial to the US. I do not think Thailand would be to keen to crack down on it's clothing industry as this would seriously harm the local economy and put many people out of work.

The current strategy against piracy using IP law enforcement, I feel is currently flawed. It is aimed at keeping retail prices high to possibly maximize profits. The discrepancy here is the difference in price between the real product and the pirated copy. The difference only encourages crime as the customers are looking for value for money. Parts of the software industry has been forced through Open Source to change to charging for services rather the software itself. Companies like Sun actually give their software away. At first this may sound like a dumb idea, how does Sun make money by making something free? There is a method to the madness. Charging $0 for Solaris encourages many more people to deploy it. This gives Sun a great opportunity to sell more hardware and services based around Solaris. Will this work for other industries? In short it depends on product. I feel that companies need to look for new approaches to getting profit from their product to make piracy less attractive. The law enforcement approach is not working. It just changes your customers into criminals.

Post Note: My favorite gem from the article - "the mass production and distribution of counterfeit goods such as optical discs". I think the counterfeit goods here is actually the content on the disc rather then the actual disc itself :)

Pirate cleanup

The Bangkok Post had an interesting report about the anti-piracy group the Business Software Alliance increasing it's presence in Thailand. The first thing I did was to find out who are the BSA. It's members consist of the usual anti-open source (or ones who pay lip service) IT corporations like Adobe, Microsoft, IBM etc. Hardly surprising is that companies which are largely focused producing open source like Sun, Red Hat are not members.

Ok, I go to the bio's of the BSA management team and find that nobody seems to ever actually written any software! Is the focus of the BSA to promote software at all, or is it just a debt collection agency focused on software.

Digging through the article they start off with the obvious and say that software piracy is rampant in Thailand. No kid Ted! The Bangkok Post does it's usual bad journalism - "Various studies have shown a direct correlation between a low piracy rate and the health of a country's IT industry". Mr Post, please, please provide a reference of where the study comes from. Without it my bullshit meter goes ballistic.....

Here we go again - "Hardee said that if Thailand were to bring down the piracy rate by 10 percentage points over the next four years, this would translate to over 5,000 highly paid IT jobs, 1.9 billion US dollars in GDP growth and over 75 million dollars in extra tax for the government". Right, are we plucking the numbers out of the air, or is there some sort of logic behind it. My assumption here is the Mr Hardee thinks that people in Thailand can actually afford to pay for the software in the first place. It would be interesting to see how much money from the software sales stays in Thailand, and how much goes overseas.

Mr Post finally provided a reference - "According to a survey by the Economist Intelligence Unit on IT competitiveness, Thailand today languishes in 41st place out of 64 countries. Perhaps not too surprisingly, the study showed that we fared worst in terms of our legal environment, lack of R&D and an inadequate IT infrastructure. The current "lack of R&D" would suggest the likelihood of the "5000 highly paid IT jobs" coming from Thailand would be remote. My hunch is that they only increase in jobs in Thailand would be to translate existing or new products into Thai. Exciting stuff eh!

"The BSA thinks we have an important role to play in helping the government and industry develop a strong IT sector". I maybe a bit harsh here, but the BSA is really only interested in getting people to pay their members for the use of the software they produce. This statement is just sugar coating. Lets call a spade a spade....

Don't get me wrong here. I am all for people paying companies like Microsoft for the use of "their" software. If this is their business model then so be it. We just need to produce competitive or better Open Source software to put this old business model out of business. In my opinion the software industry and IT literacy in countries like Thailand depend on it.

"In the first nine months of this year, the BSA has had 250 leads for corporate piracy submitted to its hotline and five of the informants have been paid a portion of the reward money of 250,000 baht". Either the exchange rate is really fluctuating or Thai's are being short changed. The reward money on the US site is $1,000,000 or ~30,000,000 baht. I wonder how much money I would get for giving them address of Pantip Plaza over the hotline.

Hey something we can agree on - "However, it was the choice of a developer to engage in software development using Open Source tools and to give it away for free, he said. Open Source is based on the principle that developers have that choice to protect their intellectual property and keep that code free.".
My question is does the BSA actively promote the use of Open Source software. I doubt it....

pkgbuild tips - 1

I have been thinking for some time that I should publish some tips for building packages using pkgbuild. If you have created more then one spec file using pkgbuild, then you will know that many things are repeatable. The general process goes like time -

1. Create an initial spec file from either a template or a spec file for a similar application.
2. Try a build.
   
3. You should expect that the above should fail. Check the log file (/tmp/.log) for where it failed. Fix the problem. Generally a typo or the build needs a patch
4. Repeat 2-3 until the prep and build works. If it actually installs the packages, then go to 7
   
5. Look in the RPM_BUILD_ROOT directory tree in /var/tmp for what was installed, and adjust the %files section of the spec file to match the installation.
6. Go back and repeat from 2 (Note: you can short-cut this with pkgbuild)
7. Publish the spec file(s), patches, ChangeLog etc and your done.

Todays tip involves adding a patch. I find that using 2 or 3 windows makes this really easy. The 1st window is where you do the pkgtool/pkgbuild commands and edit your spec files. You use the 2nd window to dig through the build area to find and fix any bugs or problems after a fail build. Have the 3rd window setup so you are in the /tmp directory. Once you have a change to make that requires a patch, just do somthing similar in following in the 3rd window.

jdsbuild@prae> cd /tmp
jdsbuild@prae> gtar fxj ~/packages/SOURCES/xfwm4-4.4.1.tar.bz2
jdsbuild@prae> mv xfwm4-4.4.1 xfwm4-4.4.1-orig
jdsbuild@prae> gtar fxj ~/packages/SOURCES/xfwm4-4.4.1.tar.bz2
#
# Fix the files which need to be patched.
#
jdsbuild@prae> diff -ur xfwm4-4.4.1-orig xfwm4-4.4.1 > ~/spec-files-xfce/patches/xfwm4-01-fixbadcode.diff

At this point add an entry for the patch xfwm4-01-fixbadcode.diff into the spec file and try again.

If you need to add another patch, just remove the patched directory, and un-tar again.

Maybe you want to merge a fix into an existing patch. Just patch the directory in /tmp and then make the changes.

jdsbuild@prae> cd /tmp
jdsbuild@prae> gtar fxj ~/packages/SOURCES/xfwm4-4.4.1.tar.bz2
jdsbuild@prae> mv xfwm4-4.4.1 xfwm4-4.4.1-orig
jdsbuild@prae> gtar fxj ~/packages/SOURCES/xfwm4-4.4.1.tar.bz2
jdsbuild@prae> cd xfwm4-4.4.1
jdsbuild@prae> gpatch -p1 < ~/spec-files-xfce/patches/xfwm4-01-fixbadcode.diff jdsbuild@prae> cd /tmp
#
# Fix the files which need to be patched.
#
jdsbuild@prae> diff -ur xfwm4-4.4.1-orig xfwm4-4.4.1 > ~/spec-files-xfce/patches/xfwm4-01-fixbadcode.diff

Using a shell like bash with command line editing, the process can be quite quick and easy.

Have Fun!!!

Sex on the Desktop with Solaris

My first draft of this blog went into half a page of crap about how Sun was originally really cool on the desktop, and then decided it would leave it all to Microsoft and Apple. Rather than dragging up the past (and boring you will what you with what you already know), I will focus on the here, now, and future.

Having been lagging by several years, Solaris Express is now hot on the heals of todays desktop. JDS is currently Gnome 2.16 (2.18.x will be Solaris Express real soon). If you really want the latest and greatest now, you can either download and install the Vermillion developers release or build it directly from the JDS spec files. The latest and greatest Xfce Desktop (4.4.1) like Vermillion can be downloaded and installed (x86 only), or built from the Xfce spec files.

While JDS and Xfce are now are rather nice now. For the real story on a sexy eye-candy Desktop you can not go past Compiz on Solaris. If you are into shimmering transparent windows etc, and you think that there is not a hope in hell that you can do this on Solaris, then you need to check out Erwann Chénedé's blog. Also if you also want transparent windows for Xfce. Install Erwann's Compiz packages and rebuild the Xfce window manager from the spec files. Re-login and turn on the "Compositor" in the Window Teaks settings.

To top this off, I have recently added the spec files for several really cool 3D Linux Games (more on the way) and wine to the SFE repository. The SFE repository has the spec files for a lot of really cool applications for Open Solaris. Check it out!

The only thing missing now is the latest KDE....

Below I have put together hopefully some useful links for a Sexy Desktop Solaris...

Name: Open Solaris Desktop Community
Homepage: http://www.opensolaris.org/os/community/desktop
Discussion List: http://www.opensolaris.org/os/community/desktop/discussions

Name: JDS
Full Name: Java Desktop System
Description: Sun's version of the Gnome Desktop
Project Homepage: http://www.opensolaris.org/os/project/jds
Download URL: http://dlc.sun.com/osol/jds/downloads/current
SVN URL: svn+ssh://anon@svn.opensolaris.org/svn/jds/spec-files/trunk

Name: Erwann Chénedé's Weblog
Description: Compiz 0.5.0 packages and patches for OpenSolaris x86
Blog URL: http://blogs.sun.com/erwann/entry/compiz_0_5_0_packages

Name: Open Solaris Xfce Project
Description: Xfce Desktop for Open Solaris
Project Homepage: http://www.opensolaris.org/os/project/xfce
Download URL: http://dlc.sun.com/osol/xfce/downloads/current
Parent Project URL 1: http://www.xfce.org
Parent Project URL 2: http://goodies.xfce.org
SVN URL 1: svn+ssh://anon@svn.opensolaris.org/svn/xfce/spec-files-xfce/trunk
SVN URL 2: svn+ssh://anon@svn.opensolaris.org/svn/xfce/xfce-goodies/trunk

Name: Spec Files Extra (SFE)
Description: A Repository for pkgbuild spec files (Not only Desktop)
Project Homepage: http://pkgbuild.sourceforge.net
SVN URL: https://svn.sourceforge.net/svnroot/pkgbuild/spec-files-extra/trunk

Name: Build Recipes Reference
Description: List of build files, scripts and patches for Open Source Software on Solaris
URL: http://www.genunix.org/wiki/index.php/BuildRecipesReference

OpenOffice icon fix for Solaris

One thing that has always bugged me when I install Solaris and add OpenOffice.org 2.x, is that the icons for OpenOffice.org never appear next to their menu entries in JDS. I know it is not life threatening, or stops the application for working. It just looks bad!

A quick and dirty fix to this is to change the "Icon" entry in the relevent .desktop files in /usr/share/applications. For a slightly quicker and cleaner fix I have written a small script which just adds symbolic links to the installed icon files. The script does this by adding the links directly to the OpenOffice.org desktop integration installed package. This ensures that the links are removed when the package is removed.

#!/bin/sh
# Usage: openoffice-icon-fix
# 
# Currently OpenOffice.org for Solaris is delivered with the wrong filenames
# for the icons in the .desktop files. This means the icons will not show
# up in the JDS menus. To work around this, this script adds a link to the
# original file the Solaris package database, which in-turn creates a
# physical link in the filesystem (via installf -f)
#

PKGINST=openofficeorg-desktop-integratn
APPS="writer printeradmin math impress draw calc base"
PIXDIR=/usr/share/pixmaps
OLD=openoffice.org-2.0
NEW=openofficeorg-20

#
# Check to see if OpenOffice.org is installed.
#
/usr/bin/pkginfo | /usr/bin/grep ${PKGINST} >/dev/null || {
    echo "${PKGINST} is not installed."
    echo "Please install OpenOffice.org 2.x, before running this command"
    exit 1
}

#
# Note: The trailing 's' on the end of the installf command is NOT a typo!
#
for i in ${APPS}
do
    /usr/sbin/installf ${PKGINST} ${PIXDIR}/${NEW}-${i}.png=${OLD}-${i}.png s
done

/usr/sbin/installf -f openofficeorg-desktop-integratn

Good Hunting!

VOLD is dead. Long live Tamarack

After a quick check of the recent ON nevada changelog, I saw that one of my pet Solaris hates "vold" has been deleted and replaced with "PSARC case 2005/399 : Tamarack: Removable Media Enhancements in Solaris". Sun THANK YOU!!!

Not only is vold gone, but you get hal, and joliet extensions thrown in to the mix. A quick download and bfu of the on-bfu-20061016 archive, and this is what it looks like to mount a DVD backup of my photographs burnt from MS Windows, and a memory stick -

root@bangkok> uname -a 
SunOS bangkok.priv 5.11 opensol-20061016 i86pc i386 i86pc
root@bangkok> df -h -F hsfs
Filesystem             size   used  avail capacity  Mounted on
/dev/dsk/c1t0d0s2      2.7G   2.7G     0K   100%    /media/Photos10042006
root@bangkok> df -h -F pcfs
Filesystem             size   used  avail capacity  Mounted on
/dev/dsk/c2t0d0p0:1    123M   173K   123M     1%    /media/MEM128

As you can see no more /vol, /cdrom, /rmdisk, just the real device name, and the /media mountpoint. Also all the filenames on the DVD are humanly readable.

INQ thinks Quad cores too many cores for games.

After speaking to "many developers including some big names", the Inquirer said that you should forget quad core CPU's for games!

What a load of rubbish. Plenty of games impress me that who ever built the game is obviously a gifted programmer. The following narrow minded statement, if it did come from real games programmers, does makes me wonder.

You can keep one core busy with the physics and collision detection, second core will have to wait for the score to move on with the Artificial intelligence while the third core could possible calculate the graphic data. In this best case scenario you have to realise that the core number two and three would always have to wait for the core number one to finish its job and pass the job to the cores two and three. In this concept there is absolutely no place for quad core as games are non parallel applications

Most games I play, I am not the only one playing. If it is not another human or humans, then you play against several computer 'AI' characters. The whole idea the each section of the game is bound to a particular core just stinks of a bad programming model. Maybe they have never heard of threading (or just scared of it). To say that "game are non parallel applications", is just grossly incorrect.

Lets take an old favorites like Galaga. Very simple... There is my spaceship at the bottom of the screen. It can fire several missiles at about 20 enemy spaceships. Now if you treat my space ship, the missiles, and the enemy space ships as different objects and assign at least one thread to each. You end up with a very parallel application, where the more cores the merrier.

Is it a case of gaming programmers still using 1980's programming tools/models, or is it just another bad INQ article. I am sure it is the later.

Its a Coup

Last night I have just sat through my first coup. All the action was only 5km's away and it just seemed like another Tuesday night. The first I knew about it was when I tried to access some of the online Thailand newspaper and they responded after a long wait with lots of PHP/MYSQL errors. A quick flick through the television channels were exactly the same. Pre-recorded music and images of the King.

Luckily, I could still access the CNN and BBC websites to find what was going on. Through the whole thing there was the sound of 'booms' in the distance. Luckily, nobody was shooting at each other, and it was just the usual Bangkok thunderstorm.

Today, some of the newspapers are saying the coup will not affect tourism. I would not be that optimistic, though if you are comming to Thailand, then still come. It is safe! Tomorrow, things should be back to normal.

What lessons could be learnt from this? Maybe the online newspapers need to upgrade some of their hardware. They didn't cope.... I am sure some T2000's would :)

Almost.... But no cookie today.

Recently one of my girlfriend's friends decided to do a computer course. So now she wants to buy a new computer. Cool lets go shopping. After walking around the many levels of the infamous Pantip Plaza in Bangkok, she decided she wanted a HP. Cool!

My first assumption was that since it is a branded computer, it will come with the usual Microsft Virusware installed. Looking through the documentation though, I find that it only comes with a "Free DOS Operating System". Wow, times have changed.... but DOS just makes it totally useless. HP, why not at least install a free Linux Distribution?.

Rather, than buying Windows, lets look at the options -

Solaris Express - That would be cool. I have the DVD...
Ubuntu - Quite nice for beginner. I also have the DVD...
Linux TLE - Very good for her since she really only speaks Thai

So, I ask the question. "What computer course are you doing?" - Windows..... DOH!!!!

JDS has new fan

When a friend came to visit the other day, she asked if she could use my computer for a while. Hmmmm, no problems. This is a Windows only person, and my laptop only has OpenSolaris, no Microsoft Windows. I won't say a word and see what happens. Ok, so I went and watched a Thai soapy on TV (Contains plenty of hate, violence, blood and guts. Absolutely no sex! - strange logic), while she typed away.

After she had finished she commented "I really like your computer. It is nice and easy to use". Wow, It looks like the JDS team have done a good job.

Chinese Pirates

According to a report in the Bangkok Post, "Nearly half of all books, films, music CDs and software sold in China are illegally copied". Now I find it very hard to believe that it could be less than 50%. The Chinese are much smarter than that!

I have always had the view that if the sticker price is much higher than the price to reproduce, than you will always have illegal copies being made. For software, I would love for Microsoft (and others) to find a secure way of protecting their licensing of their software. This would force people to look at the alternative free Open Source software such as Solaris, BSD, Linux and Open Office. Companies like Microsoft using the old "high admission fee" model will one day have to change their ways. They maybe able to hold their market share for a little longer in the developed world, but there is a much larger market in the developing world who are much more price sensitive.

The missing benchmarks

When puchasing computer hardware, unless you get your hands on the equipment before you purchase, you have to rely on past experience, experience of others, or benchmarks, to see if they have enough capacity for your users needs.

There is a large wealth of information you can use to judge server capacity. There is also plenty of information for storage capacity. One place that is almost always overlooked is network devices. I have seen many people in the past, blindly using the vendors advertised specifications to determine their needs. The assumption that they have Gig ports on a switch and the vendor claims the backplane is "HUGE", does not mean that you can assume that the actual performance will be "wire" speed.

If you are in the market for network hardware at the moment, before you go too far you should have a sober look at Simon Bullen's blog on Cisco's 3750. I have always had my doubts about Cisco equipment in the past, and the reasons why people choose it over other vendors. It would be nice if we could have somewhat neutral benchmark standards body like www.spec.org to produce a set of network equipment benchmarks. My searching so far only finds vendors (or their proxies) competitive benchmarking their opposition, and of course whipping their ass. You would not expect less.

While independent results from people like Simon are fantastic. It would be nice to see network vendors put their crown jewels on the benchmarking chopping block for all to see. The nice thing about this is if they don't post results for certain equipment, they may be selling you a brumby.

PR Nightmare

Is there a breakout of foot-in-mouth disease at IBM lately, or have do they just have a drug problem? I know the Solar System is currently being redefined, but what planet are they on?

Their recent comments on Sun open source creditials do make you wonder just what are they thinking. Due to their work on the Linux kernel, Apache, Eclipse etc, IBM have been able to enjoy the admoration of the open source community, while at the same time keeping their products very closed. To now it has been very clever marketing. The problem they have now is that some commentators are starting to see through this, and are asking why IBM does not open source its software and hardware like Sun Microsystems has been doing.

The obvious reply to this to avoid the focus on IBM's closed products, would have actually been to acknowledge Sun's effort, and focus on how long it takes to change a product of the size of Solaris and AIX from multiple closed source licenses to an open source license. Instead they have decided to attack the Open Solaris and Open Sparc projects stating that they are not truely open. This is just total bullshit!

Rather than hosing down the question, giving the commentators little to write about, they have decided to give the press a juicy story. This would be fine if they had a large swag of products based on open source. The fact is that they have not, and the press have now alerted to a renewed battle between IBM and Sun with IBM standing on very shakey ground, surrounded by mountains of closed source and hardware.

IBM's current statements are probably an indication of they think Sun's push to open source not only it's entire software stack, but also hardware, is a medium to long term threat to IBM position. It looks from what they are saying, is at the moment they are scratching themselves to find a solution. Using verbal attacks on Sun is just plain counter productive. I think the future may be tough for companies that dabble in open source while at the same time keeping a very closed product line. Hypocrisy does not further their cause.

What can IBM do to get themselves out of this mess? I think a good start would be to avoid opening their mouths, and start work on a timetable to start opening up their products. They state that their customers are not interested in AIX enough, but I find this very hard to believe. When Solaris was re-released onto x86 hardware, they stated that the was not enough interest from their customers to release software for the platform. Their stance changed very quickly when they got a tap on the shoulder from some of their largest customers. I think if they are seeing little interest from their customers, then this is a more of a worrying sign of the future of AIX. I hope this is not the case.

Don't be afraid of mdb - cont

I thought I would just quickly show why I did not use any "optimize" flags when I compiled my example code. At the same time I show the er_src program which part of Sun Studio.

First I compile the program with the '-g' flag. This will give er_src access to the source code.

doug@bangkok> cc -g -o ./makecore ./makecore.c

Next run the program and do a pstack. Everything looks fine

doug@bangkok> ./makecore
Memory fault(coredump)
doug@bangkok> /bin/pstack core
core 'core' of 102246:  ./makecore
08050775 fruitloop (8060958, 804710c, 80507bb, 8047210, 804712c, 80506ca) + 15
08050794 giveitatry (8047210, 804712c, 80506ca, 1, 8047138, 8047140) + 14
080507bb main     (1, 8047138, 8047140) + b
080506ca _start   (1, 8047278, 0, 8047283, 8047290, 80472ca) + 7a

Ok, lets have a look at what er_src does

doug@bangkok> /opt/SUNWspro/bin/er_src -func ./makecore

Functions sorted in lexicographic order

Load Object: 

   Address     Size        Name

 0x000005e0        16      @plt
 0x000005f0        16      __fpstart
 0x000006dc       123      __fsr
 0x00000620        16      _exit
 0x000007ec        27      _fini
 0x00000640        16      _get_exit_frame_monitor
 0x000007d0        27      _init
 0x00000650       139      _start
 0x00000610        16      atexit
 0x00000600        16      exit
 0x00000630        16      printf
 0x00000760        30      fruitloop
 0x00000780        38      giveitatry
 0x000007b0        29      main

doug@bangkok> /opt/SUNWspro/bin/er_src -disasm fruitloop ./makecore
Annotated disassembly
---------------------------------------
Source file: ./makecore.c
Object file: ./makecore
Load Object: ./makecore
    1. #include 
    2. #include 
    3.
    4. static void
    5. fruitloop(){
       
       [ 5]  8050760:  pushl   %ebp
       [ 5]  8050761:  movl    %esp,%ebp
       [ 5]  8050763:  subl    $4,%esp
    6.         char *p;
    7.         p=(char *)NULL;
       [ 7]  8050766:  movl    $0,-4(%ebp)
    8.         *p='c';
       [ 8]  805076d:  movl    $0x63,%eax
       [ 8]  8050772:  movl    -4(%ebp),%edx
       [ 8]  8050775:  movb    %al,0(%edx)
       [ 8]  8050778:  jmp     .+4 [ 0x805077c ]
       [ 8]  805077a:  nop
       [ 8]  805077b:  nop
    9. }
       [ 9]  805077c:  leave
       [ 9]  805077d:  ret
   10.
   11. static void
   12. giveitatry(){
       
       [12]  8050780:  pushl   %ebp
       [12]  8050781:  movl    %esp,%ebp
       [12]  8050783:  subl    $4,%esp
   13.         char *msg="Ahh we made it!\n";
       [13]  8050786:  leal    0x8060958,%eax
       [13]  805078c:  movl    %eax,-4(%ebp)
   14.
   15.         fruitloop();
       [15]  805078f:  call    fruitloop [ 0x8050760, .-0x2f ]
   16.         (void)printf(msg);
       [16]  8050794:  movl    -4(%ebp),%eax
       [16]  8050797:  pushl   %eax
       [16]  8050798:  call    printf [ 0x8050630, .-0x168 ]
       [16]  805079d:  addl    $4,%esp
       [16]  80507a0:  jmp     .+4 [ 0x80507a4 ]
       [16]  80507a2:  nop
       [16]  80507a3:  nop
   17. }
       [17]  80507a4:  leave
       [17]  80507a5:  ret
   18.
   19. int
   20. main(int argc, char **argv){
       
       [20]  80507b0:  pushl   %ebp
       [20]  80507b1:  movl    %esp,%ebp
       [20]  80507b3:  subl    $4,%esp
   21.         giveitatry();
       [21]  80507b6:  call    giveitatry [ 0x8050780, .-0x36 ]
   22.         return(0);
       [22]  80507bb:  movl    $0,-4(%ebp)
       [22]  80507c2:  jmp     .+6 [ 0x80507c8 ]
       [22]  80507c4:  jmp     .+4 [ 0x80507c8 ]
       [22]  80507c6:  nop
       [22]  80507c7:  nop
   23. }
       [23]  80507c8:  movl    -4(%ebp),%eax
       [23]  80507cb:  leave
       [23]  80507cc:  ret

Now, that is really nice. The C program is listed with the assembly code for the corresponding C code. It makes a very nice assembly language tutorial. Now, lets do the same, but compile using the "-fast" flag. (-fast is actually a macro for several other flags. It is known to generally give the best optimized code for your system with the least effort)

doug@bangkok> cc -fast -g -o ./makecore ./makecore.c
doug@bangkok> ./makecore
Memory fault(coredump)
doug@bangkok> /bin/pstack core
core 'core' of 102283:  ./makecore
08050850 main     (1, 8047278, 0, 8047283, 8047290, 80472ca)

Hmmm, this time it stopped in function main. Lets look at what -fast did to the code.

doug@bangkok> /opt/SUNWspro/bin/er_src -disasm fruitloop ./makecore
Annotated disassembly
---------------------------------------
Source file: ./makecore.c
Object file: ./makecore
Load Object: ./makecore
    1. #include 
    2. #include 
    3.
    4. static void
    5. fruitloop(){
    6.         char *p;
    7.         p=(char *)NULL;
    8.         *p='c';
       
       [ 8]  8050820:  movb    $0x63,0
    9. }
       [ 9]  8050827:  ret
       
       [ 8]  8050830:  movb    $0x63,0
   10.
   11. static void
   12. giveitatry(){
   13.         char *msg="Ahh we made it!\n";
   14.

  Function fruitloop inlined from source file ./makecore.c into the code for the following line.  0 loops inlined
   15.         fruitloop();
   16.         (void)printf(msg);
       [16]  8050837:  subl    $8,%esp
       [16]  805083a:  pushl   $0x80609f4
       [16]  805083f:  call    printf [ 0x8050630, .-0x20f ]
       [16]  8050844:  addl    $0xc,%esp
   17. }
       [17]  8050847:  ret
       
       [ 8]  8050850:  movb    $0x63,0
       [16]  8050857:  subl    $8,%esp
       [16]  805085a:  pushl   $0x80609f4
       [16]  805085f:  call    printf [ 0x8050630, .-0x22f ]
       [16]  8050864:  addl    $0xc,%esp
   18.
   19. int
   20. main(int argc, char **argv){

   Function giveitatry inlined from source file ./makecore.c into the code for the following line.  0 loops inlined
   Function fruitloop inlined from source file ./makecore.c into inline copy of function giveitatry.  0 loops inlined
   21.         giveitatry();
   22.         return(0);
       [22]  8050867:  xorl    %eax,%eax
       [22]  8050869:  ret
   23. }

As you can read from the comments, both of the functions were inlined. Therefore they are now part of the 'main' function. The 'er_src' program is really neat app. Lets see the comment change when we tell it not to inline.

doug@bangkok> cc -fast -g -xinline=no%fruitloop -o ./makecore ./makecore.c
doug@bangkok> /opt/SUNWspro/bin/er_src -disasm fruitloop ./makecore
Annotated disassembly
---------------------------------------
Source file: ./makecore.c
Object file: ./makecore
Load Object: ./makecore
.
.
    5. fruitloop(){
    6.         char *p;
    7.         p=(char *)NULL;
    8.         *p='c';
       
       [ 8]  8050820:  movb    $0x63,0
    9. }
       [ 9]  8050827:  ret
   10.
   11. static void
   12. giveitatry(){
   13.         char *msg="Ahh we made it!\n";
   14.

   Function fruitloop not inlined because user explicitly requested that it not be inlined

If you are playing around with optimizing code, then er_src is one tool you should use.

Have Fun!

Don't be afraid of mdb

Many Solaris system admins or developers would know that Solaris has some very good debugging tools. Most sysadmins would know there is a command called mdb. Sadly most would have either never used it, or was scared off when they scanned through the documentation. While using mdb does require a good knowledge of the Solaris internals, and some assembly language skills, there are times where it is probably the only (or best) tool for the job.

Consider the case where you have an application that your company has been using for a long time. Something has changed on the system, and now it crashes when it is run. Since the person who wrote the application now does not work for your company anymore and nobody knows where the source code is, you have a problem. To make things worst, when you do a pstack on the core file, you find that they have “stripped” the binary of its symbol table to save a few bytes. Your options are now really limited to do any useful debugging. Enter 'mdb'....

Now to simulate this I have created a small C program, with a null pointer buried in a couple of functions. I compile the program (not using any optimizations as the compiler will inline all of the functions as they are very small), and then run the strip command on it. During running the program we get not a very useful error message, and a core dump. Argggg!

Running a pstack on the binary, because it was stripped, pstack returns an address with “????????”, as the function name. Ah, it is now turned into a challenge.

doug@bangkok> /bin/pstack core
core 'core' of 101996:       ./makecore
080506d5 ???????? (8060898, 80470fc, 805071b, 804720c, 8047124, 805062a)
080506f4 ???????? (804720c, 8047124, 805062a, 1, 8047130, 8047138)
0805071b main     (1, 8047130, 8047138) + b
0805062a _start   (1, 8047274, 0, 804727f, 804728c, 80472c6) + 7a

You can get the similar output from the “::stack” command, within mdb.

doug@bangkok> mdb core
Loading modules: [ libc.so.1 ld.so.1 ]
> ::stack
0x80506d5(8060898, 80470fc, 805071b, 804720c, 8047124, 805062a)
(804720c, 8047124, 805062a, 1, 8047130, 8047138)
main+0xb(1, 8047130, 8047138)
_start+0x7a(1, 8047274, 0, 804727f, 804728c, 80472c6)

Since there is nothing in human readable form, at this point most people would look elsewhere or through it in the too hard basket. If you know a little assembly language (32bit x86 in this case), you should probably continue on. A good starting point would be the assembly listing of the function where it bombs out. The first address “0x80506d5” is for the instruction where we bombed out. Doing a disassemble backwards from this address is tedious, especially if this instruction is a long way from the beginning. The address on the next line “0x80506f4” is actually more useful. It is the return address of the function, which should be the next instruction after the function call. The function calling code should be immediately before this. Lets attack it with the disassembler built into 'mdb' byte by byte.

> 80506f4::dis
0x80506f4:                      movl   -0x4(%ebp),%eax
> 80506f3::dis
0x80506f3:                      decl   0xe850fc45(%ebx)
> 80506f2::dis
0x80506f2:                      ***ERROR--unknown op code***
> 80506f1::dis
0x80506f1:                      ***ERROR--unknown op code***
> 80506f0::dis
0x80506f0:                      int    $0x3
> 80506ef::dis
0x80506ef:                      call   -0x34    <0x80506c0>

Bingo! We have a winner - 0x80506c0. You will probably notice, the "call" op-code (1 byte) was followed by a 4 byte address, so we could have first tried the addess – 5. In my case the command inside of mdb would have been “80506f4-5::dis”.

Now we have an address, we can now easily list the function from the start.

> 80506c0::dis
0x80506c0:                      pushl  %ebp
0x80506c1:                      movl   %esp,%ebp
0x80506c3:                      subl   $0x4,%esp
0x80506c6:                      movl   $0x0,-0x4(%ebp)
0x80506cd:                      movl   $0x63,%eax
0x80506d2:                      movl   -0x4(%ebp),%edx
0x80506d5:                      movb   %al,0x0(%edx)
0x80506d8:                      leave
0x80506d9:                      ret
0x80506da:                      nop

We can eithen add the function into mdb's user-defined symbol table, so we can now see symbolic names, rather than hex addresses. The rough comments we added by me :)

> 80506c0::nmadd -f -e 80506da badfunc
added badfunc, value=80506c0 size=1a
> badfunc::dis
badfunc:                        pushl  %ebp            ; save frame pointer to the stack
badfunc+1:                      movl   %esp,%ebp       ; copy stack pointer to frame pointer
badfunc+3:                      subl   $0x4,%esp       ; make room for the pointer - char *p
badfunc+6:                      movl   $0x0,-0x4(%ebp) ; initialize pointer to null - p=(char*)NULL;
badfunc+0xd:                    movl   $0x63,%eax      ; copy 'c' to %eax register
badfunc+0x12:                   movl   -0x4(%ebp),%edx ; copy pointer to register %edx - now = 0
badfunc+0x15:                   movb   %al,0x0(%edx)   ; *p = 'c' - Hmmm copy 'c' to address 0 - BAD!!!
badfunc+0x18:                   leave                  ; cleanup function call
badfunc+0x19:                   ret                    ; return to calling function
> ::stack
badfunc+0x15(8060898, 80470fc, 805071b, 804720c, 8047124, 805062a)
0x80506f4(804720c, 8047124, 805062a, 1, 8047130, 8047138)
main+0xb(1, 8047130, 8047138)
_start+0x7a(1, 8047274, 0, 804727f, 804728c, 80472c6)
> 80506f4-5::dis
0x80506ef:                      call   -0x34    

From a quick look at my disassembled code, it is clear that some idiot created a null pointer , and then tried to copy a byte to there. Not very bright eh! In a real world example you would probably need to run the command in mdb and set the breakpoint to the start of the function. From there you could step through the code to see what is does. It would go something like this -

doug@bangkok> mdb ./makecore
> 80506c0::nmadd -f -e 80506da badfunc          ; Add our own symbol from above
added badfunc, value=80506c0 size=1a
> badfunc:b                                     ; Set a breakpoint at the beginning of badfunc
> :r                                            ; run ./makecore in the debugger
mdb: stop at badfunc
mdb: target stopped at:
badfunc:        pushl  %ebp
> :s                                            ; Step through code, 1 step at a time
mdb: target stopped at:
badfunc+1:      movl   %esp,%ebp
> :s
mdb: target stopped at:
badfunc+3:      subl   $0x4,%esp
> :s
mdb: target stopped at:
badfunc+6:      movl   $0x0,-0x4(%ebp)
> :s
mdb: target stopped at:
badfunc+0xd:    movl   $0x63,%eax
> ::regs                                        ; Check the registers - Hmmm. %edx = 0
%cs = 0x003b            %eax = 0x08060898
%ds = 0x0043            %ebx = 0xfeffa7c0
%ss = 0x0043            %ecx = 0xfefa9768 libc.so.1`_sse_hw
%es = 0x0043            %edx = 0x00000000
%fs = 0x0000            %esi = 0x080470e0
%gs = 0x01c3            %edi = 0x08047204

%eip = 0x080506cd badfunc+0xd
%ebp = 0x080470e4
%kesp = 0x00000000

%eflags = 0x00000202
 id=0 vip=0 vif=0 ac=0 vm=0 rf=0 nt=0 iopl=0x0
 status=

  %esp = 0x080470e0
%trapno = 0x1
  %err = 0x0

To find the best reference on the guts of Solaris and how to make the best use mdb and other Solaris tools such as DTrace. Go and purchase the just released 2nd edition of Solaris Internals and it's new companion Solaris Performance and Tools. You can save 30% by buying the through Sun. While you are stacking your bookself, you should also consider Solaris System Programming, and Sun Performance and Tuning. Some light reading :)

Filesystem Benchmarks

If you have been reading the zfs-discuss on OpenSolaris.org recently, you would have read that Robert Milkowski has been doing some benchmarks using Sun's StorageTek 3510 FC diskarrays. He has been getting some interesting results that suggest that using ZFS and the 3510 without the hardware RAID controllers is faster than using it with. This is very interesting because the cost of hardware raid controllers can be expensive. If it suits your needs, you can save some cash by using Solaris 10 and ZFS, as both are free!

Since I don't have have a 3510 sitting around to test on, I decided to do a quick benchmark on a spare partition of my laptop to compare ZFS and UFS. We have all been told that ZFS is faster than UFS, but by how much and when is a interesting question.

Using filebench as Robert did, I have started with the varmail workload using the average of three runs to produce the graph below. For each run I created the pool (ZFS) and filesystem, did the three benchmark runs for 60 seconds each, and then destroyed the partition for the next benchmark test.

For ZFS this was

root> zpool create -fm /none benchpool /dev/dsk/c0d0s4
root> zfs create benchpool/mnt
root> zfs set mountpoint=/mnt benchpool/mnt
root> # Set options zfs options e.g. zfs set atime=off benchpool/mnt
root> /opt/filebench/bin/filebench
filebench> load varmail
filebench> set $dir=/mnt/zfstest
filebench> run 60
root> zfs destroy benchpool/mnt
root> zpool destroy benchpool

For UFS -

root> newfs /dev/dsk/c0d0s4
root> mount -o noatime -F ufs /dev/dsk/c0d0s4 /mnt
root> # -o noatime is the option for this test
root> /opt/filebench/bin/filebench
filebench> load varmail
filebench> set $dir=/mnt/zfstest
filebench> run 60
root> umount /mnt

As you can see ZFS is indeed faster for this benchmark than UFS. To be fair, and to compare apples to apples, I should have combined UFS with the Solaris Volume Manager (SVM). This most likely seen a greater gap between ZFS and UFS. One thing it shows, is that a Acer Ferrari 4005 maybe a nice laptop, but it makes a horrible mailserver :(

How small can you make Open Solaris - Part 5

In the latest post I have just done a quick update of the "Quick and Dirty Solaris Installer". Version 0.3 of the installer is now able to exclude Solaris clusters and packages from the installation. With this we are now able to start reducing the size and try to get close to what was achieved in the first couple of posts.

If you read through the code you can see that I have started with the SUNWCmreq metacluster, as it is the smallest defined on the Solaris installation media. I actually had to add the package SUNWmdr, as devfsadmd started complaining during the first reboot. In the code you will see a big list of packages defined in the "exclude array". Since this list includes device drivers, you may have to add and remove driver packages for your system. To do this you will need to read through the '.clustertoc' file in the Product directory of your installation media.

Using the configuration in this script, I was able to get a bootable multi-user system which only needed /etc/nodename and /etc/hosts configured. The disk usage is down to around 149MB (still a long way to go). Solaris core packages actually have a lot of files and command which I would not call 'core'. Later on I will look at creating new packages, and pulling files directly from a Open Solaris bfu archive. This may also require some files being pulled from "running" Solaris, as the bfu archives are not complete. If we can successfully use a bfu archive, then we could later see if changing some of the compiler flags in an Open Solaris build will shrink the binary size.

In the next post, I will start reducing installation further, by adding commands to the local_install.bash script which can be run just before the filesystem is un-mounted.

#!/bin/bash
#
# Quick and Dirty Solaris installer
# Version 0.3
#
PROD=/cdrom/sol_11_x86/Solaris_11/Product
#PROD=/a/Solaris_11/Product
TOK=${PROD}/.clustertoc
ORDER=${PROD}/.order

# Current metaclusters are
# SUNWCXall - Entire Distribution plus OEM support
# SUNWCall  - Entire Distribution
# SUNWCprog - Developer System Support
# SUNWCuser - End User System Suppor
# SUNWCreq  - Core System Support
# SUNWCrnet - Reduced Networking Core System Support
# SUNWCmreq - Minimal Core System Support
METACLUSTER=SUNWCmreq
export METACLUSTER

SWAP=/dev/dsk/c0d0s1
LOG=/tmp/install.log
SVCPROFILE=generic_limited_net.xml

#
# Change these to relect your system.
#
FS=zfs
[ "${FS}" != "zfs" -a "${FS}" != "ufs" ] && {
    printf "Unknown filesystem ${FS}\n"
    exit 1
}

[ "${FS}" = "zfs" ] && {
    ROOTDEV=intdisk/snv43_zfs
    RAWROOTDEV=-
    GRUBFS=/dev/dsk/c0d0s0
    ZFSBOOTARCHIVE=/grub/boot/boot_zfs
    ZFSCOMPRESS=off
#    ZFSCOMPRESS=on
    MOB=yes
}

[ "${FS}" = "ufs" ] && {
    ROOTDEV=/dev/dsk/c0d0s3
    RAWROOTDEV=${ROOTDEV/dsk/rdsk}
    MOB=no
}

typeset -a pkgs
# extrapkgs is an array of additional packages to be installed
typeset -a extrapkgs=( SUNWmdr )
# exclude is an array of packages or clusters which should not be installed
typeset -a exclude=( SUNWChbaapi SUNWCfcadb SUNWCfca SUNWCfcadb SUNWCfct
    SUNWCfutil SUNWCiscsi SUNWCib SUNWCmpapi SUNWCtavor
    CADP160 HPFC SK98sol SUNWaac SUNWadp SUNWadpu320 SUNWamr SUNWcadp SUNWced
    SUNWchxge SUNWcqhpc SUNWlsimega SUNWmv88sx SUNWnge SUNWrge SUNWrmodr
    SUNWrmodu SUNWrtls SUNWses SUNWsi3124 SUNWuksp SUNWuedg SUNWukspfw
    SUNWuprl SUNWxge SYMhisl SKfp SUNWintgige SUNWwbsup
    SUNWCpkgcmds SUNWjss SUNWidnl SUNWbzip
    SUNWperl584core SUNWperl584usr  )

#
# check_install tests whether $1 is in the list of packages to be excluded
#
function check_install() {
    local i
    [ "$#" != "1" ] && return 0

    for i in ${exclude[@]} ; do
    [ ${i} = "$1" ] && return 1
    done
    return 0
}

#
# add_extra_pkgs will add in the extrapkgs array to the pkgs array. It will
# void adding a package twice
#
function add_extra_pkgs() {
    pkgcnt=${#pkgs[@]}
    for i in ${extrapkgs[@]} ; do
    found=0
    for j in ${pkgs[@]} ; do
        [ ${i} = ${j} ] && {
        found=1
        break;
        }
    done
    # Add package if not already found
    [ "${found}" = "0" ] && {
        printf "Adding Package - %s [%d]\n" "${i}" ${pkgcnt}
        pkgs[$(( pkgcnt++ ))]="$i"
    }
    done
}

#
# Solaris packages need to be installed in the correct order.
# The .order file contains all the packages in the correct
# installation order
#
function reorder_pkgs() {
    typeset -a pkglist=( ${pkgs[@]} )
    pkgcnt=0

    while read order_pkg ; do
    for i in ${pkglist[@]} ; do
        [ "$order_pkg" = "${i%.i}" ] &&  {
        pkgs[$(( pkgcnt++ ))]="${i}"
        printf "."
        }
    done
    done < ${ORDER}
}

#
# This function builds a list of packages in a cluster
# If there is a cluster within a cluster, it will call itself to
# resolve all the packages.
#
# Before calling make sure you initialize pkgcnt to 0
# Arg: $1 contains the cluster name
# Affected vars: pkgs, pkgcnt
#
function get_pkg_list() {
    local IFS="="
    local print_on=0
    local cluster=$1

    while read arg1 arg2
    do
        [ "${arg1}" = "END" -a "${print_on}" = "1" ] && break;
        [ -z "${arg2}" ] && continue;

        [ "${arg2}" = "${cluster}" ] && {
            print_on=1
            continue
        }

        [ "${print_on}" = "1" -a "${arg1}" = "SUNW_CSRMEMBER" ] && {
        # Test to see if package/cluster is on the exclude list
        check_install "${arg2}" || continue

        ifcluster=`expr "${arg2}" : '\(SUNWC\)'`
        if [ "${ifcluster}" = "SUNWC" ]; then
        get_pkg_list ${arg2}
        else
        [ -d ${PROD}/${arg2} ] && {
            pkgs[$(( pkgcnt++ ))]="${arg2}"
            printf "."
            continue;
        }

        arg2="${arg2}.i"
        [ -d ${PROD}/${arg2} ] && pkgs[$(( pkgcnt++ ))]="${arg2}"
        printf "."
        fi
        }
    done < ${TOK}
}

#
# Check for the installation image before proceeding
#
[ ! -d ${PROD} ] && {
    echo "Cannot find Solaris Installation"
    exit 1
}

#
# Create a pkg admin file - see man admin(4)
#
sed 's/ask/nocheck/' /var/sadm/install/admin/default > /tmp/.admin.doit

#
# Build an ordered list of packages from the Solaris installation image
#
printf "Building a list of packages  "
pkgcnt=0
get_pkg_list ${METACLUSTER}
echo
# Add extra packages before sorting
add_extra_pkgs
echo
printf "Sorting packages into the correct order for installation  "
reorder_pkgs
echo
#
# Try to create the filesystem you define at the beginning
#
case ${FS} in
    "zfs")  zfs create ${ROOTDEV} || {
                echo "Cannot create zfs filesystem!"
                exit 1
            }
        zfs set mountpoint=legacy ${ROOTDEV}
        zfs set compression=${ZFSCOMPRESS} ${ROOTDEV}
        ;;
    "ufs")  newfs ${ROOTDEV} || {
                echo "Cannot create ufs filesystem!"
                exit 1
            } ;;
    *)      echo "Cannot create ufs filesystem!"
            exit 1
            ;;
esac

mount -F ${FS} ${ROOTDEV} /mnt || {
    echo "Cannot mount filesystem!"
    exit 1
}

#
# Install packages from Solaris installation image
#
echo "Starting installation of packages"
echo
(
    for i in ${pkgs[@]} ; do
    pkgadd -n -a /tmp/.admin.doit -d ${PROD} -R /mnt $i
    done
) > ${LOG}

#
# Update /etc/vfstab with swap and root partitions
#
(
    printf "${SWAP}\t-\t-\tswap\t-\tno\t-\n"
    printf "${ROOTDEV}\t${RAWROOTDEV}\t/\t${FS}\t1\t${MOB}\t-\n"
    [ "$FS" = "zfs" ] && {
    mkdir -m 0755 /mnt/grub
    printf "${GRUBFS}\t${GRUBFS/dsk/rdsk}\t/grub\tufs\t3\tyes\t-\n"
    }
) >> /mnt/etc/vfstab

#
# Copy links for disk partitions in /dev/dsk and /dev/rdsk
# This is needed so the system can find the root partion on boot
#
( cd /dev && find dsk rdsk -depth | cpio -pdm /mnt/dev 2>/dev/null )

#
# Configure system to initialize identity on first boot
# If there is a sysidcfg file in the current directory. This will
# be copied across.
#
PROFILEDIR=/mnt/var/svc/profile
[ -f ${PROFILEDIR}/${SVCPROFILE} ] && {
    if [ -f ./sysidcfg ]; then
    cp ./sysidcfg /mnt/etc
    else
        touch /mnt/etc/.UNCONFIGURED
    fi
    cp -p ${PROFILEDIR}/${SVCPROFILE} ${PROFILEDIR}/generic.xml
}

#
# set bootpath to root filesystem.
# Also set the console to text
#
(
    [ "${FS}" = "ufs" ] && {
        BOOTPATH=$( ls -l ${ROOTDEV} | nawk '{print $11}' |
        sed -e 's#[./]*/devices/#/#' )

        printf "setprop bootpath ${BOOTPATH}\n"
    }
    printf "setprop console 'text'\n"
) >> /mnt/boot/solaris/bootenv.rc

#
# If found execute local script before /mnt is unmounted
#
[ -x ./local_install.bash ] && ./local_install.bash

#
# Finish off installation
#
[ -f /etc/zfs/zpool.cache ] && {
    cp -p /etc/zfs/zpool.cache /mnt/etc/zfs
    echo "etc/zfs/zpool.cache" >> /mnt/boot/solaris/filelist.ramdisk  
}

#
# Configure for a ZFS boot. At the moment you need a small UFS partition
# somewhere for grub.
#
[ "${FS}" = "zfs" ] && {
    (
    printf "rootfs:zfs\n"
    printf "zfsroot:${ROOTDEV}\n"
    ) >> /mnt/etc/system
}

devfsadm -r /mnt
rm -f /mnt/reconfigure
bootadm update-archive -R /mnt

[ "${FS}" = "zfs" ] && {
    cp /mnt/platform/i86pc/boot_archive ${ZFSBOOTARCHIVE}
    cp -p /mnt/sbin/bootadm /mnt/sbin/bootadm.real
    cat >/mnt/sbin/bootadm << EOM
#!/usr/bin/sh

/sbin/bootadm.real "\$@"
if [ "\$1" = "update-archive" -a -d /grub/boot/grub ]; then
        /usr/bin/cp /platform/i86pc/boot_archive ${ZFSBOOTARCHIVE}
fi
exit 0
EOM
}

echo
echo "If you have not already, you will need to configure menu.lst to"
echo "boot this partition."

umount /mnt
# eject cdrom

---

How small can you make Open Solaris - Part 4

Below, I have posted the new version of the Quick and Dirty Solaris Installer. This version will install Open Solaris directly onto a zfs filesystem. It took me a little longer than expected as I was having problems with the create_ramdisk.ksh script which is used by bootadm. The problem is that create_ramdisk.ksh does a 'du' on every file it is copying to the ramdisk to calculate the space required. If the 'du' command is used on a compressed file, it will return the compressed size. The problem comes about when a UFS filesystem is used to create the boot_archive image. Files copied from a compress zfs volume, will soon fill the ramdisk.

Not being able to create a boot archive is not good. The current work around is to turn compression off for the root filesystem. In the next update, I will modify the script to start excluding packages, so we can see what is needed and what is not.

#!/bin/bash
#
# Quick and Dirty Solaris installer
# Version 0.2
#
PROD=/cdrom/sol_11_x86/Solaris_11/Product
# PROD=/a/Solaris_11/Product
TOK=${PROD}/.clustertoc
ORDER=${PROD}/.order

# Current metaclusters are
# SUNWCXall - Entire Distribution plus OEM support
# SUNWCall  - Entire Distribution
# SUNWCprog - Developer System Support
# SUNWCuser - End User System Suppor
# SUNWCreq  - Core System Support
# SUNWCrnet - Reduced Networking Core System Support
# SUNWCmreq - Minimal Core System Support
METACLUSTER=SUNWCrnet

SWAP=/dev/dsk/c0d0s1
LOG=/tmp/install.log
SVCPROFILE=generic_limited_net.xml

#
# Change these to relect your system.
#
FS=zfs
[ "${FS}" != "zfs" -a "${FS}" != "ufs" ] && {
    printf "Unknown filesystem ${FS}\n"
    exit 1
}

[ "${FS}" = "zfs" ] && {
    ROOTDEV=intdisk/snv43_zfs
    RAWROOTDEV=-
    GRUBFS=/dev/dsk/c0d0s0
    ZFSBOOTARCHIVE=/grub/boot/boot_zfs
    ZFSCOMPRESS=off
#    ZFSCOMPRESS=on
    MOB=yes
}

[ "${FS}" = "ufs" ] && {
    ROOTDEV=/dev/dsk/c0d0s3
    RAWROOTDEV=${ROOTDEV/dsk/rdsk}
    MOB=no
}

typeset -a pkgs

#
# Solaris packages need to be installed in the correct order.
# The .order file contains all the packages in the correct
# installation order
#
function reorder_pkgs() {
    typeset -a pkglist=( ${pkgs[@]} )
    pkgcnt=0

    while read order_pkg ; do
        for i in ${pkglist[@]} ; do
            [ "$order_pkg" = "${i%.i}" ] &&  {
                pkgs[$(( pkgcnt++ ))]="${i}"
                printf "."
            }
        done
    done < ${ORDER}
}

#
# This function builds a list of packages in a cluster
# If there is a cluster within a cluster, it will call itself to
# resolve all the packages.
#
# Before calling make sure you initialize pkgcnt to 0
# Arg: $1 contains the cluster name
# Affected vars: pkgs, pkgcnt
#
function get_pkg_list() {
    local IFS="="
    local print_on=0
    local cluster=$1

    while read arg1 arg2
    do
        [ "${arg1}" = "END" -a "${print_on}" = "1" ] && break;
        [ -z "${arg2}" ] && continue;

        [ "${arg2}" = "${cluster}" ] && {
            print_on=1
            continue
        }

        [ "${print_on}" = "1" -a "${arg1}" = "SUNW_CSRMEMBER" ] && {
            ifcluster=`expr "${arg2}" : '\(SUNWC\)'`
            if [ "${ifcluster}" = "SUNWC" ]; then
                get_pkg_list ${arg2}
            else
                [ -d ${PROD}/${arg2} ] && {
                        pkgs[$(( pkgcnt++ ))]="${arg2}"
                        printf "."
                        continue;
                }

                arg2="${arg2}.i"
                [ -d ${PROD}/${arg2} ] && pkgs[$(( pkgcnt++ ))]="${arg2}"
                printf "."
            fi
        }
    done < ${TOK}
}

#
# Check for the installation image before proceeding
#
[ ! -d ${PROD} ] && {
        echo "Cannot find Solaris Installation"
        exit 1
}

#
# Create a pkg admin file - see man admin(4)
#
sed 's/ask/nocheck/' /var/sadm/install/admin/default > /tmp/.admin.doit

#
# Build an ordered list of packages from the Solaris installation image
#
printf "Building a list of packages  "
pkgcnt=0
get_pkg_list ${METACLUSTER}
echo
printf "Sorting packages into the correct order for installation  "
reorder_pkgs
echo

#
# Try to create the filesystem you define at the beginning
#
case ${FS} in
    "zfs")  zfs create ${ROOTDEV} || {
                echo "Cannot create zfs filesystem!"
                exit 1
            }
            zfs set mountpoint=legacy ${ROOTDEV}
            zfs set compression=${ZFSCOMPRESS} ${ROOTDEV}
            ;;
    "ufs")  newfs ${ROOTDEV} || {
                echo "Cannot create ufs filesystem!"
                exit 1
            } ;;
    *)      echo "Cannot create ufs filesystem!"
            exit 1
            ;;
esac

mount -F ${FS} ${ROOTDEV} /mnt || {
    echo "Cannot mount filesystem!"
    exit 1
}

#
# Install packages from Solaris installation image
#
echo "Starting installation of packages"
echo
(
    for i in ${pkgs[@]} ; do
        pkgadd -n -a /tmp/.admin.doit -d ${PROD} -R /mnt $i
    done
) > ${LOG}

#
# Update /etc/vfstab with swap and root partitions
#
(
    printf "${SWAP}\t-\t-\tswap\t-\tno\t-\n"
    printf "${ROOTDEV}\t${RAWROOTDEV}\t/\t${FS}\t1\t${MOB}\t-\n"
    [ "$FS" = "zfs" ] && {
        mkdir -m 0755 /mnt/grub
        printf "${GRUBFS}\t${GRUBFS/dsk/rdsk}\t/grub\tufs\t3\tyes\t-\n"
    }
) >> /mnt/etc/vfstab

#
# Copy links for disk partitions in /dev/dsk and /dev/rdsk
# This is needed so the system can find the root partion on boot
#
( cd /dev && find dsk rdsk -depth | cpio -pdm /mnt/dev 2>/dev/null )

#
# Configure system to initialize identity on first boot
# If there is a sysidcfg file in the current directory. This will
# be copied across.
#
PROFILEDIR=/mnt/var/svc/profile
[ -f ${PROFILEDIR}/${SVCPROFILE} ] && {
    if [ -f ./sysidcfg ]; then
        cp ./sysidcfg /mnt/etc
    else
        touch /mnt/etc/.UNCONFIGURED
    fi
    cp -p ${PROFILEDIR}/${SVCPROFILE} ${PROFILEDIR}/generic.xml
}

#
# set bootpath to root filesystem.
# Also set the console to text
#
(
    [ "${FS}" = "ufs" ] && {
        BOOTPATH=$( ls -l ${ROOTDEV} | nawk '{print $11}' |
            sed -e 's#[./]*/devices/#/#' )

        printf "setprop bootpath ${BOOTPATH}\n"
    }
    printf "setprop console 'text'\n"
) >> /mnt/boot/solaris/bootenv.rc

#
# If found execute local script before /mnt is unmounted
#
[ -x ./local_install.bash ] && ./local_install.bash

#
# Finish off installation
#
[ -f /etc/zfs/zpool.cache ] && {
    cp -p /etc/zfs/zpool.cache /mnt/etc/zfs
    echo "etc/zfs/zpool.cache" >> /mnt/boot/solaris/filelist.ramdisk
}

#
# Configure for a ZFS boot. At the moment you need a small UFS partition
# somewhere for grub.
#
[ "${FS}" = "zfs" ] && {
    (
        printf "rootfs:zfs\n"
        printf "zfsroot:${ROOTDEV}\n"
    ) >> /mnt/etc/system
}

devfsadm -r /mnt
rm -f /mnt/reconfigure
bootadm update-archive -R /mnt

[ "${FS}" = "zfs" ] && {
    cp /mnt/platform/i86pc/boot_archive ${ZFSBOOTARCHIVE}
    cp -p /mnt/sbin/bootadm /mnt/sbin/bootadm.real
    cat >/mnt/sbin/bootadm << EOM
#!/usr/bin/sh

/sbin/bootadm.real "\$@"
if [ "\$1" = "update-archive" -a -d /grub/boot/grub ]; then
        /usr/bin/cp /platform/i86pc/boot_archive ${ZFSBOOTARCHIVE}
fi
exit 0
EOM
}

echo
echo "If you have not already, you will need to configure menu.lst to"
echo "boot this partition."

umount /mnt
# eject cdrom

How small can you make Open Solaris - Part 3

In this post I have decided to focus on installing Solaris directly without using the normal Solaris installer. If you are like me and forever installing the latest Solaris Nevada version, and would like to do this without rebooting and running the Solaris installer then the script below might be for you. If you are playing around to see what packages you can remove to shrink a Solaris installation, then a modified version of this script might be what you are looking for.

Solaris already has an excellent tool for upgrading your current system called live upgrade. The script below is different, as it does a totally fresh install. Hopefully the next version will also install directly onto a ZFS filesystem rather than a UFS partition.

To start off you need a Solaris X86 system (one which uses grub) with a free Solaris partition that has enough diskspace to hold the installation. Note, this script does no checking! It assumes you have configured the settings correctly, and have allocated enough space. You also have to have a DVD (or image) of Solaris (or maybe a jumpstart type installation directory).

Going through the script, at the beginning you will find some variables to may need to change to refect your setup. Since it uses the Solaris Meta Clusters you will need to define which one you want. The functions (which is most of the code), rattle through the cluster table of contents file and produces an order list of packages for the meta cluster you selected. Once it has a list a pkgadd is executed for each package.

After the packages have been installed, there are a few system configurations and device links that are required to get the system booted correctly. You can have the system automatically configure itself on reboot if you have a valid sysidcfg file in your current directory. Otherwise, the system will ask you many questions on reboot to configure hostnames, networking etc. If you are familiar with building zones, this will not be new.

Before you start, you should read through the script so you know exactly what it does. The is very little error/sanity checks in the script, and incorrect settings could be devastating as you will need to be root (or have the correct privs) to use it. When you have finish the installation you will need to modify grub to boot off the correct partition. Below the relevent section of my /boot/grub/menu.lst file.

title Solaris Test
root (hd0,0,d)
kernel /platform/i86pc/multiboot
module /platform/i86pc/boot_archive

Enjoy, and Good Luck!!!

#!/bin/bash
#
# Quick and Dirty Solaris installer
# Version 0.1
#
PROD=/cdrom/sol_11_x86/Solaris_11/Product
TOK=${PROD}/.clustertoc
ORDER=${PROD}/.order

# Current metaclusters are
# SUNWCXall - Entire Distribution plus OEM support
# SUNWCall  - Entire Distribution
# SUNWCprog - Developer System Support
# SUNWCuser - End User System Suppor
# SUNWCreq  - Core System Support
# SUNWCrnet - Reduced Networking Core System Support
# SUNWCmreq - Minimal Core System Support
METACLUSTER=SUNWCrnet

#
# Change these to relect your system. The next version should
# use ZFS rather that UFS.
#
ROOTDEV=/dev/dsk/c0d0s4
RAWROOTDEV=/dev/rdsk/c0d0s4

SWAP=/dev/dsk/c0d0s1
LOG=/tmp/install.log

SVCPROFILE=generic_limited_net.xml

typeset -a pkgs

#
# Solaris packages need to be installed in the correct order.
# The .order file contains all the packages in the correct
# installation order
#
function reorder_pkgs() {
    typeset -a pkglist=( ${pkgs[@]} )
    pkgcnt=0

    while read order_pkg ; do
        for i in ${pkglist[@]} ; do
            [ "$order_pkg" = "${i%.i}" ] &&  {
                pkgs[$(( pkgcnt++ ))]="${i}"
                printf "."
            }
        done
    done < ${ORDER}
}

#
# This function builds a list of packages in a cluster
# If there is a cluster within a cluster, it will call itself to
# resolve all the packages.
#
# Before calling make sure you initialize pkgcnt to 0
# Arg: $1 contains the cluster name
# Affected vars: pkgs, pkgcnt
#
function get_pkg_list() {
    local IFS="="
    local print_on=0
    local cluster=$1

    while read arg1 arg2
    do
        [ "${arg1}" = "END" -a "${print_on}" = "1" ] && break;
        [ -z "${arg2}" ] && continue;

        [ "${arg2}" = "${cluster}" ] && {
            print_on=1
            continue
        }

        [ "${print_on}" = "1" -a "${arg1}" = "SUNW_CSRMEMBER" ] && {
            ifcluster=`expr "${arg2}" : '\(SUNWC\)'`
            if [ "${ifcluster}" = "SUNWC" ]; then
                get_pkg_list ${arg2}
            else
                [ -d ${PROD}/${arg2} ] && {
                        pkgs[$(( pkgcnt++ ))]="${arg2}"
                        printf "."
                        continue;
                }

                arg2="${arg2}.i"
                [ -d ${PROD}/${arg2} ] && pkgs[$(( pkgcnt++ ))]="${arg2}"
                printf "."
            fi
        }
    done < ${TOK}
}

#
# Check for the installation image before proceeding
#
[ ! -d ${PROD} ] && {
        echo "Cannot find Solaris Installation"
        exit 1
}

#
# Create a pkg admin file - see man admin(4)
#
sed 's/ask/nocheck/' /var/sadm/install/admin/default > /tmp/.admin.doit

#
# Build an ordered list of packages from the Solaris installation image
#
printf "Building a list of packages  "
pkgcnt=0
get_pkg_list ${METACLUSTER}
echo
printf "Sorting packages into the correct order for installation  "
reorder_pkgs
echo

newfs ${ROOTDEV} || exit 1
mount ${ROOTDEV} /mnt

#
# Install packages from Solaris installation image
#
echo "Starting installation of packages"
echo
(
    for i in ${pkgs[@]} ; do
        pkgadd -n -a /tmp/.admin.doit -d ${PROD} -R /mnt $i
    done
) > ${LOG}

#
# Update /etc/vfstab with swap and root partitions
#
(
    printf "${SWAP}\t-\t-\tswap\t-\tno\t-\n"
    printf "${ROOTDEV}\t${RAWROOTDEV}\t/\tufs\t1\tno\t-\n"
) >> /mnt/etc/vfstab

#
# Copy links for disk partitions in /dev/dsk and /dev/rdsk
# This is needed so the system can find the root partion on boot
#
( cd /dev && find dsk rdsk -depth | cpio -pdm /mnt/dev 2>/dev/null )

#
# Configure system to initialize identity on first boot
# If there is a sysidcfg file in the current directory. This will
# be copied across.
#
PROFILEDIR=/mnt/var/svc/profile
[ -f ${PROFILEDIR}/${SVCPROFILE} ] && {
    if [ -f ./sysidcfg ]; then
        cp ./sysidcfg /mnt/etc
    else
        touch /mnt/etc/.UNCONFIGURED
    fi
    cp -p ${PROFILEDIR}/${SVCPROFILE} ${PROFILEDIR}/generic.xml
}

#
# set bootpath to root filesystem.
# Also set the console to text
#
(
    BOOTPATH=$( ls -l ${ROOTDEV} | nawk '{print $11}' |
        sed -e 's#[./]*/devices/#/#' )

    printf "setprop bootpath ${BOOTPATH}\n"
    printf "setprop console 'text'\n"
) >> /mnt/boot/solaris/bootenv.rc

#
# If found execute local script before /mnt is unmounted
#
[ -x ./local_install.bash ] && ./local_install.bash

#
# Finish off installation
#
bootadm update-archive -R /mnt
echo "You will need to configure /boot/grub/menu.lst to boot this partition"

umount /mnt
# eject cdrom