Tag: DTrace

Linux and Solaris are Converging but Not the Way You Imagined

tux

In case you haven’t been paying attention, Linux is in a mad dash to copy everything that made Solaris 10 amazing when it launched in 2005. Everyone has recognized the power of Zones, ZFS and DTrace but licensing issues and the sheer effort required to implement the technologies has made it a long process.

ZFS

ZFS is, most probably, the most advanced file system in the world. The creators of ZFS realized, before anyone else, that file systems weren’t built to handle the amounts of data that the future would bring.

Work to port ZFS to Linux began in 2008 and a stable port of ZFS from Illumos was announced in 2013. That said, even 2 years later, the latest release still hasn’t reached feature parity with ZFS on Illumos. With developers preferring to develop OpenZFS on Illumos and the licensing issues preventing OpenZFS from being distributed as part of the Linux Kernel, it seems like ZFS on Linux (ZOL) may be doomed to playing second fiddle.

DTrace

DTrace is the most advanced tool in the world for debugging and monitoring live systems. Originally designed to help troubleshoot performance and other bugs in a live Solaris kernel, it quickly became extremely useful in debugging userland programs and run times.

Oracle has been porting DTrace since at least 2011 and while they both own the original and have prioritized the most widely used features, they still haven’t caught up to the original.

Zones

Solaris Zones are Operating System level virtual machines. They are completely isolated from each other but all running on the same kernel so there is only one operating system in memory. Zones have great integration with ZFS, DTrace, and all the standard system monitoring tools which makes it very easy to support and manage servers with hundreds of Zones running on them. Zones also natively support a mechanism called branding which allows the kernel to provide different interfaces to the guest zone. In Oracle Solaris, this is used to support running zones from older versions of Solaris on a machine running a newer OS.

Linux containers of some type or another have been around for a while, but haven’t gotten nearly as mature as Zones. Recently, the continued failure of traditional hypervisors to provide bare metal performance in the cloud, coupled with the uptake of Docker, has finally gotten the world to realize the tremendous benefits of container based virtualization like Zones.

The current state of containers in Linux is extremely fractured with at least 5 competing projects that I know of. LXC, initially released in 2008, seems to be the favorite but historically had serious privilege separation issues but has gotten a little better if you can meet all the system requirements.

Joyent has been waiting at the finish line.

While Linux users wait and wait for mature container solutions, full OS and application visibility, and a reliable and high performance file system, Joyent has been waiting to make things a whole lot easier.

About a year ago, David Mackay showed some interest in Linux Branded Zones, work which had been abandoned in Illumos. In the spring of 2014, Joyent started work on resurrecting lx-zones and in September, they presented their work. They already have working support for 32 bit and some 64 bit Linux binaries in Linux branded SmartOS Zones. As part of the process, they are porting some of the main Linux libraries and facilities to native SmartOS which will make porting Linux code to SmartOS much easier.

The upshot of it is that you can already get ZFS, Dtrace, and Linux apps inside a fully isolated, high performance, SmartOS zone. With only 9 months or so of work behind it, there are still some missing pieces to the Linux support, but, considering how long Linux has been waiting, I’m pretty sure SmartOS will reach feature parity with Linux a lot faster than Linux will reach feature parity with SmartOS.

Network Interface Utilization in Solaris

A friend asked me how he could see the network utilization in Solaris. It seems like a fairly simple request but for some reason this is not a simple command line away.

In Linux I would instinctively go straight to iptraf. I don’t know if iptraf is the tool of choice these days but I’m pretty sure it is an apt-get away if not already installed.

If you are a DTrace wizard, you could whip something up. Maybe you could get the information from one of the of the DTraceToolkit scripts if their installed. The DTraceToolkit scripts I’ve seen seem to give too much information as most of them are concentrated on not only telling you if the network is loaded but what is loading it as well.

For the sake of practice I wrote the following script:

#!/usr/bin/perl -w
print "Interface: ";
$if=<>;
chomp($if);
$max=`dladm show-dev -p $if | awk -F= '{print \$3}' | awk '{print \$1*1024*1024/8}'`;
print "Max speed: ",$max,"\n";
$if=~m/([a-z0-9]+?)(\d+)/;
($module,$instance)=($1,$2);
$last_rbytes=0;
$last_obytes=0;
while(1){
@kstat=`kstat ${module}:${instance}:mac:/[or]bytes\$/ |awk '{print \$2}'`;
chomp(@kstat);
if($last_rbytes!=0){
printf("%02d%%\n",
(($kstat[$#kstat-1]-$last_rbytes)+
($kstat[$#kstat-2]-$last_obytes))/$max*100);
}
$last_rbytes=$kstat[$#kstat-1];
$last_obytes=$kstat[$#kstat-2];
sleep 1;
};

This script will ask you which interface you want to watch and then print out the utilization percentage on a new row every ~second.

On a side note, it seems strange to me the the received bytes are stored in kstat as rbytes while the transmitted bytes are stored in obytes. The only answer I can come up with is that if they would have chosen ibytes (in bytes) instead of rbytes, then the ‘i’ and ‘o’ might become interchanged in typos since they are next to each other on the keyboard. If they would have chosen tbytes (transmitted bytes), the same situation occurs- ‘r’ next to ‘t’. Still, as a friend pointed out, they could have used sbytes (sent bytes) which makes more sense than obytes.

Top on Solaris

Recently, I was asked to give some advice on an integration project involving some Solaris web servers . One of the sides requested to install the top command.
Now I know and love top for Linux but using top on Solaris is a waste in my opinion. Solaris comes with the prstat command built in- why use something else?
Of course he answered that top was standard for him and he was used to it but I felt obliged to convince him otherwise so I dug around and found some proof 🙂

Brendan Gregg wrote up a great piece comparing top vs prstat using dtrace on his website:
http://www.brendangregg.com/DTrace/prstatvstop.html.

In summary, he finds the following:

  • Top uses more system calls than prstat
  • Top opens and closes the psinfo file over and over while prstat only open it once and saves the file handle
  • Top takes more cpu time to do its job than prstat due to the overhead in the extra system calls and code differences
  • When top uses the cpu it uses it for longer than prstat
  • Most of the issues top has compared to prstat are connected to the number of processes running on the server so the more processes running, the worse top will perform compared to prstat

Aside from the performance issues, prstat also has the ability to give you project and zone related information which I doubt top knows about.

In short top is great for Linux but if you are going to use Solaris, use prstat!