Seeing Katsumi
Inoue blogging about Oracle 10g reporting timestamps using the
output from gethrtime() reminded me that I have had on occasion
wished I had a log to map hrtime to the current time. As Katsumi
points out the output of gethrtime() is not absolutely tied to the
current time. So there is no way to take the output from it and tell
when in real time the output was generated unless you have some
reference point. To make things more complex the output is reset each
time the system reboots.

For this reason it is useful to keep a file that contains a
history of the hrtime and the real time so that any logs can be
retrospectively coerced back into a readable format.

There are lots of ways to do this but since on this blog we seem
to be in Dtrace mode here is how using dtrace

pfexec /usr/sbin/dtrace -o /var/log/hrtime.log -qn ‘BEGIN,tick-1hour,END {

printf("%d:%d.%9.9d:%Y\n",

timestamp, walltimestamp/1000000000,

walltimestamp%1000000000, walltimestamp);

}’

Then you get a nice file that contains
three columns. The hrtime, the time in seconds since January 1^st
1970 and a human readable representation of the time in the current
timezone:

: s4u-10-gmp03.eu TS 39 $; cat /var/log/hrtime.log

5638545510919736:1248443226.350000625:2009 Jul 24 14:47:06

5642145449325180:1248446826.279995332:2009 Jul 24 15:47:06

I have to confess however that using Dtrace for this does not feel
right, not least as you need to be root for this to be reliable and
also the C code is trivial to write, compile and run from cron and
send the output to syslog:

: exdev.eu FSS 39 $; cat  ./gethrtime_base.c

#include <sys/time.h>

#include <stdio.h>

int

main(int argc, char **argv)

{

hrtime_t hrt = gethrtime();

struct timeval tv;

gettimeofday(&tv, NULL);

printf("%lld:%d.%6.6d:%s", hrt, tv.tv_sec, tv.tv_usec,

ctime(&tv.tv_sec));

}

: exdev.eu FSS 40 $; make ./gethrtime_base

cc    -o gethrtime_base gethrtime_base.c

: exdev.eu FSS 41 $;  ./gethrtime_base

11013365852133078:1248444379.163215:Fri Jul 24 15:06:19 2009

: exdev.eu FSS 42 $;

./gethrtime_base | logger -p daemon.notice -t hrtime

: exdev.eu FSS 43 $;  tail -10 /var/adm/messages | grep hrtime

Jul 24 15:32:33 exdev hrtime: [ID 702911 daemon.notice] 11014939896174861:1248445953.109855:Fri Jul 24 15:32:33 2009

Jul 24 16:09:21 exdev hrtime: [ID 702911 daemon.notice] 11017148054584749:1248448161.131675:Fri Jul 24 16:09:21 2009

: exdev.eu FSS 50 $;

As I cycled home I realised there was one more thing I could do on
the exploring the limits of threads and processes on Solaris. That
would be the highest load average ever. Modifying the thread creator
program to not have each thread sleep once started but instead wait
until all the threads were set up and then go into an infinite
compute loop that should get me the highest load average possible on
a system or so you would think.

With 784001 threads the load stabilised
at:

10:16am  up 18:07,  2 users,  load average: 22114.50, 22022.68, 21245.781

Which was somewhat disappointing. However an earlier run with just
780,000 threads managed to peak the load at 1,784,593 while it was
exiting:

 7:44am  up 15:35,  2 users,  load average: 1724593.79, 477392.80, 188985.10

I’ still pondering how 780000 thread can result in a load average of
more than 1 million.

After the surprise interest in the maximum
number of processes on a system it seems rude not to try and see
how many threads I can squeeze into a single process while I have
access to a system where physical memory will not be the limiting
factor. The expectation is that this will closely match the number of
processes as each thread will have an LWP in the kernel which will in
turn consume the segkp.

A slight modification to the forker program:

: exdev.eu FSS 62 $; cat thr_creater.c

#include <stdio.h>

#include <stdlib.h>

#include <sys/types.h>

#include <unistd.h>

#include <thread.h>

int

main(int argc, char **argv)

{

pid_t pid;

int count=0;

while(count < (argc != 2 ? 100 : atoi(argv[1])) &&

(pid = thr_create(NULL, 0, (void * (*)(void *))pause,

NULL, THR_DETACHED, NULL)) != -1) {

if (pid == 0 ) {

/* Success, ) */

if (count % 1000 == 0)

printf("%d\n", count);

count++;

}

}

if (pid < 0)

perror("fork");

printf("%d\n", count);

pause();

}

and this time it has to be built as a 64 bit program:

# make "CFLAGS=-m64 -mt" thr_creater

#

Here is how it went:

$; ./thr_creater 1000000

01000

2000

3000

4000

5000

6000

7000

8000

9000

10000

11000

12000

13000

14000

15000

…..

782000

783000

784000

Here things have stopped and for some bizarre reason attaching a
debugger to see what is going on does not seem to be a good idea. I
had prstat running in another window and it reported:

  2336 cg13442  7158M 7157M cpu73    0    0   1:42:59 1.6% thr_creater/784970

Which is just a few more threads than I got processes (784956) when
running in multi user. However at this point the system is pretty
much a warm brick as if I exit any process thr_creater hoovers up the
process so I can create no more. Fortunately I had realized this
would happen and had some sleep(1) processes running so I could pause
the thr_creater and then kill one of the sleeps to allow me to run a
command:

$; ps -o pid,vsz,rss,nlwp,comm -p 2336

PID  VSZ  RSS NLWP COMMAND

2336 7329704 7329248 784972 ./thr_creater

as you can see it managed to get another two threads created since
the prstat exited.

I’ve just straightened the rear wheel on the bike of a friend of
my daughter and while I was at it checked over the rest of the bike.
This leads to my top tip for parents of children who cycle (which IMO
should be every parent of an able bodied child).

Before you start worrying about whether your child is or is not
wearing a helmet¹
make sure that the brakes on their bike work. Not just when the bike
is new but regularly.

If you have recently come into possession of a Laptop onto which
to load Solaris then here are my top tips:

1. Install OpenSolaris.
   At the time of writing the release is 2009.06, install that, parts
   of this advice may become obsolete with later releases. Do not
   install Solaris 10 or even worse Nevada. You should download the
   live CD and burn it onto a disk boot that and let it install but
   before you start the install read the next tip.

   
   

2. Before you start the install open a terminal so that you can
   turn on compression on the root pool once it it created. You have to
   keep running “zpool list” until you see the pool is
   created and then run (pfexec zfs set compression=on rpool). You may
   think that disk is big but after a few months you will be needing
   every block you can get. Also laptop drives are so slow that
   compression will probably make things faster.

   
   

3. Before you do anything after installation take a snapshot of
   the system so you can always go back (pfexec beadm create
   opensolaris@initialinstall).
   I really mean this.

   
   

4. Add the extras repository. It contains virtualbox, the flash
   plugin for firefox, true type fonts and more. All you need is a sun
   online account. See https://pkg.sun.com/register/
   and
   http://blogs.sun.com/chrisg/entry/installing_support_certificates_in_opensolaris
   

   
   

5. Decide whether you want to use the development or support
   repository. If in doubt choose the supported one. Sun employees get
   access to the support repository. Customers need to get a support
   contract. (http://www.opensolaris.com/learn/subscriptions/).
   Then update to the latest bigs (pfexec pkg image-update).

   
   

6. Add any extra packages you need. Since I am now writing this
   retrospectively there may be things missing. My starting list is:

   
   
   
   

   • OpenOffice (pfexec pkg install openoffice)

     
     

   • SunStudio (pfexec pkg install sunstudioexpress)

     
     

   • Netbeans (pfexec pkg install netbeans)

     
     

   • Flash (pkfexec pkg install flash)

     
     

   • Virtualbox (pfexec pkg install virtualbox)

     
     

   • TrueType fonts (pfxec pkg install ttf-fonts-core)

     
     
   
   

7. If you are a Sun Employee install the punchin packages so you
   can access SWAN. I actually rarely use this as I have a Solaris 10
   virtualbox image that I use for punchin so I can be both on and off
   SWAN at the same time but it is good to have the option.

   
   

8. Add you keys to firefox so that you can browse the extras and
   support repositories from firefox. See
   http://wikis.sun.com/display/OpenSolarisInfo200906/How+to+Browse+the+Support+and+Extra+Repositories.

   
   

9. Go to Fluendo
   and get and install the free mp3 decoder. They also sell a complete
   and legal set of decoders for the major video formats, I have them
   and have been very happy with them. They allow me to view the videos
   I have cycling events.

   
   

10. Got to Adobe and
    get acroread. I live in hope that at some point this will be in a
    repository either at Sun or one Adobe runs so that it can be
    installed using the standard pkg commands but until then do it by
    hand.

    
    

Enjoy.

This week we had a customer claiming that they were unable to
create more then 60,000 processes. This turned out to be due to them
tuning max_nproc, maxuprc and maxpid but not setting segkpsize
so the system would run out of “memory” before it ran
into the resource limits for process.

Tuning segkpsize to 8G resolved it but I just had to see how many
processes I could get running on an M8000.

Using these settings in /etc/system:

set segkpsize=0×300000

set pidmax=999999

set maxuprc=999990

set max_nprocs=999999

and a simple forker program:

#include <stdio.h>

#include <stdlib.h>

#include <sys/types.h>

#include <unistd.h>

int

main(int argc, char **argv)

{

pid_t pid;

int count=0;

while(count < argc == 2 ? 100 : atoi(argv[1]) &&

(pid = fork()) != -1) {

if (pid != 0 ) {

/* Parent */

if (count % 1000 == 0)

printf("%d\n", count);

count++;

} else {

pause();

exit(0);

}

}

if (pid < 0)

perror("fork");

printf("%d\n", count);

}

I was slightly disappointed at the result:

$ ./forker 100000

1000

2000

3000

…..

782000

783000

784000

fork: Resource temporarily unavailable

784956

$

Only 784956 processes, plus the ones already running when the system
booted. Trying to count them with ps obviously fails but mdb gives
the real count.

# ps -e| wc

ksh: cannot fork: too many processes

#

# echo nproc::print -d | mdb -k

0t785025

# 

Someone must have managed to get more.

In case anyone wanted to cycle from Farnborough Main railway
station to the Sun Microsystems offices at Guillemont Park here is
the route I take.

View Larger Map

I’ve been working with a customer to try and find a memory “leak”
in their application. Many things have been tried, libumem, and the
mdb ::findleaks command all with no success.

So I was, as I am sure others before me have, pondering if you
could use dtrace to do this. Well I think you can. I have a script
that puts probes into malloc et al and counts how often they are
called by this thread and when they are freed often free is called.

Then in the entry probe of the target application note away how
many calls there have been to the allocators and how many to free and
with a bit of care realloc. Then in the return probe compare the
number of calls to allocate and free with the saved values and
aggregate the results. The principle is that you find the routines
that are resulting in allocations that they don’t clear up. This
should give you a list of functions that are possible leakers which
you can then investigate¹.

Using the same technique I for getting dtrace to “follow
fork” that I described
here I ran this up on diskomizer, a program that I understand
well and I’m reasonably sure does not have systemic memory leaks. The
dtrace script reports three sets of results.

1. A count of how many times each routine and it’s descendents
   have called a memory allocator.

   
   

2. A count of how many times each routine and it’s descendents
   have called free or realloc with a non NULL pointer as the first
   argument.

   
   

3. The difference between the two numbers above.

   
   

Then with a little bit of nawk to remove all the functions for
which the counts are zero gives:

# /usr/sbin/dtrace -Z -wD TARGET_OBJ=diskomizer2 -o /tmp/out-us \

-s /tmp/followfork.d \

-Cs /tmp/allocated.d -c \

"/opt/SUNWstc-diskomizer/bin/sparcv9/diskomizer -f /devs -f background \

-o background=0 -o SECONDS_TO_RUN=1800"

dtrace: failed to compile script /tmp/allocated.d: line 20: failed to create entry probe for ‘realloc’: No such process

dtrace: buffer size lowered to 25m

dtrace: buffer size lowered to 25m

dtrace: buffer size lowered to 25m

dtrace: buffer size lowered to 25m

# nawk ‘$1 != 0 { print  $0 }’ < /tmp/out.3081

allocations

1 diskomizer`do_dev_control

1 diskomizer`set_dev_state

1 diskomizer`set_state

3 diskomizer`report_exit_reason

6 diskomizer`alloc_time_str

6 diskomizer`alloc_time_str_fmt

6 diskomizer`update_aio_read_stats

7 diskomizer`cancel_all_io

9 diskomizer`update_aio_write_stats

13 diskomizer`cleanup

15 diskomizer`update_aio_time_stats

15 diskomizer`update_time_stats

80 diskomizer`my_calloc

240 diskomizer`init_read

318 diskomizer`do_restart_stopped_devices

318 diskomizer`start_io

449 diskomizer`handle_write

606 diskomizer`do_new_write

2125 diskomizer`handle_read_then_write

2561 diskomizer`init_buf

2561 diskomizer`set_io_len

58491 diskomizer`handle_read

66255 diskomizer`handle_write_then_read

124888 diskomizer`init_read_buf

124897 diskomizer`do_new_read

127460 diskomizer`expect_signal

freecount

1 diskomizer`expect_signal

3 diskomizer`report_exit_reason

4 diskomizer`close_and_free_paths

6 diskomizer`update_aio_read_stats

9 diskomizer`update_aio_write_stats

11 diskomizer`cancel_all_io

15 diskomizer`update_aio_time_stats

15 diskomizer`update_time_stats

17 diskomizer`cleanup

160 diskomizer`init_read

318 diskomizer`do_restart_stopped_devices

318 diskomizer`start_io

442 diskomizer`handle_write

599 diskomizer`do_new_write

2125 diskomizer`handle_read_then_write

2560 diskomizer`init_buf

2560 diskomizer`set_io_len

58491 diskomizer`handle_read

66246 diskomizer`handle_write_then_read

124888 diskomizer`do_new_read

124888 diskomizer`init_read_buf

127448 diskomizer`cancel_expected_signal

mismatch_count

-127448 diskomizer`cancel_expected_signal

-4 diskomizer`cancel_all_io

-4 diskomizer`cleanup

-4 diskomizer`close_and_free_paths

1 diskomizer`do_dev_control

1 diskomizer`init_buf

1 diskomizer`set_dev_state

1 diskomizer`set_io_len

1 diskomizer`set_state

6 diskomizer`alloc_time_str

6 diskomizer`alloc_time_str_fmt

7 diskomizer`do_new_write

7 diskomizer`handle_write

9 diskomizer`do_new_read

9 diskomizer`handle_write_then_read

80 diskomizer`init_read

80 diskomizer`my_calloc

127459 diskomizer`expect_signal

#

From the above you can see that there are two functions that create
and free the majority of the allocations and the allocations almost
match each other, which is expected as they are effectively
constructor and destructor for each other. The small mismatch is not
unexpected in this context.

However it is the vast number of
functions that are not listed at all as they and their children make
no calls to the memory allocator or have exactly matching allocation
and free that are important here. Those are the functions that we
have just ruled out.

From here it is easy now to drill down
on the functions that are interesting you, ie the ones where there
are unbalanced allocations.

I’ve uploaded the files allocated.d
and followfork.d
so you can see the details. If you find it useful then let me know.

There is a ongoing request to have follow fork functionality for
the dtrace pid provider but so far no one has stood upto the plate
for that RFE. In the mean time my best workaround for this is this:

cjg@brompton:~/lang/d$ cat followfork.d

proc:::start

/ppid == $target/

{

stop();

printf("fork %d\n", pid);

system("dtrace -qs child.d -p %d", pid);

}

cjg@brompton:~/lang/d$ cat child.d

pid$target::malloc:entry

{

printf("%d %s:%s %d\n", pid, probefunc, probename, ustackdepth)

}

cjg@brompton:~/lang/d$ pfexec /usr/sbin/dtrace -qws followfork.d -s child.d -p 26758

26758 malloc:entry 22

26758 malloc:entry 15

26758 malloc:entry 18

26758 malloc:entry 18

26758 malloc:entry 18

fork 27548

27548 malloc:entry 7

27548 malloc:entry 7

27548 malloc:entry 18

27548 malloc:entry 16

27548 malloc:entry 18

Clearly you can have the child script do what ever you wish.

Better solutions are welcome!

Why a fixie¹?

A few people have asked
me this so here are the reasons for a fixie:

• They are alleged to improve you pedaling.

  
  

• They are supposed to make your legs stronger.

  
  

• People say you are more in touch with the bike on a fixie

  
  

• People say they are fun to ride. Quite why is hard to
  understand why would this be significantly different from just
  picking a gear and sticking to it. For a single speed bike with a
  free wheel I would agree except a single speed there is no way to
  give in up hills without getting off.

  
  

My additional reasons were:

• I had noticed I generally ride in 3 gears during the winter
  so wondered if I could make it on a single speed.

  
  

• Bike to work made it very affordable.

  
  

• I wanted one.

  
  

Now having one I agree they are
fun more fun than I ever expected and even though I don’t think I
have mastered it yet I do understand about it being in touch with the
bike. Going up hill there is nowhere to hide I don’t know if it is
making me stronger but it feels like it.

It certainly has improved my
ability to “spin”.