Core debugging in OpenSolaris

I use OpenSolaris nearly exclusively since snv_131 or for about six weeks now. Having used Linux for years, its quite a new world for me, although its similar. After my gnome-panel crashes from time to time I looked around how to find out where exactly it crashes. Under Linux you would have used the backtrace function in gdb, but OpenSolaris uses Sun Studio compilers and is different. I stumbled upon a couple of articles how to do this.

There seems to be three standard ways.

Using pstack

$ pstack core
core 'core' of 3853: gnome-panel --sm-config-prefix /gnome-panel-gkaWKj/ --sm-client-id 10f
fc6c2d84 ???????? (80476a0, 0, 80476c8, fed1395a)
fed139c0 _gdk_events_queue (80fd000) + 74
fed13b92 gdk_event_dispatch (8105ed8, 0, 0, fe754ddc) + 2e
fe754fa2 g_main_context_dispatch (8105f20, 0, 85bc6d8, 17) + 262
fe755653 g_main_context_iterate (8105f20, 1, 1, 80de700) + 483
fe755c7d g_main_loop_run (81ba250, 81ba250, 8047888, fe97f466) + 1dd
fe97f50f gtk_main (80478a0, feffb804, 80478d8, 8074b9d, 7, 80478e4) + b7
0807902e main (7, 80478e4, 8047904, 8074b3f) + 15a
08074b9d _start (7, 8047a14, 8047a20, 8047a33, 8047a48, 8047a57) + 7d

You can see its somewhere in _gdk_events_queue() at address fc6c2d84. The interesting thing on pstack is, that you can use this for running processes, too.

Using mdb

$ mdb core
Loading modules: [ libuutil.so.1 ld.so.1 ]
> ::status
debugging core file of gnome-panel (32-bit) from stars
file: /usr/bin/gnome-panel
initial argv: gnome-panel --sm-config-prefix /gnome-panel-gkaWKj/ --sm-client-id 10fb8f7f3338
threading model: native threads
status: process terminated by SIGSEGV (Segmentation Fault), addr=fc6c2d84
> $c1 ! head -10
0xfc6c2d84(80476a0)
libgdk-x11-2.0.so.0.1800.5`_gdk_events_queue+0x74(80fd000)
libgdk-x11-2.0.so.0.1800.5`gdk_event_dispatch+0x2e(8105ed8)
libglib-2.0.so.0.2200.3`g_main_context_dispatch+0x262(8105f20)
libglib-2.0.so.0.2200.3`g_main_context_iterate+0x483(8105f20)
libglib-2.0.so.0.2200.3`g_main_loop_run+0x1dd(81ba250)
libgtk-x11-2.0.so.0.1800.5`gtk_main+0xb7(80478a0)
main+0x15a(7)
_start+0x7d(7)

Doesn't tell you more for real, just you can see the library being called, too. Unlike pstack you are now in a real debugger. So you can investigate a bit more, although not on source level, as mdb is just an assembly level debugger.

> 80476a0::dis
0x80476a0: andl %eax,(%eax)
0x80476a2: addb %al,(%eax)
0x80476a4: leave
0x80476a5: decl %edi
0x80476a6: addb %al,(%eax)
0x80476a8: addl %eax,(%eax)
0x80476aa: addb %al,(%eax)
0x80476ac: incl %eax
0x80476ad: orl $0x20069c08,(%edi)
0x80476b3: addl %esi,0x3(%ebp)
0x80476b6: addb %al,(%eax)
> 

So you see where it crashed on assembly level and can even get the values of registers at crash time. Sadly I'm not familiar with assembly anymore - I knew a bit 8086 assembly on MS-DOS around 15 years ago - so I can't give further information on that topic. There is an extensive guide on mdb and some blog entries as well.

Using xdb

As I stated before mdb is just an assembly level debugger. When you are used to gdb, you will miss a source level debugger. There is one for Sun Studio as well: xdb

$ dbx -c 'where;quit' - core
Corefile specified executable: "/usr/bin/gnome-panel"
Reading gnome-panel
core file header read successfully
Reading ld.so.1
Reading libc.so.1
...
Reading libpixbufloader-xpm.so
t@1 (l@1) program terminated by signal SEGV (no mapping at the fault address)
0xffffffffffffffff:
current thread: t@1
[1] 0xfc6c2d84(0x80f8140, 0x120069c, 0x80476a0, 0x0), at 0xfc6c2d84
[2] XFilterEvent(0x80476a0, 0x0, 0x80476c8, 0xfed1395a), at 0xfe47ad91
[3] _gdk_events_queue(0x80fd000), at 0xfed139c0
[4] gdk_event_dispatch(0x8105ed8, 0x0, 0x0, 0xfe754ddc), at 0xfed13b92
[5] g_main_context_dispatch(0x8105f20, 0x0, 0x85bc6d8, 0x17), at 0xfe754fa2
[6] g_main_context_iterate(0x8105f20, 0x1, 0x1, 0x80de700), at 0xfe755653
[7] g_main_loop_run(0x81ba250, 0x81ba250, 0x8047888, 0xfe97f466), at
0xfe755c7d
[8] gtk_main(0x80478a0, 0xfeffb804, 0x80478d8, 0x8074b9d, 0x7, 0x80478e4), at
0xfe97f50f
[9] main(0x7, 0x80478e4, 0x8047904, 0x8074b3f), at 0x807902e

And as you can see, it even mapped one function deeper: XFilterEvent().

Conclusion

The tools on OpenSolaris are different, but not necessarily worse than Linux or GNU tools. This blog entry is just a starting point and a note to myself how to use them. Feel free to comment and supply more useful information.

Syndicate content