Thread Analysis
  Home
  Introduction
  Getting KGDB
  Documentation
  Using KGDB
  Credits
  Miscellaneous
  Troubleshooting
  FAQ
  Copyright
  About KGDB
  GEEks of Pune (GEEP)

 

GDB features analysis of application threads. GDB provides a listing of threads created by an application program. It then allows a developer to look into any of those threads. This GDB feature can be used with KGDB to look into kernel threads. GDB can provide a listing of all the threads in a kernel.  A developer can specify a particular thread to be analyzed. GDB commands like backtrace,info regi then show the information in context of the specified thread.

All threads created by an application share the same address space. Similarly all kernel threads share kernel address space. User address space for each kernel thread may be different. Hence gdb thread applies well to analysis of kernel code and data structures that reside in kernel address space.

gdb info pages give more information on using GDB thread analysis feature. An example of kernel thread analysis is shown below:

The gdb command info threads gives a listing of kernel threads.

(gdb) info thr
  21 thread 516  schedule_timeout (timeout=2147483647) at sched.c:411
  20 thread 515  schedule_timeout (timeout=2147483647) at sched.c:411
  19 thread 514  schedule_timeout (timeout=2147483647) at sched.c:411
  18 thread 513  schedule_timeout (timeout=2147483647) at sched.c:411
  17 thread 512  schedule_timeout (timeout=2147483647) at sched.c:411
  16 thread 511  schedule_timeout (timeout=2147483647) at sched.c:411
  15 thread 438  schedule_timeout (timeout=2147483647) at sched.c:411
  14 thread 420  schedule_timeout (timeout=-1013981316) at sched.c:439
  13 thread 406  schedule_timeout (timeout=-1013629060) at sched.c:439
  12 thread 392  do_syslog (type=2, buf=0x804dc20 "run/utmp", len=4095)
    at printk.c:182
  11 thread 383  schedule_timeout (timeout=2147483647) at sched.c:411
  10 thread 328  schedule_timeout (timeout=2147483647) at sched.c:411
  9 thread 270  schedule_timeout (timeout=-1011908724) at sched.c:439
  8 thread 8  interruptible_sleep_on (q=0xc02c8848) at sched.c:814
  7 thread 6  schedule_timeout (timeout=-1055490112) at sched.c:439
  6 thread 5  interruptible_sleep_on (q=0xc02b74b4) at sched.c:814
  5 thread 4  kswapd (unused=0x0) at vmscan.c:736
  4 thread 3  ksoftirqd (__bind_cpu=0x0) at softirq.c:387
  3 thread 2  context_thread (startup=0xc02e93c8) at context.c:101
  2 thread 1  schedule_timeout (timeout=-1055703292) at sched.c:439
* 1 thread 0  breakpoint () at gdbstub.c:1159
(gdb)

gdb assigns its own id to each thread as shown above. When referring to a thread inside gdb, this id is used. For example thread 7 (PID 7) has gdb id 8. To analyze the kernel thread 8, we specify thread 9 to gdb. gdb then switches to this thread for further analysis. Further commands like backtrace apply to this thread.

(gdb) thr 9
[Switching to thread 9 (thread 270)]
#0  schedule_timeout (timeout=-1011908724) at sched.c:439
439             del_timer_sync(&timer);
(gdb) bt
#0  schedule_timeout (timeout=-1011908724) at sched.c:439
#1  0xc0113f36 in interruptible_sleep_on_timeout (q=0xc11601f0, timeout=134)
    at sched.c:824
#2  0xc019e77c in rtl8139_thread (data=0xc1160000) at 8139too.c:1559
#3  0xc010564b in kernel_thread (fn=0x70617773, arg=0x6361635f,
    flags=1767859560) at process.c:491
#4  0x19 in uhci_hcd_cleanup () at uhci.c:3052
#5  0x313330 in ?? () at af_packet.c:1891
Cannot access memory at address 0x31494350
(gdb) info regi
eax            0xc38fdf7c       -1013981316
ecx            0x86     134
edx            0xc0339f9c       -1070358628
ebx            0x40f13  266003
esp            0xc3af7f74       0xc3af7f74
ebp            0xc3af7fa0       0xc3af7fa0
esi            0xc3af7f8c       -1011908724
edi            0xc3af7fbc       -1011908676
eip            0xc011346d       0xc011346d
eflags         0x86     134
cs             0x10     16
ss             0x18     24
ds             0x18     24
es             0x18     24
fs             0xffff   65535
gs             0xffff   65535
fctrl          0x0      0
fstat          0x0      0
ftag           0x0      0
fiseg          0x0      0
fioff          0x0      0
foseg          0x0      0
fooff          0x0      0
---Type <return> to continue, or q <return> to quit---
fop            0x0      0
(gdb) thr 7
[Switching to thread 7 (thread 6)]
#0  schedule_timeout (timeout=-1055490112) at sched.c:439
439             del_timer_sync(&timer);
(gdb) bt
#0  schedule_timeout (timeout=-1055490112) at sched.c:439
#1  0xc0137ef2 in kupdate (startup=0xc02e9408) at buffer.c:2826
#2  0xc010564b in kernel_thread (fn=0xc3843a64, arg=0xc3843a68,
    flags=3280222828) at process.c:491
#3  0xc3843a60 in ?? ()
Cannot access memory at address 0x1f4
(gdb)

Process information macros ps and psname available from the downloads page are useful for thread analysis. The ps macro provides a names and ids of threads running in a kernel.
(gdb) ps
0 swapper
1 init
2 keventd
3 ksoftirqd_CPU0
4 kswapd
5 bdflush
6 kupdated
8 khubd
270 eth0
328 portmap
383 syslogd
392 klogd
406 atd
420 crond
438 inetd
511 mingetty
512 mingetty
513 mingetty
514 mingetty
515 mingetty
516 mingetty
(gdb)
 

The psname macro can be used to get name of a thread when it's id is known.
(gdb) psname 8
8 khubd
(gdb) psname 7
(gdb)

Caveats in using gdb thread model with kernel threads:

  • GDB may require a really large amount of time to show thread listing on a slow serial line. It can go even over few tens of seconds if the number of threads is over hundred.
  • GDB assumes that thread ids always increase. GDB queries KGDB for threads that have been created with ids greater than the largest thread id in previous thread listing. Because thread ids wrap around, this condition may be violated in case of a heavily loaded which has been running for a long time. If you think that the test kernel you are using may have wrapped around thread ids, it's necessary to quit from GDB and restart it when you want to get a thread listing. This gets around the problem of GDB remembering the largest thread id from previous thread listing.