Information printed in a GDB backtrace is usually sufficient to find out what a function call hierarchy is at the point where the execution stopped. It may not be enough when one of the stack frames is inside an expanded inline function. Since inline functions are expanded inline, if execution stopped inside an inline function, or if a call to an inner function was made from an inline function, GDB shows source code file name and line number of the statement in the inline function. It may be possible to know which function the inline function was called from and where it was called by looking at the outer function. If the inline function was called two times, or if it is not possible to know which function the inline function was called from, following procedure can be used to find out this information.

gdb also shows code addresses alongwith function names in a backtrace. The statement where an inline function was called can be found out from these code addresses. The disasfun.sh script can be used here to disassemble with source code references a kernel function from the vmlinux file. The vmlinux file contains absolute addresses of kernel functions, hence the addresses seen in the assembly code are the addresses in memory. An example of how this can be done is given below.

KGDB thread analysis (CONFIG_KGDB_THREAD) is enabled while configuring the kernel. GDB is connected to the target kernel.

Let's break it using Ctrl+C, place a breakpoint in function __break and continue.

Program received signal SIGTRAP, Trace/breakpoint trap.
breakpoint () at gdbstub.c:1160
1160    }

(gdb) break __down
Breakpoint 1 at 0xc0105a43: file semaphore.c, line 62.
(gdb) c
Continuing.

To hit the breakpoint, run man lilo on the target machine. The breakpoint will be hit and gdb will go in command mode.

Breakpoint 1, __down (sem=0xc7393f90) at semaphore.c:62
62              add_wait_queue_exclusive(&sem->wait, &wait);
(gdb) backtrace
#0  __down (sem=0xc7393f90) at semaphore.c:62
#1  0xc0105c70 in __down_failed () at af_packet.c:1878
#2  0xc011433b in do_fork (clone_flags=16657, stack_start=3221199556,
    regs=0xc7393fc4, stack_size=0)
    at /mnt/work/build/old-pc/linux-2.4.6-kgdb/include/asm/semaphore.h:120
#3  0xc010594b in sys_vfork (regs={ebx = 1074823660, ecx = 1074180970,
      edx = 1074823660, esi = -1073767732, edi = 134744856, ebp = -1073767712,
      eax = 190, xds = 43, xes = 43, orig_eax = 190, eip = 1074437320,
      xcs = 35, eflags = 518, esp = -1073767740, xss = 43}) at process.c:719

The line number in function sys_vfork is correctly shown 719 in file process.c We can confirm that by listing the code around this line number.

(gdb) list process.c:719
714      * do not have enough call-clobbered registers to hold all
715      * the information you need.
716      */
717     asmlinkage int sys_vfork(struct pt_regs regs)
718     {
719             return do_fork(CLONE_VFORK | CLONE_VM | SIGCHLD, regs.esp, &regs, 0);
720     }
721
722     /*
723      * sys_execve() executes a new program.

As shown by gdb, do_fork function is called from sys_vfork function. Let's consider frame 2 in the stack trace. gdb shows that it's on line number 120 in file semaphore.h This is correct, though not very useful.

(gdb) list semaphore.h:118
113      */
114     static inline void down(struct semaphore * sem)
115     {
116     #if WAITQUEUE_DEBUG
117             CHECK_MAGIC(sem->__magic);
118     #endif
119
120             __asm__ __volatile__(                                              <-----
121                     "# atomic down operation\n\t"
122                     LOCK "decl %0\n\t"     /* --sem->count */

The only information we get is that it's inside an inline expanded down function in do_fork at the statement indicated by an arrow. gdb has also printed the absolute address of the code in do_fork from next function was called: 0xc011433b. Here we use the disasfun script to find which line of code this address corresponds to. Part  of the output of the command  disasfun vmlinux do_fork is shown below:

        if ((clone_flags & CLONE_VFORK) && (retval > 0))
c011431d:       8b 7d 08                mov    0x8(%ebp),%edi
c0114320:       f7 c7 00 40 00 00       test   $0x4000,%edi
c0114326:       74 13                   je     c011433b <do_fork+0x707>
c0114328:       83 7d d4 00             cmpl   $0x0,0xffffffd4(%ebp)
c011432c:       7e 0d                   jle    c011433b <do_fork+0x707>
#if WAITQUEUE_DEBUG
        CHECK_MAGIC(sem->__magic);
#endif

        __asm__ __volatile__(
c011432e:       8b 4d d0                mov    0xffffffd0(%ebp),%ecx
c0114331:       f0 ff 4d ec             lock decl 0xffffffec(%ebp)
c0114335:       0f 88 68 95 13 00       js     c024d8a3 <stext_lock+0x7bf>
                down(&sem);
        return retval;
c011433b:       8b 45 d4                mov    0xffffffd4(%ebp),%eax              <-----
c011433e:       e9 8d 00 00 00          jmp    c01143d0 <do_fork+0x79c>

Looking at the code in fork.c we know where above code is: