10.2. Documentation and Debugging¶

10.2.2.1. Breakpoints and Watchpoints¶

In the systems programming field, particularly with languages like C, the lack of exception handling makes debugging challenging. Often, the only indication that the error has occurred is the notoriously vague “Segmentation fault.” To get started with GDB, we will use it to examine Code Listing A.1.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

/* Code Listing A.1:
   Code for tracing watchpoints and breakpoints in GDB 
 */

#include <stdio.h>

int
helper (int input)
{
  return input + 1;
}

int
main (int argc)
{
  int x = argc;
  int y = helper (x);
  x = 10;
  printf ("x = %d; y = %d\n", x, y);

  return 0;
}

To get started with GDB, we must compile our program with the -g flag for gcc, indicating that we want to include debugging symbols. Without debugging symbols, we cannot refer to variables or (in certain circumstances) functions by their names. In the example code above, the steps to compile it, run it (without GDB), and run it again (with GDB) are as follows (note that GDB requires passing the --args option if there are command-line arguments):

$ gcc -g -o watch watch.c
$ ./watch 5
x = 10; y = 3
$ gdb --args ./watch 5
GNU gdb (Ubuntu 8.1-0ubuntu3.2) 8.1.0.20180409-git

The following GDB session illustrates how to use watchpoints and breakpoints. In GDB, we can set a watchpoint on a variable and be notified any time that variable’s value changes. (In fact, we can set watchpoints for any arbitrary memory location, but that is beyond our current scope.) We can set a breakpoint for a function name and be notified any time that function is called. In this first GDB session, we set a breakpoint for the helper() function and a watchpoint for the x variable in main().

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

GNU gdb (Ubuntu 8.1-0ubuntu3.2) 8.1.0.20180409-git
[GDB license info omitted...]
Reading symbols from ./watch...done.
(gdb) start
Temporary breakpoint 1 at 0x661: file watch.c, line 12.
Starting program:  /home/csf/watch 

Temporary breakpoint 1, main (argc=2) at watch.c:12
13	  int x = 5;
(gdb) break helper
Breakpoint 2 at 0x555555554651: file watch.c, line 6.
(gdb) watch x
Hardware watchpoint 3: x
(gdb) watch input
No symbol "input" in current context.
(gdb)

For simplicity, we have omitted the several lines of introductory text that GDB shows regarding its license and documentation. The GDB command prompts are shown as bolded lines, where (gdb) indicates the prompt and our input is shown afterwards. On line 4 of this session, we use the start command to begin the program’s execution, which will pause when the main() function begins execution. When GDB executes lines of code as the result of a command, it displays the next line of code that will be executed in the future. So on line 9, GDB is indicating that it paused just before line 12 of Code Listing A.1, which initializes the x variable.

Lines 10 and 12 of the GDB session set up the breakpoint and watchpoint with their respective commands, break and watch. Note that these commands require the target symbols to be visible by GDB when they are run. That means that the program must be compiled with debugging symbols included and the variable must be in scope. On line 14 of this GDB session, we cannot set a watchpoint for the input variable, which is only defined within the scope of the helper() function; GDB is currently executing inside the main() function scope.

The main commands to execute code in GDB are step, next, and continue. The step and next commands (not shown here) would simply execute the next line of code. In the session above, running either of these commands would execute the line 12 of Code Listing A.1, which sets the variable x to 5. The difference between these two commands arises when the next line of code is a function call. The step command would step into the called function’s body, whereas the next command treats the function call as an opaque box, executing the entire function as one step. Based on the previous session, we use the continue command, which allows the program to run until it is interrupted.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

(gdb) continue
Continuing.

Hardware watchpoint 3: x

Old value = 0
New value = 2
main () at watch.c:13
14	  int y = helper (x);
(gdb) continue
Continuing.

Breakpoint 2, helper (input=2) at watch.c:6
7	  return input + 1;
(gdb) continue
Continuing.

Hardware watchpoint 3: x

Old value = 2
New value = 10
main () at watch.c:15
16	  printf ("x = %d; y = %d\n", x, y);
(gdb) print x
$1 = 10
(gdb)

In this session, we can observe the effects of the watchpoint and breakpoint that we set previously. When GDB encounters code that changes the value of a variable being observed with a watchpoint, it will pause the execution to indicate the old and new values (x has changed from 0 to 5), as well as the next line of code when execution resumes (line 13 to call the helper() function). By continuing a second time (line 10), this session encounters the breakpoint when the execution of helper() begins. Using breakpoints like this makes it easy to skip over large chunks of code and pausing right before the execution of a function that we wish to focus on. Line 15 of this session performs another continue, pausing on line 15 of the code; as x’s value changes from 5 to 10, the watchpoint is triggered again. Finally, note that we can also print in-scope variables at any point to observe their current value (line 24).

To illustrate the end of a GDB session, we execute the continue command two more times. The first continue causes an interrupt to arise when the return from main() occurs. Once this return happens, the variable x no longer exists, so the watchpoint can be deleted (lines 5 and 6). Note that this deletion implies GDB can distinguish between the variable being temporarily out of scope (as x is not in scope while helper() is executing) and being no longer needed. Lines 7 – 9 of this session can be ignored here, as it is simply GDB informing us (indirectly) that the source code containing the next line to execute cannot be found; this typically happens when GDB pauses during the executing of the C standard library, as these libraries are not compiled with debugging symbols included. The last continue (line 10) runs the program until it finishes, with line 12 informing us that the process has been terminated.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

(gdb) continue
Continuing.
x = 10; y = 6

Watchpoint 3 deleted because the program has left the block in
which its expression is valid.
__libc_start_main (main=0x555555554659 <main>, argc=2, argv=0x7fffffffea18, init=<optimized out>, 
    fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffea08) at ../csu/libc-start.c:344
344	../csu/libc-start.c: No such file or directory.
(gdb) continue
Continuing.
[Inferior 1 (process 16005) exited normally]
(gdb)

10.2.2.2. Backtrace¶

One significant source of frustration for systems programming is the lack of information regarding segmentation faults. Backtrace is an essential tool for debugging segmentation faults, such as the code shown in Code Listing A.2.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

/* Code Listing A.2:
   Dereferencing a NULL pointer to trigger a segmentation fault
 */

void
segfault (int value)
{
  int *nullptr = (int *) value;
  *ptr = 5;
}

int
main (void)
{
  segfault ();
  return 0;
}

Compiling and running this program would produce the following unhelpful results: [3]

$ gcc -g -o segfault segfault.c
segfault.c: In function ‘segfault’:
segfault.c:4:18: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
   int *nullptr = (int *) value;
                  ^
$ ./segfault
Segmentation fault

Diagnosing and fixing segmentation faults can be very difficult. One reason is that the message above provides no information about what line of code was executing when the segmentation fault occurred. Many programmers, particularly novices, try to address this with “printf() debugging,” the ad hoc approach of adding printf() statements to the code. This approach is inefficient and ineffective for a number of reasons. First, it requires re-editing and re-compiling the program. Second, adding any code, including printf(), changes the program. In some cases, this seemingly innocuous change will cause the segmentation fault to disappear—but only if the printf() stays in the code; removing the printf() brings the segmentation fault back. Lastly, adding code increases the opportunity for bugs to cause misleading results. For instance, changing the main() of Code Listing A.2 to that shown in Code Listing A.3 will produce identical results; without a '\n', the STDOUT buffer will not be flushed and the message in the printf() will never be displayed to the screen.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

/* Code Listing A.3:
   Printing without newlines may prevent the display of the message
 */

int
main (void)
{
  printf ("Hello from main()");
  segfault ();
  return 0;
}

GDB’s backtrace utility provides a more efficient and effective approach to debugging. Line 10 of this GDB session informs us that GDB encountered a segmentation fault (SIGSEGV) while executing line 5 of Code Listing A.2. Line 11 also provides information about the function that was executing at the time (segfault()), and what its current input arguments were (value=0). That information might be enough if the function is only called once or if the relationship between the arguments and the segmentation fault is clear. However, it is often the case that more context is needed; in order to debug the segmentation fault, we need to know what specific call—including the arguments passed and where it was called from—produced these results. This information is provided by backtrace, which shows the sequence of function calls that led to the segmentation fault.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

(gdb) start
Temporary breakpoint 1 at 0x61b: file segfault.c, line 11.
Starting program: /home/csf/segfault 

Temporary breakpoint 1, main () at segfault.c:11
11	  segfault (0);
(gdb) continue
Continuing.

Program received signal SIGSEGV, Segmentation fault.
0x000055555555460e in segfault (value=0) at segfault.c:5
5	  *nullptr = 5;
(gdb) backtrace
#0  0x000055555555460e in segfault (value=0) at segfault.c:5
#1  0x0000555555554625 in main () at segfault.c:11
(gdb)

Lines 14 and 15 show that the segmentation fault occurred while executing line 5 of Code Listing A.2. This line exists within the segfault() function, which was specifically called with the argument value=0. The call to segfault(0) occurred on line 11 of the code, which exists as part of the call to main(). Unless the contents of the stack have become corrupted (which go beyond our discussion here), this history will show the full trace [4] back to the main() context. That information provides more context that the programmer can use to determine what might be the root cause of the segmentation fault.

10.2.2.3. Tracing multiple processes¶

Debugging segmentation faults—and other bugs—with multiple processes can be particularly challenging. For one thing, the asynchronous timing of the execution might intersperse output messages in odd ways. Even more frustrating, messages written to the STDERR stream are also lost unless the output streams are specifically linked. Consider the example shown in Code Listing A.4, which uses a slight variant on the same segfault() function from Code Listing A.2.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

/* Code Listing A.4:
   The child's "Segmentation fault" will not appear on screen
 */

#include <assert.h>
#include <stdio.h>
#include <sys/wait.h>
#include <unistd.h>

void
segfault (int *ptr)
{
  *ptr = 5;
}

int
main (void)
{
  int *nullptr = NULL;
  pid_t child = fork ();
  assert (child >= 0);

  if (child == 0)
    segfault (nullptr);
  else
    wait (NULL);

  printf ("Goodbye\n");
  return 0;
}

As with the previous cases, running this code should result in a segmentation fault. However, the results would actually look as follows:

$ gcc -g -o forkfault forkfault.c
$ ./forkfault
Goodbye

In this case, the parent process calls fork() to create a child process. Based on the nondeterministic timing of process scheduling, either the parent continues executing or the child runs first. This nondeterminism is not an issue here, though; if the parent runs, it calls wait() until the child runs to completion. Consequently, when the parent executes the printf() call on line 24, the child has been guaranteed to run to completion; that child process definitely terminated with a segmentation fault because of the code on line 13. The reason the child’s error message does not appear on the terminal is that the child process’s STDERR output stream is thrown away.

To debug these kinds of errors in multiple processes, GDB allows you to specify which process to follow when a fork() occurs. Specifically, the default behavior is that GDB will follow the parent; if you use next or continue to move past a fork(), GDB will show a message about a new process but continue executing the parent. To change this behavior, the set command can be used to change GDB’s follow-fork-mode setting as shown below. Based on this new setting, GDB will switch to the child process after the call to fork(); additional GDB commands will be sent to the child process rather than the parent.

(gdb) start
Temporary breakpoint 2 at 0x6f7: file forkfault.c, line 15.
Starting program:  /home/csf/forkfault 

Breakpoint 1, main () at forkfault.c:15
15	  int *nullptr = NULL;
(gdb) set follow-fork-mode child
(gdb) continue
Continuing.
[New process 9666]

Thread 2.1 "forkfault" received signal SIGSEGV, Segmentation fault.
[Switching to process 9666]
0x0000555555554726 in segfault (ptr=0x0) at forkfault.c:9
9	  *ptr = 5;
(gdb) backtrace
#0  0x0000555555554726 in segfault (ptr=0x0) at forkfault.c:9
#1  0x000055555555477e in main () at forkfault.c:20
(gdb) 

The full GDB manual provides much more information about debugging multiple processes. One particularly useful command is info proc, which can be used to examine information about the status of a process, including any memory map regions, memory usage, number of threads, and many others. Information on info proc can be found at:

• https://sourceware.org/gdb/current/onlinedocs/gdb/Process-Information.html