Hey guys, I’m sure, at least once in your life, you got an application crash with an non understanding message like “segmentation fault”, don’t you ?

So, what you are supposed to do  ?

Get a complete “core” file and launch your favorite tool : “gdb” (GNU Debugger) !

Software requirement

Install “gcore” & “gdb” :

[root@redhat ~]# yum --disablerepo="*" --enablerepo="RPMs" --nogpgcheck provides gcore
Loaded plugins: product-id, search-disabled-repos, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
gdb-7.6.1-80.el7.x86_64 : A GNU source-level debugger for C, C++, Fortran, Go and other languages
Repo        : RPMs
Matched from:
Filename    : /usr/bin/gcore
[root@redhat ~]# yum -y --disablerepo="*" --enablerepo="RPMs" --nogpgcheck install gdb

For this lab, we need “gcc” compiler, so install it :

[root@redhat ~]# yum -y --disablerepo="*" --enablerepo="RPMs" --nogpgcheck install gcc

Let’s go with examples

First example : try to divide an integer by zero (interger value of zero)

In this first example, we’ll put a function (where the “divide by zero” bad instruction will be) inside a shared library.

This program will have 1 thread only.

We use a non “root” user account :

[root@redhat ~]# su - wou

Create an executable

We create a directory :

[wou@redhat ~]$ mkdir division_by_zero
[wou@redhat ~]$ cd division_by_zero
  • The include file :
[wou@redhat division_by_zero]$ cat division_by_zero.h
#ifndef try_to_divide_by_zero_h__
#define try_to_divide_by_zero_h__
extern void try_to_divide_by_zero(void);
#endif
  • The shared library source :
[wou@redhat division_by_zero]$ cat division_by_zero.c
#include "division_by_zero.h"

void try_to_divide_by_zero(void) {
  const int my_mumber = 1,
            zero = 0;
  static int my_expected_result;

/* following line will abort the execution
   and generate a core dump */
  my_expected_result = my_mumber % zero ;
}
  • The main program :
[wou@redhat division_by_zero]$ cat main_program.c
#include "division_by_zero.h"

#include <stdio.h>
#include <string.h>

int main(void) {
  const char hello_message[] ="here an \"hello\" message for you\n";
  char message[sizeof(hello_message)];

  strcpy(&message[0],&hello_message[0]);
  puts(&message[0]);
  printf("we are going to divide by zero !!! Let's try ...\n");
  try_to_divide_by_zero();
  return(0);
}
  • The “make” file :
[wou@redhat division_by_zero]$ cat Makefile
all: main_program

clean:
        rm -f division_by_zero.o division_by_zero.so

main_program: main_program.c division_by_zero.so
        gcc -L. -Wall -o main_program main_program.c -l:division_by_zero.so

division_by_zero.so: division_by_zero.o
        gcc -shared -o division_by_zero.so division_by_zero.o

division_by_zero.o: division_by_zero.c division_by_zero.h
        gcc -c -Wall -fpic division_by_zero.c

Compile to get a binary executable :

[wou@redhat division_by_zero]$ make
gcc -c -Wall -fpic division_by_zero.c
division_by_zero.c: In function ‘try_to_divide_by_zero’:
division_by_zero.c:6:14: warning: variable ‘my_expected_result’ set but not used [-Wunused-but-set-variable]
   static int my_expected_result;
              ^
gcc -shared -o division_by_zero.so division_by_zero.o
gcc -L. -Wall -o main_program main_program.c -l:division_by_zero.so

We get :

  • An object :
[wou@redhat division_by_zero]$ file division_by_zero.o
division_by_zero.o: ELF 64-bit LSB relocatable, x86-64, version 1 (SYSV), not stripped
  • An shared library :
[wou@redhat division_by_zero]$ file division_by_zero.so
division_by_zero.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=7f06ac5f5851480fe92c2cf43c91d8b6ddac6b52, not stripped
  • A binary executable
main_program: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=81e09c3ea970bd7f6c5b5a92e46df1de3f6f99ec, not stripped
[wou@redhat division_by_zero]$ ldd main_program
        linux-vdso.so.1 =>  (0x00007ffecdcfd000)
        division_by_zero.so => not found
        libc.so.6 => /lib64/libc.so.6 (0x00007f75cedd7000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f75cf1a5000)

It’s time to execute this application

We would like to get a complete “core” file, so extend apllicable limits for this session :

[wou@redhat division_by_zero]$ ulimit -c unlimited

Then, start the application :

[wou@redhat division_by_zero]$ LD_LIBRARY_PATH=. ./main_program
here an "hello" message for you

we are going to divide by zero !!! Let's try ...
Floating point exception (core dumped)

As expected, we got a core dump :

[wou@redhat division_by_zero]$ file core.3951
core.3951: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from './main_program'

Core analysis

Launch “gdb” :

[wou@redhat division_by_zero]$ gdb
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
(gdb)

Using “gdb”,

  • Load the core file :
(gdb) core core.3951
[New LWP 3951]
Missing separate debuginfo for the main executable file
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/81/e09c3ea970bd7f6c5b5a92e46df1de3f6f99ec
Core was generated by `./main_program'.
Program terminated with signal 8, Arithmetic exception.
#0 0x00007fdc04a3869e in ?? ()
  • Load the binary executable :
(gdb) exec-file main_program
  • 1 thead only :
(gdb) info threads
  Id   Target Id         Frame
* 1    LWP 3951          0x00007fdc04a3869e in ?? ()
  • loaded shared libraries :
(gdb) info sharedlibrary
From                To                  Syms Read   Shared Object Library
0x00007fdc04a385a0  0x00007fdc04a386ac  No          ./division_by_zero.so
0x00007fdc046963e0  0x00007fdc047d9b60  No          /lib64/libc.so.6
0x00007fdc04c3aae0  0x00007fdc04c5527a  No          /lib64/ld-linux-x86-64.so.2
  • In which binary file, this application stopped ?
(gdb) where
#0  0x00007fdc04a3869e in ?? () from ./division_by_zero.so
#1  0x00007fffdd4c4c30 in ?? ()
#2  0x000000000040077c in ?? ()
#3  0x206e612065726568 in ?? ()
#4  0x20226f6c6c656822 in ?? ()
#5  0x206567617373656d in ?? ()
#6  0x0a756f7920726f66 in ?? ()
#7  0x0000000000000000 in ?? ()
  • which assembly instruction caused the program interruption ?
(gdb) x/i 0x00007fdc04a3869e
=> 0x7fdc04a3869e:      idivl  -0x8(%rbp)

A signed division instruction ! We tried to divide the content of registers dx:ax by the content addressed by bp 

  • What was the content of dx:ax ?
(gdb) info register rdx
rdx            0x0      0
(gdb) info register rax
rax            0x1      1

We tried to divide “1”

  • What was the content of the “rbp”register ?
(gdb) inf r rbp
rbp            0x7fffdd4c4bc0   0x7fffdd4c4bc0
  • What was the content of the memory at this address ?
(gdb) x/8xb 0x7fffdd4c4bc0
0x7fffdd4c4bc0: 0x30    0x4c    0x4c    0xdd    0xff    0x7f    0x00    0x00
(gdb) x/8xb $rbp
0x7fffdd4c4bc0: 0x30    0x4c    0x4c    0xdd    0xff    0x7f    0x00    0x00

We tried to divide “1” by zero (integer 0) !

Second example : try to access to 0x000000000000 memory address

For this example, we’ll create a multithreading application (10 threads), and one of those threads will try to copy a byte at NULL address ….

Create an executable

We create a directory :

[wou@redhat ~]$ mkdir threads
[wou@redhat ~]$ cd threads
  • The main program :
#include <stdio.h>
#include <stdlib.h>
#include <pthread.h>
#include <unistd.h>

#define NUMBER_OF_THREADS 10

void *print_message_function(void *pointer)
{
void *pointer_with_null_address = NULL;

  printf("we are runing thread #%d\n",*(int *)pointer);
  sleep(1);
  /* the thread #5 will abort the program and create a core dump */
  if (*(int *)pointer == 5)
    /* following instruction will create a core dump */
    *(char *)pointer_with_null_address = 'X';
}

int main(void)
{
pthread_t thread_handle[NUMBER_OF_THREADS];

int value[NUMBER_OF_THREADS], /* one parameter per thread !!! */
    thread_number;

/* we create NUMBER_OF_THREADS threads */
  for (thread_number = 0;thread_number != NUMBER_OF_THREADS;thread_number++)
  {
    value[thread_number] = thread_number + 1;
    if (pthread_create(&thread_handle[thread_number],NULL,print_message_function,(void*)&value[thread_number]) != 0)
    {
      perror("pthread_create");
      exit(EXIT_FAILURE);
    }
  }

/* we are waiting for all NUMBER_OF_THREADS threads to finish */
  for (thread_number = 0;thread_number != NUMBER_OF_THREADS;thread_number++)
    pthread_join(thread_handle[thread_number],NULL);
  return(EXIT_SUCCESS);
}
  • The “make” file :
[wou@redhat threads]$ cat Makefile
multi_threads: multi_threads.c
        gcc -g -o multi_threads -D_REETRANT -pthread multi_threads.c

(In this second example, we added “-g” option, that will help us to analyse).

Compile to get a binary executable :

[wou@redhat threads]$ make
gcc -g -o multi_threads -D_REETRANT -pthread multi_threads.c

It’s time to execute this second application

We would like to get a complete “core” file, so extend apllicable limits for this session :

[wou@redhat threads]$ ulimit -c unlimited

Then, start the application :

[wou@redhat threads]$ ./multi_threads
we are runing thread #1
we are runing thread #2
we are runing thread #3
we are runing thread #4
we are runing thread #5
we are runing thread #10
we are runing thread #6
we are runing thread #7
we are runing thread #8
we are runing thread #9
Segmentation fault (core dumped)

As expected, we got a core dump :

[wou@redhat threads]$ file core.3826
core.3826: ELF 64-bit LSB core file x86-64, version 1 (SYSV), SVR4-style, from './multi_threads'

Core analysis

Launch “gdb” :

[wou@redhat threads]$ gdb
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-80.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
(gdb)
  • Load the core file :
(gdb) core core.3826
[New LWP 3831]
[New LWP 3826]
[New LWP 3834]
[New LWP 3833]
[New LWP 3835]
Missing separate debuginfo for the main executable file
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/8c/fbeec7eb575736e04360335aa4c59f9a0b2388
Core was generated by `./multi_threads'.
Program terminated with signal 11, Segmentation fault.
#0 0x00000000004007b4 in ?? ()
  • Load the binary executable :
(gdb) file ./multi_threads
Reading symbols from /home/wou/threads/multi_threads...done.
  • A multi threading application :
(gdb) info threads
  Id   Target Id         Frame
  5    LWP 3835          0x00007f7996ae041d in ?? ()
  4    LWP 3833          0x00007f7996ae041d in ?? ()
  3    LWP 3834          0x00007f7996ae041d in ?? ()
  2    LWP 3826          0x00007f7996decef7 in ?? ()
* 1    LWP 3831          0x00000000004007b4 in print_message_function (pointer=0x7ffeb58283c0) at multi_threads.c:17
  • Which shared libraries were loaded ?
(gdb) info sharedlibrary
From                To                  Syms Read   Shared Object Library
0x00007f7996de98a0  0x00007f7996df4514  No          /lib64/libpthread.so.0
0x00007f7996a423e0  0x00007f7996b85b60  No          /lib64/libc.so.6
0x00007f7997000ae0  0x00007f799701b27a  No          /lib64/ld-linux-x86-64.so.2
  • Dump the stack trace for all threads :
(gdb) thread apply all bt

Thread 5 (LWP 3835):
#0  0x00007f7996ae041d in ?? () from /lib64/libc.so.6
#1  0x0000000000000000 in ?? ()

Thread 4 (LWP 3833):
#0  0x00007f7996ae041d in ?? () from /lib64/libc.so.6
#1  0x0000000000000000 in ?? ()

Thread 3 (LWP 3834):
#0  0x00007f7996ae041d in ?? () from /lib64/libc.so.6
#1  0x0000000000000000 in ?? ()

Thread 2 (LWP 3826):
#0  0x00007f7996decef7 in ?? () from /lib64/libpthread.so.0
#1  0x00007f7996dece30 in ?? () from /lib64/libpthread.so.0
#2  0x00007f7994a1ed28 in ?? ()
#3  0x00007f7996a22700 in ?? ()
#4  0x0000000000000000 in ?? ()

Thread 1 (LWP 3831):
#0  0x00000000004007b4 in print_message_function (pointer=0x7ffeb58283c0) at multi_threads.c:17
#1  0x00007f7996debdc5 in ?? () from /lib64/libpthread.so.0
#2  0x0000000000000000 in ?? ()
  • Analyse thread #1 (which had stopped this program) :
(gdb) thread 1
[Switching to thread 1 (LWP 3831)]
#0  0x00000000004007b4 in print_message_function (pointer=0x7ffeb58283c0) at multi_threads.c:17
17          *(char *)pointer_with_null_address = 'X';
  • What was the corresponding binary instruction ?
(gdb) x/i 0x00000000004007b4
=> 0x4007b4 <print_message_function+68>:        movb   $0x58,(%rax)

We tried to write 1 byte (0X58 = ‘X’ character) to memory addressed by “ax” register.

  • What was the address stored in “ax” register ?
(gdb) info r ax
ax             0x0      0

We tried to store a byte to memory 0x0000000000000000 which is not permited !

 

en.pdf24.org    Send article as PDF   

Leave a Reply

Your email address will not be published. Required fields are marked *


*