How Android tombstone files are generated

In this section, we focus on androidQ and analyze a function for debugging in android, that is tombstone, commonly known as "tombstone". In real life, the tombstone is generally prepared for the dead, while the "tombstone" in the android system is prepared for the process.

Why did Android design such a thing? Because the android system is running on top of the Linux Kernel kernel, when the kernel is abnormal, the kernel exception mechanism will recognize what is the cause and cannot directly handle panic. For an Android system running on top of the Linux Kernel, if an exception occurs, the Android layer is automatically restarted, which makes it difficult to reproduce the problem and locate the debug. When the Android layer is abnormal, the context information of the process is usually displayed Saved to tombstone (tombstone) for easy debugging.

               

The above picture is a classic Android system architecture diagram, and our tombstone is mainly prepared for the process of the Native layer, which is mainly used to analyze NativeCrash. Because the entire system of Kernel Crash is panic directly, the kernel will print out the corresponding call trace, and there will be a corresponding exception thrown for the code error of the Java layer. Therefore, the tombstone is mainly prepared for the process of the Native layer.

Tombstone

What does tombstone look like? When an exception occurs in the android system, the corresponding tombstone file will be generated in the / data / tombstones directory

root:/data/tombstones # ls -l
-rw-r----- 1 tombstoned system 3454991 2020-03-13 18:10 tombstone_00
-rw-rw-rw- 1 root       root         0 2020-03-14 10:28 tombstone_01
-rw-r----- 1 root       root   3454991 2020-03-14 10:29 tombstone_02
-rw-r----- 1 root       root   3454991 2020-03-14 10:29 tombstone_03

Open a file and see what tombstone looks like

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
Build fingerprint: 'dev/good_dev/2020.3.6_china_dev_test:user/test-keys'
Revision: '0'
ABI: 'arm'
Timestamp: 2020-03-07 02:46:27+0800
pid: 23051, tid: 23051, name: .tencent.qqlive  >>> com.tencent.qqlive <<<
uid: 10256
signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0xdb3fb000
    r0  db3f9000  r1  00000000  r2  00023780  r3  db3fb000
    r4  cd5ef340  r5  db3f9000  r6  e7a1e260  r7  cd62c6e0
    r8  cd668190  r9  ffc417a8  r10 00000001  r11 d1ffa760
    ip  00000000  sp  ffc41768  lr  b9466590  pc  e79bfd58

backtrace:
      #00 pc 0005dd58  /apex/com.android.runtime/lib/bionic/libc.so (memset_a7+48) (BuildId: dcf0e174e93e33d22f35a631ba9c0de5)
      #01 pc 0001258c  /data/app/com.tencent.qqlive-NGSZiLGPko8T894J1J65kQ==/lib/arm/libmarsxlog.so (LogBuffer::__Clear()+36) (BuildId: 05f64491a5d2ace3d88e1623fefbc695c32396ba)
      #02 pc 0000ff68  /data/app/com.tencent.qqlive-NGSZiLGPko8T894J1J65kQ==/lib/arm/libmarsxlog.so (appender_open(TAppenderMode, char const*, char const*, char const*)+560) (BuildId: 05f64491a5d2ace3d88e1623fefbc695c32396ba)
      #03 pc 00010ca8  /data/app/com.tencent.qqlive-NGSZiLGPko8T894J1J65kQ==/lib/arm/libmarsxlog.so (appender_open_with_cache(TAppenderMode, std::string const&, std::string const&, char const*, char const*)+1652) (BuildId: 05f64491a5d2ace3d88e1623fefbc695c32396ba)
      #04 pc 000051cc  /data/app/com.tencent.qqlive-NGSZiLGPko8T894J1J65kQ==/lib/arm/libmarsxlog.so (Java_com_tencent_mars_xlog_Xlog_appenderOpen+428) (BuildId: 05f64491a5d2ace3d88e1623fefbc695c32396ba)
      #05 pc 000db673  /data/app/com.tencent.qqlive-NGSZiLGPko8T894J1J65kQ==/oat/arm/base.odex (art_jni_trampoline+210)

This file is relatively long, we currently only post a part, the content of this section is not to analyze the meaning of tombstone content, this section focuses on the process of generating this file, understand how it is generated, and then analyze what the content of this file means And how to analyze and solve such problems.

NULL pointer example review

In the wonderful tour of the NULL pointer in the previous section, we explained in detail the series of activities within the operating system when the CPU accesses the NULL pointer, and finally printed the familiar "Segmetation Faule" on the console. Here we review, because this process can be applied to today's tombstone production process.

• When the CPU accesses a virtual address, it will definitely go through the MMU to check the corresponding virtual-real relationship
• Once the virtual address is illegal, the MMU hardware unit will trigger an exception, and the CPU will go to the exception vector table to execute the corresponding exception
• After processing, the Linux kernel's exception to the userspace is signaled to the corresponding process.
• Once the process receives the signal, it will execute the corresponding signal processing function.
• The installation of signal processing functions is generally done in glibc, and glibc will do default processing for all common signals.

Back to the Android system, when a Native process triggers a NULL pointer, first the CPU will receive the corresponding exception, and then execute the exception, then the SIGSEGV signal will occur, and the signal processing function will process the signal In the process, the scene of the process will be saved, and finally the tombstone will be left for future generations to worship.

From the above description, we have probably guessed the general implementation process of tombstone, and then we will verify the conjecture.

 

How the process works

Here is a brief description of how a process runs in android. Here we use the WeChat app as an example

• WeChat app is first stored on UFS, EMMC instruction storage device
• When the user clicks the WeChat app icon, the operating system will load the WeChat app from Flash to the main memory
  • Definitely have to go through the fork similar command to create the corresponding process
  • After the process is created, you need to load the content of WeChat through an exec-like command
• Finally, the / system / bin / linker program is responsible for loading some shared libraries used by the WeChat program,
• Finally jump to the entrance of the WeChat program to execute

The above is a simple description of how to run a program. Let ’s directly look at the / system / bin / linker code in the android system

代码路径： /bionic/linker/arch/arm64/begin.S

ENTRY(_start)
  // Force unwinds to end in this function.
  .cfi_undefined x30

  mov x0, sp
  bl __linker_init

  /* linker init returns the _entry address in the main image */
  br x0
END(_start)

First of all, it must jump to the linker code segment to execute, jump to the __linker_init function

/*
 * This is the entry point for the linker, called from begin.S.
 */
extern "C" ElfW(Addr) __linker_init(void* raw_args) {

  tmp_linker_so.base = linker_addr;
  tmp_linker_so.size = phdr_table_get_load_size(phdr, elf_hdr->e_phnum);
  tmp_linker_so.load_bias = get_elf_exec_load_bias(elf_hdr);
  tmp_linker_so.dynamic = nullptr;
  tmp_linker_so.phdr = phdr;
  tmp_linker_so.phnum = elf_hdr->e_phnum;
  tmp_linker_so.set_linker_flag();

  return __linker_init_post_relocation(args, tmp_linker_so);
}

In Linker_init, according to the link address and elf header information, to recalculate whether relocation or the like is needed

static ElfW(Addr) __attribute__((noinline))
__linker_init_post_relocation(KernelArgumentBlock& args, soinfo& tmp_linker_so) {

  // Initialize the linker's static libc's globals
  __libc_init_globals();

  // Initialize the linker's own global variables
  tmp_linker_so.call_constructors();

  ElfW(Addr) start_address = linker_main(args, exe_to_load);

  INFO("[ Jumping to _start (%p)... ]", reinterpret_cast<void*>(start_address));
}

Return the start address of the executable program through the linker_main function, and then jump to the execution. Let's focus on the linker_main function

static ElfW(Addr) linker_main(KernelArgumentBlock& args, const char* exe_to_load) {
  ProtectedDataGuard guard;

  // Register the debuggerd signal handler.
#ifdef __ANDROID__
  debuggerd_callbacks_t callbacks = {
    .get_abort_message = []() {
      return __libc_shared_globals()->abort_msg;
    },
    .post_dump = &notify_gdb_of_libraries,
  };
  debuggerd_init(&callbacks);
#endif

Focus on here, at that time, if the android system, it will initialize debggerd_init, this function will follow the signal's default processing function

void debuggerd_init(debuggerd_callbacks_t* callbacks) {

  struct sigaction action;
  memset(&action, 0, sizeof(action));
  sigfillset(&action.sa_mask);
  action.sa_sigaction = debuggerd_signal_handler;
  action.sa_flags = SA_RESTART | SA_SIGINFO;

  // Use the alternate signal stack if available so we can catch stack overflows.
  action.sa_flags |= SA_ONSTACK;
  debuggerd_register_handlers(&action);
}

static void __attribute__((__unused__)) debuggerd_register_handlers(struct sigaction* action) {
  sigaction(SIGABRT, action, nullptr);
  sigaction(SIGBUS, action, nullptr);
  sigaction(SIGFPE, action, nullptr);
  sigaction(SIGILL, action, nullptr);
  sigaction(SIGSEGV, action, nullptr);
#if defined(SIGSTKFLT)
  sigaction(SIGSTKFLT, action, nullptr);
#endif
  sigaction(SIGSYS, action, nullptr);
  sigaction(SIGTRAP, action, nullptr);
  sigaction(DEBUGGER_SIGNAL, action, nullptr);
}

It can be seen that some abnormal signals are registered here, and the action processing function of the signal is debuggerd_signal_handler.

When an exception occurs

For example, when the Native process has a null pointer problem, the signal is generated by the linux kernel, and the final signal is processed by the debuggerd_signal_handler function

debuggerd_signal_handler

static void debuggerd_signal_handler(int signal_number, siginfo_t* info, void* context) {

  // Only allow one thread to handle a signal at a time.
  int ret = pthread_mutex_lock(&crash_mutex);
  if (ret != 0) {
    async_safe_format_log(ANDROID_LOG_INFO, "libc", "pthread_mutex_lock failed: %s", strerror(ret));
    return;
  }

  log_signal_summary(info);
}

Use pthread_mutex_lock to prevent multiple threads from processing signals at the same time, then call the log_signal_summary function to print some information

  async_safe_format_log(ANDROID_LOG_FATAL, "libc",
                        "Fatal signal %d (%s), code %d (%s%s)%s in tid %d (%s), pid %d (%s)",
                        info->si_signo, get_signame(info), info->si_code, get_sigcode(info),
                        sender_desc, addr_desc, __gettid(), thread_name, self_pid, main_thread_name)

The purpose of doing this here is to prevent the subsequent actions from making mistakes, and ultimately it is impossible to determine which process made the mistake. Here, some key information is printed first. You can find the corresponding information from logcat

 libc    : Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0xdb3fb000 in tid 23051 (.tencent.qqlive), pid 23051 (.tencent.qqlive)

• Signal num, for example signal 11 represents SIGSEGV
• The signal code, SEGV_MAPERR, means that the mapping is wrong
• fault addr, the address of the error
• tid: corresponding thread ID
• pid: The corresponding process ID. If there are multiple threads in a process, the ID of each thread is different.

  pid_t child_pid =
    clone(debuggerd_dispatch_pseudothread, pseudothread_stack,
          CLONE_THREAD | CLONE_SIGHAND | CLONE_VM | CLONE_CHILD_SETTID | CLONE_CHILD_CLEARTID,
          &thread_info, nullptr, nullptr, &thread_info.pseudothread_tid);
  if (child_pid == -1) {
    fatal_errno("failed to spawn debuggerd dispatch thread");
  }

  // Wait for the child to start...
  futex_wait(&thread_info.pseudothread_tid, -1);

  // and then wait for it to terminate.
  futex_wait(&thread_info.pseudothread_tid, child_pid);

Next is through the clone system call, clone out a pseudothread pseudothread thread, to dispatch to process the signal, here waiting for the start and end of the child process

debuggerd_dispatch_pseudothread

  pid_t crash_dump_pid = __fork();
  if (crash_dump_pid == -1) {
    async_safe_format_log(ANDROID_LOG_FATAL, "libc",
                          "failed to fork in debuggerd signal handler: %s", strerror(errno));
  } else if (crash_dump_pid == 0) {
   
    async_safe_format_buffer(main_tid, sizeof(main_tid), "%d", thread_info->crashing_tid);
    async_safe_format_buffer(pseudothread_tid, sizeof(pseudothread_tid), "%d",
                             thread_info->pseudothread_tid);
    async_safe_format_buffer(debuggerd_dump_type, sizeof(debuggerd_dump_type), "%d",
                             get_dump_type(thread_info));

    execle(CRASH_DUMP_PATH, CRASH_DUMP_NAME, main_tid, pseudothread_tid, debuggerd_dump_type,
           nullptr, nullptr);
    async_safe_format_log(ANDROID_LOG_FATAL, "libc", "failed to exec crash_dump helper: %s",
                          strerror(errno));
    return 1;
  }

Create a child thread through the fork in the pseudo thread, the newly created child thread is transferred to execute the crash_dump64 program through the execle system call, and the parent process is here to wait for the crash_dump process to exit

crash_dump process

  pid_t forkpid = fork();
  if (forkpid == -1) {
    PLOG(FATAL) << "fork failed";
  } else if (forkpid == 0) {
    fork_exit_read.reset();
  } else {
    // We need the pseudothread to live until we get around to verifying the vm pid against it.
    // The last thing it does is block on a waitpid on us, so wait until our child tells us to die.
    fork_exit_write.reset();
    char buf;
    TEMP_FAILURE_RETRY(read(fork_exit_read.get(), &buf, sizeof(buf)));
    _exit(0);
  }

The crash_dump process directly forks out a new process, the parent process waits for the child process through read, and the child process continues to execute the task of crash_dump

  // Get the process name (aka cmdline).
  std::string process_name = get_process_name(g_target_thread);

  // Collect the list of open files.
  OpenFilesList open_files;
  {
    ATRACE_NAME("open files");
    populate_open_files_list(&open_files, g_target_thread);
  }

• Get the name of the process, get the name of the process through / proc / PID / cmdline
• Get multiple files opened by this process, you can get how many files are opened by / proc / PID / fd /, each file has a file descriptor fd

{
    ATRACE_NAME("ptrace");
    for (pid_t thread : threads) {
      // Trace the pseudothread separately, so we can use different options.
      if (thread == pseudothread_tid) {
        continue;
      }

      if (!ptrace_seize_thread(target_proc_fd, thread, &error)) {
        bool fatal = thread == g_target_thread;
        LOG(fatal ? FATAL : WARNING) << error;
      }

      ThreadInfo info;
      info.pid = target_process;
      info.tid = thread;
      info.uid = getuid();
      info.process_name = process_name;
      info.thread_name = get_thread_name(thread);

      if (!ptrace_interrupt(thread, &info.signo)) {
        PLOG(WARNING) << "failed to ptrace interrupt thread " << thread;
        ptrace(PTRACE_DETACH, thread, 0, 0);
        continue;
      }

      if (thread == g_target_thread) {
        // Read the thread's registers along with the rest of the crash info out of the pipe.
        ReadCrashInfo(input_pipe, &siginfo, &info.registers, &abort_msg_address,
                      &fdsan_table_address);
        info.siginfo = &siginfo;
        info.signo = info.siginfo->si_signo;
      } else {
        info.registers.reset(unwindstack::Regs::RemoteGet(thread));
        if (!info.registers) {
          PLOG(WARNING) << "failed to fetch registers for thread " << thread;
          ptrace(PTRACE_DETACH, thread, 0, 0);
          continue;
        }
      }

      thread_info[thread] = std::move(info);
    }
  }

The for loop traverses all threads in this process, reads the pinfo operation of the thread process in each process, and reads the crashinfo of the target thread.

  // Detach from all of our attached threads before resuming.
  for (const auto& [tid, thread] : thread_info) {
    int resume_signal = thread.signo == DEBUGGER_SIGNAL ? 0 : thread.signo;
    if (wait_for_gdb) {
      resume_signal = 0;
      if (tgkill(target_process, tid, SIGSTOP) != 0) {
        PLOG(WARNING) << "failed to send SIGSTOP to " << tid;
      }
    }

    LOG(DEBUG) << "detaching from thread " << tid;
    if (ptrace(PTRACE_DETACH, tid, 0, resume_signal) != 0) {
      PLOG(ERROR) << "failed to detach from thread " << tid;
    }
  }

After reading crashinfo, detach each thead

tombstoned_connect

  {
    ATRACE_NAME("tombstoned_connect");
    LOG(INFO) << "obtaining output fd from tombstoned, type: " << dump_type;
    g_tombstoned_connected =
        tombstoned_connect(g_target_thread, &g_tombstoned_socket, &g_output_fd, dump_type);
  }

Connected to tombstone process, connected through socket.

  // si_value is special when used with DEBUGGER_SIGNAL.
  //   0: dump tombstone
  //   1: dump backtrace
  if (!fatal_signal) {
    int si_val = siginfo.si_value.sival_int;
    if (si_val == 0) {
      backtrace = false;
    } else if (si_val == 1) {
      backtrace = true;
    } else {
      LOG(WARNING) << "unknown si_value value " << si_val;
    }
  }

Make different judgments based on the parameters passed by tombstone. When the parameter = 0, it means dump tombstone. When it is equal to 1, only the backtrace is dumped.

  if (backtrace) {
    ATRACE_NAME("dump_backtrace");
    dump_backtrace(std::move(g_output_fd), &unwinder, thread_info, g_target_thread);
  } else {
    {
      ATRACE_NAME("fdsan table dump");
      populate_fdsan_table(&open_files, unwinder.GetProcessMemory(), fdsan_table_address);
    }

    {
      ATRACE_NAME("engrave_tombstone");
      engrave_tombstone(std::move(g_output_fd), &unwinder, thread_info, g_target_thread,
                        abort_msg_address, &open_files, &amfd_data);
    }
  }

The final tombstone is generated by engrave_tombstone.

engrave_tombstone

void engrave_tombstone(unique_fd output_fd, unwindstack::Unwinder* unwinder,
                       const std::map<pid_t, ThreadInfo>& threads, pid_t target_thread,
                       uint64_t abort_msg_address, OpenFilesList* open_files,
                       std::string* amfd_data) {
  // don't copy log messages to tombstone unless this is a dev device
  bool want_logs = android::base::GetBoolProperty("ro.debuggable", false);

  log_t log;
  log.current_tid = target_thread;
  log.crashed_tid = target_thread;
  log.tfd = output_fd.get();
  log.amfd_data = amfd_data;

  _LOG(&log, logtype::HEADER, "*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***\n");
  dump_header_info(&log);
  dump_timestamp(&log, time(nullptr));

  auto it = threads.find(target_thread);
  if (it == threads.end()) {
    LOG(FATAL) << "failed to find target thread";
  }
  dump_thread(&log, unwinder, it->second, abort_msg_address, true);

  if (want_logs) {
    dump_logs(&log, it->second.pid, 50);
  }

  for (auto& [tid, thread_info] : threads) {
    if (tid == target_thread) {
      continue;
    }

    dump_thread(&log, unwinder, thread_info, 0, false);
  }

  if (open_files) {
    _LOG(&log, logtype::OPEN_FILES, "\nopen files:\n");
    dump_open_files_list(&log, *open_files, "    ");
  }

  if (want_logs) {
    dump_logs(&log, it->second.pid, 0);
  }

tombstone file example analysis

tombstone's iconic log starts: "*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** * ** *** "

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***

dump_header_info print head information

static void dump_header_info(log_t* log) {
  auto fingerprint = GetProperty("ro.build.fingerprint", "unknown");
  auto revision = GetProperty("ro.revision", "unknown");

  _LOG(log, logtype::HEADER, "Build fingerprint: '%s'\n", fingerprint.c_str());
  _LOG(log, logtype::HEADER, "Revision: '%s'\n", revision.c_str());
  _LOG(log, logtype::HEADER, "ABI: '%s'\n", ABI_STRING);
}

Examples are as follows:

Build fingerprint: 'dev/good_dev/2020.3.6_china_dev_test:user/test-keys'
Revision: '0'
ABI: 'arm'

dump_timestamp print time information

static void dump_timestamp(log_t* log, time_t time) {
  struct tm tm;
  localtime_r(&time, &tm);

  char buf[strlen("1970-01-01 00:00:00+0830") + 1];
  strftime(buf, sizeof(buf), "%F %T%z", &tm);
  _LOG(log, logtype::HEADER, "Timestamp: %s\n", buf);
}

Print the time of business trip, the example is as follows:

Timestamp: 2020-03-07 02:46:27+0800

dump_thread_info print thread information

static void dump_thread_info(log_t* log, const ThreadInfo& thread_info) {
  // Blacklist logd, logd.reader, logd.writer, logd.auditd, logd.control ...
  // TODO: Why is this controlled by thread name?
  if (thread_info.thread_name == "logd" ||
      android::base::StartsWith(thread_info.thread_name, "logd.")) {
    log->should_retrieve_logcat = false;
  }

  _LOG(log, logtype::HEADER, "pid: %d, tid: %d, name: %s  >>> %s <<<\n", thread_info.pid,
       thread_info.tid, thread_info.thread_name.c_str(), thread_info.process_name.c_str());
  _LOG(log, logtype::HEADER, "uid: %d\n", thread_info.uid);
}

Examples are as follows:

pid: 23051, tid: 23051, name: .tencent.qqlive  >>> com.tencent.qqlive <<<
uid: 10256

dump_signal_info print signal information

  _LOG(log, logtype::HEADER, "signal %d (%s), code %d (%s%s), fault addr %s\n",
       thread_info.siginfo->si_signo, get_signame(thread_info.siginfo),
       thread_info.siginfo->si_code, get_sigcode(thread_info.siginfo), sender_desc, addr_desc);

Examples are as follows:

signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0xdb3fb000

dump_probable_cause print possible cause information

      cause = StringPrintf("null pointer dereference");
    } else if (si->si_addr == reinterpret_cast<void*>(0xffff0ffc)) {
      cause = "call to kuser_helper_version";
    } else if (si->si_addr == reinterpret_cast<void*>(0xffff0fe0)) {
      cause = "call to kuser_get_tls";
    } else if (si->si_addr == reinterpret_cast<void*>(0xffff0fc0)) {
      cause = "call to kuser_cmpxchg";
    } else if (si->si_addr == reinterpret_cast<void*>(0xffff0fa0)) {
      cause = "call to kuser_memory_barrier";
    } else if (si->si_addr == reinterpret_cast<void*>(0xffff0f60)) {
      cause = "call to kuser_cmpxchg64";
    }

if (!cause.empty()) _LOG(log, logtype::HEADER, "Cause: %s\n", cause.c_str());

Examples are as follows:

Cause: null pointer dereference

dump_registers print register information

    r0  db3f9000  r1  00000000  r2  00023780  r3  db3fb000
    r4  cd5ef340  r5  db3f9000  r6  e7a1e260  r7  cd62c6e0
    r8  cd668190  r9  ffc417a8  r10 00000001  r11 d1ffa760
    ip  00000000  sp  ffc41768  lr  b9466590  pc  e79bfd58

log_backtrace prints backtrace information

backtrace:
      #00 pc 0005dd58  /apex/com.android.runtime/lib/bionic/libc.so (memset_a7+48) (BuildId: dcf0e174e93e33d22f35a631ba9c0de5)
      #01 pc 0001258c  /data/app/com.tencent.qqlive-NGSZiLGPko8T894J1J65kQ==/lib/arm/libmarsxlog.so (LogBuffer::__Clear()+36) (BuildId: 05f64491a5d2ace3d88e1623fefbc695c32396ba)
      #02 pc 0000ff68  /data/app/com.tencent.qqlive-NGSZiLGPko8T894J1J65kQ==/lib/arm/libmarsxlog.so (appender_open(TAppenderMode, char const*, char const*, char const*)+560) (BuildId: 05f64491a5d2ace3d88e1623fefbc695c32396ba)
      #03 pc 00010ca8  /data/app/com.tencent.qqlive-NGSZiLGPko8T894J1J65kQ==/lib/arm/libmarsxlog.so (appender_open_with_cache(TAppenderMode, std::string const&, std::string const&, char const*, char const*)+1652) (BuildId: 05f64491a5d2ace3d88e1623fefbc695c32396ba)
      #04 pc 000051cc  /data/app/com.tencent.qqlive-NGSZiLGPko8T894J1J65kQ==/lib/arm/libmarsxlog.so (Java_com_tencent_mars_xlog_Xlog_appenderOpen+428) (BuildId: 05f64491a5d2ace3d88e1623fefbc695c32396ba)
      #05 pc 000db673  /data/app/com.tencent.qqlive-NGSZiLGPko8T894J1J65kQ==/oat/arm/base.odex (art_jni_trampoline+210)

dump_stack print stack information

stack:
         ffc41728  cd422400  [anon:libc_malloc]
         ffc4172c  b946cc2b  /data/app/com.tencent.qqlive-NGSZiLGPko8T894J1J65kQ==/lib/arm/libmarsxlog.so
         ffc41730  ffc417b0  [stack]
         ffc41734  db3f9000  /data/data/com.tencent.qqlive/files/log/QQLiveLog.mmap2
         ffc41738  db3f9000  /data/data/com.tencent.qqlive/files/log/QQLiveLog.mmap2
         ffc4173c  e7a1e260  [anon:.bss]
         ffc41740  ffc417b0  [stack]
         ffc41744  00001252
         ffc41748  e7a1e260  [anon:.bss]
         ffc4174c  b946cc7b  /data/app/com.tencent.qqlive-NGSZiLGPko8T894J1J65kQ==/lib/arm/libmarsxlog.so
         ffc41750  00000000

dump_memory_and_code print memory information

memory near r0 (/data/data/com.tencent.qqlive/files/log/QQLiveLog.mmap2):
    db3f8fe0 -------- -------- -------- --------  ................
    db3f8ff0 -------- -------- -------- --------  ................
    db3f9000 02000109 00096a02 00000000 00000000  .....j..........
    db3f9010 bd0aa200 5ef280e3 000000cd 00000000  .......^........
    db3f9020 00000000 00000000 00000000 c6e85800  .............X..
    db3f9030 00000012 00000000 00000000 00000000  ................
    db3f9040 00000000 d5ef2800 838b8a0c 74d2c701  .....(.........t
    db3f9050 05132305 73430323 ccd0cf20 cadcd4ca  .#..#.Cs .......
    db3f9060 1d2e0cd0 6c646d17 03a86a60 622dcd24  .....mdl`j..$.-b
    db3f9070 8c0c8da3 8c0d740c 15cc0d75 0c2c0db4  .....t..u.....,.
    db3f9080 8c0c140c accc4cac b963cc8c 00000000  .....L....c.....
    db3f9090 494affff c92851cd c8554dcc 48512d2f  ..JI.Q(..MU./-QH
    db3f90a0 2d49cccb 14ad7306 0000b80c ffff0000  ..I-.s..........
    db3f90b0 512d4f4a 2c4dcdc8 72180a80 00000001  JO-Q..M,...r....
    db3f90c0 75f2ffff 0f8e0a0c 52b1f20d 294fcdc8  ...u.......R..O)
    db3f90d0 f7d549cd e62a2c4d 00000002 8b02ffff  .I..M,*.........

dump_all_maps print map information

memory map (2172 entries): (fault address prefixed with --->)
    0a125000-0a126fff r--         0      2000  /system/bin/app_process32 (BuildId: 4b3fcbf21f6cb09225ed60ef016d3f87)
    0a127000-0a129fff r-x      2000      3000  /system/bin/app_process32 (BuildId: 4b3fcbf21f6cb09225ed60ef016d3f87)
    0a12a000-0a12afff r--      5000      1000  /system/bin/app_process32 (BuildId: 4b3fcbf21f6cb09225ed60ef016d3f)
    0a12b000-0a12bfff rw-         0      1000
    12c00000-15fbffff rw-         0   33c0000  [anon:dalvik-main space (region space)]
    15fc0000-161bffff rw-         0    200000  [anon:dalvik-main space (region space)]

  dump_log_file (log, pid, "system", tail); print system log information
  dump_log_file (log, pid, "main", tail); print mainlog information

--------- tail end of log main
03-07 02:46:27.072 23051 23051 W System.err: 	at com.tencent.qqlive.ona.base.QQLiveApplication.attachBaseContext(QQLiveApplication.java:1223)
03-07 02:46:27.072 23051 23051 W System.err: 	at com.tencent.qqlive.ona.base.QQLiveApplicationWrapper.attachBaseContext(QQLiveApplicationWrapper.java:197)
03-07 02:46:27.073 23051 23051 W System.err: 	at android.app.Application.attach(Application.java:376)

to sum up

• When an exception occurs in the Native process, such as a NULL pointer
• The operating system will go to the address of the exception vector table to handle the exception, and then send a signal
• The signal processing function registered in debugged_init will be processed
• Create a pseudo thread to start the crash_dump process, crash_dump will get the crash information of each thread in the current process
• The tombstoned process is started when the computer is turned on, and the socket is registered when the computer is turned on, waiting for monitoring
• When connecting to tombstoned process in crash_dump, a file descriptor under / data / tombstones / will be returned based on the dump_type passed
• The crash_dump process subsequently writes the details of all threads to the tombstone file through the engrave_tombstone function
• Then the corresponding tombstone_XX file is generated under / data / tombstones