This article outlines how to use Breakpad in a C++ executable program or dynamic link library on the Mac platform.
As explained in Getting Started with Breakpad , the entire breakpad system includes not only the breakpad client library connected in the C++ program, but also the tool dump_syms for generating symbols, the minidump_stackwalk tool for converting the generated minidump into symbols, and the tool for uploading minidump files. This article will briefly explain how to compile and use all these binaries.
Build Breadpad
Here is an explanation of how to build the GN + ninja build system based on WebRTC / OpenRTCClient .
Download the source code of OpenRTCClient through Git . Then execute the following command in the source code root directory of OpenRTCClient ./build_system/webrtc_pack mac debug crash_catch_system. This will generate the client static library file build/mac/debug/libbreakpad_client.a , which contains static libraries for the two CPU architectures arm64 and x64. It will also generate Symbol files and tools to convert generated minidumps to symbols are located in build/mac/x64/debug .
Integrate Breakpad into your application
First, configure your build process to add the directory address of libbreakpad_client.a to the search path for the link library file, add the breakpad_client library for the link, link libbreakpad_client.a into your binary file, and then set the include path to include the breakpad source tree src directory .
Although the class interfaces provided by Google breakpad itself ExceptionHandlerare not very different on various platforms, they are not exactly the same. This puts a certain burden on breakpad access.
In OpenRTCClient , the author implemented a simple encapsulation, namely open_rtc::InstallCrashHandler()and open_rtc::UninstallCrashHandler(). To use this interface, include the crash handler header file:
#include "client/crash_handler.h"
Call open_rtc::InstallCrashHandler()to install the crash handler. open_rtc::UninstallCrashHandler()Exception handling is active until after calling , so you should call it as early as possible in your program's startup process open_rtc::InstallCrashHandler()and keep it active as close to shutdown as possible. To do anything useful, open_rtc::InstallCrashHandler()you need a path to which the minidump can be written, and a callback function to receive information about the minidump that has been written:
#include <stdlib.h>
#include "client/crash_handler.h"
bool minidumpCallback(const char* dump_dir,
const char* minidump_id,
void* context,
bool succeeded) {
printf("Dump path: %s, minidump_id %s\n", dump_dir, minidump_id);
return succeeded;
}
static void crashfunc() {
volatile int* a = (int*)(NULL);
*a = 1;
}
int main(int argc, const char *argv[]) {
open_rtc::InstallCrashHandler(".", nullptr, minidumpCallback, nullptr);
crashfunc();
return 0;
}
Compile and run this sample program, which should generate a minidump file in the current directory, and it should print out the directory path and id of the minidump file before exiting. The id is the file name of the minidump file. The actual minidump file contains Has the file extension ".dmp".
You can also directly use the class provided by Google breakpad ExceptionHandlerto access breakpad. At this time, the header file is included first:
#include "client/mac/handler/exception_handler.h"
Then create ExceptionHandlerthe class object. ExceptionHandlerException handling is active during the entire life cycle of the object, so it should be instantiated as early as possible during program startup and kept active as close to the shutdown state as possible. To do anything useful, ExceptionHandlerthe constructor needs a path to which the minidump can be written, and a callback function to receive information about the minidump that has been written:
#include <stdio.h>
#include "client/mac/handler/exception_handler.h"
bool MinidumpCallback(const char* dump_dir,
const char* minidump_id,
void* context,
bool succeeded) {
printf("Dump path: %s, minidump_id %s\n", dump_dir, minidump_id);
return succeeded;
}
int main(int argc, const char *argv[]) {
google_breakpad::ExceptionHandler eh(".", NULL, MinidumpCallback, NULL,
true, nullptr);
crashfunc();
}
Compiling and running this program produces the same results as the previous program.
According to the WebRTC build configuration, USE_PROTECTED_ALLOCATIONS=1macro definition is enabled ( webrtc/third_party/breakpad/BUILD.gn). But there is a piece of code like the following in breakpad ( webrtc/third_party/breakpad/breakpad/src/client/mac/handler/exception_handler.cc):
bool ExceptionHandler::InstallHandler() {
// If a handler is already installed, something is really wrong.
if (gProtectedData.handler != NULL) {
return false;
}
if (!IsOutOfProcess()) {
struct sigaction sa;
memset(&sa, 0, sizeof(sa));
sigemptyset(&sa.sa_mask);
sigaddset(&sa.sa_mask, SIGABRT);
sa.sa_sigaction = ExceptionHandler::SignalHandler;
sa.sa_flags = SA_SIGINFO;
scoped_ptr<struct sigaction> old(new struct sigaction);
if (sigaction(SIGABRT, &sa, old.get()) == -1) {
return false;
}
old_handler_.swap(old);
gProtectedData.handler = this;
#if USE_PROTECTED_ALLOCATIONS
assert(((size_t)(gProtectedData.protected_buffer) & PAGE_MASK) == 0);
mprotect(gProtectedData.protected_buffer, PAGE_SIZE, PROT_READ);
#endif
}
try {
#if USE_PROTECTED_ALLOCATIONS
previous_ = new (gBreakpadAllocator->Allocate(sizeof(ExceptionParameters)) )
ExceptionParameters();
#else
previous_ = new ExceptionParameters();
#endif
}
catch (std::bad_alloc) {
return false;
}
This code ExceptionHandleris executed when the object is created. The global variables gBreakpadAllocatorare actually webrtc/third_party/breakpad/breakpad/src/client/mac/Framework/Breakpad.mminitialized in , and this code BreakpadCreate()will only be executed by connecting the breakpad client in the OC code. In this way, when the breakpad client is directly accessed through breakpad's C++ interface, the gBreakpadAllocatorvalue of the global variable is empty, which in turn causes ExceptionHandlerthe program to crash due to a null pointer when the object is created.
In the OpenRTCClient project, USE_PROTECTED_ALLOCATIONS=1the problem caused by accessing the breakpad client through the C++ interface is solved by removing the definition of macros. Of course checking gBreakpadAllocatorthe value of can also solve this problem.
Generate symbols for programs
As explained above, building breakpad through OpenRTCClient will generate various tools together. Here we take OpenRTCClient 's smoke_test as an example to illustrate this process.
Build smoke_test:
OpenRTCClient % ./build_system/webrtc_build build:smoke_test mac x64 debug
Generate symbol files:
OpenRTCClient % ./build/mac/x64/debug/dump_syms -g build/mac/x64/debug/smoke_test.dSYM build/mac/x64/debug/smoke_test > build/mac/x64/debug/smoke_test.sym
webrtc/build/toolchain/apple/linker_driver.pyThere seems to be some problems with the M98 version of the WebRTC code that OpenRTCClient is based on
warning: (x86_64) could not find object file symbol for symbol __ZNSt3__17forwardIPN6webrtc20AudioEncoderG722ImplEEEOT_RNS_16remove_referenceIS4_E4typeE
warning: (x86_64) could not find object file symbol for symbol __ZNSt3__122__compressed_pair_elemIPN6webrtc12AudioEncoderELi0ELb0EEC2IPNS1_20AudioEncoderG722ImplEvEEOT_
warning: (x86_64) could not find object file symbol for symbol __ZNSt3__122__compressed_pair_elemINS_14default_deleteIN6webrtc12AudioEncoderEEELi1ELb1EEC2INS1_INS2_20AudioEncoderG722ImplEEEvEEOT_
warning: (x86_64) could not find object file symbol for symbol __ZNSt3__114default_deleteIN6webrtc12AudioEncoderEEC2INS1_20AudioEncoderG722ImplEEERKNS0_IT_EEPNS_9enable_ifIXsr14is_convertibleIPS6_PS2_EE5valueEvE4typeE
warning: (x86_64) could not find object file symbol for symbol __ZN6webrtc16AudioEncoderOpus11SdpToConfigERKNS_14SdpAudioFormatE
warning: (x86_64) could not find object file symbol for symbol __ZN6webrtc16AudioEncoderOpus16MakeAudioEncoderERKNS_22AudioEncoderOpusConfigEiN4absl8optionalINS_16AudioCodecPairIdEEE
warning: (x86_64) could not find object file symbol for symbol __ZN6webrtc16AudioEncoderOpus17QueryAudioEncoderERKNS_22AudioEncoderOpusConfigE
warning: (x86_64) could not find object file symbol for symbol __ZN6webrtc16AudioEncoderOpus23AppendSupportedEncodersEPNSt3__16vectorINS_14AudioCodecSpecENS1_9allocatorIS3_EEEE
. . . . . .
warning: (x86_64) could not find object file symbol for symbol _dav1d_itx_dsp_init_x86_8bpc
warning: (x86_64) could not find object file symbol for symbol _iclip_u8
warning: (x86_64) could not find object file symbol for symbol _dav1d_loop_filter_dsp_init_8bpc
warning: (x86_64) could not find object file symbol for symbol _dav1d_loop_filter_dsp_init_x86_8bpc
warning: (x86_64) could not find object file symbol for symbol _iclip_u8
warning: (x86_64) could not find object file symbol for symbol _dav1d_loop_restoration_dsp_init_8bpc
warning: (x86_64) could not find object file symbol for symbol _dav1d_loop_restoration_dsp_init_x86_8bpc
warning: (x86_64) could not find object file symbol for symbol _iclip_u8
warning: (x86_64) could not find object file symbol for symbol _dav1d_mc_dsp_init_8bpc
warning: (x86_64) could not find object file symbol for symbol _dav1d_mc_dsp_init_x86_8bpc
warning: (x86_64) could not find object file symbol for symbol _dav1d_get_cpu_flags_x86
build success
This issue is also discussed on the Chromium mailing list, macOS build warnings . This issue seems to cause symbol file generation to fail:
smoke_test.dSYM/Contents/Resources/DWARF/smoke_test: the DIE at offset 0x134c917 has a DW_AT_specification attribute referring to the DIE at offset 0x134c8c6, which was not marked as a declaration
smoke_test.dSYM/Contents/Resources/DWARF/smoke_test: the DIE at offset 0x134ca01 has a DW_AT_specification attribute referring to the DIE at offset 0x134c9b1, which was not marked as a declaration
smoke_test.dSYM/Contents/Resources/DWARF/smoke_test: the DIE at offset 0x134caeb has a DW_AT_specification attribute referring to the DIE at offset 0x134ca9b, which was not marked as a declaration
smoke_test.dSYM/Contents/Resources/DWARF/smoke_test: the DIE at offset 0x134cc0b has a DW_AT_specification attribute referring to the DIE at offset 0x134cbc9, which was not marked as a declaration
Assertion failed: (it_debug == externs_.end() || (*it_debug)->address >= range.address + range.size), function AddFunction, file module.cc, line 175.
Linking the generated binary causes build/mac/x64/debug/dump_symsa crash and execution failure.
In the OpenRTCClient project, this problem is solved by replacing webrtc/build/toolchain/apple/linker_driver.pywith M88's webrtc/build/toolchain/mac/linker_driver.py. This will successfully generate the symbol file.
In order to use these symbol files through minidump_stackwalktools, they need to be placed in a specific directory structure. The first line of the symbol file contains the information needed to generate this directory structure, such as (your output may be different):
OpenRTCClient % ./build/mac/x64/debug/dump_syms -g build/mac/x64/debug/smoke_test.dSYM build/mac/x64/debug/smoke_test > build/mac/x64/debug/smoke_test.sym
OpenRTCClient % head build/mac/x64/debug/smoke_test.sym
MODULE mac x86_64 4C4C44D755553144A13B271E21EDCA370 smoke_test
FILE 0 /Users/henryhan/Projects/opensource/OpenRTCClient/build/mac/x64/debug/../../../../../../../../../Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.0.sdk/System/Library/Frameworks/CoreFoundation.framework/Headers/CFBase.h
FILE 1 /Users/henryhan/Projects/opensource/OpenRTCClient/build/mac/x64/debug/../../../../../../../../../Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.0.sdk/System/Library/Frameworks/CoreFoundation.framework/Headers/CFByteOrder.h
FILE 2 /Users/henryhan/Projects/opensource/OpenRTCClient/build/mac/x64/debug/../../../../../../../../../Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.0.sdk/usr/include/_ctype.h
FILE 3 /Users/henryhan/Projects/opensource/OpenRTCClient/build/mac/x64/debug/../../../../../../../../../Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.0.sdk/usr/include/_wctype.h
FILE 4 /Users/henryhan/Projects/opensource/OpenRTCClient/build/mac/x64/debug/../../../../../../../../../Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.0.sdk/usr/include/architecture/byte_order.h
FILE 5 /Users/henryhan/Projects/opensource/OpenRTCClient/build/mac/x64/debug/../../../../../../../../../Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.0.sdk/usr/include/dispatch/once.h
FILE 6 /Users/henryhan/Projects/opensource/OpenRTCClient/build/mac/x64/debug/../../../../../../../../../Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.0.sdk/usr/include/dispatch/queue.h
FILE 7 /Users/henryhan/Projects/opensource/OpenRTCClient/build/mac/x64/debug/../../../../../../../../../Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.0.sdk/usr/include/libkern/i386/_OSByteOrder.h
FILE 8 /Users/henryhan/Projects/opensource/OpenRTCClient/build/mac/x64/debug/../../../../../../../../../Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.0.sdk/usr/include/math.h
OpenRTCClient % mkdir -p build/mac/x64/debug/symbols/smoke_test/4C4C44D755553144A13B271E21EDCA370
OpenRTCClient % mv build/mac/x64/debug/smoke_test.sym build/mac/x64/debug/symbols/smoke_test/4C4C44D755553144A13B271E21EDCA370
Process minidump to generate stack trace
Here we first simulate a crash. There are several test cases in smoke_test of the OpenRTCClient project that can produce crashes:
OpenRTCClient % ./build/mac/x64/debug/smoke_test '--gtest_filter=CrashCatchTest.*' --gtest_also_run_disabled_tests
Note: Google Test filter = CrashCatchTest.*
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from CrashCatchTest
[ RUN ] CrashCatchTest.DISABLED_crash_catch_common
Dump path: ~/OpenRTCClient/build/mac/x64/debug, minidump_id A25EA989-A9EC-400F-B44F-AF4B517101A1
When a crash occurs, some information related to the captured crash minidump file will be spit out. Based on this information we can know that the full path of the crash file is ~/OpenRTCClient/build/mac/x64/debug/A25EA989-A9EC-400F-B44F-AF4B517101A1.dmp.
Breakpad includes a minidump_stackwalktool called , which takes a minidump and its corresponding symbol in text format and generates a symbolic stack trace. If you compiled the Breakpad source code according to the instructions above, it should be located OpenRTCClient/build/mac/x64/debugin the directory. Simply pass in the minidump and symbolic path as command line arguments:
OpenRTCClient % build/mac/x64/debug/minidump_stackwalk ./build/mac/x64/debug/A25EA989-A9EC-400F-B44F-AF4B517101A1.dmp ./build/mac/x64/debug/symbols
Operating system: Mac OS X
11.6.7 20G630
CPU: amd64
family 6 model 158 stepping 10
12 CPUs
GPU: UNKNOWN
Crash reason: EXC_BAD_ACCESS / KERN_INVALID_ADDRESS
Crash address: 0x0
Process uptime: 0 seconds
Thread 0 (crashed)
0 smoke_test!crashfunc() [crash_catch_test.cpp : 48 + 0x0]
rax = 0x0000000000000000 rdx = 0x00000000000fb340
rcx = 0x0000000000000000 rbx = 0x0000000000000000
rsi = 0x00000000d587bb72 rdi = 0x0000000107d11080
rbp = 0x00007ffeeebff1c0 rsp = 0x00007ffeeebff1c0
r8 = 0x00000000000000d7 r9 = 0x0000000000000008
r10 = 0x00007ff972a00000 r11 = 0x0000000000000000
r12 = 0x0000000000000000 r13 = 0x0000000000000000
r14 = 0x0000000000000000 r15 = 0x0000000000000000
rip = 0x000000010101744e
Found by: given as instruction pointer in context
1 smoke_test!CrashCatchTest_DISABLED_crash_catch_common_Test::TestBody() [crash_catch_test.cpp : 53 + 0x5]
rbp = 0x00007ffeeebff200 rsp = 0x00007ffeeebff1d0
rip = 0x00000001010173ff
Found by: previous frame's frame pointer
2 smoke_test!void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) [gtest.cc : 2631 + 0x2]
rbp = 0x00007ffeeebff260 rsp = 0x00007ffeeebff210
rip = 0x0000000101095e4b
Found by: previous frame's frame pointer
3 smoke_test!void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) [gtest.cc : 2686 + 0x15]
rbp = 0x00007ffeeebff2d0 rsp = 0x00007ffeeebff270
rip = 0x00000001010719c7
Found by: previous frame's frame pointer
4 smoke_test!testing::Test::Run() [gtest.cc : 2706 + 0x24]
rbp = 0x00007ffeeebff330 rsp = 0x00007ffeeebff2e0
rip = 0x0000000101071911
Found by: previous frame's frame pointer
5 smoke_test!testing::TestInfo::Run() [gtest.cc : 2885 + 0x5]
rbp = 0x00007ffeeebff3b0 rsp = 0x00007ffeeebff340
rip = 0x0000000101072455
Found by: previous frame's frame pointer
6 smoke_test!testing::TestSuite::Run() [gtest.cc : 3044 + 0x5]
rbp = 0x00007ffeeebff430 rsp = 0x00007ffeeebff3c0
rip = 0x00000001010732fb
Found by: previous frame's frame pointer
7 smoke_test!testing::internal::UnitTestImpl::RunAllTests() [gtest.cc : 5915 + 0x5]
rbp = 0x00007ffeeebff510 rsp = 0x00007ffeeebff440
rip = 0x000000010107f763
Found by: previous frame's frame pointer
8 smoke_test!bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) [gtest.cc : 2631 + 0x2]
rbp = 0x00007ffeeebff570 rsp = 0x00007ffeeebff520
rip = 0x000000010109bccb
Found by: previous frame's frame pointer
9 smoke_test!bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) [gtest.cc : 2686 + 0x15]
rbp = 0x00007ffeeebff5e0 rsp = 0x00007ffeeebff580
rip = 0x000000010107f277
Found by: previous frame's frame pointer
10 smoke_test!testing::UnitTest::Run() [gtest.cc : 5482 + 0x27]
rbp = 0x00007ffeeebff650 rsp = 0x00007ffeeebff5f0
rip = 0x000000010107f15f
Found by: previous frame's frame pointer
11 smoke_test!RUN_ALL_TESTS() [gtest.h : 2497 + 0x5]
rbp = 0x00007ffeeebff660 rsp = 0x00007ffeeebff660
rip = 0x000000010104ce41
Found by: previous frame's frame pointer
12 smoke_test!main [main.cpp : 12 + 0x5]
rbp = 0x00007ffeeebff690 rsp = 0x00007ffeeebff670
rip = 0x000000010104cdf2
Found by: previous frame's frame pointer
13 libdyld.dylib + 0x15f3d
rbp = 0x00007ffeeebff6a0 rsp = 0x00007ffeeebff6a0
rip = 0x00007fff2036bf3d
Found by: previous frame's frame pointer
Loaded modules:
0x101000000 - 0x104299fff smoke_test ??? (main)
0x7fff20088000 - 0x7fff20089fff libsystem_blocks.dylib ???
0x7fff2008a000 - 0x7fff200bffff libxpc.dylib ???
0x7fff200c0000 - 0x7fff200d7fff libsystem_trace.dylib ???
0x7fff200d8000 - 0x7fff20175fff libcorecrypto.dylib ???
0x7fff20176000 - 0x7fff201a2fff libsystem_malloc.dylib ???
0x7fff201a3000 - 0x7fff201e7fff libdispatch.dylib ???
0x7fff201e8000 - 0x7fff20221fff libobjc.A.dylib ???
It produces verbose output on stderr and a stack trace on stdout, so you may need to redirect stderr.
Reference documentation
Debugging using Breakpad’s dump file under mac
Using breakpad in Linux programs
Correct compilation and general usage of breakpad