docs/linux/debugging_minidump.md

# Linux Minidump Code Lab

[TOC]

## About Minidumps

Minidump is a file format for storing parts of a program's state for later
inspection. [Microsoft's
Documentation](https://docs.microsoft.com/en-us/windows/desktop/api/minidumpapiset/)
defines the format though the [Rust
Documentation](https://docs.rs/minidump/latest/minidump/format/index.html)
is sometimes easier to navigate. The minidump implementation and tools used by
Chrome are
[Breakpad](https://chromium.googlesource.com/breakpad/breakpad/+/refs/heads/main/README.md)
and
[Crashpad](https://chromium.googlesource.com/crashpad/crashpad/+/main/README.md).
However, the tools of interest here are from the Breakpad project.

## Create a Minidump

When Chrome crashes it writes out a minidump file. The minidump file is written
under the application product directory. On Linux this is
`<XDG_CONFIG_HOME>/<app-name>/Crash Reports`. The default for `XDG_CONFIG_HOME`
is `~.config`. Common `<app-name>`s are `chromium`, `google-chrome`,
`google-chrome-beta`, and `google-chrome-unstable`. A typical example is
`~.config/google-chrome/Crash Reports`. When a minidump is uploaded it will be
moved between the `new`, `pending`, and `completed` subdirectories. The minidump
file is named something like `<uuid>.dmp`. If the minidump is uploaded to the
crash reporting system, the `<uuid>.meta` file will contain the crash report id.
Those with access can find the uploaded report at `go/crash/<report-id>`, where
the minidump file will be available with a name like
`upload_file_minidump-<report-id>.dmp`.

To create a minidump, you can use a local build of Chromium or a release
version of Chrome. Run the browser with the environment variable
`CHROME_HEADLESS=1`, which enables crash dumping but prevents crash dumps from
being uploaded and deleted. Something like `$ env CHROME_HEADLESS=1
./out/debug/chrome-wrapper` or `$ env CHROME_HEADLESS=1
/opt/google/chrome/google-chrome`. Navigate to `chrome://crash` to trigger a
crash in the renderer process or reproduce your current crash bug. A crash dump
file should appear in the `Crash Reports` directory.

## Inspect the Minidump

To get an idea about what is in a minidump file, install the Okteta hex editor
and add the [Minidump Structure
Definition](https://github.com/bungeman/structures/tree/main/okteta-minidump).
Open the minidump previously created and explore the information it contains.

One quirk to notice is that there is a `ThreadListStream` which contains
`MINIDUMP_THREAD`s which contain a `MINIDUMP_THREAD_CONTEXT` and an
`ExceptionStream` which also contains a `MINIDUMP_THREAD_CONTEXT`. The thread
list contains the thread contexts as they existed when the crash reporter was
running. The exception's thread context is the state of the crashing thread at
the time that it crashed, which is generally the most interesting thread
context. When using the Breakpad tools for Linux (like `minidump_stackwalk` and
`minidump-2-core`) the thread context from the exception record is used in place
of the thread context associated with the corresponding thread.

Each `MINIDUMP_THREAD` contains a `StackMemoryRva` which is a reference to to a
copy of the stack on that thread at the time the crash handler was running.
Parsing a stack usefully requires additional debug information.
`minidump_stackwalk` or a debugger may be used to parse the stack memory to
create a usable trace.

## Get the Tools

From a Chromium checkout `ninja -C out/release minidump-2-core
minidump_stackwalk dump_syms`. From a [Breakpad
checkout](https://chromium.googlesource.com/breakpad/breakpad/) `make`. It can
be useful to use Breakpad directly on machines where one does not already have
a Chromium checkout.

When working at this level, one will also want to have `readelf` and
`objdump` available, which are available from most distributions.

## Get Executables and Symbols

In addition to the minidump, you will need the exact executables of Chromium or
Chrome which produced the minidump and those executable's symbols. If the
minidump was created locally, you already have the executables. Symbols for
Google Chrome's official builds are available from
`https://edgedl.me.gvt1.com/chrome/linux/symbols/google-chrome-debug-info-linux64-${VERSION}.zip`
where `${VERSION}` is any version of Google Chrome that has recently been served
to Stable, Beta, or Unstable (Dev) channels on Linux, like `114.0.5696.0`. Those
with access can find both executables and symbols for unreleased builds at
`go/chrome-symbols`.

For symbols outside of Chrome (like when the crash is happening in a shared
library) then symbols for the files of interest must be found. If the minidump
was created locally then install the symbol packages from your distribution.  If
not, you will need to track down the exact symbol files, which can be an
interesting exercise. For some distributions using the
[debuginfod](https://sourceware.org/elfutils/Debuginfod.html) system can be
quite helpful.

To ensure the correct binaries and debug symbols are used, the minidump contains
the build-id for each loaded module in the `ModuleListStream` in the
`CvRecordRva`'s `Signature`. This build-id is matched against a note
section of type `NT_GNU_BUILD_ID`, usually named `.note.gnu.build-id` in the
executable and symbol files. This note can be inspected with `readelf -n
<file>` like `readelf -n chrome` or `readelf -n chrome.debug` and looking for
the `.note.gnu.build-id` section. `readelf` reports the `Build ID` as the flat
bytes in the note, but Breakpad binaries like `stackwalk_minidimp` and
`dump_syms` will report and expect this truncated to a formatted Type 2 GUID
(without dashes). This means `readelf` will output a `<build-id>` like
33221100554477668899AABBCCDDEEFFXXXXXXXX but crashpad binaries will expect and
report this as a `<build-uuid>` of 00112233445566778899AABBCCDDEEFF.

The `.gnu_debuglink` section states which debug symbol file to use with a
striped binary. For example `readelf --string-dump=.gnu_debuglink chrome`
produces `chrome.debug`. This can be helpful to know for libraries with
interesting debug symbol setup, like libc.so.6.

## Create Symbolized Stack

Given a minidump with the name `mini.dmp`

`minidump_stackwalk mini.dmp > mini.stackwalk.nosym`

This will produce a mostly unsymbolized summary of the crash. To symbolize, look
toward the bottom of the output for `WARNING: No symbols, <file>, <build-uuid>`.
For each `<file>` which is of interest, `mkdir -p symbols/<file>/<build-uuid>` then
`dump_syms <file> <directory-with-file.debug> >
symbols/<file>/<build-uuid>/<file>.sym`. Ensure this output `<file>.sym`
contains the expected `<build-uuid>`. Then re-run `minidump_stackwalk` but
with the symbols directory, like `minidump_stackwalk mini.dmp symbols/ >
mini.stackwalk`.

The output of `minidump_stackwalk` is often quite useful and enough to track
down many issues. However, it does not fully use all of the information from
DWARF, so it is possible sometimes to get much better stack traces from a full
debugger like gdb. This is particularly true when functions have been
aggressively inlined.

## Create Core File

`minidump-2-core mini.dmp > mini.core`

## Loading into GDB

This works best if the binaries, symbols, and core files are all in different
directories to prevent gdb from automatically loading them into the wrong
locations. This is also generally necessary when using a system installed
version of Chrome. For full details see [the gdb
manual](https://sourceware.org/gdb/onlinedocs/gdb/Separate-Debug-Files.html).
The easiest way is to rename and move the .debug files to a directory structure
like `<debugdir>/debug/.build-id/nn/nnnnnnnn.debug` where `nn` are the first
two hex characters of the `build-id`, and `nnnnnnnn` are the rest of the hex
characters of the `build-id`. Note that this `build-id` is exactly what is
reported by `readelf -n <binary> | grep "Build ID"` and not the `build-uuid`
used by Breakpad. Then in gdb use `show debug-file-directory` to get the
`<previous-directories>` and `set debug-file-directory
<previous-directories>:<debugdir>/debug`.

The `offset`s used here are the offsets of the corresponding module from the
output of `minidump_stackwalk` or (equivalently) the value of
`ModuleListStream::Modules[]::BaseOfImage` from the minidump file (which can be
read with the structure definition).

```
$ gdb
(gdb) file <executable>
(gdb) show debug-file-directory
<previous directories>
(gdb) set debug-file-directory <previous directories>:<debugdir>/debug
(gdb) symbol-file <executable> -o <executable-offset>
(gdb) core-file <mini.core>
```

Running the commands in this order avoids needing to load the symbols twice and
maps the `<executable>` to the expected location.

To add an additional shared library it is possible to
`(gdb) add-symbol-file <shared-library> -o <shared-library-offset>`

Source paths in Chrome builds are relative to the `out/<build>` directory. If you
have a Chromium checkout at or around when the Chrome build was created, it can
be added to the debugger search path, like

```
(gdb) directory <path-to-chromium>/chromium/src/out/<build>/
```