Overview of the Mozilla symbols server
Presented in September, 2020
Will Kahn-Greene
When we compile Firefox, the compiler generates binaries with debugging symbols in them.
We use dump_syms to extract the debugging symbols from the binaries and put them in SYM files.
A symbol is something like "This set of bytes in this binary correspond to function JammyJamJam on line 200 in source file jampocalypso.c."
Let's do a quick example.
We can create a new Rust project, build it, then extract symbols from it.
That creates a SYM file like this:
MODULE Linux x86_64 D48F191186D67E69DF025AD71FB91E1F0 testproj
...
FILE 5 /home/willkg/projects/testproj/src/main.rs
...
FUNC 5380 44 0 testproj::main
5380 9 1 5
5389 36 2 5
53bf 5 3 5
(This file is 6,809 lines for a "Hello, world!" binary.)
Given an array of modules that are loaded and an array of memory offsets for a stack:
We can use the SYM files, look up the memory addresses, and find the symbols for them.
Tada!
That's enough about symbols, SYM files, and symbolication.
(Stop me if you've heard this before.)
A long time ago, someone threw together a project named Snappy to serve SYM files stored in an AWS S3 bucket for released builds. 😀
That person left. ðŸ˜
Snappy was running under their desk. 🥵
🔥🔥🔥 Snappy died. Panic ensued. 🔥🔥🔥
Someone was tasked with getting Snappy to run. 😬
And also put it on a server with monitoring. 😖
That effort was fraught with pain and suffering.
In another galaxy, Socorro managed uploading SYM files to an AWS S3 bucket.
Socorro uses SYM files to symbolicate stacks in crash reports.
Rather than continue toiling on Snappy and in an effort to reduce the scope of Socorro, we decided to start a new project that centralized all the symbol things.
Thus was born Tecken.
"Tecken" is the Swedish word for "symbol".
Tecken does four things:
Tecken is used by:
You can read more about Tecken:
https://tecken.readthedocs.io/