TECKEN

Overview of the Mozilla symbols server

Presented in September, 2020

Will Kahn-Greene

What is a symbol?

When we compile Firefox, the compiler generates binaries with debugging symbols in them.

We use dump_syms to extract the debugging symbols from the binaries and put them in SYM files.

A symbol is something like "This set of bytes in this binary correspond to function JammyJamJam on line 200 in source file jampocalypso.c."

Let's do a quick example.

We can create a new Rust project, build it, then extract symbols from it.

That creates a SYM file like this:


          MODULE Linux x86_64 D48F191186D67E69DF025AD71FB91E1F0 testproj
          ...
          FILE 5 /home/willkg/projects/testproj/src/main.rs
          ...
          FUNC 5380 44 0 testproj::main
          5380 9 1 5
          5389 36 2 5
          53bf 5 3 5
          

(This file is 6,809 lines for a "Hello, world!" binary.)

Given an array of modules that are loaded and an array of memory offsets for a stack:

We can use the SYM files, look up the memory addresses, and find the symbols for them.

Tada!

That's enough about symbols, SYM files, and symbolication.

Short history of Tecken

(Stop me if you've heard this before.)

A long time ago, someone threw together a project named Snappy to serve SYM files stored in an AWS S3 bucket for released builds. 😀

That person left. 😭

Snappy was running under their desk. 🥵

🔥🔥🔥 Snappy died. Panic ensued. 🔥🔥🔥

Someone was tasked with getting Snappy to run. 😬

And also put it on a server with monitoring. 😖

That effort was fraught with pain and suffering.

🤕

In another galaxy, Socorro managed uploading SYM files to an AWS S3 bucket.

Socorro uses SYM files to symbolicate stacks in crash reports.

Rather than continue toiling on Snappy and in an effort to reduce the scope of Socorro, we decided to start a new project that centralized all the symbol things.

Thus was born Tecken.

"Tecken" is the Swedish word for "symbol".

What does Tecken do?

Tecken does four things:

  • API for uploading symbols
  • API for downloading symbols
  • API for symbolication
  • webapp for investigating and debugging symbols issues and also granting permissions for uploading

Tecken is used by:

  • release engineering uploads symbols for product builds
  • engineers download symbols when debugging
  • Socorro downloads symbols to symbolicate stacks in crash reports
  • Firefox devtools profile symbolicates stacks during profiling
  • (soon) we will symbolicate stacks in crash pings and generate signatures
  • etc

You can read more about Tecken:

https://tecken.readthedocs.io/

El Fin!