Tecken/Socorro: Code info lookup: retrospective (2023)

Project

time:

6 weeks

impact:

• improved visibility on set of crash reports by fixing symbolication and signatures

• better understanding of consequences of sampling Firefox / Windows < 8.1 / ESR crash reports

Summary

In November, 2021, we wrote up a bug in the Tecken product to support download symbols files using the code file and code id.

In July, 2023, Mozilla migrated users for Windows 7, 8, and 8.1 from Firefox release channel to ESR channel. Firefox / Windows / release is sampled by the Socorro collector, so the system only accepts and processes 10% of incoming crash reports. When the users were migrated, their crash reports moved to an unsampled group, so then we were getting 100% of those incoming crash reports. That caused a volume increase of 30k.

I looked into adding another sampling rule for Firefox / Windows < 8.1 / ESR, but many of the crash reports had a xul module where there wasn't a debug file and debug id in the module list stream in the minidump, so we couldn't get symbols files for them. Because of that, we didn't have much visibility into this group of crash reports.

I looked at [bug 1746940] and worked out how to fix it. I thought it would be relatively straight-forward, so I prioritized working on it with the assumption it'd take a week to do.

I hit a bunch of road bumps and it took me 6 weeks to work through several attempts, settle on a final architecture, implement it, test it, and push all the pieces to production. I finished the work on October 24th, 2023.

The end result is a radically reduced number of crash reports where the stackwalker couldn't symbolicate xul.dll addresses because of missing debug file and debug id.

What is Socorro and Tecken?

Socorro is the crash ingestion pipeline for Mozilla products like Firefox, Fenix, Thunderbird, and MozillaVPN.

When Firefox crashes, the crash reporter asks the user if the user would like to send a crash report. If the user answers "yes!", then the crash reporter collects data related to the crash, generates a crash report, and submits that crash report as an HTTP POST to Socorro. Socorro saves the submitted crash report, processes it, and has tools for viewing and analyzing crash data.

Tecken is the Mozilla Symbols Server where we store Breakpad-style symbols files for binaries.

rust-minidump is the set of Rust crates that make up the stackwalker that the Socorro processor uses.

Problem

If a crash report had modules where we had no debug file or debug id, but did have a code file and code id, then those modules would not get symbolicated.

For example, here's the output from the stackwalker for a module from the module list stream in the minidump that has all the information we need for finding the symbols file:

{
    "base_addr": "0x01930000",
    "cert_subject": "Mozilla Corporation",
    "code_id": "652de0ed706d000",
    "corrupt_symbols": false,
    "debug_file": "xul.pdb",
    "debug_id": "569E0A6C6B88C1564C4C44205044422E1",
    "end_addr": "0x0899d000",
    "filename": "xul.dll",
    "loaded_symbols": true,
    "missing_symbols": false,
    "symbol_url": "https://symbols.mozilla.org/try/xul.pdb/569E0A6C6B88C1564C4C44205044422E1/xul.sym",
    "version": "115.4.0.8689"
}

Take note of debug_file and debug_id. The stackwalker uses that to request the symbols file using this url:

https://symbols.mozilla.org/try/xul.pdb/569E0A6C6B88C1564C4C44205044422E1/xul.sym
                                ^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                                |       |
                                |       debug id
                                debug file

Here's the output from the stackwalker for a module that's missing the debug file and debug id:

{
    "base_addr": "0x69920000",
    "cert_subject": "Mozilla Corporation",
    "code_id": "64e782c570c4000",
    "corrupt_symbols": false,
    "debug_file": "",
    "debug_id": "000000000000000000000000000000000",
    "end_addr": "0x709e4000",
    "filename": "xul.dll",
    "loaded_symbols": false,
    "missing_symbols": true,
    "symbol_url": null,
    "version": "117.0.0.8636"
}

Here, the debug file is the empty string and the debug id is null represented by 33 0s. Because the Tecken download API requires the debug file and debug id, the stackwalker doesn't have the information it needs to request the symbols file. So it doesn't. So it marks the file as missing.

Why does the minidump lack the debug file and debug id information?

Gabriele says this:

We don't know for sure, but most of the crashes which were missing those bits of information were OOMs. I suspect that windbg.dll failed to extract those values when memory was tight. This happened very often to xul.dll in particular, so I wonder if the fact that it's a very large library made this more likely.

That's about all we know.

How does this affect us?

The stackwalker can't find the symbols file for the xul module which affects stack unwinding and symbolication. That in turn affects signature generation.

Here's the top 6 lines of the October 23rd, 2023 weekly missing symbols report email:

xul.dll with no debug id accounts for like 10k crash reports out of like 300k crash reports.

Here's the top 10 signatures for the week ending October 22nd where xul.dll/000000000000000000000000000000000 is a module in the stack:

Not particularly helpful.

How do we tie code file and code id to debug file and debug id?

With our current system, the stackwalker has no way to figure out the debug file and debug id using the code file and code id. We need to maintain a map of code file / code id -> debug file / debug id somewhere.

Symbols files for Windows modules have this header:

MODULE windows arm64 46A0ADB3F299A70B4C4C44205044422E1 xul.pdb
INFO CODE_ID 64EC878F867C000 xul.dll
INFO GENERATOR mozilla/dump_syms 2.2.0

If Tecken reads and parses the headers of uploaded symbols files, it can save that information in the database. Then we can wrap that in some kind of API that the stackwalker in the Socorro processor can access.

A learning experience

My initial understanding suggested that to fix this, I would just need to make changes to Tecken and the stackwalker and Socorro wouldn't need any changes. That's what led me to believe I could fix this in a week. I was wrong.

Implementation decisions

Using the upload_fileupload table

There were two options for where to put the information about the symbols file:

1. the upload_fileupload table which had a record per symbol file upload attempt

2. a separate table with a foreign key to the upload_fileupload table

I decided to go with option 1. That meant we had to do a migration of a large table which meant we had to schedule an outage.

This seemed like a better idea than have to worry about data in another table. It's easier to build views. It's easier to maintain the data some of which expires after 3 months and some which expires after 2 years.

Overloading the download API

There were three options here:

1. use a completely separate API to do the code info lookup

2. create a new version of the download API that includes the code info lookup feature and also fixes issues with the download API (it's rooted at /, it's not versioned, ...)

3. overload the existing download API with the new code info lookup feature

I talked with Gabriele about this for a while especially how it impacts the undocumented symbols server protocol that is implemented by other symbols servers. We decided it made more sense to "generalize" the API such that it supports fetching symbols files by debug info (debug file / debug id) and code info (code file / code id).

With Tecken, the files are stored in AWS S3 by their debug info path. We could have stored these files in a code info path as well, but since these are the xul.dll files which are hundreds of mb in size, I decided to instead store the information in a database and implement a db lookup.

Checking all symbols suppliers

The stackwalker can be configured with multiple symbols suppliers. This is helpful if you're an engineer and you're running your own symbols supplier on your machine. This is helpful if you're some other company running your own Socorro and have multiple symbols servers you want to check.

In the case of multiple symbols suppliers, the stackwalker doesn't know which support the code info lookup feature. Because symbols suppliers are configured by a single url and the stackwalker process is ephemeral (it runs, processes a minidump, and then terminates), it seemed best to assume that any of the symbols suppliers could support a code info lookup and we should first figure out a debug file and debug id before then querying the cache and symbols suppliers for the symbols file.

Results

In the past, if a crash report had modules where we had no debug file or debug id, but did have a code file and code id, then those modules would not get symbolicated. There was no way in our system for something to take a code file/code id and figure out the debug file/debug id in order to find the symbols file in the symbols bucket.

Bug 1746940 will fix that going forward. Now the Mozilla Symbols Server captures information in the header of the symbols file and stores it in the database. Additionally, the Socorro stackwalker will request symbols files using the code file/code id and the Mozilla Symbols Server will look that up and figure out the debug file/debug id (if it exists) and return that. This allows the stackwalker to symbolicate symbols in modules which we couldn't get symbols files for before.

You'll see evidence of this in a couple of places:

1. crash reports that used to have "xul.dll" or other unsymbolicated things in them will now have symbolicated things--we'll see changes in signature reports and maybe topcrashers

2. the cases of module/000000000000000000000000000000000 will drop in the missing symbols report

Change in weekly missing symbols report

October 23rd, 2023:

October 30th, 2023:

The xul.dll/000000000000000000000000000000000 items don't show up until much later in the list now.

Here's the top 10 signatures for the week ending October 23nd where xul.dll/000000000000000000000000000000000 is a module in the stack:

Here's the top 10 signatures for the week ending October 30nd where xul.dll/000000000000000000000000000000000 is a module in the stack:

There are still a fair number of out-of-memory errors, but the signatures are much better now.

Skimming the last missing symbols report, I'm guessing this affects 10k out of like 300k crash reports. I'm not sure if those are all actionable or meaningful crash reports, but signatures like this (#3 in Top Crashers for Firefox 115.4.0esr):

OOM | large | mozalloc_abort | xul.dll | _PR_NativeRunThread | pr_root

switch to something like this:

OOM | large | mozalloc_abort | webrender::renderer::Renderer::render_impl

Random thoughts

I've been working on Socorro and Tecken alone for a while. I do self-review because there isn't anyone else to review the work I do. Historically, this has been good enough. I occasionally make mistakes, but they're caught in CI or stage environments. In the case where it gets to production, it's usually minor and I can fix it and push out a fix.

In this case, I made some big errors that probably would have been caught in code review. I definitely feel bad about them. I added a couple more things to think about when I do self-review so I don't do something like this again.

Further, starting October 16th, I'll have a team with other people who can review, so this will further reduce the likelihood of this occurrence.

Along with the changes for this project, I fixed a bunch of minor issues in the Tecken, socorro-stackwalker, and Socorro repositories to make it easier to implement and test these changes.

I also built a glossary for Tecken and one for Socorro that adds definitions for many of the terms used as well as links to references, specifications, etc.

Tecken glossary: https://tecken.readthedocs.io/en/latest/glossary.html

Socorro glossary: https://socorro.readthedocs.io/en/latest/glossary.html

Conclusion and where we could go from here

One caveat is that because we have to capture information from the symbol file and store it in the database in the Mozilla Symbols Server, we won't have code file/code id -> debug file/debug id information for symbols files that were uploaded before October 2023. We can backfill this information as we discover other files it'd be useful for.

That's it!

That's the story of how I implemented code info lookup which required changes in multiple systems.

Many thanks to Gabriele who helped me understand the problem, figure out how to fix it, and reviewed (and vastly improved) the rust-minidump changes. Also, to Markus who helped with understanding the problem and figuring out how to fix it. Also to Harold and Mikael who helped me work through the production outage I caused.

If you have any questions or bump into bugs, I hang out on #crashreporting on chat.mozilla.org. You can also write up a bug for Socorro.

Hopefully this helps. If not, let me know!