Will's Blog

Bleach 6.4.0 releases -- final release

Will Kahn-Greene — Fri, 05 Jun 2026 13:00:00 GMT

What is it?

Bleach is a Python library for sanitizing and linkifying text from untrusted sources for safe usage in HTML.

Bleach v6.4.0 released!

Bleach 6.4.0 includes two security fixes, a fix to tinycss2 dependency requirements, and some other things.

See the changes here:

https://bleach.readthedocs.io/en/latest/changes.html#version-6-4-0-june-5th-2026

Bleach v6.4.0 is the final release

I haven't used Bleach on a project in years, but I still had some time to maintain it. That changed about a year ago when I got re-orged into a new role and I haven't had time to do any Bleach work since then.

To recap, Bleach sits on top of html5lib which hasn't been actively maintained in years. It is dangerous to maintain Bleach in that context.

We vendored html5lib so we could make adjustments to the library to keep Bleach going. This is not a sustainable approach, but it was ok for the short term.

Over the years, we've talked about other options:

find another library to switch to
take over html5lib development
fork html5lib and vendor and maintain our fork
write a new HTML parser
etc

None of those are feasible for me.

Bleach has been a solo-maintained project for a while now. The world is crazy and it's much harder to build a team of trusted maintainers now than it was (or at least, it sure feels that way). I don't see any possibility of increasing the maintenance team or passing it to someone else responsibly.

Switching contexts from my regular work to Bleach is really hard. Bleach is complicated, the problem domain is complicated, and there's a lot of nuanced context. I can't just switch gears, spend 15 minutes on Bleach to do something, and then switch back to the rest of my day. I periodically get nag messages about this which are entirely valid, but there's nothing I can do about it. It doesn't feel great.

Then in 2025, Emil, a long-time Bleach contributor, built justhtml which gives us an easy migration path off of Bleach. He even took the time to write a migration guide.

Thoughts and statistics

In 2019, when I stepped down the first time, I wrote a post on stepping down.

In 2023, when I deprecated the project, I wrote a post on Bleach 6.0.0 and deprecation.

From the first commit on 2010-02-18 to today's final commit on 2026-06-05, the Bleach project lasted 16 years, 3 months — 5,951 days, or about 16.29 years.
There were 64 releases.
There were roughly 960 commits.
- From 80 roughly contributors
- Top 3:
  - Will Kahn-Greene: 462
  - James Socol: 182
  - Greg Guthe: 133
Roughly 5,040 lines of Python code excluding the vendored html5lib.
I was maintainer from October 2015 to now--that's a little under 11 years.

It feels weird to end a project that's outlived many of the Mozilla sites and Python web frameworks it was designed to protect.

What happens now?

This is the end of the project.

Bleach. Last release.

If you're still using Bleach, I think you have three options:

End your project. Maybe you don't need to be maintaining your thing anymore? Use Bleach as your reason to exit and do something different with your time on Earth.
Switch to the sanitizer API. Rework your project to use the sanitizer API.
- Spec: https://wicg.github.io/sanitizer-api/
- Docs: https://developer.mozilla.org/en-US/docs/Web/API/Element/setHTML
Swap Bleach out for justhtml. Emil provided a migration guide for switching from Bleach to justhtml.

Good luck with whatever option you choose!

Thanks!

Many thanks to James who created Bleach and gave it a set of first principles that guided our choices for 16 years.

Many thanks to Greg who I worked with on Bleach for a long while and maintained Bleach for several years. Working with Greg was always easy and his reviews were thoughtful and spot-on.

Many thanks to Emil who was a contributor to Bleach for a long while and created justhtml providing Bleach users a migration path.

Many thanks to Jonathan who, over the years, provided a lot of insight into how best to solve some of Bleach's more squirrely problems.

Many thanks to Sam who was an indispensible resource on HTML parsing and sanitizing text in the context of HTML.

Many thanks to all the users and contributors of Bleach!

Where to go for more

For more specifics on this release, see here: https://bleach.readthedocs.io/en/latest/changes.html#version-6-4-0-june-5th-2026

Documentation and quickstart here: https://bleach.readthedocs.io/en/latest/

Source code and issue tracker here: https://github.com/mozilla/bleach/

Open Source Project Maintenance 2025

Will Kahn-Greene — Tue, 28 Oct 2025 14:00:00 GMT

Every October, I do a maintenance pass on all my projects. At a minimum, that involves dropping support for whatever Python version is no longer supported and adding support for the most recently released Python version. While doing that, I go through the issue tracker, answer questions, and fix whatever I can fix. Then I release new versions. Then I think about which projects I should deprecate and figure out a deprecation plan for them.

This post covers the 2025 round.

TL;DR

sphinx-js -- transferred to pyodide organization
crashstats-tools and siggen -- transferred to the Mozilla crash ingestion team, which I'm no longer on
paul-mclendahand -- deprecated and archived
pip-stale -- deprecated and archived
everett -- released v3.5.0, then deprecated and archived
fillmore -- released v2.2.0, then deprecated and archived
kent -- released v2.2.0
markus -- released v5.2.0
bleach -- released v6.3.0

Switching from pyenv to uv

Will Kahn-Greene — Thu, 12 Sep 2024 16:00:00 GMT

Premise

The 0.4.0 release of uv does everything I currently do with pip, pyenv, pipx, pip-tools, and pipdeptree. Because of that, I'm in the process of switching to uv.

This blog post covers switching from pyenv to uv.

History

2024-08-29: Initial writing.
2024-09-12: Minor updates and publishing.
2024-09-20: Rename uv-sync (which is confusing) to uv-python-symlink.

Start state

I'm running Ubuntu Linux 24.04. I have pyenv installed using the the automatic installer. pyenv is located in $HOME/.pyenv/bin/.

I have the following Pythons installed with pyenv:

$ pyenv versions
  system
  3.7.17
  3.8.19
  3.9.19
* 3.10.14 (set by /home/willkg/mozilla/everett/.python-version)
  3.11.9
  3.12.3

I'm not sure why I have 3.7 still installed. I don't think I use that for anything.

My default version is 3.10.14 for some reason. I'm not sure why I haven't updated that to 3.12, yet.

In my 3.10.14, I have the following Python packages installed:

$ pip freeze
appdirs==1.4.4
argcomplete==3.1.1
attrs==22.2.0
cffi==1.15.1
click==8.1.3
colorama==0.4.6
diskcache==5.4.0
distlib==0.3.8
distro==1.8.0
filelock==3.14.0
glean-parser==6.1.1
glean-sdk==50.1.4
Jinja2==3.1.2
jsonschema==4.17.3
MarkupSafe==2.0.1
MozPhab==1.5.1
packaging==24.0
pathspec==0.11.0
pbr==6.0.0
pipx==1.5.0
platformdirs==4.2.1
pycparser==2.21
pyrsistent==0.19.3
python-hglib==2.6.2
PyYAML==6.0
sentry-sdk==1.16.0
stevedore==5.2.0
tomli==2.0.1
userpath==1.8.0
virtualenv==20.26.2
virtualenv-clone==0.5.7
virtualenvwrapper==6.1.0
yamllint==1.29.0

That probably means I installed the following in the Python 3.10.14 Python environment:

MozPhab
pipx
virtualenvwrapper

Maybe I installed some other things for some reason lost in the sands of time.

Then I had a whole bunch of things installed with pipx.

I have many open source projects all of which have a .python-version file listing the Python versions the project uses.

I think that covers the start state.

Steps

First, I made a list of things I had.

I listed all the versions of Python I have installed so I know what I need to reinstall with uv.
```
$ pyenv versions
```
I listed all the packages I have installed in my 3.10.14 environment (the default one).
```
$ pip freeze
```
I listed all the packages I installed with pipx.
```
$ pipx list
```

I uninstalled all the packages I installed with pipx.

$ pipx uninstall PACKAGE

Then I uninstalled pyenv and everything it uses. I followed the pyenv uninstall instructions:

$ rm -rf $(pyenv root)

Then I removed the bits in my shell that add to the PATH and set up pyenv and virtualenvwrapper.

Then I started a new shell that didn't have all the pyenv and virtualenvwrapper stuff in it.

Then I installed uv using the uv standalone installer.

Then I ran uv --version to make sure it was installed.

Then I installed the shell autocompletion.

$ echo 'eval "$(uv generate-shell-completion bash)"' >> ~/dotfiles/bash.d/20-uv.bash

Then I started a new shell to pick up those changes.

Then I installed Python versions:

$ uv python install 3.8 3.9 3.10 3.11 3.12
Searching for Python versions matching: Python 3.10
Searching for Python versions matching: Python 3.11
Searching for Python versions matching: Python 3.12
Searching for Python versions matching: Python 3.8
Searching for Python versions matching: Python 3.9
Installed 5 versions in 8.14s
 + cpython-3.8.19-linux-x86_64-gnu
 + cpython-3.9.19-linux-x86_64-gnu
 + cpython-3.10.14-linux-x86_64-gnu
 + cpython-3.11.9-linux-x86_64-gnu
 + cpython-3.12.5-linux-x86_64-gnu

When I type "python", I want it to be a Python managed by uv. Also, I like having "pythonX.Y" symlinks, so I created a uv-python-symlink-sync script which creates symlinks to uv-managed Python versions:

https://github.com/willkg/dotfiles/blob/main/dotfiles/bin/uv-python-symlink

Then I installed all my tools using uv tool install.

$ uv tool install PACKAGE

For tox, I had to install the tox-uv package in the tox environment:

$ uv tool install --with tox-uv tox

Now I've got everything I do mostly working.

So what does that give me?

I installed uv and I can upgrade uv using uv self update.

Python interpreters are managed using uv python. I can create symlinks to interpreters using uv-sync script. Adding new interpreters and removing old ones is pretty straight-forward.

When I type python, it opens up a Python shell with the latest uv-managed Python version. I can type pythonX.Y and get specific shells.

I can use tools written in Python and manage them with uv tool including ones where I want to install them in an "editable" mode.

I can write scripts that require dependencies and it's a lot easier to run them now.

I can create and manage virtual environments with uv venv.

Next steps

Delete all the .python-version files I've got.

Update documentation for my projects and add a uv tool install PACKAGE option to installation instructions.

Probably discover some additional things to add to this doc.

Thanks

Thank you to the Astral crew who wrote uv.

Thank you to Rob Hudson who goaded me into posting this finally rather than sit on it another month.

crashstats-tools v2.0.0 released!

Will Kahn-Greene — Thu, 25 Apr 2024 16:00:00 GMT

What is it?

crashstats-tools is a set of command-line tools for working with Crash Stats (https://crash-stats.mozilla.org/).

crashstats-tools comes with four commands:

supersearch: for performing Crash Stats Super Search queries
supersearchfacet: for performing aggregations, histograms, and cardinality Crash Stats Super Search queries
fetch-data: for fetching raw crash, dumps, and processed crash data for specified crash ids
reprocess: for sending crash report reprocess requests

v2.0.0 released!

There have been a lot of improvements since the last blog post for the v1.0.1 release. New commands, new features, improved cli ui, etc.

v2.0.0 focused on two major things:

improving supersearchfacet to support nested aggregation, histogram, and cardinality queries
moving some of the code into a crashstats_tools.libcrashstats module improving its use as a library

Improved supersearchfacet

The other day, Alex and team finished up the crash reporter Rust rewrite. The crash reporter rewrite landed and is available in Firefox, nightly channel, where build_id >= 20240321093532.

The crash reporter is one of the clients that submits crash reports to Socorro which is now maintained by the Observability Team. Firefox has multiple crash reporter clients and there are many ways that crash reports can get submitted to Socorro.

One of the changes we can see in the crash report data now is the change in User-Agent header. The new rewritten crash reporter sends a header of crash-reporter/1.0.0. That gets captured by the collector and put in the raw crash metadata.user_agent field. It doesn't get indexed, so we can't search on it directly.

We can get a sampling of the last 100 crash reports, download the raw crash data, and look at the user agents.

$ supersearch --num=100 --product=Firefox --build_id='>=20240321093532' \
    --release_channel=nightly > crashids.txt
$ fetch-data --raw --no-dumps --no-processed crashdata < crashids.txt
$ jq .metadata.user_agent crashdata/raw_crash/*/* | sort | uniq -c
     16 "crashreporter/1.0.0"
      2 "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:127.0) Gecko/20100101 Firefox/127.0"
      1 "Mozilla/5.0 (Windows NT 10.0; rv:127.0) Gecko/20100101 Firefox/127.0"
      2 "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:126.0) Gecko/20100101 Firefox/126.0"
     63 "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:127.0) Gecko/20100101 Firefox/127.0"
      1 "Mozilla/5.0 (X11; Linux x86_64; rv:126.0) Gecko/20100101 Firefox/126.0"
     12 "Mozilla/5.0 (X11; Linux x86_64; rv:127.0) Gecko/20100101 Firefox/127.0"
      3 "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:127.0) Gecko/20100101 Firefox/127.0"

16 out of 100 crash reports were submitted by the new crash reporter. We were surprised there are so many Firefox user agents. We discussed this on Slack. I loosely repeat it here because it's a great way to show off some of the changes of supersearchfacet in v2.0.0.

First, the rewritten crash reporter only affects the parent (aka main) process. The other processes have different crash reporters that weren't rewritten.

How many process types are there for Firefox crash reports in the last week? We can see that in the ProcessType annotation (docs) which is processed and saved in the process_type field (docs).

$ supersearchfacet --product=Firefox --build_id='>=20240321093532' --release_channel=nightly
    --_facets=process_type
process_type
 process_type | count
--------------|-------
 content      | 3664
 parent       | 2323
 gpu          | 855
 utility      | 225
 rdd          | 60
 plugin       | 18
 socket       | 2
 total        | 7147

Judging by that output, I would expect to see a higher percentage of crashreporter/1.0.0 in our sampling of 100 crash reports.

Turns out that Firefox uses different code to submit crash reports not just by process type, but also by user action. That's in the SubmittedFrom annotation (docs) which is processed and saved in the submitted_from field (docs).

$ supersearchfacet --product=Firefox --build_id='>=20240321093532' --release_channel=nightly \
    --_facets=submitted_from
submitted_from
 submitted_from | count
----------------|-------
 Auto           | 3477
 Client         | 1741
 CrashedTab     | 928
 Infobar        | 792
 AboutCrashes   | 209
 total          | 7147

What is "Auto"? The user can opt-in to auto-send crash reports. When Firefox upgrades and this setting is set, then Firefox will auto-send any unsubmitted crash reports. The nightly channel has two updates a day, so there's lots of opportunity for this event to trigger.

What're the counts for submitted_from/process_type pairs?

$ supersearchfacet --product=Firefox --build_id='>=20240321093532' --release_channel=nightly \
    --_aggs.process_type=submitted_from
process_type / submitted_from
 process_type / submitted_from | count
-------------------------------|-------
 content / Auto                | 2214
 content / CrashedTab          | 926
 content / Infobar             | 399
 content / AboutCrashes        | 125
 parent / Client               | 1741
 parent / Auto                 | 450
 parent / Infobar              | 107
 parent / AboutCrashes         | 25
 gpu / Auto                    | 565
 gpu / Infobar                 | 236
 gpu / AboutCrashes            | 54
 utility / Auto                | 198
 utility / Infobar             | 25
 utility / AboutCrashes        | 2
 rdd / Auto                    | 34
 rdd / Infobar                 | 23
 rdd / AboutCrashes            | 3
 plugin / Auto                 | 14
 plugin / CrashedTab           | 2
 plugin / Infobar              | 2
 socket / Auto                 | 2
 total                         | 7147

We can spot check these different combinations to see what the user-agent looks like.

For brevity, we'll just look at parent / Client in this blog post.

$ supersearch --num=100 --product=Firefox --build_id='>=20240321093532' --release_channel=nightly \
    --process_type=parent --submitted_from='~Client' > crashids_clarified.txt
$ fetch-data --raw --no-dumps --no-processed crashdata_clarified < crashids_clarified.txt
$ jq .metadata.user_agent crashdata_clarified/raw_crash/*/* | sort | uniq -c
    100 "crashreporter/1.0.0"

Seems like the crash reporter rewrite only affects crash reports where ProcessType=parent and SubmittedFrom=Client. All the other process_type/submitted_from combinations get submitted a different way where the user agent is the browser itself.

How many crash reports has the new crash reporter submitted over time?

$ supersearchfacet --_histogram.date=product --_histogram.interval=1d --denote-weekends \
    --date='>=2024-03-20' --date='<=2024-04-25' \
    --release_channel=nightly --product=Firefox --build_id='>=20240321093532' \
    --submitted_from='~Client' --process_type=parent
histogram_date.product
 histogram_date | Firefox | total
----------------|---------|-------
 2024-03-21     | 58      | 58
 2024-03-22     | 124     | 124
 2024-03-23 **  | 189     | 189
 2024-03-24 **  | 289     | 289
 2024-03-25     | 202     | 202
 2024-03-26     | 164     | 164
 2024-03-27     | 199     | 199
 2024-03-28     | 187     | 187
 2024-03-29     | 188     | 188
 2024-03-30 **  | 155     | 155
 2024-03-31 **  | 146     | 146
 2024-04-01     | 201     | 201
 2024-04-02     | 226     | 226
 2024-04-03     | 236     | 236
 2024-04-04     | 266     | 266
 2024-04-05     | 259     | 259
 2024-04-06 **  | 227     | 227
 2024-04-07 **  | 214     | 214
 2024-04-08     | 259     | 259
 2024-04-09     | 257     | 257
 2024-04-10     | 223     | 223
 2024-04-11     | 250     | 250
 2024-04-12     | 235     | 235
 2024-04-13 **  | 154     | 154
 2024-04-14 **  | 162     | 162
 2024-04-15     | 207     | 207
 2024-04-16     | 201     | 201
 2024-04-17     | 346     | 346
 2024-04-18     | 270     | 270
 2024-04-19     | 221     | 221
 2024-04-20 **  | 190     | 190
 2024-04-21 **  | 183     | 183
 2024-04-22     | 266     | 266
 2024-04-23     | 303     | 303
 2024-04-24     | 308     | 308

There are more examples in the crashstats-tools README.

crashstats_tools.libcrashstats library

Starting with v2.0.0, you can use crashstats_tools.libcrashstats as a library for Python scripts.

For example:

from crashstats_tools.libcrashstats import supersearch

results = supersearch(params={"_columns": "uuid"}, num_results=100)

for result in results:
    print(f"{result}")

libcrashstats makes using the Crash Stats API a little more ergonomic.

See the crashstats_tools.libcrashstats library documentation.

Be thoughtful about using data

Make sure to use these tools in compliance with our data policy:

https://crash-stats.mozilla.org/documentation/protected_data_access/

Where to go for more

See the project on GitHub which includes a README which contains everything about the project including examples of usage, the issue tracker, and the source code:

https://github.com/willkg/crashstats-tools

Let me know whether this helps you!

Observability Team Newsletter (2024q1)

Will Kahn-Greene — Tue, 16 Apr 2024 13:00:00 GMT

Observability Team is a team dedicated to the problem domain and discipline of Observability at Mozilla.

We own, manage, and support monitoring infrastructure and tools supporting Mozilla products and services. Currently this includes Sentry and crash ingestion related services (Crash Stats (Socorro), Mozilla Symbols Server (Tecken), and Mozilla Symbolication Service (Eliot)).

In 2024, we'll be working with SRE to take over other monitoring services they are currently supporting like New Relic, InfluxDB/Grafana, and others.

This newsletter covers an overview of 2024q1. Please forward it to interested readers.

Highlights

🤹 Observability Services: Change in user support
🏆 Sentry: Change in ownership
‼️ Sentry: Please don't start new trials
⏲️ Sentry: Cron monitoring trial ending April 30th
⏱️ Sentry: Performance monitoring pilot
🤖 Socorro: Improvements to Fenix support
🐛 Socorro: Support guard page access information

See details below.

Blog posts

None this quarter.

Detailed project updates

Observability Services: Change in user support

We overhauled our pages in Confluence, started an #obs-help Slack channel, created a new Jira OBSHELP project, built out a support rotation, and leveled up our ability to do support for Observability-owned services.

See our User Support Confluence page for:

where to get user support
documentation for common tasks (get protected data access, create a Sentry team, etc)
self-serve instructions

Hop in #obs-help in Slack to ask for service support, help with monitoring problems, and advice.

Sentry: Change in ownership

The Observability team now owns Sentry service at Mozilla!

We successfully completed Phase 1 of the transition in Q1. If you're a member of the Mozilla Sentry organization, you should have received a separate email about this to the sentry-users Google group.

We've overhauled Sentry user support documentation to improve it in a few ways:

easier to find "how to" articles for common tasks
best practices to help you set up and configure Sentry for your project needs

Check out our Sentry user guide.

There's still a lot that we're figuring out, so we appreciate your patience and cooperation.

Sentry: Please don't start new trials

Sentry sends marketing and promotional emails to Sentry users which often include links to start a new trial. Please contact us before starting any new feature trials in Sentry.

Starting new trials may prevent us from trialing those features in the future when we’re in a better position to evaluate the feature. There's no way for admins to prevent users from starting a trial.

Sentry: Cron monitoring trial ending April 30th

The Cron Monitoring trial that was started a couple of months ago will end April 30th.

Based on feedback so far and other factors, we will not be enabling this feature once the trial ends.

This is a good reminder to build in redundancy in your monitoring systems. Don't rely solely on trial or pilot features for mission critical information!

Once the trial is over, we'll put together an evaluation summary.

Sentry: Performance monitoring pilot

Performance Monitoring is being piloted by a couple of teams; it is not currently available for general use.

In the meantime, if you are not one of these pilot teams, please do not use Performance Monitoring. There is a shared transaction event quota for the entire Mozilla Sentry organization. Once we hit that quota, events are dumped.

If you have questions about any of this, please reach out.

Once the trial is over, we'll put together an evaluation summary.

Socorro: Improvements to Fenix support

We worked on improvements to crash ingestion and the Crash Stats site for the Fenix project:

1812771: Fenix crash reporter's Socorro crash reports for Java exceptions have "Platform" = "Unknown" instead of "Android"

Previously, the platform would be "Unknown". Now the platform for Fenix crash reports is "Android". Further, the platform_pretty_version includes the Android ABI version.

Figure 1: Screenshot of Crash Stats Super Search results showing Android versions for crash reports.

1819628: reject crash reports for unsupported Fenix forks

Forks of Fenix outside of our control periodically send large swaths of crash reports to Socorro. When these sudden spikes happened, Mozillians would spend time looking into them only to discover they're not related to our code or our users. This is a waste of our time and resources.

We implemented support for the Android_PackageName crash annotation and added a throttle rule to the collector to drop crash reports from any non-Mozilla releases of Fenix.

From 2024-01-18 to 2024-03-31, Socorro accepted 2,072,785 Fenix crash reports for processing and rejected 37,483 unhelpful crash reports with this new rule. That's roughly 1.7%. That's not a huge amount, but because they sometimes come in bursts with the same signature, they show up in Top Crashers wasting investigation time.

1884041: fix create-a-bug links to work with java_exception

A long time ago, in an age partially forgotten, Fenix crash reports from a crash in Java code would send a crash report with a JavaStackTrace crash annotation. This crash annotation was a string representation of the Java exception. As such, it was difficult-to-impossible to parse reliably.

In 2020, Roger Yang and Will Kahn-Greene spec'd out a new JavaException crash annotation. The value is a JSON-encoded structure mirroring what Sentry uses for exception information. This structure provides more information than the JavaStackTrace crash annotation did and is much easier to work with because we don't have to parse it first.

Between 2020 and now, we have been transitioning from crash reports that only contained a JavaStackTrace to crash reports that contained both a JavaStackTrace and a JavaException. Once all Fenix crash reports from crashes in Java code contained a JavaException, we could transition Socorro code to use the JavaException value for Crash Stats views, signature generation, generate-create-bug-url, and other things.

Recently, Fenix dropped the JavaStackTrace crash annotation. However, we hadn't yet gotten to updating Socorro code to use--and prefer--the JavaException values. This broke the ability to generate a bug for a Fenix crash with the needed data added to the bug description. Work on bug 1884041 fixed that.

Comments for Fenix Java crash reports went from:

Crash report: https://crash-stats.mozilla.org/report/index/eb6f852b-4656-4cf5-8350-fd91a0240408

to:

Crash report: https://crash-stats.mozilla.org/report/index/eb6f852b-4656-4cf5-8350-fd91a0240408

Top 10 frames:

0  android.database.sqlite.SQLiteConnection  nativePrepareStatement  SQLiteConnection.java:-2
1  android.database.sqlite.SQLiteConnection  acquirePreparedStatement  SQLiteConnection.java:939
2  android.database.sqlite.SQLiteConnection  executeForString  SQLiteConnection.java:684
3  android.database.sqlite.SQLiteConnection  setJournalMode  SQLiteConnection.java:369
4  android.database.sqlite.SQLiteConnection  setWalModeFromConfiguration  SQLiteConnection.java:299
5  android.database.sqlite.SQLiteConnection  open  SQLiteConnection.java:218
6  android.database.sqlite.SQLiteConnection  open  SQLiteConnection.java:196
7  android.database.sqlite.SQLiteConnectionPool  openConnectionLocked  SQLiteConnectionPool.java:503
8  android.database.sqlite.SQLiteConnectionPool  open  SQLiteConnectionPool.java:204
9  android.database.sqlite.SQLiteConnectionPool  open  SQLiteConnectionPool.java:196

This both fixes the bug and also vastly improves the bug comments from what we were previously doing with JavaStackTrace.

Between 2024-03-31 and 2024-04-06, there were 158,729 Fenix crash reports processed. Of those, 15,556 have the circumstances affected by this bug: a JavaException but don't have a JavaStackTrace. That's roughly 10% of incoming Fenix crash reports.

While working on this, we refactored the code that generates these crash report bugs, so it's in a separate module that's easier to copy and use in external systems in case others want to generate bug comments from processed crash data.

Further, we changed the code so that instead of dropping arguments in function signatures, it now truncates them at 80 characters.

We're hoping to improve signature generation for Java crashes using JavaException values in 2024q2. That work is tracked in bug #1541120.

Socorro: Support guard page access information

1830954: Expose crashes which were likely accessing a guard page

We updated the stackwalker to pick up the changes for determining is_likely_guard_page. Then we exposed that in crash reports in the has_guard_page_access field. We added this field to the Details tab in crash reports and made it searchable. We also added this to the signature report.

This helps us know if a crash is possibly due to a bug with memory access that could be a possible security vulnerability vector--something we want to prioritize fixing.

Since this field is security sensitive, it requires protected data access to view and search with.

Socorro misc

crashstats-tools 2.0.0 release
socorro-siggen 2.1.20240412 release
4 signature generation changes. Thank you Andrew McCreight and Jim Blandy!
Maintenance and documentation improvements.
6 production deploys. Created 71 issues. Resolved 61 issues.

Tecken/Eliot misc

Maintenance and documentation improvements.
5 production deploys. Created 21 issues. Resolved 28 issues.

More information

Find us:

Confluence page: Observability Team
User support hub: User Support
Support: #obs-help (Slack)
Crash ingestion: #crashreporting (Matrix)

Thank you for reading!

Observability Team Newsletter (2023q4)

Will Kahn-Greene — Fri, 22 Dec 2023 14:26:00 GMT

Observability Team is a team dedicated to the problem domain and discipline of Observability at Mozilla. We will own, manage, and support infrastructure and tools supporting Mozilla products and services. Currently this includes crash ingestion related services: Crash Stats and crash ingestion pipeline (Socorro), Mozilla Symbols Server (Tecken), and Mozilla Symbolication Service (Eliot). In 2024, we'll be working with SRE to take over many of the observability tools that they are currently supporting like Sentry, Grafana, New Relic, and others.

This newsletter covers an overview of 2023q4. Please forward it to interested readers.

Highlights

🎉 Team Changes: Socorro Engineering becomes Observability Team and picks up new members.
📄 Documentation: Overhauled support documentation for crash ingestion services.
❤️‍🩹 Socorro: Stability: Fixed ongoing Socorro processor stability problem. [bug 1795017]
🏆 Socorro: Code-info lookup: Implemented code-info lookup for symbols files. [bug 1746940] [Retro]
🔒 Tecken: Removed private symbols bucket support.[bug 1843356]
📚 Tecken: Removed missing symbols bookkeeping. [bug 1774004]
📱 Evolving SRE: Took over application support for crash ingestion services.

See details below.

Blog posts

Detailed project updates

Team changes

Prior to October, 2023, the Socorro Engineering team maintained crash ingestion systems and related services: Crash Stats and the crash ingestion pipeline, Mozilla Symbols Server, and Mozilla Symbolication Service.

In October 2023, that team picked up a couple of new people--Bianca and Sven--and changed names to become the Observability Team. In mid-December, Observability Team picked up a fourth teammate: Relud.

As we move into 2024, we expect to pick up other observability related services and work on service stability, support, and building out documentation of best practices across them supporting Mozilla products and services.

See our Confluence page for contact information, roughly what we're working on, how to do various things (add crash annotations, get protected data access, etc), and service/support documentation.

Overhauled support documentation for crash ingestion services

Documentation for crash ingestion services has been kind of all over the place. Going forward, we're working to make it clearer and easier to find.

We're moving some "how to" documentation into this tree in Confluence. Some interesting ones:

We'll add to that and improve it as time goes on. We'll be looking at centralizing API, tools, data dictionary, and other documentation over the next year as well.

If there are things you have questions about and can't find documentation for it, please let us know.

Socorro: Fixed ongoing Socorro processor stability problem

In September 2022, Mozilla began adding inline function data into symbols files. This increased the size of symbols files significantly. For example, the symbols file for xul.dll files went from around 200mb to 700mb. The increase in file size increased the time it takes for the stackwalker to download and parse symbols files, reduced the number of files the processor could store in the on-disk symbols cache, and caused the processor instances to suddenly slow down in periods of high load. This in turn would cause the processing queue to back up and page SRE causing work disruption as we scrambled to manually add more processor instances to increase throughput and reduce the queue.

We spent a lot of time analyzing the situation, adding new metrics, rewriting portions of the code based on our theories at the time, and ended up with several mitigations that reduced the likelihood that the processing queue backed up and sat with that for several months while we worked on other things.

One of the first things the Observability Team did was revisit the issue. New minds brought new theories, one of which was to change the instance type to one with a local ssd. That eliminated the disk io throttling the processors were incurring from using EBS for the symbols cache.

Now the Socorro processors are performing much like they did prior to September 2022, we've removed all the mitigations we had in place, and the processor queue isn't backing up anymore during periods of high load due to increased crash report volume and reprocessing. [bug 1795017]

Socorro: Stackwalker will use code id when debug id isn't available to fetch the symbols file

This allows symbolication of stacks where the debug id for modules is unknown. This improves crash signatures. Better signatures gives us better visibility into what crashes our users are encountering and how often.

For example, one of the problem signatures (#3 in Top Crashers at the time) looked like this:

OOM | large | mozalloc_abort | xul.dll | _PR_NativeRunThread | pr_root

and now looks like this:

OOM | large | mozalloc_abort | webrender::renderer::Renderer::render_impl

Rough estimate is that this significantly improved the crash signatures for 10k out of the 300k Firefox Windows crash reports we get a week.

See`Code info lookup: retrospective <https://bluesock.org/~willkg/blog/mozilla/socorro_tecken_code_info_retro.html>`__ for details. [bug 1746940]

Socorro misc

socorro-siggen v2.0.20231009 release. [v2.0.20231009]
11 signature generation changes most of which were self-serve.
Lots of maintenance and documentation improvements.
11 production deploys. Created 61 issues. Resolved 58 issues.

Tecken: Remove support for private symbols bucket

The Mozilla Symbols Server stored uploaded symbols in several places: a default storage bucket for build symbols, a "try" storage bucket for symbols from try builds, and a private symbols bucket. Mozilla primarily used the private symbols bucket for Flash symbols. However, we don't support Flash anymore, so we removed the private symbols bucket and all the code to support it. Removing this simplified symbols upload/download code significantly. [bug 1843356]

Tecken: Remove missing symbols bookkeeping

Mozilla Symbols Server used to keep track of symbols that were requested but didn't exist in the symbols buckets. Tecken had an API for querying this data which was used for reporting on which symbols Mozilla is missing. This helps us understand which symbols files we're missing when unwinding and symbolication stacks in crash ingestion.

There are better ways to get this data and keeping track of missing symbols in Tecken isn't helpful. We migrated users of this API and removed the data and code from Tecken. Removing this reduced the size of the database and simplified the download API code. [bug 1774004]

Tecken misc

fx-crash-sig v1.0.1 and v1.0.2 releases. [v1.0.1, v1.0.2]
Lots of maintenance and documentation improvements.
16 production deploys. Created 58 issues. Resolved 56 issues.

Prototyping Evolving SRE

In December, we finished the work to transition the application support role from Data SRE to the Observability Team making us an engineering team that also owns application support for the services we maintain.

We've accrued a lot of experience in how to migrate from the separate Engineering team and SRE team model to the combined Engineering and SRE team model. If you're thinking about transitioning to a combined Engineering and SRE team model and have questions, come find us.

More information

Find us:

Confluence page: Observability Team
Matrix: #crashreporting

Thank you for reading!

Tecken: The long windy journey to reproducing a problem remove_orphaned_files fixes

Will Kahn-Greene — Thu, 30 Nov 2023 15:00:17 GMT

Summary

This post talks about a stability problem we have with Tecken wherein the instance runs out of disk and then becomes unhealthy, some work we're doing to make it better, and the steps to reproduce the problem in a local dev environment so we can test possible fixes.

Tecken/Socorro: Code info lookup: retrospective (2023)

Will Kahn-Greene — Mon, 30 Oct 2023 18:33:05 GMT

Project

time:

6 weeks

impact:

improved visibility 3% (10k / 300k) of Firefox crash reports from Windows users by fixing symbolication and signatures
better understanding of consequences from sampling Firefox / Windows < 8.1 / ESR crash reports

Summary

In November 2021, we wrote up a bug in the Tecken product to support download symbols files using the code file and code id.

In July 2023, Mozilla migrated users for Windows 7, 8, and 8.1 from Firefox release channel to ESR channel. Firefox / Windows / release is sampled by the Socorro collector, so the system only accepts and processes 10% of incoming crash reports. When the users were migrated, their crash reports moved to an unsampled group, so then we were getting 100% of those incoming crash reports. That caused a volume increase of 30k.

While looking into adding a sampling rule for Firefox / Windows < 8.1 / ESR, I noticed many crash reports listed a xul module without a debug file and debug id. Because of that, the stackwalker isn't able to get symbols and we end up with crash reports with generic signatures that we have no visibility into.

I looked at [bug 1746940] and worked out how to fix it. I thought it would be relatively straight-forward to implement and it would solve our visibility problem, so I prioritized working on it with the assumption it'd take a week to do.

Work wasn't as straight-forward as I predicted--I hit a bunch of road bumps and it took me 6 weeks to work through several attempts, settle on a final architecture, implement it, test it, and push all the pieces to production. I finished the work on October 24th, 2023.

The end result is improved visibility for 3% of Firefox Windows crash reports and a reduction in crash reports with generic signatures because the stackwalker couldn't find the symbols file for xul.dll.

Socorro Engineering: 2022 retrospective

Will Kahn-Greene — Mon, 23 Jan 2023 21:00:00 GMT

Summary

2022 took forever. At the same time, it kind of flew by. 2023 is already moving along, so this post is a month late. Here's the retrospective of Socorro engineering in 2022.

Bleach 6.0.0 release and deprecation

Will Kahn-Greene — Mon, 23 Jan 2023 16:55:31 GMT

What is it?

Bleach is a Python library for sanitizing and linkifying text from untrusted sources for safe usage in HTML.

Bleach v6.0.0 released!

Bleach 6.0.0 cleans up some issues in linkify and with the way it uses html5lib so it's easier to reason about. It also adds support for Python 3.11 and cleans up the project infrastructure.

There are several backwards-incompatible changes, hence the 6.0.0 version.

https://bleach.readthedocs.io/en/latest/changes.html#version-6-0-0-january-23rd-2023

I did some rough testing with a corpus of Standup messages data and it looks like bleach.clean is slightly faster with 6.0.0 than 5.0.0.

Using Python 3.10.9:

5.0.0: bleach.clean on 58,630 items 10x: minimum 2.793s
6.0.0: bleach.clean on 58,630 items 10x: minimum 2.304s

The other big change in 6.0.0 is that I've deprecated the project and planning to move to a minimum-maintenance mode for the foreseeable future.

Bleach is deprecated

Bleach sits on top of html5lib which is not actively maintained. It is increasingly difficult to maintain Bleach in that context and I think it's nuts to build a security library on top of a library that's not in active development.

Over the years, we've talked about other options:

find another library to switch to
take over html5lib development
fork html5lib and vendor and maintain our fork
write a new HTML parser
etc

With the exception of option 1, they greatly increase the scope of the work for Bleach. They all feel exhausting to me.

Given that, I think Bleach has run its course and this journey is over.

What happens now?

Possibilities:

Pass it to someone else?

No, I won't be passing Bleach to someone else to maintain. Bleach is a security-related library, so making a mistake when passing it to someone else would be a mess. I'm not going to do that.
Switch to an alternative?

I'm not aware of any alternatives to Bleach. I don't plan to work on coordinating the migration for everyone from Bleach to something else.
Oh my goodness--you're leaving us with nothing?

Sort of.

I'm going to continue doing minimal maintenance:

security updates
support for new Python versions
fixes for egregious bugs (begrudgingly)

I'll do that for at least a year. At some point, I'll stop doing that, too.

I think that gives the world enough time for either something to take Bleach's place, or for the sanitizing web api to kick in, or for everyone to come to the consensus that they never really needed Bleach in the first place.

Bleach. Tired. At the end of its journey.

Thanks!

Many thanks to Greg who I worked with on Bleach for a long while and maintained Bleach for several years. Working with Greg was always easy and his reviews were thoughtful and spot-on.

Many thanks to Jonathan who, over the years, provided a lot of insight into how best to solve some of Bleach's more squirrely problems.

Many thanks to Sam who was an indispensible resource on HTML parsing and sanitizing text in the context of HTML.

Where to go for more

For more specifics on this release, see here: https://bleach.readthedocs.io/en/latest/changes.html#version-6-0-0-january-23rd-2023

Documentation and quickstart here: https://bleach.readthedocs.io/en/latest/

Source code and issue tracker here: https://github.com/mozilla/bleach