Will's Blog (Posts about retrospective)

Tecken/Socorro: Code info lookup: retrospective (2023)

Will Kahn-Greene — Mon, 30 Oct 2023 18:33:05 GMT

Project

time:

6 weeks

impact:

improved visibility 3% (10k / 300k) of Firefox crash reports from Windows users by fixing symbolication and signatures
better understanding of consequences from sampling Firefox / Windows < 8.1 / ESR crash reports

Summary

In November 2021, we wrote up a bug in the Tecken product to support download symbols files using the code file and code id.

In July 2023, Mozilla migrated users for Windows 7, 8, and 8.1 from Firefox release channel to ESR channel. Firefox / Windows / release is sampled by the Socorro collector, so the system only accepts and processes 10% of incoming crash reports. When the users were migrated, their crash reports moved to an unsampled group, so then we were getting 100% of those incoming crash reports. That caused a volume increase of 30k.

While looking into adding a sampling rule for Firefox / Windows < 8.1 / ESR, I noticed many crash reports listed a xul module without a debug file and debug id. Because of that, the stackwalker isn't able to get symbols and we end up with crash reports with generic signatures that we have no visibility into.

I looked at [bug 1746940] and worked out how to fix it. I thought it would be relatively straight-forward to implement and it would solve our visibility problem, so I prioritized working on it with the assumption it'd take a week to do.

Work wasn't as straight-forward as I predicted--I hit a bunch of road bumps and it took me 6 weeks to work through several attempts, settle on a final architecture, implement it, test it, and push all the pieces to production. I finished the work on October 24th, 2023.

The end result is improved visibility for 3% of Firefox Windows crash reports and a reduction in crash reports with generic signatures because the stackwalker couldn't find the symbols file for xul.dll.

Socorro: Schema-Based Overhaul of Crash Ingestion: Retrospective (2022)

Will Kahn-Greene — Wed, 18 Jan 2023 18:00:00 GMT

Project

time:

2+ years

impact:

radically reduces the risk of data leaks due to misconfigured permissions
centralizes and simplifies configuration and management of fields
normalization and validation are performed during processing
documentation of data reviews, data caveats, etc.
reduces the risk of bugs when adding new fields—testing is done in CI
new crash reporting data dictionary with Markdown-formatted descriptions, real examples, and relevant links

Summary

I've been working on Socorro, the crash ingestion pipeline at Mozilla, since the beginning of 2016. During that time, I've focused on streamlining maintenance of the project, paying down technical debt, reducing risk, and improving crash analysis tooling.

Early on, I observed that the crash ingestion pipeline was difficult to reason about, poorly documented, and full of risk. What did the incoming data look like? What did the processed data look like? Was it valid? Which fields contained data anyone could look at? Which fields contained data that was sensitive and required access controls to access? How do we add support for new crash annotations? What happens when data is invalid or malformed? At a given point in the system, what did we know about the data?

In 2020, Socorro moved into the Data Org, which has multiple data pipelines. After spending some time looking at how their pipelines work, I decided to rework the crash ingestion pipeline to be schema-driven and to move data validation and normalization earlier in the processor.

The end result of this project is that:

the project is easier to maintain:
- adding support for new crash annotations is done in a couple of schema files and possibly a processor rule
risk of security issues and data breaches is lower:
- typos, bugs, and mistakes when adding support for a new crash annotation are caught in CI
- permissions are specified in a central location; changing permission for fields is trivial and takes effect in the next deploy; setting permissions supports complex data structures in easy-to-reason-about ways; and mistakes are caught in CI
the data is easier to use and reason about:
- normalization and validation of crash annotation data happens during processing, and downstream uses of the data can expect it to be valid; further, we get a signal when the data isn't valid, which can indicate product bugs
- schemas describing incoming and processed data
- a crash reporting data dictionary documenting incoming data fields, processed data fields, descriptions, sources, data gotchas, examples, and permissions

What is Socorro?

Socorro is the crash ingestion pipeline for Mozilla products like Firefox, Fenix, Thunderbird, and MozillaVPN.

When Firefox crashes, the crash reporter asks the user if they would like to send a crash report. If the user answers "yes!", the crash reporter collects data related to the crash, generates a crash report, and submits that crash report as an HTTP POST to Socorro. Socorro saves the submitted crash report, processes it, and has tools for viewing and analyzing crash data.

State of crash ingestion at the beginning

The crash ingestion system was working and it was usable, but it was in a bad state.

Poor data management

Normalization and validation of data was all over the codebase and not consistent:
- processor rule code
- AWS S3 crash storage code
- Elasticsearch indexing code
- Telemetry crash storage code
- Super Search querying and result rendering code
- report view and template code
- signature report code and template code
- crontabber job code
- any scripts that used the data
- tests -- many of which had bad test data so who knows what they were really testing
Naive handling of minidump stackwalker output meant that any changes in the stackwalker output were predominantly unnoticed, and there was no indication as to whether changed output created issues in the system.

Further, since it was all over the place, there were no guarantees for data validity when downloading it using the RawCrash, ProcessedCrash, and SuperSearch APIs. Anyone writing downstream systems would also have to normalize and validate the data.
Poor permissions management

Permissions were defined in multiple places:
- Elasticsearch json redactor
- Super Search fields
- RawCrash API allow list
- ProcessedCrash API allow list
- report view and template code
- Telemetry crash storage code
- and other places
We couldn't effectively manage the permissions of fields in the stackwalker output because we had no idea what was there.
Poor documentation

No documentation of crash annotation fields other than CrashAnnotations.yaml, which didn't enforce anything in crash ingestion (process, valid type, data correctness, etc.) and was missing important information like data gotchas, data review URLs, and examples.

No documentation of processed crash fields at all.
Making changes was high risk

Changing fields from public to protected was high risk because you had to find all the places it might show up which was intractable. Adding support for new fields often took multiple passes over several weeks because we'd miss things. Server errors happened with some regularity due to weirdness with crash annotation values affecting the Crash Stats site.
Tangled concerns across the codebase

Lots of tangled concerns where things defined in one place affected other places that shouldn't be related. For example, the Super Search fields definition was acting as a "schema" for other parts of the system that had nothing to do with Elasticsearch or Super Search.
Difficult to maintain

It was difficult to support new products.

It was difficult to debug issues in crash ingestion and crash reporting.

The Crash Stats web app contained lots of if/then/else bits to handle weirdness in the crash annotation values. Nulls, incorrect types, different structures, etc.

Socorro contained lots of vestigial code from half-done field removal, deprecated fields, fields that were removed from crash reports, etc. These vestigial bits were all over the code base. Discovering and removing these bits was time consuming and error prone.

The code for exporting data to Telemetry built the export data using a list of fields to exclude rather than a list of fields to include. This is backward and impossible to maintain—we never should have been doing this. Further, it pulled data from the raw crash, for which we had no validation guarantees, which would cause issues downstream in the Telemetry import code.

There was no way to validate the data used in the unit tests, which meant that a lot of it was invalid. We had no way to validate the test data, which meant that CI would pass, but we'd see errors in our stage and production environments.
Different from other similar systems

In 2020, Socorro was moved to the Data Org in Mozilla which had a set of standards and conventions for collecting, storing, analyzing, and providing access to data. Socorro didn't follow any of it, which made it difficult to work on, to connect with, and to staff. Things the Data Org has that Socorro didn't:
- a schema covering specifying fields, types, and documentation
- data flow documentation
- data review policy, process, and artifacts for data being collected and how to add new data
- a data dictionary for fields for users including documentation, data review URLs, and data gotchas

In summary, we had a system that took a lot of effort to maintain, wasn't serving our users' needs, and was at high risk of a security/data breach.

Project plan

Many of these issues can be alleviated and reduced by moving to a schema-driven system where we:

define a schema for annotations and a schema for the processed crash
change crash ingestion and the Crash Stats site to use those schemas

When designing this schema-driven system, we should be thinking about:

how easy is it to maintain the system?
how easy is it to explain?
how flexible is it for solving other kinds of problems in the future?
what kinds of errors will likely happen when maintaining the system, and how can we avert them in CI?
what kinds of errors can happen and how much risk do they pose for data leaks? what of those can we avert in CI?
how flexible is the system, which needs to support multiple products potentially with different needs?

I worked out a minimal version of that vision that we could migrate to and then work with going forward.

The crash annotations schema should define:

what annotations are in the crash report?
which permissions are required to view a field
field documentation (provenance, description, data review, related bugs, gotchas, analysis tips, etc)

The processed crash schema should define:

what's in the processed crash?
which permissions are required to view a field
field documentation (provenance, description, related bugs, gotchas, analysis tips, etc)

Then we make the following changes to the system:

write a processor rule to copy, normalize, and validate data from the raw crash based on the processed crash schema
switch the Telemetry export code to using the processed crash for data to export
switch the Telemetry export code to using the processed crash schema for permissions
switch Super Search to using the processed crash for data to index
switch Super Search to using the processed crash schema for documentation and permissions
switch the Crash Stats site to using the processed crash for data to render
switch the Crash Stats site to using the processed crash schema for documentation and permissions
switch the RawCrash, ProcessedCrash, and SuperSearch APIs to using the crash annotations and processed crash schemas for documentation and permissions

After doing that, we have:

field documentation is managed in the schemas
permissions are managed in the schemas
data is normalized and validated once in the processor and everything uses the processed crash data for indexing, searching, and rendering
adding support for new fields and changing existing fields is easier and problems are caught in CI

Implementation decisions

Use JSON Schema.

Data Org at Mozilla uses JSON Schema for schema specification. The schema is written using YAML.

https://mozilla.github.io/glean_parser/metrics-yaml.html

The metrics schema is used to define metrics.yaml files which specify the metrics being emitted and collected.

For example:

https://searchfox.org/mozilla-central/source/toolkit/mozapps/update/metrics.yaml

One long-term goal for Socorro is to unify standards and practices with the Data Ingestion system. Towards that goal, it's prudent to build out a crash annotation and processed crash schemas using whatever we can take from the equivalent metrics schemas.

We'll also need to build out tooling for verifying, validating, and testing schema modifications to make ongoing maintenance easier.

Use schemas to define and drive everything.

We've got permissions, structures, normalization, validation, definition, documentation, and several other things related to the data and how it's used throughout crash ingestion spread out across the codebase.

Instead of that, let's pull it all together into a single schema and change the system to be driven from this schema.

The schema will include:

structure specification
documentation including data gotchas, examples, and implementation details
permissions
processing instructions

We'll have a schema for supported annotations and a schema for the processed crash.

We'll rewrite existing parts of crash ingestion to use the schema:

processing
1. use processing instructions to validate and normalize annotation data
super search
1. field documentation
2. permissions
3. remove all the normalization and validation code from indexing
crash stats
1. field documentation
2. permissions
3. remove all the normalization and validation code from page rendering

Only use processed crash data for indexing and analysis.

The indexing system has its own normalization and validation code since it pulls data to be indexed from the raw crash.

The crash stats code has its own normalization and validation code since it renders data from the raw crash in various parts of the site.

We're going to change this so that all normalization and validation happens during processing, the results are stored in the processed crash, and indexing, searching, and crash analysis only work on processed crash data.

By default, all data is protected.

By default, all data is protected unless it is explicitly marked as public. This has some consequences for the code:

any data not specified in a schema is treated as protected
all schema fields need to specify permissions for that field
any data in a schema is either:
- marked public, OR
- lists the permissions required to view that data
for nested structures, any child field that is public has public ancestors

We can catch some of these issues in CI and need to write tests to verify them.

This is slightly awkward when maintaining the schema because it would be more reasonable to have "no permissions required" mean that the field is public. However, it's possible to accidentally not specify the permissions, and we don't want to be in that situation. Thus, we decided to go with explicitly marking public fields as public.

Work done

Phase 1: cleaning up

We had a lot of work to do before we could start defining schemas and changing the system to use those schemas.

remove vestigial code (some of this work was done in other phases as it was discovered)
- [bug 1724933]: remove unused/obsolete annotations (2021-08)
- [bug 1743487]: remove total_frames (2021-11)
- [bug 1743704]: remove jit crash classifier (2022-02)
- [bug 1762000]: remove vestigial Winsock_LSP code (2022-03)
- [bug 1784485]: remove vestigial exploitability code (2022-08)
- [bug 1784095]: remove vestigial contains_memory_report code (2022-08)
- [bug 1787933]: exorcise flash things from the codebase (2022-09)
fix signature generation
- [bug 1753521]: use fields from processed crash (2022-02)
- [bug 1755523]: fix signature generation so it only uses processed crash data (2022-02)
- [bug 1762207]: remove hang_type (2022-04)
fix Super Search
- [bug 1624345]: stop saving random data to Elasticsearch crashstorage (2020-06)
- [bug 1706076]: remove dead Super Search fields (2021-04)
- [bug 1712055]: remove system_error from Super Search fields (2021-07)
- [bug 1712085]: remove obsolete Super Search fields (2021-08)
- [bug 1697051]: add crash_report_keys field (2021-11)
- [bug 1736928]: remove largest_free_vm_block and tiny_block_size (2021-11)
- [bug 1754874]: remove unused annotations from Super Search (2022-02)
- [bug 1753521]: stop indexing items from raw crash (2022-02)
- [bug 1762005]: migrate to lower-cased versions of Plugin* fields in processed crash (2022-03)
- [bug 1755528]: fix flag/boolean handling (2022-03)
- [bug 1762207]: remove hang_type (2022-04)
- [bug 1763264]: clean up super search fields from migration (2022-07)
fix data flow and usage
- [bug 1740397]: rewrite CrashingThreadInfoRule to normalize crashing thread (2021-11)
- [bug 1755095]: fix TelemetryBotoS3CrashStorage so it doesn't use Super Search fields (2022-03)
- [bug 1740397]: change webapp to pull crashing_thread from processed crash (2022-07)
- [bug 1710725]: stop using DotDict for raw and processed data (2022-09)
clean up the raw crash structure
- [bug 1687987]: restructure raw crash (2021-01 through 2022-10)

Phase 2: define schemas and all the tooling we needed to work with them

After cleaning up the code base, removing vestigial code, fixing Super Search, and fixing Telemetry export code, we could move on to defining schemas and writing all the code we needed to maintain the schemas and work with them.

[bug 1762271]: rewrite json schema reducer (2022-03)
[bug 1764395]: schema for processed crash, reducers, traversers (2022-08)
[bug 1788533]: fix validate_processed_crash to handle pattern_properties (2022-08)
[bug 1626698]: schema for crash annotations in crash reports (2022-11)

Phase 3: fix everything to use the schemas

That allowed us to fix a bunch of things:

[bug 1784927]: remove elasticsearch redactor code (2022-08)
[bug 1746630]: support new threads.N.frames.N.unloaded_modules minidump-stackwalk fields (2022-08)
[bug 1697001]: get rid of UnredactedCrash API and model (2022-08)
[bug 1100352]: remove hard-coded allow lists from RawCrash (2022-08)
[bug 1787929]: rewrite Breadcrumbs validation (2022-09)
[bug 1787931]: fix Super Search fields to pull permissions from processed crash schema (2022-09)
[bug 1787937]: fix Super Search fields to pull documentation from processed crash schema (2022-09)
[bug 1787931]: use processed crash schema permissions for super search (2022-09)
[bug 1100352]: remove hard-coded allow lists from ProcessedCrash models (2022-11)
[bug 1792255]: add telemetry_environment to processed crash (2022-11)
[bug 1784558]: add collector metadata to processed crash (2022-11)
[bug 1787932]: add data review urls for crash annotations that have data reviews (2022-11)

Phase 4: improve

With fields specified in schemas, we can write a crash reporting data dictionary:

[bug 1803558]: crash reporting data dictionary (2023-01)
[bug 1795700]: document raw and processed schemas and how to maintain them (2023-01)

Then we can finish:

[bug 1677143]: documenting analysis gotchas (ongoing)
[bug 1755525]: fixing the report view to only use the processed crash (future)
[bug 1795699]: validate test data (future)

Random thoughts

This was a very very long-term project with many small steps and some really big ones. Getting large projects done is futile and the only way to do it successfully is to break it into a million small steps each of which stand on their own and don't create urgency for getting the next step done.

Any time I changed field names or types, I'd have to do a data migration. Data migrations take 6 months to do because I have to wait for existing data to expire from storage. On the one hand, it's a blessing I could do migrations at all--you can't do this with larger data sets or with data sets where the data doesn't expire without each migration becoming a huge project. On the other hand, it's hard to juggle being in the middle of multiple migrations and sometimes the contortions one has to perform are grueling.

If you're working on a big project that's going to require changing data structures, figure out how to do migrations early with as little work as possible and use that process as often as you can.

Conclusion and where we could go from here

This was such a huge project that spanned years. It's so hard to finish projects like this because the landscape for the project is constantly changing. Meanwhile, being mid-project has its own set of complexities and hardships.

I'm glad I tackled it and I'm glad it's mostly done. There are some minor things to do, still, but this new schema-driven system has a lot going for it. Adding support for new crash annotations is much easier, less risky, and takes less time.

It took me about a month to pull this post together.

That's it!

That's the story of the schema-based overhaul of crash ingestion. There's probably some bits missing and/or wrong, but the gist of it is here.

If you have any questions or bump into bugs, I hang out on #crashreporting on chat.mozilla.org. You can also write up a bug for Socorro.

Hopefully this helps. If not, let us know!

Eliot: retrospective (2021)

Will Kahn-Greene — Mon, 15 Nov 2021 18:10:24 GMT

Project

time:

1 year

impact:

reduced risk of a Mozilla Symbols Server outage, which affects symbols uploads from the build system
improves the maintainability of the symbolication service by offloading parsing of Breakpad symbols files and symbol lookups to an external library that's used by other tools in the crash reporting ecosystem at Mozilla
opens up possible futures around supporting inline functions and using other debug file types

Problem statement

Tecken is the Mozilla Symbols Service. This service manages several things:

symbol upload API for uploading and storing debugging symbols generated by build systems for products like Firefox, Fenix, etc.
download API for downloading symbols, which could be located in a variety of different places, supporting tools like Visual Studio, stackwalkers, profilers, symbolicators, etc.
symbolication API for finding symbols for memory addresses

It also has a webapp for querying symbols, debugging symbols problems, managing API tokens, and granting permissions for uploading symbols.

All of those functions are currently handled by a single webapp service.

There are a few problems here.

First, these three different functions of the Mozilla Symbols Service have vastly different runtime properties.

Symbol upload API requests can take a long time to process (and it's done during the HTTP request cycle (which is a different issue for a different day)). Symbol uploads are critical to the Firefox build system which uploads symbols from builds.

We want to reduce the risk of an outage for uploading symbols. When we have service outages, the build systems can't upload symbols. It tries really hard to upload symbols, so this increases the build times for Firefox and other products. On top of that, if the build system doesn't successfully upload symbols, any crashes in tests or channels result in unsymbolicated stacks, which obscures the details of the crash.

Symbol download API requests are fast to process. Symbol downloads are used by the Firefox profiler, the Socorro stackwalker during crash report processing, VS Code, and other systems.

Symbolication API requests take a long time to process, but aren't currently used by a critical system. There are several projects that are waiting in the wings to dramatically increase their use of the symbolication API, which increases the likelihood of an outage with the service that affects symbol uploads.

Second, the existing symbolication API implementation is an independent implementation of sym file parsing, lookups, and symbolication. Whenever we make adjustments to how sym files are built, structured, or the lookup algorithms, we have to additionally update the symbolication API code.

Mozilla is in the process of rewriting crash reporting related code in Rust. It behooves us greatly to switch from our independent implementation to a shared library.

Third, the symbolication API is missing some critical features, like support for line numbers and inline functions. The existing architecture can't be changed to support either line numbers or inline functions--we need to rewrite this code.

In September 2020, I embarked on a project to break out the symbolication API as a separate microservice and implement it using the Symbolic library. That had the following effects:

eases the risk of an outage due to increasing usage of the symbolication API,
adds support for line numbers and sets us up for adding support for inline functions, and
reduces the maintenance work because we'll be using a library used by other parts of the crash reporting ecosystem

This post covers that project.

Data Org Working Groups: retrospective (2020)

Will Kahn-Greene — Thu, 25 Feb 2021 15:00:00 GMT

Project

time:: 1 month
impact:: established cross-organization groups as a tool for collaboration

Problem statement

The Data Org architects, builds, and maintains a data ingestion system and the ecosystem around it. The group covers a swath of engineering and data science disciplines and problem domains. Many of us are generalists and have expertise and interests in multiple areas. Many projects cut across disciplines, problem domains, and organizational structures. Some projects, disciplines, and problem domains benefit from the participation of other stakeholders who aren't in the Data Org.

In order to succeed in tackling the projects of tomorrow, we need to formalize the creation, maintenance, and disbanding of groups composed of interested stakeholders focusing on specific missions. Further, we need a set of best practices to help make these groups successful.

Socorro Smooth Mega-Migration: retrospective (2018)

Will Kahn-Greene — Wed, 04 Apr 2018 16:00:00 GMT

Project

period:

1.5 years

impact:

vastly reduced technical debt
vastly improved developer efficiency
reduced infrastructure security risks

Summary

Socorro is the crash ingestion pipeline for Mozilla's products like Firefox. When Firefox crashes, the Breakpad crash reporter asks the user if they would like to send a crash report. If the user answers "yes!", then the Breakpad crash reporter collects data related to the crash, generates a crash report, and submits that crash report as an HTTP POST to Socorro. Socorro collects and saves the crash report, processes it, and provides an interface for aggregating, searching, and looking at crash reports.

Over the last year and a half, we've been working on a new infrastructure for Socorro and migrating the project to it. It was a massive undertaking and involved changing a lot of code and some architecture and then redoing all the infrastructure scripts and deploy pipelines.

On Thursday, March 28th, 2018, we pushed the button and switched to the new infrastructure. The transition was super smooth. Now we're on the new infra!

This blog post talks a little about the old and new infrastructures and the work we did to migrate.

html5lib-python 1.0 released!: retrospective (2017)

Will Kahn-Greene — Fri, 08 Dec 2017 17:00:00 GMT

Project

time:

3 months

impact:

reduced technical debt and maintenance friction for html5lib which impacts a variety of projects like PyPI, pip, readme_renderer, Jupyter, TensorFlow
reduced security risks for Bleach

html5lib-python v1.0 released!

Yesterday, Sam released html5lib 1.0 [1]! The changes aren't wildly interesting for users, but are important for the health of the project.

The more interesting part for me is how the release happened and experimenting with interim maintainers to get projects going again. I'm going to spend the rest of this post talking about that.

The story of Bleach and html5lib

I work on Bleach which is a Python library for sanitizing and linkifying text from untrusted sources for safe usage in HTML. It relies heavily on another library called html5lib-python. Most of the work that I do on Bleach consists of figuring out how to make html5lib do what I need it to do.

Over the last few years, maintainers of the html5lib library have been working towards a 1.0. Those well-meaning efforts got them into a versioning model which had some unenthusing properties. I would often talk to people about how I was having difficulties with Bleach and html5lib 0.99999999 (8 9s) and I'd have to mentally count how many 9s I had said. It was goofy [2].

In an attempt to deal with the effects of the versioning, there's a parallel set of versions that start with 1.0b. Because there are two sets of versions, it was a total pain in the ass to correctly specify which versions of html5lib that Bleach worked with.

While working on Bleach 2.0, I bumped into a few bugs and upstreamed a patch for at least one of them. That patch sat in the PR queue for months. That's what got me wondering--what's going on with html5lib?

I tracked down Sam and talked with her a bit on IRC. She seems to be the only active maintainer. She was really busy with other things, html5lib doesn't pay, there's a ton of stuff to do, she's burned out, and recently there have been spats of negative comments in the issues and PRs. Generally the project had a lot of stop energy.

Some time in August, I offered to step up as an interim maintainer and shepherd html5lib to 1.0. The goals being:

land or close as many old PRs as possible
triage, fix, and close as many issues as possible
clean up testing and CI
clean up documentation
ship 1.0 which ends the versioning issues

Thoughts on being an interim maintainer

I see a lot of open source projects that are in trouble in the sense that they don't have a critical mass of people and energy. When the sole part-time volunteer maintainer burns out, the project languishes. Then users show up, complain, demand changes, and talk about how horrible the situation is and everyone should be ashamed. It's tough--people are frustrated and then do a bunch of things that make everything so much worse. How do projects escape the raging inferno death spiral?

For a while now, I've been thinking about a model for open source projects where someone else pops in as an interim maintainer for a short period of time with specific goals, works to achieve those goals, and then steps down. Maybe this alleviates users' frustrations? Maybe this gives the part-time volunteer burned-out maintainer a breather? Maybe this can get the project moving again? Maybe the temporary interim maintainer can make some of the hard decisions that a regular long-term maintainer just can't?

I wondered if I should try that model out here. In the process of convincing myself that stepping up as an interim maintainer was a good idea [3], I looked at projects that rely on html5lib [4]:

pip vendors it
Bleach relies upon it heavily, so anything that uses Bleach uses html5lib (jupyter, hypermark, readme_renderer, TensorFlow, ...)
anything that uses readme_renderer like PyPI and tools around Python packages
most web browsers (Firefox, Chrome, servo, etc) have it in their repositories because web-platform-tests uses it

I talked with Sam and offered to step up with these goals in mind.

I started with cleaning up the milestones in GitHub. I decided the 0.9999999999 (10 9s) milestone was going to be 1.0. I bumped everything from the 0.9999999999 (10 9s) milestone to the 1.0 milestone. I went through all the issues and PRs and threw any that piqued my interest in the 1.0 milestone bucket.

Then I went through the issue tracker and triaged all the issues. I tried to get steps to reproduce and any other data that would help resolve the issue. I closed some issues I didn't think would ever get resolved.

I triaged all the pull requests. Some of them had been open for a long time. I apologized to people who had spent their time to upstream a fix that sat around for years. In some cases, the changes had bitrotted severely they had to be re-done [5].

Then I plugged away at issues and pull requests for a couple of months landing and fixing, and pushed anything out of the milestone that wasn't well-defined or something we couldn't fix in a week.

At the end of all that, Sam released version 1.0 and here we are today!

Conclusion and more thoughts

I finished up as interim maintainer for html5lib. I don't think I'm going to continue actively as a maintainer. Yes, Bleach uses it, but I've got other things I should be doing.

I think this was an interesting experiment. I also think it was a successful experiment in regards to achieving my stated goals, but I don't know if it gave the project much momentum to continue forward.

I'd love to see other examples of interim maintainers stepping up, achieving specific goals, and then stepping down again. Does it bring in new people to the community? Does it affect the raging inferno death spiral at all? What kinds of projects would benefit from this the most? What kinds of projects wouldn't benefit at all?

Socorro signature generation overhaul and command line interface: retrospective (2017)

Will Kahn-Greene — Fri, 06 Oct 2017 13:00:00 GMT

Project

time:

6 months

impact:

improved ease of contribution for signature generation changes
improved ability to experiment with signatures
improved ability to use Socorro-style crash signatures in other projects

Summary

This quarter I worked on creating a command line interface for signature generation and in doing that extracted it from the processor into a standalone-ish module.

The end result of this work is that:

anyone making changes to signature generation can can test the changes out on their local machine using a Socorro local development environment
I can trivially test incoming signature generation changes--this both saves me time and gives me a much higher confidence of correctness without having to merge the code and test it in our -stage environment [1]
we can research and experiment with changes to the signature generation algorithm and how that affects existing crash signatures
it's a step closer to being usable by other groups

This blog post talks about that work briefly and then talks about some of the things I've been able to do with it.

Socorro local development environment: retrospective (2017)

Will Kahn-Greene — Wed, 20 Sep 2017 16:34:00 GMT

Project

time:

1 year

impact:

vastly reduced time-to-onboard for new developers and contributors from months to days
vastly improved developer efficacy

Summary

Socorro is the crash ingestion pipeline for Mozilla's products like Firefox. When Firefox crashes, the Breakpad crash reporter asks the user if they would like to send a crash report. If the user answers "yes!", then the Breakpad crash reporter collects data related to the crash, generates a crash report, and submits it as an HTTP POST to Socorro. Socorro saves the crash report, processes it, and provides an interface for aggregating, searching, and viewing crash reports.

This (long-ish) blog post talks about how, when I started on Socorro, there wasn't really a local development environment, and how I went on a magical journey through dark forests and craggy mountains to find one.

If you do anything with Socorro at Mozilla, you definitely want to at least read the "Tell me more about this local development environment" part.

Antenna: retrospective (2017)

Will Kahn-Greene — Mon, 17 Jul 2017 16:00:00 GMT

Project

time:

6 months

impact:

reduced risk for deployments
improved reliability of collector
reduced technical debt
improved developer efficacy

Problem statement

Socorro is the crash ingestion pipeline for Mozilla's products like Firefox. When Firefox crashes, the Breakpad crash reporter asks the user if the user would like to send a crash report. If the user answers "yes!", then the Breakpad crash reporter collects data related to the crash, generates a crash report, and submits that crash report as an HTTP POST to Socorro--specifically the Socorro collector.

The Socorro collector is one of several components that comprise Socorro. Each of the components has different uptime requirements and different security risk profiles. However, all the code is maintained in a single repository and we deploy everything every time we do a deploy. This is increasingly inflexible and makes it difficult for us to make architectural changes to Socorro without affecting everything and incurring uptime risk for components that have high uptime requirements.

Because of that, in early 2016, we embarked on a re-architecture to split out some components of Socorro into separate services. The first component to get split out was the Socorro collector since it needs has the highest uptime requirements of all the Socorro components, but rarely changes, so it'd be a lot easier to meet those requirements if it was separate from the rest of Socorro.

Thus I was tasked with splitting out the Socorro collector and this blog post covers that project. It's a bit stream-of-consciousness, because I think there's some merit to explaining the thought process behind how I did the work over the course of the project for other people working on projects.

Input: Moving to Django 1.8: retrospective (2015)

Will Kahn-Greene — Thu, 01 Oct 2015 20:00:00 GMT

Project

time:

1 year

impact:

upgraded Input to Django 1.8
paved the road for AMO, MDN, and other Mozilla sites
wrapped up jingo, jingo-minify, and other playdough-era projects
built migration path from Tower to Puente and other l10n options

Summary

Over the course of 2015, we've been reworking large parts of the Fjord codebase to do the following:

ditch jingo and friends and other libraries that deviate from typical Django and aren't active projects
reduce complexity by moving closer to a "default/typical Django project"
upgrade to Django 1.8

This blog post covers many grueling details, including the order we did things, the design decisions we made, and some anecdotes.