Standup report: June 8th, 2018

, | Tweet this

What is Standup?

Standup is a system for capturing standup-style posts from individuals making it easier to see what's going on for teams and projects. It has an associated IRC bot standups for posting messages from IRC.

Project report

Over the last six months, we've done:

  • monthly library updates
  • revamped static assets management infrastructure
  • service maintenance
  • fixed the textarea to be resizeable (Thanks, Arai!)

The monthly library updates have helped with reducing technical debt. That takes a few hours each month to work through.

Paul redid how Standup does static assets. We no longer use django-pipeline, but instead use gulp. It works muuuuuuch better and makes it possible to upgrade to Djagno 2.0 soon. That was a ton of work over the course of a few days for both of us.

We've been keeping the Standup service running. That includes stage and production websites as well as stage and production IRC bots. That also includes helping users who are stuck--usually with accounts management. That's been a handful of hours.

Arai fixed the textareas so they're resizeable. That helps a ton! I'd love to get more help with UI/UX fixing.

Some GitHub stats:

GitHub
======

  mozilla/standup: 15 prs

    Committers:
             pyup-bot :     6  (  +588,   -541,   20 files)
               willkg :     5  (  +383,   -169,   27 files)
                 pmac :     2  ( +4179,   -223,   58 files)
               arai-a :     1  (    +2,     -1,    1 files)
                  g-k :     1  (    +3,     -3,    1 files)

                Total :        ( +5155,   -937,   89 files)

    Most changed files:
      requirements.txt (11)
      requirements-dev.txt (7)
      standup/settings.py (5)
      docker-compose.yml (4)
      standup/status/jinja2/base.html (3)
      standup/status/models.py (3)
      standup/status/tests/test_views.py (3)
      standup/status/urls.py (3)
      standup/status/views.py (3)
      standup/urls.py (3)

    Age stats:
          Youngest PR : 0.0d: 466: Add site-wide messaging
       Average PR age : 2.3d
        Median PR age : 0.0d
            Oldest PR : 10.0d: 459: Scheduled monthly dependency update for May


  All repositories:

    Total merged PRs: 15


Contributors
============

  arai-a
  g-k
  pmac
  pyup-bot
  willkg

That's it for the last six months!

Switching to swag-driven development

Do you use Standup?

Did you use Standup, but the glacial pace of fixing issues was too much so you switched to something else?

Do you want to use Standup?

We think there's still some value in having Standup around and there are still people using it. There's still some technical debt to fix that makes working on it harder than it should be. We've been working through that glacially.

As a project, we have the following problems:

  1. The bulk of the work is being done by Paul and Will.
  2. We don't have time to work on Standup.
  3. There isn't anyone else contributing.

Why aren't users contributing? Probably a lot of reasons. Maybe everyone has their own reason! Have I spent a lot of time to look into this? No, because I don't have a lot of time to work on Standup.

Instead, we're just going to make some changes and see whether that helps. So we're doing the following:

  1. Will promises to send out Standup project reports every 6 months before the All Hands and in doing this raise some awareness of what's going on and thank people who contributed.
  2. We're fixing the Standup site to be clearer on who's doing work and how things get fixed so it's more likely your ideas come to fruition rather than get stale.
  3. We're switching Standup to swag-driven development!

What's that you say? What's swag-driven development?

I mulled over the idea in my post on swag-driven development.

It's a couple of things, but mainly an explicit statement that people work on Standup in our spare time at the cost of not spending that time on other things. While we don't feel entitled to feeling appreciated, it would be nice to feel appreciated sometimes. Not feeling appreciated makes me wonder whether I should spend the time elsewhere. (And maybe that's the case--I have no idea.) Maybe other people would be more interested in spending their spare time on Standup if they knew there were swag incentives?

So what does this mean?

It means that we're encouraging swag donations!

  • If your team has stickers at the All Hands and you use Standup, find Paul and Will and other Standup contributors and give them one!
  • If there are features/bugs you want fixed and they've been sitting in the queue forever, maybe bribing is an option.

For the next quarter

Paul and I were going to try to get together at the All Hands and discuss what's next.

We don't really have an agenda. I know I look at the issue tracker and go, "ugh" and that's about where my energy level is these days.

Possible things to tackle in the next 6 months off the top of my head:

If you're interested in meeting up with us, toss me an email at willkg at mozilla dot com.

AWS Lambda dev with Python

, | Tweet this

A story of a pigeon

I work on Socorro which is the crash ingestion pipeline for Mozilla's products.

The pipeline starts at the collector which handles incoming HTTP POST requests, pulls out the payload, futzes with it a little, and then saves it to AWS S3. Socorro then processes some of those crashes in the processor. The part that connects the two is called Pigeon. It was intended as a short-term solution to bridge the collector and the processor, but it's still around a year later and the green grass grows all around all around and the green grass grows all around.

Pigeon is an AWS Lambda function that triggers on S3 ObjectCreated:Put events, looks at the filename, and then adds things to the processing queue depending on the filename structure. We called it Pigeon for various hilarious reasons that are too mundane to go into in this blog post.

It's pretty basic. It doesn't do much. It was a short term solution we thought we'd throw away pretty quickly. I wrote some unit tests for the individual parts of it and a "client" that invoked the function in a faux AWS Lambda like way. That was good enough.

But then some problems

Pigeon was written with Python 2 because at the time AWS Lambda didn't have a Python 3 runtime. That changed--now there's one with Python 3.6.

In January, I decided to update Pigeon to work with Python 3.6. I tweaked the code, tweaked the unit tests, and voila--it was done! Then we deployed it to our -stage environment where it failed epically in technicolor glory (but no sound!) and we had to back it out and return to the Python 2 version.

What happened? I'll tell you what happened--we had a shit testing environment. Sure, we had tests, but they lacked several things:

  1. At no point do we test against the build artifact for Pigeon. The build artifact for AWS Lambda jobs in Python is a .zip file that includes the code and all the libraries that it uses.
  2. The tests "invoke" Pigeon with a "client", but it was pretty unlike the AWS Lambda Python 3.6 runtime.
  3. Turns out I had completely misunderstood how I should be doing exception handling in AWS Lambda.

So our tests tested some things, but missed some important things and a big bug didn't get caught before going to -stage.

It sucked. I felt chagrinned. I like to think I have a tolerance for failure since I do it a lot, but this felt particularly faily and some basic safeguards would have prevented it from happening.

Fleshing out AWS Lambda in Python project

We were thinking of converting another part of the Socorro pipeline to AWS Lambda, but I put that on hold until I had wrapped my head around how to build a development environment that included scaffolding for testing AWS Lambda functions in a real runtime.

Miles or Brian mentioned aws-sam-local. I looked into that. It's written in Go, they suggest installing it with npm, it does a bunch of things, and it has some event generation code. But for the things I needed, it seemed like it would just be a convenience cli for docker-lambda.

I had been aware of docker-lambda for a while, but hadn't looked at the project recently. They added support for passing events via stdin. Their docs have examples of invoking Lambda functions. That seemed like what I needed.

I took that and built the developer environment scaffolding that we've got in Pigeon now. Further, I decided to use this same model for future AWS Lambda function development.

How does it work?

Pigeon is a Python project, so it uses Python libraries. I maintain those requirements in a requirements.txt file.

I install the requirements into a ./build directory:

$ pip install --ignore-installed --no-cache-dir -r requirements.txt -t build/

I copy the Pigeon source into that directory, too:

$ cp pigeon.py build/

That's all I need for the runtime to use.

The tests are in the tests/ directory. I'm using pytest and in the conftest.py file have this at the top:

import os
import sys

# Insert build/ directory in sys.path so we can import pigeon
sys.path.insert(
    0,
    os.path.join(
        os.path.dirname(os.path.dirname(__file__)),
        'build'
    )
)

I'm using Docker and docker-compose to aid development. I use a test container which is a python:3.6 image with the test requirements installed in it.

In this way, tests run against the ./build directory.

Now I want to be able to invoke Pigeon in an AWS Lambda runtime so I can debug issues and also write an integration test.

I set up a lambda-run container that uses the lambci/lambda:python3.6 image. I mount ./build as /var/task since that's where the AWS Lambda runtime expects things to be.

I created a shell script for invoking Pigeon:

#!/bin/bash

docker-compose run \
    --rm \
    -v "$PWD/build":/var/task \
    --service-ports \
    -e DOCKER_LAMBDA_USE_STDIN=1 \
    lambda-run pigeon.handler $@

That's based on the docker-lambda invoke examples.

Let's walk through that:

  1. It runs the lambda-run container with the services it depends on as defined in my docker-compose.yml file.
  2. It mounts the ./build directory as /var/task because that's where the runtime expectes the code it's running to be.
  3. The DOCKER_LAMBDA_USE_STDIN=1 environment variable causes it to look at stdin for the event. That's pretty convenient.
  4. It runs invokes pigeon.handler which is the handler function in the pigeon Python module.

I have another script that generates fake AWS S3 ObjectCreated:Put events. I cat the result of that into the invoke shell script. That runs everything nicely:

$ ./bin/generate_event.py --key v2/raw_crash/000/20180313/00007bd0-2d1c-4865-af09-80bc00180313 > event.json
$ cat event.json | ./bin/run_invoke.sh
Starting socorropigeon_rabbitmq_1 ... done
START RequestId: 921b4ecf-6e3f-4bc1-adf6-7d58e4d41f47 Version: $LATEST
{"Timestamp": 1523588759480920064, "Type": "pigeon", "Logger": "antenna", "Hostname": "300fca32d996", "EnvVersion": "2.0", "Severity": 4, "Pid": 1, "Fields": {"msg": "Please set PIGEON_AWS_REGION. Returning original unencrypted data."}}
{"Timestamp": 1523588759481024512, "Type": "pigeon", "Logger": "antenna", "Hostname": "300fca32d996", "EnvVersion": "2.0", "Severity": 4, "Pid": 1, "Fields": {"msg": "Please set PIGEON_AWS_REGION. Returning original unencrypted data."}}
{"Timestamp": 1523588759481599232, "Type": "pigeon", "Logger": "antenna", "Hostname": "300fca32d996", "EnvVersion": "2.0", "Severity": 6, "Pid": 1, "Fields": {"msg": "number of records: 1"}}
{"Timestamp": 1523588759481796864, "Type": "pigeon", "Logger": "antenna", "Hostname": "300fca32d996", "EnvVersion": "2.0", "Severity": 6, "Pid": 1, "Fields": {"msg": "looking at key: v2/raw_crash/000/20180313/00007bd0-2d1c-4865-af09-80bc00180313"}}
{"Timestamp": 1523588759481933056, "Type": "pigeon", "Logger": "antenna", "Hostname": "300fca32d996", "EnvVersion": "2.0", "Severity": 6, "Pid": 1, "Fields": {"msg": "crash id: 00007bd0-2d1c-4865-af09-80bc00180313 in dev_bucket"}}
MONITORING|1523588759|1|count|socorro.pigeon.accept|#env:test
{"Timestamp": 1523588759497482240, "Type": "pigeon", "Logger": "antenna", "Hostname": "300fca32d996", "EnvVersion": "2.0", "Severity": 6, "Pid": 1, "Fields": {"msg": "00007bd0-2d1c-4865-af09-80bc00180313: publishing to socorrodev.normal"}}
END RequestId: 921b4ecf-6e3f-4bc1-adf6-7d58e4d41f47
REPORT RequestId: 921b4ecf-6e3f-4bc1-adf6-7d58e4d41f47 Duration: 101 ms Billed Duration: 200 ms Memory Size: 1536 MB Max Memory Used: 28 MB

null

Then I wrote an integration test that cleared RabbitMQ queue, ran the invoke script with a bunch of different keys, and then checked what was in the processor queue.

Now I've got:

  • tests that test the individual bits of Pigeon
  • a way to run Pigeon in the same environment as -stage and -prod
  • an integration test that runs the whole setup

A thing I hadn't mentioned was that Pigeon's documentation is entirely in the README. The docs cover setup and development well enough that I can hand this off to normal people and future me. I like simple docs. Building scaffolding such that docs are simple makes me happy.

Summary

You can see the project at https://github.com/mozilla-services/socorro-pigeon.

Socorro Smooth Mega-Migration 2018

, | Tweet this

Summary

Socorro is the crash ingestion pipeline for Mozilla's products like Firefox. When Firefox crashes, the Breakpad crash reporter asks the user if the user would like to send a crash report. If the user answers "yes!", then the Breakpad crash reporter collects data related to the crash, generates a crash report, and submits that crash report as an HTTP POST to Socorro. Socorro collects and saves the crash report, processes it, and provides an interface for aggregating, searching, and looking at crash reports.

Over the last year and a half, we've been working on a new infrastructure for Socorro and migrating the project to it. It was a massive undertaking and involved changing a lot of code and some architecture and then redoing all the infrastructure scripts and deploy pipelines.

On Thursday, March 28th, we pushed the button and switched to the new infrastructure. The transition was super smooth. Now we're on new infra!

This blog post talks a little about the old and new infrastructures and the work we did to migrate.

Read more…

Side projects and swag-driven development

, | Tweet this

Summary

I work at Mozilla. I work on a lot of stuff:

  • a main project I do a ton of work on and maintain: Socorro
  • a bunch of projects related to that project which I work on and maintain: Antenna, Everett, Markus
  • some projects that I work on occasionally but don't maintain: mozilla-django-oidc
  • one project that many Mozilla sites use that somehow I ended up with but has no relation to my main project: Bleach
  • some projects I'm probably forgetting about
  • a side-project that isn't related to anything else I do that I "maintain": Standups

For most of those projects, they're either part of my main job or I like working on them or I get some recognition for owning them. Whatever the reason, I don't work on them because I feel bad. Then there's Standups which I work on solely because I feel bad.

This blog post talks about me and Standups, pontificates about some options I've talked with others about, and then lays out the concept of swag-driven development.

Read more…

Socorro in 2017

, | Tweet this

Summary

Socorro is the crash ingestion pipeline for Mozilla's products like Firefox. When Firefox crashes, the Breakpad crash reporter asks the user if the user would like to send a crash report. If the user answers "yes!", then the Breakpad crash reporter collects data related to the crash, generates a crash report, and submits that crash report as an HTTP POST to Socorro. Socorro saves the crash report, processes it, and provides an interface for aggregating, searching, and looking at crash reports.

2017 was a big year for Socorro. In this blog post, I opine about our accomplishments.

Read more…

html5lib-python 1.0 released!

, | Tweet this

html5lib-python v1.0 released!

Yesterday, Geoffrey released html5lib 1.0 [1]! The changes aren't wildly interesting.

The more interesting part for me is how the release happened. I'm going to spend the rest of this post talking about that.

[1] Technically there was a 1.0 release followed by a 1.0.1 release because the 1.0 release had issues.

The story of Bleach and html5lib

I work on Bleach which is a Python library for sanitizing and linkifying text from untrusted sources for safe usage in HTML. It relies heavily on another library called html5lib-python. Most of the work that I do on Bleach consists of figuring out how to make html5lib do what I need it to do.

Over the last few years, maintainers of the html5lib library have been working towards a 1.0. Those well-meaning efforts got them into a versioning model which had some unenthusing properties. I would often talk to people about how I was having difficulties with Bleach and html5lib 0.99999999 (8 9s) and I'd have to mentally count how many 9s I had said. It was goofy [2].

In an attempt to deal with the effects of the versioning, there's a parallel set of versions that start with 1.0b. Because there are two sets of versions, it was a total pain in the ass to correctly specify which versions of html5lib that Bleach worked with.

While working on Bleach 2.0, I bumped into a few bugs and upstreamed a patch for at least one of them. That patch sat in the PR queue for months. That's what got me wondering--is this project dead?

I tracked down Geoffrey and talked with him a bit on IRC. He seems to be the only active maintainer. He was really busy with other things, html5lib doesn't pay at all, there's a ton of stuff to do, he's burned out, and recently there have been spats of negative comments in the issues and PRs. Generally the project had a lot of stop energy.

Some time in August, I offered to step up as an interim maintainer and shepherd html5lib to 1.0. The goals being:

  1. land or close as many old PRs as possible
  2. triage, fix, and close as many issues as possible
  3. clean up testing and CI
  4. clean up documentation
  5. ship 1.0 which ends the versioning issues
[2] Many things in life are goofy.

Thoughts on being an interim maintainer

I see a lot of open source projects that are in trouble in the sense that they don't have a critical mass of people and energy. When the sole part-time volunteer maintainer burns out, the project languishes. Then the entitled users show up, complain, demand changes, and talk about how horrible the situation is and everyone should be ashamed. It's tough--people are frustrated and then do a bunch of things that make everything so much worse. How do projects escape the raging inferno death spiral?

For a while now, I've been thinking about a model for open source projects where someone else pops in as an interim maintainer for a short period of time with specific goals and then steps down. Maybe this alleviates users' frustrations? Maybe this gives the part-time volunteer burned-out maintainer a breather? Maybe this can get the project moving again? Maybe the temporary interim maintainer can make some of the hard decisions that a regular long-term maintainer just can't?

I wondered if I should try that model out here. In the process of convincing myself that stepping up as an interim maintainer was a good idea [3], I looked at projects that rely on html5lib [4]:

  • pip vendors it
  • Bleach relies upon it heavily, so anything that uses Bleach uses html5lib (jupyter, hypermark, readme_renderer, tensorflow, ...)
  • most web browsers (Firefox, Chrome, servo, etc) have it in their repositories because web-platform-tests uses it

I talked with Geoffrey and offered to step up with these goals in mind.

I started with cleaning up the milestones in GitHub. I bumped everything from the 0.9999999999 (10 9s) milestone which I determined will never happen into a 1.0 milestone. I used this as a bucket for collecting all the issues and PRs that piqued my interest.

I went through the issue tracker and triaged all the issues. I tried to get steps to reproduce and any other data that would help resolve the issue. I closed some issues I didn't think would ever get resolved.

I triaged all the pull requests. Some of them had been open for a long time. I apologized to people who had spent their time to upstream a fix that sat around for years. In some cases, the changes had bitrotted severely and had to be redone [5].

Then I plugged away at issues and pull requests for a couple of months and pushed anything out of the milestone that wasn't well-defined or something we couldn't fix in a week.

At the end of all that, Geoffrey released version 1.0 and here we are today!

[3] I have precious little free time, so this decision had sweeping consequences for my life, my work, and people around me.
[4] Recently, I discovered libraries.io--it's pretty amazing project. They have a page for html5lib. I had written a (mediocre) tool that does vaguely similar things.
[5] This is what happens on projects that don't have a critical mass of energy/people. It sucks for everyone involved.

Conclusion and thoughts

I finished up as interim maintainer for html5lib. I don't think I'm going to continue actively as a maintainer. Yes, Bleach uses it, but I've got other things I should be doing.

I think this was an interesting experiment. I also think it was a successful experiment in regards to achieving my stated goals, but I don't know if it gave the project much momentum to continue forward.

I'd love to see other examples of interim maintainers stepping up, achieving specific goals, and then stepping down again. Does it bring in new people to the community? Does it affect the raging inferno death spiral at all? What kinds of projects would benefit from this the most? What kinds of projects wouldn't benefit at all?

Markus v1.0 released! Better metrics API for Python projects.

, | Tweet this

What is it?

Markus is a Python library for generating metrics.

Markus makes it easier to generate metrics in your program by:

  • providing multiple backends (Datadog statsd, statsd, logging, logging roll-up, and so on) for sending data to different places
  • sending metrics to multiple backends at the same time
  • providing a testing framework for easy testing
  • providing a decoupled architecture making it easier to write code to generate metrics without having to worry about making sure creating and configuring a metrics client has been done--similar to the Python logging Python logging module in this way

I use it at Mozilla in the collector of our crash ingestion pipeline. Peter used it to build our symbols lookup server, too.

v1.0 released!

This is the v1.0 release. I pushed out v0.2 back in April 2017. We've been using it in Antenna (the collector of the Firefox crash ingestion pipeline) since then. At this point, I think the API is sound and it's being used in production, ergo it's production-ready.

This release also adds Python 2.7 support.

Why you should take a look at Markus

Markus does three things that make generating metrics a lot easier.

First, it separates creating and configuring the metrics backends from generating metrics.

Let's create a metrics client that sends data nowhere:

import markus

markus.configure()

That's not wildly helpful, but it works and it's 2 lines.

Say we're doing development on a laptop on a speeding train and want to spit out metrics to the Python logging module so we can see what's being generated. We can do this:

import markus

markus.configure(
    backends=[
        {
            'class': 'markus.backends.logging.LoggingMetrics'
        }
    ]
)

That will spit out lines to Python logging. Now I can see metrics getting generated while I'm testing my code.

I'm ready to put my code in production, so let's add a statsd backend, too:

import markus

markus.configure(
    backends=[
        {
            # Log metrics to the logs
            'class': 'markus.backends.logging.LoggingMetrics',
        },
        {
            # Log metrics to statsd
            'class': 'markus.backends.statsd.StatsdMetrics',
            'options': {
                'statsd_host': 'statsd.example.com',
                'statsd_port': 8125,
                'statsd_prefix': '',
            }
        }
    ]
)

That's it. Tada!

Markus can support any number of backends. You can send data to multiple statsd servers. You can use the LoggingRollupBackend which will generate statistics every flush_interval of count, current, min, and max for incr stats and count, min, average, median, 95%, and max for timing/histogram stats for metrics data.

If Markus doesn't have the backends you need, writing your own metrics backend is straight-forward.

For more details, see the usage documentation and the backends documentation.

Second, writing code to generate metrics is straight-forward and easy to do.

Much like the Python logging module, you add import markus at the top of the Python module and get a metrics interface. The interface can be module-level or in a class. It doesn't matter.

Here's a module-level metrics example:

import markus

metrics = markus.get_metrics(__name__)

Then you use it:

@metrics.timer_decorator('chopping_vegetables')
def some_long_function(vegetable):
    for veg in vegetable:
        chop_vegetable()
        metrics.incr('vegetable', 1)

That's it. No bootstrapping problems, nice handling of metrics key prefixes, decorators, context managers, and so on. You can use multiple metrics interfaces in the same file. You can pass them around. You can reconfigure the metrics client and backends dynamically while your program is running.

For more details, see the metrics overview documentation.

Third, testing metrics generation is easy to do.

Markus provides a MetricsMock to make testing easier:

import markus
from markus.testing import MetricsMock


def test_something():
    with MetricsMock() as mm:
        # ... Do things that might publish metrics

        # This helps you debug and write your test
        mm.print_records()

        # Make assertions on metrics published
        assert mm.has_metric(markus.INCR, 'some.key', {'value': 1})

I use it with pytest on my projects, but it is testing-system agnostic.

For more details, see the testing documentation.

Why not use statsd directly?

You can definitely use statsd/dogstatsd libraries directly, but using Markus is a lot easier.

With Markus you don't have to worry about the order in which you create/configure the statsd client versus using the statsd client. You don't have to pass around the statsd client. It's a lot easier to use in Dango and Flask where bootstrapping the app and passing things around is tricky sometimes.

With Markus you get to degrade to sending metrics data to the Python logging library which helps surface issues in development. I've had a few occasions when I thought I wrote code to send data, but it turns out I hadn't or that I had messed up the keys or tags.

With Markus you get a testing mock which lets you write tests guaranteeing that your code is generating metrics the way you're expecting.

If you go with using the statsd/dogstatsd libraries directly, that's fine, but you'll probably want to write some/most of these things yourself.

Where to go for more

For more specifics on this release, see here: https://markus.readthedocs.io/en/latest/history.html#october-30th-2017

Documentation and quickstart here: https://markus.readthedocs.io/en/latest/index.html

Source code and issue tracker here: https://github.com/willkg/markus

Let me know whether this helps you!

rob-bugson 1.0: or how I wrote a webextension

, | Tweet this

I work on Socorro and other projects which use GitHub for version control and code review and use Mozilla's Bugzilla for bug tracking.

After creating a pull request in GitHub, I attach it to the related Bugzilla bug which is a contra-dance of clicking and copy-and-paste. Github tweaks for Bugzilla simplified that by adding a link to the GitHub pull request page that I could click on, edit, and then submit the resulting form. However, that's a legacy addon and I use Firefox Nightly and it doesn't look like anyone wrote a webextension version of it, so I was out-of-luck.

Today, I had to bring in my car for service and was sitting around at the dealership for a few hours. I figured instead of working on Socorro things, I'd take a break and implement an attach-pr-to-bug webextension.

I've never written a webextension before. I had written a couple of addons years ago using the SDK and then Jetpack (or something like that). My JavaScript is a bit rusty, especially ES6 stuff. I figured this would be a good way to learn about webextensions.

It took me about 4 hours of puzzling through docs, writing code, and debugging and then I had something that worked. Along the way, I discovered exciting things like:

  • host permissions let you run content scripts in web pages
  • content scripts can't access browser.tabs--you need a background script for that
  • you can pass messages from content scripts to background scripts
  • seems like everything returns a promise, but async/await make that a lot easier to work with
  • the attachment page on Bugzilla isn't like the create-bug page and ignores querystring params

The MDN docs for writing webextensions and the APIs involved are fantastic. The webextension samples are also great--I started with them when I was getting my bearings.

I created a new GitHub repository. I threw the code into a pull request making it easier for someone else to review it. Mike Cooper kindly skimmed it and provided insightful comments. I fixed the issues he brought up.

TheOne helped me resurrect my AMO account which I created in 2012 back when Gaia apps were the thing.

I read through Publishing your webextension, generated a .zip, and submitted a new addon.

About 10 minutes later, the addon had been reviewed and approved.

Now it's a thing and you can install rob-bugson.

Socorro signature generation overhaul and command line interface

, | Tweet this

Summary

This quarter I worked on creating a command line interface for signature generation and in doing that extracted it from the processor into a standalone-ish module.

The end result of this work is that:

  1. anyone making changes to signature generation can can test the changes out on their local machine using a Socorro local development environment
  2. I can trivially test incoming signature generation changes--this both saves me time and gives me a much higher confidence of correctness without having to merge the code and test it in our -stage environment [1]
  3. we can research and experiment with changes to the signature generation algorithm and how that affects existing crash signatures
  4. it's a step closer to being usable by other groups

This blog post talks about that work briefly and then talks about some of the things I've been able to do with it.

[1] I can't overstate how awesome this is.

Read more…

Socorro local development environment

, | Tweet this

Summary

Socorro is the crash ingestion pipeline for Mozilla's products like Firefox. When Firefox crashes, the Breakpad crash reporter asks the user if the user would like to send a crash report. If the user answers "yes!", then the Breakpad crash reporter collects data related to the crash, generates a crash report, and submits that crash report as an HTTP POST to Socorro. Socorro saves the crash report, processes it, and provides an interface for aggregating, searching, and looking at crash reports.

This (long-ish) blog post talks about how when I started on Socorro, there wasn't really a local development environment and how I went on a magical journey through dark forests and craggy mountains to find one.

If you do anything with Socorro at Mozilla, you definitely want to at least read the "Tell me more about this local development environment" part.

Read more…