Socorro: 2018q3 review

, Will Kahn-Greene | share this (mastodon)
Note: This is an old post in a blog with a lot of posts. The world has changed, technologies have changed, and I've changed. It's likely this is out of date and not representative. Let me know if you think this is something that needs updating.

Summary

Socorro is the crash ingestion pipeline for Mozilla's products like Firefox. When Firefox crashes, the Breakpad crash reporter asks the user if the user would like to send a crash report. If the user answers "yes!", then the Breakpad crash reporter collects data related to the crash, generates a crash report, and submits that crash report as an HTTP POST to Socorro. Socorro saves the crash report, processes it, and provides an interface for aggregating, searching, and looking at crash reports.

2018q3 was a busy quarter. This blog post covers what happened.

Staff changes in 2018q3

At the beginning of 2018q3, there were two engineers and two interns. Ced and Alexis ended their internship in August. Mike got slid over to a tiger team which he'll be on until November. For most of September, it was just me.

Here's a handy table showing engineering staff over time:

Month

Engineering Team

July

Mike, Alexis, Ced, me

August

Mike (part time), Alexis (until August 10th), Ced (until August 10th), me

September

me

One person isn't a critical mass for a project, so we had to make changes in development process and priorities. It's been tough. If you want to talk more about that, toss me an email, ping me on IRC, or catch me at the All Hands in December.

Starting a monthly newsletter

I've been solo on other projects at Mozilla before. This is the first time I've been the solo engineer with an amorphous mass of stakeholders. When I worked on Input, we had weekly meetings to work out what was going on and for me to get feedback on engineering work and for other people to learn about upcoming changes.

With Socorro, it's really different. I don't have a good feel for who the stakeholders are, what they're doing, what their current/future plans are, or whether they find anything I'm doing helpful or not. At first it was just inconvenient because it made it hard to prioritize things. Now I find it somewhat demoralizing. I feel like it's just me and Wilson on an island somewhere.

Other projects are doing monthly blog post updates. Maybe if I do that, it'll help.

This is a 2018q3 rollup. After this, I'll start doing monthly updates.

Highlights in 2018q3

We accomplished a lot in 2018 quarter 3.

  • Lots of improvements to Socorro-style signature generation:

  • Reworked Crash Stats to support new products that don't follow the same build methodologies as Firefox. That work allowed us to add GeckoView, Focus and FirefoxReality.

  • Changed the throttling code to reduce the number of crash reports coming into Socorro.

    • Socorro no longer accepts crash reports for unsupported products.

    • Socorro no longer accepts crash reports that it doesn't plan to process.

    Previously, Socorro was saving roughly 70% of incoming crashes. Now it's saving roughly 20% and all those crashes are searchable in the Crash Stats interface. Socorro is no longer saving crash data that we don't know anything about.

  • Redesigned the about:crashes page to fix a bunch of UI/UX issues.

  • Removed ADI and all ADI-requiring things.

  • Improvements to make Crash Stats more service-y:

  • Added a bunch of fields to supersearch.

  • Removed Postgres from the processor. This is a big deal because it was one of the things blocking a processor rewrite.

  • Audited who had access and removed access from a bunch of people. Auditing now happens monthly.

  • Made a lot of progress with transitioning to Python 3. Linting is fine. Fixing code is ongoing. Dropped some libraries that don't support Python 3 or were unmaintained. Vendored Crontabber.

  • Redid Docker infrastructure so we have a single Docker image that all the services use. This simplified a lot of things and makes Socorro development easier.

Thank you!

Thank you to Alexis and Ced! I hope you got as much out of your internship experiences as you could have. Thank you for being part of the team and the work you did!

Thank you to Ted, Kartikaya, Andrew, Jeff, Nicholas, Nathan, and others for helping to guide fixing signature generation to work better with Rust!

Thank you to Chris, Snorp, and others for helping to guide new product support on Socorro!

Thank you to Marcia, Ted, Liz, Philipp, dbaron, Gabriele, and others for helping to guide the reduction in crash report accepting work!

Bugzilla and GitHub stats for 2018q3

Quarter 2018q3 (2018-07-01 -> 2018-09-30)
=========================================


Bugzilla
========

  Bugs created: 149
  Creators: 21

       Will Kahn-Greene [:willkg] ET  : 85
       Alexis Deschamps [:alexisdesch : 13
       Marcia Knous [:marcia - needin : 5
                      Kartikaya Gupta : 4
           Calixte Denizet (:calixte) : 4
       Michael Kelly [:mkelly,:Osmose : 3
          Chris Peterson [:cpeterson] : 3
       Jan Andre Ikenmeyer [:darkspir : 3
                          Brian Pitts : 2
                           [:philipp] : 2
       Ted Mielczarek [:ted] [:ted.mi : 2
                 Tim Smith [:tdsmith] : 1
             Daosheng Mu[:daoshengmu] : 1
                  Wayne Mery (:wsmwk) : 1
       Liz Henry (:lizzard) (needinfo : 1
                      Rob Wu [:robwu] : 1
               Randell Jesup [:jesup] : 1
              Hiroyuki Ikezoe (:hiro) : 1
               Cristi Fogel [:cfogel] : 1
           Jeff Muizelaar [:jrmuizel] : 1
            Brian Hackett (:bhackett) : 1

  Bugs resolved: 173

                           INCOMPLETE : 2
                              WONTFIX : 16
                                FIXED : 138
                           WORKSFORME : 2
                            DUPLICATE : 4
                              INVALID : 9

  Resolvers: 21

       Will Kahn-Greene [:willkg] ET  : 136
                          Brian Pitts : 8
           Peter Bengtsson [:peterbe] : 6
       Miles Crabill [:miles] [also m : 5
       Alexis Deschamps [:alexisdesch : 5
       Michael Kelly [:mkelly,:Osmose : 3
                               Lonnen : 2
                                 kats : 1
          Chris Peterson [:cpeterson] : 1
               Sotaro Ikeda [:sotaro] : 1
              Stephen A Pohl [:spohl] : 1
                              mstange : 1
       Ted Mielczarek [:ted] [:ted.mi : 1
                  mozillamarcia.knous : 1
                    mozilla+bugcloser : 1

  Commenters: 76

                               willkg : 589
                    mozilla+bugcloser : 104
                              peterbe : 74
                               bpitts : 52
                               lonnen : 22
                           adeschamps : 21
                                miles : 20
                               mkelly : 17
                  mozillamarcia.knous : 17
                                  ted : 16
                        mcastelluccio : 14
                                  dmu : 13
                            madperson : 10
                                  jan : 10
                             vseerror : 9
                            cpeterson : 9
                   spohl.mozilla.bugs : 8
                                 kats : 7
                             cdenizet : 6
                            and others...

  Tracker bugs: 4

      1346883: [tracker] remove postgres usage from processor
      1478110: [tracker] stop saving crash data we aren't processing
      1478351: [tracker] support rust
      1478353: [tracker] support new products on Socorro

  Statistics

      Youngest bug : 0.0d: 1473935: grant :tdsmith access to rawdumps
   Average bug age : 115.4d
    Median bug age : 6.0d
        Oldest bug : 1295.0d: 1140930: Crash Reports SUMO search field is outdated, li...

GitHub
======

  mozilla-services/antenna: 12 prs

    Committers:
               willkg :    12  (  +719,   -847,   21 files)

                Total :        (  +719,   -847,   21 files)

    Most changed files:
      antenna/throttler.py (7)
      tests/unittest/test_throttler.py (5)
      tests/unittest/test_breakpad_resource.py (3)
      antenna/breakpad_resource.py (2)
      tests/unittest/test_s3_crashstorage.py (2)
      requirements/constraints.txt (2)
      requirements/default.txt (2)
      tests/unittest/conftest.py (1)
      docs/requirements.txt (1)
      docs/breakpad_reporting.rst (1)

    Age stats:
          Youngest PR : 0.0d: 277: bug 1474037: add support for FirefoxReality
       Average PR age : 1.3d
        Median PR age : 0.0d
            Oldest PR : 16.0d: 268: fix bug 1480936: change throttle default to REJECT

  mozilla-services/socorro: 115 prs

    Committers:
               willkg :    99  (+11830, -10498,  265 files)
      AlexisDeschamps :     7  ( +7613,  -2457,   72 files)
             pyup-bot :     3  (  +403,   -337,    3 files)
             jcristau :     1  (    +1,     -1,    1 files)
          sotaroikeda :     1  (    +1,     -0,    1 files)
               lonnen :     1  (   +53,  -4402,   39 files)
             cpeterso :     1  (    +1,     -0,    1 files)
               Osmose :     1  ( +6295,   -589,   20 files)
           ceddy-cedd :     1  (   +47,    -18,    6 files)

                Total :        (+26244, -18302,  361 files)

    Most changed files:
      webapp-django/crashstats/crashstats/models.py (11)
      socorro/signature/rules.py (10)
      webapp-django/crashstats/settings/base.py (10)
      socorro/processor/mozilla_transform_rules.py (9)
      webapp-django/crashstats/crashstats/utils.py (8)
      webapp-django/crashstats/crashstats/views.py (8)
      socorro/external/es/super_search_fields.py (8)
      socorro/signature/tests/test_utils.py (7)
      socorro/signature/utils.py (7)
      socorro/signature/tests/test_rules.py (7)

    Age stats:
          Youngest PR : 0.0d: 4624: fix bug 1494995: show graphics vendor/adapter n...
       Average PR age : 0.8d
        Median PR age : 0.0d
            Oldest PR : 18.0d: 4506: fix bug 1344864: added startup stats to the sig...

  mozilla-services/socorro-pigeon: 0 prs



  All repositories:

    Total merged PRs: 127


Contributors
============

  [:philipp]
  acrichton
  adityamotwani
  adrian
  aklotz
  Alexis Deschamps [:alexisdeschamps]
  aryx.bugmail
  ayumiqmazaky
  bbirtles
  bewu
  bhackett1024
  Brian Hackett (:bhackett)
  Brian Pitts
  Calixte Denizet (:calixte)
  ceddy-cedd
  chartjes
  Chris Peterson [:cpeterson]
  continuation
  cpeterson
  Cristi Fogel [:cfogel]
  Daosheng Mu[:daoshengmu]
  dave.hunt
  dbaron
  dbrown
  ddurst
  dmu
  dveditz
  ehsan
  fbraun
  hikezoe
  Hiroyuki Ikezoe (:hiro)
  hkirschner
  Jan Andre Ikenmeyer [:darkspirit]
  jcristau
  Jeff Muizelaar [:jrmuizel]
  John99-bugs
  jschneider
  kairo
  Kartikaya Gupta (email:kats@mozilla.com)
  kats
  kbrosnan
  larsberg
  Liz Henry (:lizzard) (needinfo? me)
  lonnen
  Marcia Knous [:marcia - needinfo? me]
  matt.woodrow
  mbrandt
  mcastelluccio
  merwin
  mh+mozilla
  Michael Kelly [:mkelly,:Osmose]
  miket
  Miles Crabill [:miles] [also mcrabill
  mstange
  n.nethercote
  ncsoregi
  nfroyd
  nthomas
  orangefactor
  pbone
  Peter Bengtsson [:peterbe]
  ptheriault
  pulgasaur
  Randell Jesup [:jesup]
  rares.doghi
  rbarker
  rkothari
  Rob Wu [:robwu]
  sledru
  Sotaro Ikeda [:sotaro]
  Stephen A Pohl [:spohl]
  stephen.donner
  susingh
  svoisen
  tdsmith
  Ted Mielczarek [:ted] [:ted.mielczarek]
  Tim Smith 👨‍🔬 [:tdsmith]
  viveknegi1
  vnegi
  vseerror
  Wayne Mery (:wsmwk)
  Will Kahn-Greene [:willkg] ET needinfo? me
  willkg
  yor
Want to comment? Send an email to willkg at bluesock dot org. Include the url for the blog entry in your comment so I have some context as to what you're talking about.