Socorro: March 2019 happenings

, Will Kahn-Greene | share this (mastodon)
Note: This is an old post in a blog with a lot of posts over a long span of time. The world has changed, technologies have changed, and I've changed. It's likely this is out of date, the code doesn't work, the ideas haven't aged well, or the ideas were terrible to begin with. Let me know if you think this is something that needs updating.

Summary

Socorro is the crash ingestion pipeline for Mozilla's products like Firefox. When Firefox crashes, the crash reporter collects data about the crash, generates a crash report, and submits that report to Socorro. Socorro saves the crash report, processes it, and provides an interface for aggregating, searching, and looking at crash reports.

This blog post summarizes Socorro activities in March.

Staff in March

Me.

Highlights of March

  • Pub/Sub migration

    • Changed the collector, processor, and webapp to use Google Cloud Platform Pub/Sub instead of RabbitMQ for queuing for crash processing ([bug 1527343], [bug 1527346], [bug 1527345])

      This is on stage, but will go to prod very soon.

  • Development and infrastructure-related changes

    • Redo tests and configuration in Antenna to match Socorro ([bug 1531865])

    • Update Docker image in CI. ([bug 1540248])

    • Update base Docker image for all containers. ([bug 1539272])

    • Rewrite archivescraper to use multi-processing which makes it SOOOOO much faster. ([bug 1536147])

  • Upgraded dependencies

Bugzilla and GitHub stats for March

Period (2019-03-01 -> 2019-03-31)
=================================


Bugzilla
========

  Bugs created: 32
  Creators: 9

       Will Kahn-Greene [:willkg] ET  : 20
       Marcia Knous [:marcia - needin : 1
           Calixte Denizet (:calixte) : 1
          Andrew Overholt [:overholt] : 1
                Edouard Oger [:eoger] : 1
       Robert Strong (Robert they/the : 1
       Adam Gashlin (he/him) [:agashl : 1
         Myk Melez [:myk] [@mykmelez] : 1
       Kartikaya Gupta (email:kats@mo : 1

  Bugs resolved: 22

                                FIXED : 17
                              WONTFIX : 3
                           WORKSFORME : 1
                              INVALID : 1

  Resolvers: 9

       Will Kahn-Greene [:willkg] ET  : 15
          Marco Castelluccio [:marco] : 1
            Andrew McCreight [:mccr8] : 1
                                miles : 1
           Peter Bengtsson [:peterbe] : 1
                          Brian Pitts : 1
                               nfroyd : 1
                        mcastelluccio : 1

  Commenters: 25

                               willkg : 84
                              peterbe : 6
                                eoger : 6
                         continuation : 5
                               gguthe : 4
                        mcastelluccio : 4
                                  amk : 4
                                  ted : 3
                               nfroyd : 3
                               sledru : 3
                                  myk : 3
                   robert.strong.bugs : 2
                             jmathies : 2
                           viveknegi1 : 2
                             nehagarw : 2
                                 kats : 2
                    mozilla+bugcloser : 1
                               mkelly : 1
                  mozillamarcia.knous : 1
                                miles : 1
                           nalexander : 1
                              gsvelto : 1
                            cpeterson : 1
                             agashlin : 1
                              jwalker : 1

  Tracker bugs: 0


  Statistics

      Youngest bug : 0.0d: 1533007: antenna deploy: 28
   Average bug age : 45.1d
    Median bug age : 3.0d
        Oldest bug : 551.0d: 1400076: running antenna with socorro

GitHub
======

  mozilla-services/antenna: 16 prs

    Merged PRs:

    * 310: bug 1538202: fix the docker CMD line to use the new worker (willkg)
    * 309: Add code of conduct (willkg)
    * 308: bug 1538202: fix SystemExit (attempt 3) (willkg)
    * 307: bug 1538202: add monkey.patch_all() calls to guarantee patching (willkg)
    * 306: bug 1538202: remove preload (willkg)
    * 305: Add mozilla/socorro_collector:latest image (willkg)
    * 304: bug 1527343: switch to use CRASHPUBLISH_SERVICE_ACCOUNT_FILE (willkg)
    * 303: Update docs regarding RabbitMQ -> Pub/Sub (willkg)
    * 301: fix bug 1533732: add support for JSON-encoded fields (willkg)
    * 302: bug 1533732: fix miniposter to support JSON blobs (willkg)
    * 299: fix bug 1527343: implement publishing to Pub/Sub (willkg)
    * 300: fix bug 1359147: fix s3 and pubsub startup and health checks (willkg)
    * 298: Update dependencies (willkg)
    * 297: bug 1531865: fix image name in build/push step (willkg)
    * 296: bug 1517807: Fix s3 test class name; remove pytest-env (willkg)
    * 295: fix bug 1531865: redo tests and configuration (willkg)

    Committers:
               willkg :    16  ( +1662,   -752,   46 files)

                Total :        ( +1662,   -752,   46 files)

    Most changed files:
      antenna/ext/pubsub/crashpublish.py (5)
      docker/Dockerfile (4)
      requirements/default.txt (4)
      requirements/constraints.txt (3)
      testlib/mini_poster.py (3)
      tests/systemtest/conftest.py (3)
      tests/unittest/conftest.py (3)
      .circleci/config.yml (3)
      docker/config/local_dev.env (3)
      docker/run_tests.sh (3)

    Age stats:
          Youngest PR : 0.0d: 310: bug 1538202: fix the docker CMD line to use the...
       Average PR age : 0.1d
        Median PR age : 0.0d
            Oldest PR : 1.0d: 299: fix bug 1527343: implement publishing to Pub/Sub

  mozilla-services/socorro: 27 prs

    Merged PRs:

    * 4867: bug 1540248: fix setting up docker-compose in CI (willkg)
    * 4862: bug 1539233: fix table and add missing processed crash check to verifyprocessed (willkg)
    * 4863: bug 1539272: switch to 3.6.8-slim-stretch (willkg)
    * 4861: Add code of coduct file (willkg)
    * 4860: bug 1527345: reduce Pub/Sub pulling (willkg)
    * 4859: bug 1536903: fix queryset for processing in admin (willkg)
    * 4858: bug 1527345: fix PubSubCrashQueue iterator (willkg)
    * 4857: bug 1536903: improve bulk processing performance (willkg)
    * 4856: Switch to readthedocs config file (willkg)
    * 4855: bug 1527346: use credentials file when creating PublisherClient (willkg)
    * 4854: Minor fixes (willkg)
    * 4853: bug 1527345: fix handling of "test" crash ids (willkg)
    * 4852: bug 1400076: rework collector service to work right (willkg)
    * 4850: bug 1536903: add bulk process for missing processed crashes (willkg)
    * 4851: bug 1527346: configure service_account_file for webapp (willkg)
    * 4848: fix bug 1527346: switch webapp to use Pub/Sub (willkg)
    * 4849: bug 1536147: make archivescraper faster (willkg)
    * 4847: bug 1527345: implement Pub/Sub queue and scaffolding (willkg)
    * 4846: Change Buildhub base URL (peterbe)
    * 4845: bug 1527345: redo FetchTransformSaveApp and queuing (willkg)
    * 4842: Update Python dependencies (willkg)
    * 4844: bug 1534617: add "is_processed" check (willkg)
    * 4843: bug 1534617: add "report url link" to Django admin page (willkg)
    * 4841: fix bug 1534402: fix error handling in multiprocessing code (willkg)
    * 4840: fix bug 1528243: verify crashes are processed (willkg)
    * 4839: Fix db setup (willkg)
    * 4838: fix typos in fetch_crashids documentation (froydnj)

    Committers:
               willkg :    25  ( +2899,  -1591,   71 files)
              peterbe :     1  (    +1,     -1,    1 files)
              froydnj :     1  (    +2,     -2,    1 files)

                Total :        ( +2902,  -1594,   72 files)

    Most changed files:
      webapp-django/crashstats/crashstats/admin.py (7)
      socorro/external/pubsub/crashqueue.py (7)
      webapp-django/crashstats/crashstats/models.py (5)
      socorro/cron/jobs/verify_processed.py (4)
      socorro/unittest/cron/jobs/test_verify_processed.py (3)
      docker/config/local_dev.env (3)
      webapp-django/crashstats/settings/base.py (3)
      socorro/unittest/conftest.py (2)
      docker/run_tests.sh (2)
      docker/run_tests_in_docker.sh (2)

    Age stats:
          Youngest PR : 0.0d: 4867: bug 1540248: fix setting up docker-compose in CI
       Average PR age : 0.0d
        Median PR age : 0.0d
            Oldest PR : 1.0d: 4848: fix bug 1527346: switch webapp to use Pub/Sub

  mozilla-services/socorro-pigeon: 1 prs

    Merged PRs:

    * 43: Add code of conduct (willkg)

    Committers:
               willkg :     1  (   +17,     -0,    1 files)

                Total :        (   +17,     -0,    1 files)

    Most changed files:
      CODE_OF_CONDUCT.rst (1)

    Age stats:
          Youngest PR : 0.0d: 43: Add code of conduct
       Average PR age : 0.0d
        Median PR age : 0.0d
            Oldest PR : 0.0d: 43: Add code of conduct


  All repositories:

    Total merged PRs: 44


Contributors
============

  Adam Gashlin (he/him) [:agashlin]
  amk
  Andrew McCreight [:mccr8]
  Andrew Overholt [:overholt]
  Brian Pitts
  Calixte Denizet (:calixte)
  continuation
  cpeterson
  Edouard Oger [:eoger]
  eoger
  froydnj
  gguthe
  gsvelto
  jmathies
  jwalker
  Kartikaya Gupta (email:kats@mozilla.com)
  Marcia Knous [:marcia - needinfo? me]
  Marco Castelluccio [:marco]
  miles
  mkelly
  Myk Melez [:myk] [@mykmelez]
  nalexander
  nehagarw
  nfroyd
  Peter Bengtsson [:peterbe]
  Robert Strong (Robert they/them) [:rstrong] (use needinfo to contact me)
  sledru
  ted
  viveknegi1
  Will Kahn-Greene [:willkg] ET needinfo? me
Want to comment? Send an email to willkg at bluesock dot org. Include the url for the blog entry in your comment so I have some context as to what you're talking about.