Socorro Engineering: June 2019 happenings

Summary

Socorro Engineering team covers several projects:

This blog post summarizes our activities in June.

Highlights of June

  • Socorro: Fixed the collector's support of a single JSON-encoded field in the HTTP POST payload for crash reports. This is a big deal because we'll get less junk data in crash reports going forward.
  • Socorro: Reworked how Crash Stats manages featured versions: if the product defines a product_details/PRODUCTNAME.json file, it'll pull from that. Otherwise it calculates featured versions based on the crash reports it's received.
  • Buildhub: deprecated Buildhub in favor of Buildhub2. Current plan is to decommission Buildhub in July.
  • Across projects: Updated tons of dependencies that had security vulnerabilities. It was like a hamster wheel of updates, PRs, and deploys.
  • Tecken: Worked on GCS emulator for local dev environment.
  • All hands discussions:
    • GCP migration plan for Tecken and figure out what needs to be done.
    • Possible GCP migration schedule for Tecken and Socorro.
    • Migrating applications using Buildhub to Buildhub2 and decommissioning Buildhub in July.
    • What would happen if we switched from Elasticsearch to BigQuery?
    • Switching from Socorro's minidump-stackwalk to minidump-analyzer.
    • Re-implementing the Socorro Top Crashers and Signature reports using Telemetry tools and data.
    • Writing a symbolicator and Socorro-style signature generator in Rust that can be used for crash reports in Socorro and crash pings in Telemetry.
    • The crash ping vs. crash report situation (blog post coming soon).

Bugzilla and GitHub stats for June

Bugzilla
========

  Bugs created: 42
  Creators: 19

       Will Kahn-Greene [:willkg] ET  : 23
       Marcia Knous [:marcia - needin : 2
              Thom Chiovoloni [:tcsc] : 1
         Dylan Roeh (:droeh) (he/him) : 1
                      janet  [:janet] : 1
       Cornel Ionce [:cornel_ionce],  : 1
       Cristian Baica [:cbaica], Rele : 1
       Timea Zsoldos [:zstimi/tzsoldo : 1
       Camelia Badau [:cbadau], Relea : 1
       Anca Soncutean [:Anca], Deskto : 1
          Alexandru Trif, QA [:atrif] : 1
          Catalin Sasca, QA [:csasca] : 1
       Ciprian Georgiu [:ciprian_geor : 1
         Cristian Fogel, QA [:cfogel] : 1
          Vlad Lucaci, QA  (:vlucaci) : 1
       Oana Botisan, Desktop Release  : 1
       Daniel Cicas [:dcicas], Releas : 1
       Gabi Cheta [:Gabi] Release Des : 1
             Jeff Gilbert [:jgilbert] : 1

  Bugs resolved: 29

                                FIXED : 27
                              WONTFIX : 1
                              INVALID : 1

  Resolvers: 3

       Will Kahn-Greene [:willkg] ET  : 22
           John Whitlock [:jwhitlock] : 7

  Commenters: 24

                               willkg : 115
                            jwhitlock : 12
                           tgrabowski : 5
                            wlachance : 3
                        catalin.sasca : 3
                               fitojb : 2
                               lhenry : 2
                               ryanvm : 2
                         bogdan.maris : 2
                                droeh : 2
                              dbolter : 2
                       anca.soncutean : 2
                       alexandru.trif : 2
                                   me : 1
                              peterbe : 1
                            pulgasaur : 1
                         mihai.boldan : 1
                  mozillamarcia.knous : 1
                            s.kaspari : 1
                               bpitts : 1
                          tchiovoloni : 1
                          jdragojevic : 1
                               tmaity : 1
                              gasofie : 1

  Tracker bugs: 0


  Statistics

      Youngest bug : 0.0d: 1556802: socorro deploy: 382
   Average bug age : 22.8d
    Median bug age : 5.0d
        Oldest bug : 397.0d: 1460709: Fonts are extremely thin

GitHub
======

  mozilla-services/socorro:

    Merged PRs: 14

    * 4973: bug 1561697: add mozilla::ipc::WriteIPDLParam to prefix list (willkg)
    * 4972: Update docs (willkg)
    * 4971: bug 1560998: make RemoteType private (willkg)
    * 4969: Update Fennec release version to 67.0.3 (rvandermeulen)
    * 4968: bug 1559153: remove depcheck cronrun cmd (willkg)
    * 4967: Update Fennec Nightly version to 68.0a1 (rvandermeulen)
    * 4966: bug 1552898: fix product details url (willkg)
    * 4964: bug 1552898: add product_details and validation test (willkg)
    * 4965: bug 1552898: implement manual featured versions (willkg)
    * 4958: bug 1555744: support --host and --processed in fetch_crash_data (willkg)
    * 4963: bug 1557012: add Allocator<T>::malloc to prefix list (willkg)
    * 4962: bug 1545446: fix input box widths in super search (willkg)
    * 4931: bug 1545446: Remove Fira-Sans, reduce font list (jwhitlock)
    * 4961: Update requirements, June 2019 (jwhitlock)

    Committers:
               willkg :    10  (  +364,   -579,   25 files)
        rvandermeulen :     2  (    +2,     -2,    1 files)
            jwhitlock :     2  (   +67,    -55,    5 files)

                Total :        (  +433,   -636,   29 files)

    Most changed files:
      product_details/FennecAndroid.json (4)
      webapp-django/crashstats/settings/base.py (3)
      socorro/signature/siglists/prefix_signature_re.txt (2)
      requirements/default.txt (2)
      product_details/README.rst (2)
      webapp-django/crashstats/crashstats/tests/test_utils.py (2)
      CONTRIBUTING.rst (1)
      docs/howto.rst (1)
      socorro-cmd (1)
      socorro/external/es/super_search_fields.py (1)

    Age stats:
          Youngest PR : 0.0d: 4972: Update docs
       Average PR age : 2.4d
        Median PR age : 0.0d
            Oldest PR : 27.0d: 4931: bug 1545446: Remove Fira-Sans, reduce font list

  mozilla-services/antenna:

    Merged PRs: 7

    * 349: Update dependencies (willkg)
    * 348: bug 1560168: ignore querystring params in HTTP POSTs (willkg)
    * 347: bug 1560168: add logging for json-and-kv situation (willkg)
    * 346: bug 1559151: add pyup and set to monthly updates (willkg)
    * 318: bug 1559151: remove pyup configuration (willkg)
    * 317: bug 1557706: fix error string (willkg)
    * 316: bug 1557706: specify why crash report was discarded (willkg)

    Committers:
               willkg :     7  (  +220,   -180,    5 files)

                Total :        (  +220,   -180,    5 files)

    Most changed files:
      antenna/breakpad_resource.py (4)
      .pyup.yml (2)
      requirements/constraints.txt (1)
      requirements/default.txt (1)
      tests/unittest/test_breakpad_resource.py (1)

    Age stats:
          Youngest PR : 0.0d: 349: Update dependencies
       Average PR age : 0.0d
        Median PR age : 0.0d
            Oldest PR : 0.0d: 349: Update dependencies

  mozilla-services/tecken:

    Merged PRs: 14

    * 1835: Fix shields in README (willkg)
    * 1834: Update mozillaparsys/oidc_testprovider Docker digest to 876268f (renovate[bot])
    * 1832: Update node:10.16.0-slim Docker digest to 9afe43a (renovate[bot])
    * 1831: Update dependency Sphinx to v2.1.1 (renovate[bot])
    * 1828: Upgrade js-yaml (willkg)
    * 1823: Update node:10.16.0-slim Docker digest to f37262e (renovate[bot])
    * 1820: Update dependencies 20190601 (willkg)
    * 1826: Upgrade handlebars to 4.1.2 (willkg)
    * 1825: Overhaul make-tag (willkg)
    * 1821: bug 1556775: add "make setup" (willkg)
    * 1815: Update Node.js to v10.16.0 (renovate[bot])
    * 1819: Update dependency Sphinx to v2.1.0 (renovate[bot])
    * 1812: Update python:3.6-slim Docker digest to bab4801 (renovate[bot])
    * 1811: Update node:10.15.3-slim Docker digest to 5177e5d (renovate[bot])

    Committers:
        renovate[bot] :     8  (   +12,    -12,    4 files)
               willkg :     6  (  +352,   -227,   16 files)

                Total :        (  +364,   -239,   20 files)

    Most changed files:
      Dockerfile (5)
      frontend/Dockerfile (4)
      docs-requirements.txt (2)
      frontend/yarn.lock (2)
      docs/dev.rst (2)
      README.rst (1)
      docker/images/oidcprovider/Dockerfile (1)
      frontend/package.json (1)
      requirements-constraints.txt (1)
      requirements.txt (1)

    Age stats:
          Youngest PR : 0.0d: 1835: Fix shields in README
       Average PR age : 2.6d
        Median PR age : 0.0d
            Oldest PR : 12.0d: 1812: Update python:3.6-slim Docker digest to bab4801

  mozilla-services/buildhub2:

    Closed issues: 1

    Merged PRs: 7

    * 594: Fix shields in README (willkg)
    * 593: Update to prettier 1.18.2 (willkg)
    * 592: Update eslint (willkg)
    * 590: Overhaul docs (willkg)
    * 591: Fix docs building and theme (willkg)
    * 589: Fix make-tag to support other remote names (willkg)
    * 588: Updates for 20190601 (willkg)

    Committers:
               willkg :     7  ( +1379,  -1955,   31 files)

                Total :        ( +1379,  -1955,   31 files)

    Most changed files:
      requirements.txt (3)
      README.rst (2)
      ui/package.json (2)
      ui/yarn.lock (2)
      .circleci/config.yml (2)
      Makefile (2)
      docs/conf.py (2)
      bin/build-docs-locally.sh (1)
      docker-compose.ci.yml (1)
      docker-compose.yml (1)

    Age stats:
          Youngest PR : 0.0d: 594: Fix shields in README
       Average PR age : 0.0d
        Median PR age : 0.0d
            Oldest PR : 0.0d: 594: Fix shields in README

  mozilla-services/buildhub:

    Merged PRs: 7

    * 540: Update eslint to 4.18.2 (willkg)
    * 538: Add deprecation notice to README (willkg)
    * 537: Update ui deploy (willkg)
    * 536: Redo ui deploys (willkg)
    * 535: Fix release process (willkg)
    * 534: Add deprecation notice (willkg)
    * 533: Update deps (willkg)

    Committers:
               willkg :     7  ( +2506,  -2442,   10 files)

                Total :        ( +2506,  -2442,   10 files)

    Most changed files:
      README.md (4)
      ui/package.json (3)
      ui/yarn.lock (3)
      bin/make-release.py (1)
      ui/src/App.js (1)
      ui/src/index.css (1)
      Makefile (1)
      jobs/buildhub/inventory_to_records.py (1)
      jobs/requirements/constraints.txt (1)
      kinto/Dockerfile (1)

    Age stats:
          Youngest PR : 0.0d: 540: Update eslint to 4.18.2
       Average PR age : 0.0d
        Median PR age : 0.0d
            Oldest PR : 0.0d: 540: Update eslint to 4.18.2

  mozilla/PollBot:
    Closed issues: 3

    Merged PRs: 2

    * 248: Pin requirements, nix tox (willkg)
    * 245: project scaffolding fixes (willkg)

    Committers:
               willkg :     2  (  +509,   -261,   19 files)

                Total :        (  +509,   -261,   19 files)

    Most changed files:
      Dockerfile (2)
      requirements.txt (2)
      scripts/run-tests.sh (2)
      MANIFEST.in (1)
      constraints.txt (1)
      setup.cfg (1)
      tests/test_views.py (1)
      tox.ini (1)
      .gitignore (1)
      API_CHANGELOG.rst (1)

    Age stats:
          Youngest PR : 0.0d: 248: Pin requirements, nix tox
       Average PR age : 0.0d
        Median PR age : 0.0d
            Oldest PR : 0.0d: 248: Pin requirements, nix tox


  All repositories:

    Total closed issues: 4
    Total merged PRs: 51


Contributors
============

  Alexandru Trif, QA [:atrif]
  Anca Soncutean [:Anca], Desktop Release QA
  bogdan.maris
  bpitts
  Camelia Badau [:cbadau], Release Desktop QA
  Catalin Sasca, QA [:csasca]
  Ciprian Georgiu [:ciprian_georgiu], Release Desktop QA
  Cornel Ionce [:cornel_ionce], Desktop Release QA
  Cristian Baica [:cbaica], Release Desktop QA
  Cristian Fogel, QA [:cfogel]
  Daniel Cicas [:dcicas], Release QA
  dbolter
  Dylan Roeh (:droeh) (he/him)
  fitojb
  Gabi Cheta [:Gabi] Release Desktop QA
  gasofie
  janet  [:janet]
  Jeff Gilbert [:jgilbert]
  John Whitlock [:jwhitlock]
  lhenry
  Marcia Knous [:marcia - needinfo? me]
  me
  mihai.boldan
  Oana Botisan, Desktop Release QA
  peterbe
  pulgasaur
  rvandermeulen
  s.kaspari
  tgrabowski
  Thom Chiovoloni [:tcsc]
  Timea Zsoldos [:zstimi/tzsoldos], Desktop Release QA
  tmaity
  Vlad Lucaci, QA  (:vlucaci)
  Will Kahn-Greene [:willkg] ET needinfo? me
  Will Lachance
Want to comment? Send an email to willkg at bluesock dot org. Include the url for the blog entry in your comment so I have some context as to what you're talking about.