What is it?

Markus is a Python library for generating metrics.

Markus makes it easier to generate metrics in your program by:

• providing multiple backends (Datadog statsd, statsd, logging, logging roll-up, and so on) for sending metrics data to different places
• sending metrics to multiple backends at the same time
• providing a testing framework for easy metrics generation testing
• providing a decoupled architecture making it easier to write code to generate metrics without having to worry about making sure creating and configuring a metrics client has been done--similar to the Python logging module in this way

We use it at Mozilla on many projects.

v2.0.0 released!

I released v2.0.0 just now. Changes:

Features

• Use time.perf_counter() if available. Thank you, Mike! (#34)
• Support Python 3.7 officially.
• Add filters for adjusting and dropping metrics getting emitted. See documentation for more details. (#40)

Backwards incompatible changes

• tags now defaults to [] instead of None which may affect some expected test output.

• Adjust internals to run .emit() on backends. If you wrote your own backend, you may need to adjust it.

• Drop support for Python 3.4. (#39)

• Drop support for Python 2.7.

  If you're still using Python 2.7, you'll need to pin to <2.0.0. (#42)

Bug fixes

• Document feature support in backends. (#47)
• Fix MetricsMock.has_record() example. Thank you, John!

Where to go for more

Changes for this release: https://markus.readthedocs.io/en/latest/history.html#september-19th-2019

Documentation and quickstart here: https://markus.readthedocs.io/en/latest/index.html

Source code and issue tracker here: https://github.com/willkg/markus

Let me know whether this helps you!

Summary

Socorro Engineering team covers several projects:

• Socorro is the crash ingestion pipeline and Crash Stats web service for Mozilla's products like Firefox.
• Tecken is the symbols server for uploading, downloading, and symbolicating stacks.
• Buildhub2 is the build information index.
• Buildhub is the previous iteration of Buildhub2 that's currently deprecated and will get decommissioned soon.
• PollBot and Delivery Dashboard are something something.

This blog post summarizes our activities in July.

Highlights of July

• Socorro: Added modules_in_stack field to super search allowing people to search the set of module/debugid for functions that are in teh stack of the crashing thread.

  This lets us reprocess crash reports that have modules for which symbols were just uploaded.

• Socorro: Added PHC related fields, dom_fission_enabled, and bug_1541161 to super search.

• Socorro: Fixed some things further streamlining the local dev environment.

• Socorro: Reformatted Python code with Black.

• Socorro: Extracted supersearch and fetch-data commands as a separate Python library: https://github.com/willkg/crashstats-tools

• Tecken: Upgraded to Python 3.7 and adjusted storage bucket code to work better for multiple storage providers.

• Tecken: Added GCS emulator for local development environment.

• PollBot: Updated to use Buildhub2.

Hiatus and project changes

In April, we picked up Tecken, Buildhub, Buildhub2, and PollBot in addition to working on Socorro. Since then, we've:

• audited Tecken, Buildhub, Buildhub2, and PollBot
• updated all projects, updated dependencies, and performed other necessary maintenance
• documented deploy procedures and basic runbooks
• deprecated Buildhub in favor of Buildhub2 and updated projects to use Buildhub2

Buildhub is decomissioned now and is being dismantled.

We're passing Buildhub2 and PollBot off to another team. They'll take ownership of those projects going forward.

Socorro and Tecken are switching to maintenance mode as of last week. All Socorro/Tecken related projects are on hold. We'll continue to maintain the two sites doing "keep the lights on" type things:

• granting access to memory dumps
• adding new products
• adding fields to super search
• making changes to signature generation and updating siggen library
• responding to outages
• fixing security issues

All other non-urgent work will be pushed off.

As of August 1st, we've switched to Mozilla Location Services. We'll be auditing that project, getting it back into a healthy state, and bringing it in line with current standards and practices.

Given that, this is the last Socorro Engineering status post for a while.

Read more…

What is it?

crashstats-tools is a set of command-line tools for working with Crash Stats (https://crash-stats.mozilla.org/).

crashstats-tools comes with two commands:

• supersearch: for performing Crash Stats Super Search queries
• fetch-data: for fetching raw crash, dumps, and processed crash data for specified crash ids

v1.0.1 released!

I extracted two commands we have in the Socorro local dev environment as a separate Python project. This allows anyone to use those two commands without having to set up a Socorro local dev environment.

The audience for this is pretty limited, but I think it'll help significantly for testing analysis tools.

Say I'm working on an analysis tool that looks at crash report minidump files and does some additional analysis on it. I could use supersearch command to get me a list of crash ids to download data for and the fetch-data command to download the requisite data.

$ export CRASHSTATS_API_TOKEN=foo
$ mkdir crashdata
$ supersearch --product=Firefox --num=10 | \
    fetch-data --raw --dumps --no-processed crashdata

Then I can run my tools on the dumps in crashdata/upload_file_minidump/.

Be thoughtful about using data

Make sure to use these tools in compliance with our data policy:

https://crash-stats.mozilla.org/documentation/memory_dump_access/

Where to go for more

See the project on GitHub which includes a README which contains everything about the project including examples of usage, the issue tracker, and the source code:

https://github.com/willkg/crashstats-tools

Let me know whether this helps you!

Summary

We (Mozilla) have two different data sets for crashes: crash pings in Telemetry and crash reports in Socorro/Crash Stats. When Firefox crashes, the crash reporter collects information about the crash and this results in crash ping and crash report data. From there, the two different data things travel two different paths and end up in two different systems.

This blog post covers these two different journeys, their destinations, and the resulting properties of both data sets.

This blog post specifically talks about Firefox and not other products which have different crash reporting stories.

Updates

Updates and changes to this blog post since I wrote it:

• 2019-07-04: Crash ping data is not publicly available. Blog post updated accordingly. Thank you, Chutten!
• 2019-11-07: Added note that you can download symbols files using any http client and that the file you get back is gzipped. Thank you, Steven!

CRASH!

Firefox crashes. It happens.

The crash reporter kicks in. It uses the Breakpad library to collect data about the crashed process, package it up into a minidump. The minidump has information about the registers, what's in memory, the stack of the crashing thread, stacks of other threads, what modules are in memory, and so on.

• https://firefox-source-docs.mozilla.org/toolkit/crashreporter/crashreporter/index.html
• https://chromium.googlesource.com/breakpad/breakpad/+/master/docs/getting_started_with_breakpad.md
• https://chromium.googlesource.com/breakpad/breakpad/+/master/docs/exception_handling.md

Additionally, the crash reporter collects a set of annotations for the crash. Annotations like ProductName, Version, ReleaseChannel, BuildID and others help us group crashes for the same product and build.

• https://hg.mozilla.org/mozilla-central/file/tip/toolkit/crashreporter/CrashAnnotations.yaml

The crash reporter assembles the portions of the crash report that don't have personally identifiable information (PII) in them into a crash ping. It uses minidump-analyzer to unwind the stack. The crash ping with this stack is sent via the crash ping sender to Telemetry.

• https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/internals/pingsender.html
• https://firefox-source-docs.mozilla.org/toolkit/components/telemetry/telemetry/data/crash-ping.html

If Telemetry is disabled, then the crash ping will not get sent to Telemetry.

The crash reporter will show a crash report dialog informing the user that Firefox crashed. The crash report dialog includes a comments field for additional data about the crash and an email field. The user can choose to send the crash report or not.

If the user chooses to send the crash report, then the crash report is sent via HTTP POST to the collector for the crash ingestion pipeline. The entire crash ingestion pipeline is called Socorro. The website part is called Crash Stats.

• https://socorro.readthedocs.io/en/latest/overview.html

If the user chooses not to send the crash report, then the crash report never goes to Socorro.

If Firefox is unable to send the crash report, it keeps it on disk. It might ask the user to try to send it again later. The user can access about:crashes and send it explicitly.

Relevant backstory

  "modules": [
    ...
    {
      "debug_file": "xul.pdb",
      "base_addr": "0x7fecca50000",
      "version": "69.0.0.7091",
      "debug_id": "4E1555BE725E9E5C4C4C44205044422E1",
      "filename": "xul.dll",
      "end_addr": "0x7fed32a9000",
      "code_id": "5CF2591C6859000"
    },
    ...
  ],
  "threads": [
    {
      "frames": [
        {
          "trust": "context",
          "module_index": 8,
          "ip": "0x7feccfc3337"
        },
        {
          "trust": "cfi",
          "module_index": 8,
          "ip": "0x7feccfb0c8f"
        },
        {
          "trust": "cfi",
          "module_index": 8,
          "ip": "0x7feccfae0af"
        },
        {
          "trust": "cfi",
          "module_index": 8,
          "ip": "0x7feccfae1be"
        },
...

{
  "memoryMap": [
    [
      "xul.pdb",
      "44E4EC8C2F41492B9369D6B9A059577C2"
    ],
    [
      "wntdll.pdb",
      "D74F79EB1F8D4A45ABCD2F476CCABACC2"
    ]
  ],
  "stacks": [
    [
      [0, 11723767],
      [1, 65802]
    ]
  ]
}

What is it and where does it go?

Crash pings vs. Crash reports

Because of the above, there are big differences in collection of crash data between the two systems and what you can do with it.

Conclusions and future plans

Thanks!

Many thanks to everyone who helped with this: Will Lachance, W Chris Beard, Gabriele Svelto, Nathan Froyd, and Chutten.

Also, many thanks to Chutten and Alessio who write fantastic blog posts about Telemetry things. Those are gold.

Summary

Socorro Engineering team covers several projects:

• Socorro is the crash ingestion pipeline and Crash Stats web service for Mozilla's products like Firefox.
• Tecken is the symbols server for uploading, downloading, and symbolicating stacks.
• Buildhub2 is the build information index.
• Buildhub is the previous iteration of Buildhub2 that's currently deprecated and will get decommissioned soon.
• PollBot and Delivery Dashboard are something something.

This blog post summarizes our activities in June.

Highlights of June

• Socorro: Fixed the collector's support of a single JSON-encoded field in the HTTP POST payload for crash reports. This is a big deal because we'll get less junk data in crash reports going forward.
• Socorro: Reworked how Crash Stats manages featured versions: if the product defines a product_details/PRODUCTNAME.json file, it'll pull from that. Otherwise it calculates featured versions based on the crash reports it's received.
• Buildhub: deprecated Buildhub in favor of Buildhub2. Current plan is to decommission Buildhub in July.
• Across projects: Updated tons of dependencies that had security vulnerabilities. It was like a hamster wheel of updates, PRs, and deploys.
• Tecken: Worked on GCS emulator for local dev environment.
• All hands discussions:
  • GCP migration plan for Tecken and figure out what needs to be done.
  • Possible GCP migration schedule for Tecken and Socorro.
  • Migrating applications using Buildhub to Buildhub2 and decommissioning Buildhub in July.
  • What would happen if we switched from Elasticsearch to BigQuery?
  • Switching from Socorro's minidump-stackwalk to minidump-analyzer.
  • Re-implementing the Socorro Top Crashers and Signature reports using Telemetry tools and data.
  • Writing a symbolicator and Socorro-style signature generator in Rust that can be used for crash reports in Socorro and crash pings in Telemetry.
  • The crash ping vs. crash report situation (blog post coming soon).

Read more…

CODE_OF_CONDUCT.md

This week, Mozilla added PRs to all the repositories that Mozilla has on GitHub that aren't forks, Servo, or Rust. The PRs add a CODE_OF_CONDUCT.md file and also include some instructions on what projects can do with it. This standardizes inclusion of the code of conduct text in all projects.

I'm a proponent of codes of conduct. I think they're really important. When I was working on Bleach with Greg, we added code of conduct text in September of 2017. We spent a bunch of time thinking about how to do that effectively and all the places that users might encounter Bleach.

I spent some time this week trying to figure out how to do what we did with Bleach in the context of the Mozilla standard. This blog post covers those thoughts.

This blog post covers Python-centric projects. Hopefully, some of this applies to other project types, too.

What we did in Bleach in 2017 and why

In September of 2017, Greg and I spent some time thinking about all the places the code of conduct text needs to show up and how to implement the text to cover as many of those as possible for Bleach.

PR #314 added two things:

• a CODE_OF_CONDUCT.rst file
• a copy of the text to the README

In doing this, the code of conduct shows up in the following places:

• the contents of the repository at https://github.com/mozilla/bleach
• the rendered README text also at https://github.com/mozilla/bleach
• the docs at https://bleach.readthedocs.io/
• the PyPI page at https://pypi.org/project/bleach
• in the source tarball of all releases
• on the "new issue" page in GitHub [1]

In this way, users could discover Bleach in a variety of different ways and it's very likely they'll see the code of conduct text before they interact with the Bleach community.

The Mozilla standard

The Mozilla standard applies to all repositories in Mozilla spaces on GitHub and is covered in the Repository Requirements wiki page.

It explicitly requires that you add a CODE_OF_CONDUCT.md file with the specified text in it to the root of the repository.

This makes sure that all repositories for Mozilla things have a code of conduct specified and also simplifies the work they need to do to enforce the requirement and update the text over time.

This week, a bot added PRs to all repositories that didn't have this file. Going forward, the bot will continue to notify repositories that are missing the file and will update the file's text if it ever gets updated.

How to work with the Mozilla standard

Let's go back and talk about Bleach. We added a file and a blurb to the README and that covered the following places:

• the contents of the repository at https://github.com/mozilla/bleach
• the rendered README text also at https://github.com/mozilla/bleach
• the docs at https://bleach.readthedocs.io/
• the PyPI page at https://pypi.org/project/bleach
• in the source tarball of all releases

With the new standard, we only get this:

• the contents of the repository at https://github.com/mozilla/bleach
• (maybe) in the source tarball of all releases

In order to make sure the file is in the source tarball, you have to make sure it gets added. The bot doesn't make any changes to fix this. You can use check-manifest to help make sure that's working. You might have to adjust your MANIFEST.in file or something else in your build pipeline--hence the maybe.

Because the Mozilla standard suggests they may change the text of the CODE_OF_CONDUCT.md file, it's a terrible idea to copy the contents of the file around your repository because that's a maintenance nightmare--so that idea is out.

It's hard to include .md files in reStructuredText contexts. You can't just add this to the long description of the setup.py file and you can't include it in a Sphinx project [2].

Greg and I chatted about this a bit and I think the best solution is to add minimal text that points to the CODE_OF_CONDUCT.md in GitHub to the README. Something like this:

Code of Conduct
===============

This project and repository is governed by Mozilla's code of conduct and
etiquette guidelines. For more details please see the `CODE_OF_CONDUCT.md
file <https://github.com/mozilla/bleach/blob/master/CODE_OF_CONDUCT.md>`_.

In Bleach, the long description set in setup.py includes the README:

def get_long_desc():
    desc = codecs.open('README.rst', encoding='utf-8').read()
    desc += '\n\n'
    desc += codecs.open('CHANGES', encoding='utf-8').read()
    return desc

...

setup(
    name='bleach',
    version=get_version(),
    description='An easy safelist-based HTML-sanitizing tool.',
    long_description=get_long_desc(),
    ...

In Bleach, the index.rst of the docs also includes the README:

.. include:: ../README.rst

Contents
========

.. toctree::
   :maxdepth: 2

   clean
   linkify
   goals
   dev
   changes


Indices and tables
==================

* :ref:`genindex`
* :ref:`search`

In this way, the README continues to have text about the code of conduct and the link goes to the file which is maintained by the bot. The README is included in the long description of setup.py so this code of conduct text shows up on the PyPI page. The README is included in the Sphinx docs so the code of conduct text shows up on the front page of the project documentation.

So now we've got code of conduct text pointing to the CODE_OF_CONDUCT.md file in all these places:

• the contents of the repository at https://github.com/mozilla/bleach
• the rendered README text also at https://github.com/mozilla/bleach
• the docs at https://bleach.readthedocs.io/
• the PyPI page at https://pypi.org/project/bleach
• in the source tarball of all releases

Plus the text will get updated automatically by the bot as changes are made.

Excellent!

Future possibilities

GitHub has a Community Insights page for each project. This is the one for Bleach. There's a section for "Code of conduct", but you only get a green checkmark if and only if you use one of GitHub's pre-approved code of conduct files.

There's a discussion about that in their forums.

Is this checklist helpful to people? Does it mean something to have all these items checked off? Is there someone checking for this sort of thing? If so, then maybe we should get the Mozilla text approved?

Hope this helps!

I hope to roll this out for the projects I maintain on Monday.

I hope this helps you!