What is it?

Everett is a configuration library for Python apps.

Goals of Everett:

1. flexible configuration from multiple configured environments
2. easy testing with configuration
3. easy documentation of configuration for users

From that, Everett has the following features:

• is composeable and flexible
• makes it easier to provide helpful error messages for users trying to configure your software
• supports auto-documentation of configuration with a Sphinx autocomponent directive
• has an API for testing configuration variations in your tests
• can pull configuration from a variety of specified sources (environment, INI files, YAML files, dict, write-your-own)
• supports parsing values (bool, int, lists of things, classes, write-your-own)
• supports key namespaces
• supports component architectures
• works with whatever you're writing--command line tools, web sites, system daemons, etc

v1.0.0 released!

This release fixes many sharp edges, adds a YAML configuration environment, and fixes Everett so that it has no dependencies unless you want to use YAML or INI.

It also drops support for Python 2.7--Everett no longer supports Python 2.

Why you should take a look at Everett

At Mozilla, I'm using Everett for Antenna which is the edge collector for the crash ingestion pipeline for Mozilla products including Firefox and Fennec. It's been in production for a little under a year now and doing super. Using Everett makes it much easier to:

1. deal with different configurations between local development and server environments
2. test different configuration values
3. document configuration options

It's also used in a few other places and I plan to use it for the rest of the components in the crash ingestion pipeline.

First-class docs. First-class configuration error help. First-class testing. This is why I created Everett.

If this sounds useful to you, take it for a spin. It's almost a drop-in replacement for python-decouple and os.environ.get('CONFIGVAR', 'default_value') style of configuration.

Enjoy!

Thank you!

Thank you to Paul Jimenez who helped fixing issues and provided thoughtful insight on API ergonomics!

Where to go for more

For more specifics on this release, see here: https://everett.readthedocs.io/en/latest/history.html#january-7th-2019

Documentation and quickstart here: https://everett.readthedocs.io/en/latest/

Source code and issue tracker here: https://github.com/willkg/everett

What is it?

Bleach is a Python library for sanitizing and linkifying text from untrusted sources for safe usage in HTML.

Bleach v3.0.0 released!

Bleach 3.0.0 focused on easing the problems with the html5lib dependency and fixing regressions created in the Bleach 2.0 rewrite

For the first, I vendored html5lib 1.0.1 into Bleach and wrote a shim module. Bleach code uses things in the shim module which import things from html5lib. In this way I:

1. keep the two separated to some exten
2. the shim is easy to test on its own
3. it shouldn't be too hard to update html5lib versions
4. we don't have to test Bleach against multiple versions of html5lib (which took a lot of time)
5. no one has to deal with Bleach requiring one version of html5lib and other libraries requiring other versions

I think this is a big win for all of us.

The second was tricky. The Bleach 2.0 rewrite changed clean and linkify from running in the tokenizing step of HTML parsing to running after parsing is done. The parser (un)helpfully would clean up the HTML before passing it to Bleach. Because of that, the cleaned text would end up with all this extra stuff.

For example, with Bleach 2.1.4, you'd have this:

>>> import bleach
>>> bleach.clean('This is terrible.<sarcasm>')
'This is terrible.&lt;sarcasm&gt;&lt;/sarcasm&gt;'

The tokenizer would parse out things that looked like HTML tags, the parser, would see an end tag that didn't have a start tag and would add the start tag, then clean would escape the start and end tags because they weren't in the list of allowed tags. Blech.

Bleach 3.0.0 fixes that by tweaking the tokenizer to know about the list of allowed tags. With this knowledge, it can see a start, end, or empty tag and strip or escape it during tokenization. Then the parser doesn't try to fix anything.

With Bleach 3.0.0, we get this:

>>> import bleach
>>> bleach.clean('This is terrible.<sarcasm>')
'This is terrible.&lt;sarcasm&gt;'

What I could use help with

I could use help with improving the documentation. I think it's dense and all over the place focus-wise. I find it difficult to read.

If you're good with documentation, I sure could use your help. See issue 397 for more.

Where to go for more

For more specifics on this release, see here: https://bleach.readthedocs.io/en/latest/changes.html#version-3-0-0-october-3rd-2018

Documentation and quickstart here: https://bleach.readthedocs.io/en/latest/

Source code and issue tracker here: https://github.com/mozilla/bleach

Siggen

Siggen (sig-gen) is a Socorro-style signature generator extracted from Socorro and packaged with pretty bows and wrapping paper in a Python library. Siggen generates Socorro-style signatures from your crash data making it easier for you to bucket your crash data using the same buckets that Socorro uses.

The story

Back in June of 2017, the signature generation code was deeply embedded in Socorro's processor. I spent a couple of weeks extracting it and adding tooling so as to:

1. make it easier for others to make and test signature generation changes
2. make it easier for me to review signature generation changes
3. make it easier to experiment with algorithm changes and understand how it affects existing signatures
4. make it easier for other groups to use with their crash data

I wrote a blog post about extracting signature generation. That project went really well and as a result, we've made many big changes to signature generation with full confidence about how they would affect things. I claim this was a big success.

The fourth item in that list was a "hope", but wasn't meaningfully true. While it was theoretically possible, because while the code was in its own Python module, it was still all tied up with the rest of Socorro and effectively impossible for other people to use.

A year passed....

Early this summer, Will Lachance took on Ben Wu as an intern to look at Telemetry crash ping data. One of the things Ben wanted to do was generate Socorro-style signatures from the data. Then we could do analysis on crash ping data using Telemetry tools and then do deep dives on specific crashes in Socorro.

I forked the Socorro signature generation code and created Siggen and released it on PyPI. Ben and I fixed some realllly rough edges and did a few releases. We documented parts of signature generation that had never been documented before.

Ben wrote some symbolication code to convert the frames to symbols, then ran that through Siggen to generate a Socorro style signature. That's in fix-crash-sig. He did some great things with his internship project!

So then I had this problem where I had two very different versions of Socorro's signature generation code. I did several passes at unifying the two versions and fixing both sides so the code worked inside of Socorro as well as outside of Socorro. It was effectively a rewrite of the code.

The result of that work is Siggen v0.2.0.

Usage

Siggen can be installed using pip:

$ pip install siggen

Siggen comes with two command line tools.

You can generate a signature from crash data on Socorro given crashids:

$ signature <CRASHID> [<CRASHID> ...]

This is the same as doing socorro-cmd signature in the Socorro local development environment.

You can also generate a signature from crash data in JSON format:

$ signify <JSONFILE>

You can use it as a library in your Python code:

from siggen.generator import SignatureGenerator

generator = SignatureGenerator()

crash_data = {
    ...
}

ret = generator.generate(crash_data)
print(ret['signature'])

The schema is "documented" in the README which can be viewed online at https://github.com/willkg/socorro-siggen#crash-data-schema.

There's more Siggen documentation in the README though that's one area where this project is sort of lacking. There's also more documentation about the signature generation algorithm in the Socorro docs on signature generation.

What's the future of this library

This is alpha-quality software. It's possible the command line tools and API bits will change as people use it and issues pop up. Having said that, it's in use in a couple of places now, so it probably won't change much.

Some people want different kinds of signature generation. That's cool--this neither helps nor hinders that.

This doesn't solve everyone's Socorro signature generation problems, but I think it gives us a starting point for some of them and it was a doable first step.

Some people want to produce Socorro-style signatures from their crash data. This will help with that. Unless you need the code in some other language in which case this is probably not helpful.

I wrote some tools to update Siggen from changes in Socorro. That was how I built v0.2.0. I think that worked well and it's pretty easy to do, so I plan to keep this going for a while.

If you use this library, pleeeeeeease tell me where you're using it. That's how I'll know it's being used and that the time and effort to maintain it are worth while. Even better, add a star in GitHub so I have a list of you all and can contact you later. Plus it's a (terrible) indicator of library popularity.

If no one uses this library or if no one tells me (I can't tell the difference), then I'll probably stop maintaining it.

If there's interest in this algorithm, but implemented with a different language, pleeeeeeeease let me know. I'm interested in helping to build a version in Rust. Possibly other languages.

If there's interest in throwing a webapp with an API around this, chime in with specifics in [bug 828452].

Hopefully this helps. If so, let me know! If not, let me know!

What is Standup?

Standup is a system for capturing standup-style posts from individuals making it easier to see what's going on for teams and projects. It has an associated IRC bot standups for posting messages from IRC.

Short version

Standups project is done and we're going to decommission all the parts in the next couple of months.

Long version: End of days

Paul and I run Standups and have for a couple of years now. We do the vast bulk of work on it covering code changes, feature implementing, bug fixing, site administration and user support. Neither of us use it.

There are occasional contributions from other people, but not enough to keep the site going without us.

Standups has a lot of issues and a crappy UI/UX. Standups continues to accrue technical debt.

The activity seems to be dwindling over time. Groups are going elsewhere.

In June, I wrote Standup report: June 8th, 2018 in which I talked about us switching to swag-driven development as a way to boost our energy level in the project, pull in contributors, etc. We added a link to the site. It was a sort of last-ditch attempt to get the project going again.

Nothing happened. I heard absolutely nothing from anyone in any medium about the post or any of the thoughts therein.

Sometimes, it's hard to know when a project is dead. You sometimes have metrics that could mean something about the health of a project, but it's hard to know for sure. Sometimes it's hard to understand why you're sitting in a room all by yourself. Will something happen? Will someone show up? What if we just wait 15 minutes more?

I don't want to wait anymore. The project is dead. If it's not actually really totally dead, it's such a fantastic fascimile of dead that I can't tell the difference. There's no point in me waiting anymore; nothing's going to change and no one is going to show up.

Sure, maybe I could wait another 15 minutes--what's the harm since it's so easy to just sit and wait? The harm is that I've got so many things on my plate that are more important and have more value than this project. Also, I don't really like working on this project. All I've experienced in the last year was the pointy tips of bug reports most of them related to authentication. The only time anyone appreciates me spending my very very precious little free time on Standups is when I solicit it.

Also, I don't even know if it's "Standups" with an "s" or "Standup" without the "s". It might be both. I'm tired of looking it up to avoid embarrassment.

Timeline for shutdown

Shutting down projects like Standups is tricky and takes time and energy. I think tentatively, it'll be something like this:

• August 29th -- Announce end of Standups. Add site message (assuming site comes back up). Adjust irc bot reply message.
• October 1st -- Shut down IRC bots.
• October 15th -- Decommission infrastructure: websites, DNS records, Heroku infra; archive github repositories, etc.

I make no promises on that timeline. Maybe things will happen faster or slower depending on circumstances.

What we're not going to do

There are a few things we're definitely not going to do when decommissioning:

First, we will not be giving the data to anyone. No db dumps, no db access, nothing. If you want the data, you can slurp it down via the existing APIs on your own. [1]

Second, we're not going to keep the site going in read-only mode. No read-only mode or anything like that unless someone gives us a really really compelling reason to do that and has a last name that rhymes with Honnen. If you want to keep a historical mirror of the site, you can do that on your own.

Third, we're not going to point www.standu.ps or standu.ps at another project. Pretty sure the plan will be to have those names point to nothing or something like that. IT probably has a process for that.

Alternatives for people using it

If you're looking for alternatives or want to discuss alternatives with other people, check out issue 476.

But what if you want to save it!

Maybe you want to save it and you've been waiting all this time for just the right moment? If you want to save it, check out issue 477.

Many thanks!

Thank you to all the people who worked on Standups in the early days! I liked those days--they were fun.

Thank you to everyone who used Standups over the years. I hope it helped more than it hindered.

Update August 30, 2018: Added note about GET API not existing.