Project

time:

3 months

impact:

• fixed l10n-related HTTP 500 errors for SUMO and Input

• paved road for MDN, AMO and other Mozilla sites to use the same strategy

Problem statement

When we deploy support.mozilla.org (SUMO) and Input [1], it fetches the most recent localized strings .po files from SVN, compiles them to .mo files, and ships them with the site. Because SUMO supports many languages and the deploy pulls down the most recent changes, there's no way to effectively test the entire site for all languages before deploying to production. Because of that, users experience HTTP 500 server errors on pages that have bad strings.

When there are server errors, we get notified, write up a bug, and then have to go fix it and push the fix out as soon as possible. Fixing issues is difficult since we don't know most of the languages the site is translated in and our l10n community spans many timezones, so getting help can take many hours.

That was covered roughly in [bug 841412] for SUMO and [bug 875313] for Input.

On top of that, SUMO sends emails to SUMO users and if the localized strings are bad, then emails don't get sent. That's covered in [bug 850215].

Why are there problems in localizations? Judging by the strings we were seeing, we think we had a few issues:

1. The localizer changes the formatting token.

   We're using gettext and Python and several different token formats. For example, %s, %(name)s, and {name}. If the localizer changes the structure of the token, that will cause a server error.

2. The localizer translates the token.

   For example, the token is {name}, but the translated token is {nombre}.

3. The localizer copies a string from somewhere else with different tokens.

   For example, the string has {url}, but the translated string has {helpurl}.

Thus we're faced with:

1. a series of problems with strings that happen semi-frequently

2. server errors preventing users from seeing certain parts of SUMO

3. we can't suss out these errors by testing the whole site for every locale before every prod deploy

4. each server error is a priority 1 interrupt

SUMO and Input weren't the only sites that has this problem---it's a problem for MDN, AMO, and other sites, too.

Solution

I wondered if we could effectively lint .po files during deploy and only compile the .po files that didn't have problems. If we did that, then the problem would stop.

I wrote a localelinter.py script in SUMO [2] that let me experiment with writing a linter and tying it to the deploy process.

That went well and I wanted to use the same system for two different projects plus I suspected others would want to use it, too. Further, we had a poxx.py script that let us debug layout problems related to translated strings and I wanted to merge both of these into a new library.

I created Dennis that let you lint and fake-translate strings.

On July 29th, I released Dennis v0.3.3 which I felt was good enough for us and other people to use. We switched SUMO and Input to use it.

Our server errors from localization issues mostly ended. Periodically, we'd encounter a new kind of issue and would improve Dennis to catch it.

We made the deploy logs viewable so linting errors could be seen, bugs could be written, and errors could get fixed.

I talked about how we use it on Input and SUMO along with the bash script we used that compiled .po files that linted successfully to .mo files. MDN, AMO, and other sites switched to using Dennis thus eliminating the errors for them, too.

Impact

This eliminated a frequent cause of HTTP 500 errors which caused downtime for the site, prevented users from getting Firefox support, and created frequent interruptions for site engineers.

Building it as a library allowed other Mozilla sites to use it eliminating the problem for them, too.

Alternatives

What is pyvideo.org

pyvideo.org is an index of Python-related conference and user-group videos on the Internet. Saw a session you liked and want to share it? It's likely you can find it, watch it, and share it with pyvideo.org.

Status

Lot of stuff has happened since the last status report, but there are four things of note:

1. Sheila is now a co-admin of pyvideo.org. She has been for a couple of months. I need to update the site to reflect this.

   I'm really psyched about this. It's a ton of work and I'm just not managing it well. Splitting the work should make it more manageable.

2. Back in July, Sheila poked me about a tweet Jesse wrote suggesting Rackspace was interested in sponsoring Open Source projects. She contacted Jesse and set everything up.

   I'm psyched that Rackspace agreed to sponsor pyvideo.org by providing free hosting. Several months later, I moved pyvideo.org from where it was before to a vm at Rackspace.

   I'm really excited about this! It makes a bunch of problems that I was trying to figure out what to do about go away.

   Thank you, Rackspace!

   I need to update the site to reflect this.

3. Sheila discovered that blip.tv was expiring a bunch of accounts that held conference videos and that those videos would go away. She and I scrambled to download all the files from blip and move them to Rackspace cloudfiles. It's about 600 videos and around 250gb of data.

   In the process of doing that, we saved videos for DjangoCon EU 2010, DjangoCon EU 2011 and PyGotham 2012. I added these to pyvideo.org today. These videos have pages that are stubs with no metadata. I've got that in my queue of things to fix.

   Also, the thumbnails for all the videos on blip.tv are on my laptop which isn't very helpful. I need to move those and update the videos in pyvideo.org.

   As a side note, if we didn't have hosting from Rackspace, we'd have been totally screwed. Thank you, Jesse Noller and Rackspace!

4. I've been working on the richard codebase fixing architectural problems, reducing the complexities and trying to clean it up so it's in a better state. That work is almost done. When it is, I'll update pyvideo.org with the new site. At this rate, I think I can finish the work this year, but that assumes there aren't any more emergencies.

5. I've been thinking about how to build a better communication channel for pyvideo.org so people can more easily follow what's going on so they can act on things they're interested in.

   pyvideo.org has a "site news" section. It's a pain in the ass to use and it's not syndicated anywhere and it's likely no one sees it.

   Blogging status reports like this on my blog is better, but I don't think my blog is very widely read. Making my blog more widely-read seems like a lot of work and I'm not sure I can do it effectively anyhow.

   So I've decided to ditch the "site news" section of pyvideo.org and switch to Twitter. I started a @PyvideoOrg account.

   I'll tweet site updates, calls for help and newly posted conferences. I'm tossing around tweeting new videos when they get posted, but videos tend to get posted in huge batches and getting > 40 tweets all at once is a total drag. I'll have to think about that some more.

   Follow @PyvideoOrg if you're interested! Also, feel free to tweet at that account.

   I need to update the site to reflect this.

Also, in my life things are pretty nuts. I have a new kid and juggling everything was impossible for a while. I think that should easy up now and I can spend more time on pyvideo.org going forward.

That's the state of things!

Also, thank you thank you thank you thank you Rackspace!

What is it?

Dennis is a Python command line utility (and library) for working with localization. It includes:

• a linter for finding problems in strings in .po files

• a statuser for seeing the high-level status of your .po files

• a translator for strings .po files

v0.3.10 released!

v0.3.8 fixed mismatched errors in plural strings. Thanks Mike!

v0.3.9 fixed two false positives in error detection.

v0.3.10 adds the status subcommand and the Zombie transform which, like the dubstep transform, is silly but fun.

45 out of 47 Djangonauts use the Zombie transform to make their site accessible to those who have departed. This could open up your app to millions of new users. Truth.

What is it?

Dennis is a Python command line utility (and library) for working with localization. It includes:

• a linter for finding problems in strings in .po files

• a translator for strings .po files

v0.3.7 released!

v0.3.6 fixed a goof where the linter was skipping errors. Oops.

v0.3.7 adds a dubstep translator (which is just plain silly, but awesome).

Truth: 9 out of 10 experts agree SUMO is extra helpful in dubstep.

v0.3.7 also adds Django command shims to make it easier to use Dennis in your Django project.

Use these instructions to set up Dennis so you can use its commands with ./manage.py.

If you aren't using Dennis, yet, it's worth taking a look at. l10n tools are the best!

What is it?

Dennis is a Python command line utility (and library) for working with localization. It includes:

• a translator for strings .po files

• a linter for finding problems in strings in .po files

v0.3.5 released!

0.3.4 fixed an issue with the linter so it skips fuzzy strings.

0.3.5 fixes the rules default for the linter so that it includes the malformed lint rules. It also adds detection of formatting tokens like {0] where it doesn't end in a curly brace. This kicks up a ValueError in Python:

>>> '{0]'.format(1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: unmatched '{' in format
>>>

If you're using Dennis---especially to detect errors in .po files before you push them to production---you should upgrade.

Talk like a pirate day!

Tomorrow, September 19th, is Talk like a Pirate Day. Dennis can help you celebrate with its built-in Pirate translator which works on .po files, but also works on any input from command line arguments or stdin.

Translate your HTML pages:

(cat < "$1" | dennis-cmd translate --pipeline=html,pirate -) > "pirate_$1"

Translate all your git commit messages with this hooks/commit-msg:

#!/bin/bash

# Pipe the contents of the commit message file through dennis to
# a temp file, then copy it back.
(cat < $1 | dennis-cmd translate - > $1.tmp) && mv $1.tmp $1

# We always exit 0 even if the dennis-cmd fails. If the dennis-cmd
# fails, you get your original commit message. No one likes it when
# shenanigans break your stuff for realz.
exit 0;

If you forget about this blog post, these two recipes are in the recipes section of the documentation. If you have other recipes, I'd love to hear about them!

Also, the Pirate! translator can always be improved. If there are improvements you want to make, please submit a pull request!

What is it?

ElasticUtils is a Python library for building and executing Elasticsearch searches.

v0.8 and v0.8.1 released!

I missed the announcement for v0.8, so I'll cover both v0.8 and v0.8.1 here.

Roughly:

• ElasticUtils now requires at least pyelasticsearch 0.6

• adds range query and filter

• adds S.filter_raw

• changes the Indexable.index arguments dropping force_insert and picking up overwrite_existing

For the complete list of what's new, What's new in Version 0.8.1

Many thanks to everyone who helped out: Jannis Leidel, Rob Hudson and Grégoire Vigneron.

If you have any questions, let us know! We hang out on #elasticutils on irc.mozilla.org.

tl;dr

We just landed the bits that switch us from Schematic---the migration system we were using---to South. This is my account of that journey in case it helps others.

Context

Kitsune is the Django project that runs Mozilla Support. The project was started many years ago. For as long as I've worked on Kitsune, we used a migration system called Schematic.

Schematic has the nicety of being very very raw. You can do anything: raw SQL, raw shell, raw Python, raw fish---whatevs. This was nice because we could write whatever we wanted.

Schematic is a total pain in the ass because it hasn't been touched in 2 years, doesn't work well with recent versions of MySQL or MariaDB and makes it really difficult to write migrations that continue to work over time. Further, there's no way to do backwards migrations in Schematic even if you wanted to. We were constantly getting bit by these issues.

The switch

Switching from Schematic to South turned out to be pretty easy. I did it Monday afternoon. I essentially followed these steps:

1. Added South as a dependency for our Django project.

2. Initialized South migrations for all the apps we use in Kitsune which was a whole bunch of:

   $ ./manage.py schemamigration <appname> --initial

3. Wrote a last Schematic migration that adds all the South bookkeeping which entailed dumping the output of this to a file:

   $ mysqldump <database> south_migrationhistory

   and then editing that file by hand.

   That creates the south_migrationhistory table and populates it with the bookkeeping for the initial commits for the apps initialized in step 2.

4. Added ./manage.py migrate to our deploy script.

5. Do a happy dance!

The relevant commits for Kitsune are here:

• fa412aa2 [bug 911831] Add initial South migrations

• 0884893b [bug 911831] Update deployment script with South

• cde04df1 [bug 911831] Update db and migration docs for South

• 609144f2 Quell south logging during tests

That's pretty much it.

Update: September 11, 2013

This was with Django 1.4.7 and South 0.8.2. If you're using different versions, you may experience different things.

Update: September 13, 2013

In Schematic, one thing we would do after creating new models is add a content type and the permissions.

This walks through setting that up automatically with South and post_migrate:

http://devwithpassion.com/felipe/south-django-permissions/

What is it?

Dennis is a Python command line utility (and library) for working with localization. It includes:

• a translator for strings .po files

• a linter for finding problems in strings in .po files

v0.3.3 released!

This is the first blog-post-announced release. I think Dennis is good enough for wider use. I've been using it for development work on both kitsune (which drives Support) and fjord (which drives Input with great success.

Why Dennis?

It fills two basic needs I had:

1. translate .po files so I can find problems during development related to localized strings, layout issues, unicode support, etc

2. lint translated .po files so that errors in translated strings don't make it to production where they cause fires, make users angry and make me very sad and tired

There's another project called Translate Toolkit that you could use for item 1, but it doesn't have Pirate! and I like my pipeline architecture since it's more "pluggable" (whatever that means). Plus it didn't have a linter that covered my specific issues nor does it return a non-zero exit status so I can't use it for selective compiling.

Therefore I decided to write my own tool to meet my needs.

The ultra-basics

$ pip install dennis
$ pip install blessings  # Optional for prettier output

$ dennis-cmd lint locale/fr/LC_MESSAGES/messages.po

(dennis) saturn ~/mozilla/fjord> dennis-cmd lint locale/fr/LC_MES
SAGES/messages.po
dennis-cmd version 0.3.4.dev
>>> Working on: /home/willkg/mozilla/fjord/locale/fr/LC_MESSAGES/
messages.po
Error: mismatched: invalid variables: {count}
msgid: Most Recent Message
msgstr[0]: Les {count} derniers messages

Error: mismatched: invalid variables: {count}
msgid: Most Recent Message
msgid_plural: Last %(count)s Messages
msgstr[1]: Les {count} derniers messages

Warning: mismatched: missing variables: %(count)s
msgid: Most Recent Message
msgid_plural: Last %(count)s Messages
msgstr[1]: Les {count} derniers messages

Error: mismatched: invalid variables: {count}
msgid: {0} similar messages
msgstr: Les {count} derniers messages

Warning: mismatched: missing variables: {0}
msgid: {0} similar messages
msgstr: Les {count} derniers messages

Totals
  Warnings:     2
  Errors:       3

$ dennis-cmd lint --errorsonly locale/

$ dennis-cmd translate --pipeline=html,pirate \
    locale/xx/LC_MESSAGES/messages.po

$ dennis-cmd translate --pipeline=html,pirate,pirate,shouty,anglequote \
    locale/xx/LC_MESSAGES/messages.po

Summary

That's the gist of it. In the Dennis documentation is a list of Dennis recipes covering linting, translating, etc.

Yay for Dennis!