Will >> Will's blog

purpose: Will Kahn-Greene's blog of Miro, PyBlosxom, Python, GNU/Linux, random content, PyBlosxom, Miro, and other projects mixed in there ad hoc, half-baked, and with a twist of lemon
Page 1 of 16  >> (less recent)

Wed, 09 Apr 2014

pyvideo status: April 9th, 2014

What is pyvideo.org

pyvideo.org is an index of Python-related conference and user-group videos on the Internet. Saw a session you liked and want to share it? It's likely you can find it, watch it, and share it with pyvideo.org.

Status

I fixed a few issues and finally (finally) pushed out major site updates. Some of them are implemented in the worst possible way (e.g. facet filters for the search page), but some of them are great (e.g. Amara subtitle support).

I'm still struggling with a lot of technical debt on the site and a lack of time to really focus on it. That's mostly what's been making fixing the issues, improving the site and adding conferences take so long.

Sheila and I will be at PyCon US and hanging around for sprint days. If anyone is interested in sprinting, we'll be there. Even if we don't get any coding done, figuring out how to solve some of the bigger problems and planning what should be done in the next year would be a huge accomplishment.

If you're at PyCon and see either of us, feel free to give us a piece of your mind in regards to how you use PyVideo and what could be better.

Sat, 15 Feb 2014

pyvideo status: February 15th, 2014

What is pyvideo.org

pyvideo.org is an index of Python-related conference and user-group videos on the Internet. Saw a session you liked and want to share it? It's likely you can find it, watch it, and share it with pyvideo.org.

Status

Over the last year, a number of things have led to a tangled mess of tasks that need to be done that were blocked on other tasks that were complicated by the fact that I had half-done a bunch of things. I've been chipping away at various aspects of things, but most of them were blocked on me finishing infrastructure changes I started in November when we moved everything to Rackspace.

I finally got my local pyvideo environment working and a staging environment working. I finally sorted out my postgres issues, so I've got backups and restores working (yes--I test restores). I finally fixed all the problems with my deploy script so I can deploy when I want to and can do it reliably.

Now that I've got all that working, I pushed changes to the footer recognizing that Sheila and I are co-adminning (and have been for some time) and that Rackspace is graciously hosting pyvideo.

In the queue of things to do:

  • finish up some changes to richard and then update pyvideo to the latest richard
  • re-encode all the .flv files I have from blip.tv into something more HTML5-palatable (I could use help with this--my encoding-fu sucks)
  • fix other blip.tv metadata fallout--for example most of the PyGotham videos have terrible metadata (my fault)
  • continue working on process and tools to make pyvideo easier to contribute to

That about covers it for this status report.

Questions, comments, thoughts, etc--send me email or twart me at @PyvideoOrg or or @willcage.

Fri, 20 Dec 2013

Me: 2013 retrospective

I had a pretty intense 2013, but overall I think it was a good year. I say that with full knowledge that I had 2013 goals, but I have no idea where I wrote them down. Oh, well.

First off, I started or helped start a lot of new projects this year:

Fredrik is obsolete. Victor was a failed project. Captain Shove and Hy are doing well and other than helping to start them off, I haven't touched either in some time. I work on Dennis, Douglas and Ernest regularly. I'm thinking about Harold, but probably won't do anything with it for a while.

I also spent a ton of time working on existing projects:

I'm probably missing a bunch there. It's hard to keep track of what one did on other peoples' projects.

I got a lot accomplished at work this year:

Challenges I worked on this year:

In 2014, I want to:

It's been a busy year and there are things I should be doing better, but generally speaking, I think I did pretty ok.

Sun, 24 Nov 2013

pyvideo status: November 24th, 2013

What is pyvideo.org

pyvideo.org is an index of Python-related conference and user-group videos on the Internet. Saw a session you liked and want to share it? It's likely you can find it, watch it, and share it with pyvideo.org.

Status

Lot of stuff has happened since the last status report, but there are four things of note:

  1. Sheila is now a co-admin of pyvideo.org. She has been for a couple of months. I need to update the site to reflect this.

    I'm really psyched about this. It's a ton of work and I'm just not managing it well. Splitting the work should make it more manageable.

  2. Back in July, Sheila poked me about a tweet Jesse wrote suggesting Rackspace was interested in sponsoring Open Source projects. She contacted Jesse and set everything up.

    I'm psyched that Rackspace agreed to sponsor pyvideo.org by providing free hosting. Several months later, I moved pyvideo.org from where it was before to a vm at Rackspace.

    I'm really excited about this! It makes a bunch of problems that I was trying to figure out what to do about go away.

    Thank you, Rackspace!

    I need to update the site to reflect this.

  3. Sheila discovered that blip.tv was expiring a bunch of accounts that held conference videos and that those videos would go away. She and I scrambled to download all the files from blip and move them to Rackspace cloudfiles. It's about 600 videos and around 250gb of data.

    In the process of doing that, we saved videos for DjangoCon EU 2010, DjangoCon EU 2011 and PyGotham 2012. I added these to pyvideo.org today. These videos have pages that are stubs with no metadata. I've got that in my queue of things to fix.

    Also, the thumbnails for all the videos on blip.tv are on my laptop which isn't very helpful. I need to move those and update the videos in pyvideo.org.

    As a side note, if we didn't have hosting from Rackspace, we'd have been totally screwed. Thank you, Jesse Noller and Rackspace!

  4. I've been working on the richard codebase fixing architectural problems, reducing the complexities and trying to clean it up so it's in a better state. That work is almost done. When it is, I'll update pyvideo.org with the new site. At this rate, I think I can finish the work this year, but that assumes there aren't any more emergencies.

  5. I've been thinking about how to build a better communication channel for pyvideo.org so people can more easily follow what's going on so they can act on things they're interested in.

    pyvideo.org has a "site news" section. It's a pain in the ass to use and it's not syndicated anywhere and it's likely no one sees it.

    Blogging status reports like this on my blog is better, but I don't think my blog is very widely read. Making my blog more widely-read seems like a lot of work and I'm not sure I can do it effectively anyhow.

    So I've decided to ditch the "site news" section of pyvideo.org and switch to Twitter. I started a @PyvideoOrg account.

    I'll tweet site updates, calls for help and newly posted conferences. I'm tossing around tweeting new videos when they get posted, but videos tend to get posted in huge batches and getting > 40 tweets all at once is a total drag. I'll have to think about that some more.

    Follow @PyvideoOrg if you're interested! Also, feel free to tweet at that account.

    I need to update the site to reflect this.

Also, in my life things are pretty crazy. I have a new kid and juggling everything was impossible for a while. I think that should easy up now and I can spend more time on pyvideo.org going forward.

That's the state of things!

Also, thank you thank you thank you thank you Rackspace!

Fri, 25 Oct 2013

Dennis v0.3.10 released! Fixes, status subcommand and Zombie!

What is it?

Dennis is a Python command line utility (and library) for working with localization. It includes:

  • a linter for finding problems in strings in .po files
  • a statuser for seeing the high-level status of your .po files
  • a translator for strings .po files

v0.3.10 released!

v0.3.8 fixed mismatched errors in plural strings. Thanks Mike!

v0.3.9 fixed two false positives in error detection.

v0.3.10 adds the status subcommand and the Zombie transform which, like the dubstep transform, is silly but fun.

http://bluesock.org/~willkg/blog/static/images/thumb_sumo_zombie1.png

45 out of 47 Djangonauts use the Zombie transform to make their site accessible to those who have departed. This could open up your app to millions of new users. Truth.

Tue, 15 Oct 2013

Dennis v0.3.7 released! Dubstep and Django!

What is it?

Dennis is a Python command line utility (and library) for working with localization. It includes:

  • a linter for finding problems in strings in .po files
  • a translator for strings .po files

v0.3.7 released!

v0.3.6 fixed a goof where the linter was skipping errors. Oops.

v0.3.7 adds a dubstep translator (which is just plain silly, but awesome).

http://bluesock.org/~willkg/blog/static/images/thumb_sumo_dubstep1.png

Truth: 9 out of 10 experts agree SUMO is extra helpful in dubstep.

v0.3.7 also adds Django command shims to make it easier to use Dennis in your Django project.

Use these instructions to set up Dennis so you can use its commands with ./manage.py.

If you aren't using Dennis, yet, it's worth taking a look at. l10n tools are the best!

Wed, 18 Sep 2013

Dennis v0.3.5 released!

What is it?

Dennis is a Python command line utility (and library) for working with localization. It includes:

  • a translator for strings .po files
  • a linter for finding problems in strings in .po files

v0.3.5 released!

0.3.4 fixed an issue with the linter so it skips fuzzy strings.

0.3.5 fixes the rules default for the linter so that it includes the malformed lint rules. It also adds detection of formatting tokens like {0] where it doesn't end in a curly brace. This kicks up a ValueError in Python:

>>> '{0]'.format(1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: unmatched '{' in format
>>>

If you're using Dennis---especially to detect errors in .po files before you push them to production---you should upgrade.

Talk like a pirate day!

Tomorrow, September 19th, is Talk like a Pirate Day. Dennis can help you celebrate with its built-in Pirate translator which works on .po files, but also works on any input from command line arguments or stdin.

Translate your HTML pages:

(cat < "$1" | dennis-cmd translate --pipeline=html,pirate -) > "pirate_$1"

Translate all your git commit messages with this hooks/commit-msg:

#!/bin/bash

# Pipe the contents of the commit message file through dennis to
# a temp file, then copy it back.
(cat < $1 | dennis-cmd translate - > $1.tmp) && mv $1.tmp $1

# We always exit 0 even if the dennis-cmd fails. If the dennis-cmd
# fails, you get your original commit message. No one likes it when
# shenanigans break your stuff for realz.
exit 0;

If you forget about this blog post, these two recipes are in the recipes section of the documentation. If you have other recipes, I'd love to hear about them!

Also, the Pirate! translator can always be improved. If there are improvements you want to make, please submit a pull request!

Wed, 11 Sep 2013

Switching to South

tl;dr

We just landed the bits that switch us from Schematic---the migration system we were using---to South. This is my account of that journey in case it helps others.

Context

Kitsune is the Django project that runs Mozilla Support. The project was started many years ago. For as long as I've worked on Kitsune, we used a migration system called Schematic.

Schematic has the nicety of being very very raw. You can do anything: raw SQL, raw shell, raw Python, raw fish---whatevs. This was nice because we could write whatever we wanted.

Schematic is a total pain in the ass because it hasn't been touched in 2 years, doesn't work well with recent versions of MySQL or MariaDB and makes it really difficult to write migrations that continue to work over time. Further, there's no way to do backwards migrations in Schematic even if you wanted to. We were constantly getting bit by these issues.

The switch

Switching from Schematic to South turned out to be pretty easy. I did it Monday afternoon. I essentially followed these steps:

  1. Added South as a dependency for our Django project.

  2. Initialized South migrations for all the apps we use in Kitsune which was a whole bunch of:

    $ ./manage.py schemamigration <appname> --initial
    
  3. Wrote a last Schematic migration that adds all the South bookkeeping which entailed dumping the output of this to a file:

    $ mysqldump <database> south_migrationhistory
    

    and then editing that file by hand.

    That creates the south_migrationhistory table and populates it with the bookkeeping for the initial commits for the apps initialized in step 2.

  4. Added ./manage.py migrate to our deploy script.

  5. Do a happy dance!

The relevant commits for Kitsune are here:

That's pretty much it.

Update: September 11, 2013
This was with Django 1.4.7 and South 0.8.2. If you're using different versions, you may experience different things.
Update: September 13, 2013

In Schematic, one thing we would do after creating new models is add a content type and the permissions.

This walks through setting that up automatically with South and post_migrate:

http://devwithpassion.com/felipe/south-django-permissions/

Mon, 29 Jul 2013

Dennis v0.3.3 released!

What is it?

Dennis is a Python command line utility (and library) for working with localization. It includes:

  • a translator for strings .po files
  • a linter for finding problems in strings in .po files

v0.3.3 released!

This is the first blog-post-announced release. I think Dennis is good enough for wider use. I've been using it for development work on both kitsune (which drives Support) and fjord (which drives Input with great success.

Why Dennis?

It fills two basic needs I had:

  1. translate .po files so I can find problems during development related to localized strings, layout issues, unicode support, etc
  2. lint translated .po files so that errors in translated strings don't make it to production where they cause fires, make users angry and make me very sad and tired

There's another project called Translate Toolkit that you could use for item 1, but it doesn't have Pirate! and I like my pipeline architecture since it's more "pluggable" (whatever that means). Plus it didn't have a linter that covered my specific issues nor does it return a non-zero exit status so I can't use it for selective compiling.

Therefore I decided to write my own tool to meet my needs.

The ultra-basics

Install
$ pip install dennis
$ pip install blessings  # Optional for prettier output
Linting

Lint a single .po file for problems including mismatched/malformed Python variables in translated strings:

$ dennis-cmd lint locale/fr/LC_MESSAGES/messages.po

Produces output like this:

(dennis) saturn ~/mozilla/fjord> dennis-cmd lint locale/fr/LC_MES
SAGES/messages.po
dennis-cmd version 0.3.4.dev
>>> Working on: /home/willkg/mozilla/fjord/locale/fr/LC_MESSAGES/
messages.po
Error: mismatched: invalid variables: {count}
msgid: Most Recent Message
msgstr[0]: Les {count} derniers messages

Error: mismatched: invalid variables: {count}
msgid: Most Recent Message
msgid_plural: Last %(count)s Messages
msgstr[1]: Les {count} derniers messages

Warning: mismatched: missing variables: %(count)s
msgid: Most Recent Message
msgid_plural: Last %(count)s Messages
msgstr[1]: Les {count} derniers messages

Error: mismatched: invalid variables: {count}
msgid: {0} similar messages
msgstr: Les {count} derniers messages

Warning: mismatched: missing variables: {0}
msgid: {0} similar messages
msgstr: Les {count} derniers messages

Totals
  Warnings:     2
  Errors:       3

If you have blessings installed, it'll colorize that output.

You can also lint a directory structure of .po files:

$ dennis-cmd lint --errorsonly locale/

I use this to compile only the error-free .po files to .mo files and tell us which .po files have problems so we can fix them.

Translating

You can translate a .po file in place into Pirate! to help find l10n issues in your code:

$ dennis-cmd translate --pipeline=html,pirate \
    locale/xx/LC_MESSAGES/messages.po

This takes into account that the strings have HTML in them that should be ignored when translating. It uses a pipeline architecture where the output of one transform is fed as input to the next, so you can string them along and get shouty extra-pirate with anglequotes:

$ dennis-cmd translate --pipeline=html,pirate,pirate,shouty,anglequote \
    locale/xx/LC_MESSAGES/messages.po

Summary

That's the gist of it. In the Dennis documentation is a list of Dennis recipes covering linting, translating, etc.

Yay for Dennis!

Wed, 03 Apr 2013

pyvideo status: April 3rd, 2013

What is pyvideo.org

pyvideo.org is an index of Python-related conference and user-group videos on the Internet. Saw a session you liked and want to share it? It's likely you can find it, watch it, and share it with pyvideo.org.

Status

  • Videos for PyCon US 2013 are still going up. There are 115 posted and live now. There are around 30 that are waiting for presenters to look at the metadata and tell Carl whether the metadata is good or not. More on that later.

  • Several new people submitted patches to richard! Several of the patches were fixes to broken things they saw on pyvideo.org. I've applied the fixes to the site directly, but have been waiting on making any non-critical updates to the site until after things have cooled off. I think I'll do a site update in the next week or so.

  • PyData 2013 was recorded. When videos are posted, they'll be in the PyData category. I don't know what the posting schedule is.

  • I was contacted a couple of times by the inimitable Montréal Python to post their videos. They're going to test out steve which is the tool I've been writing for the last 6 months to make it possible for other folks to generate the video metadata needed by pyvideo.org.

    I eagerly look forward to their progress and to their videos getting on the site.

    If it works out well, I'll blog more about steve and look for volunteers to use steve to generate the video metadata for the ever increasing backlog.

  • Several people are gittip'ing me. It's not a lot of money, but that and the many emails I've gotten over the last few weeks about the site have been really great. I work on pyvideo.org in my free time of which I don't have a lot. It's nice to know that prioritizing pyvideo.org work over other things helps you.

That's the gist of things!

Most of the PyCon US 2013 videos that aren't live are waiting for presenters to tell Carl at NextDayVideo (carl at nextdayvideo dot com) whether the metadata is good.

  • If you see your name on this list and you've told Carl the metadata is fine already, please send him a friendly reminder.
  • If you see your name on this list and you haven't told Carl anything, please send him a "yes, this is great!" or the list of things you need corrected.
  • If you see a friend on this list, tell your friend to do one of the above.

I'll update this list as I'm aware of changes. However, I don't work for NextDayVideo, so it's entirely possible my list is not current and/or there are errors. If so, please let me know.

Here's the list (last updated 2013-04-12 7:13am -0400):

  • Digital signal processing through speech, hearing, and Python -- Mel Chua
  • Faster Python Programs through Optimization -- Mike Müller
  • Python beyond the CPU -- Andy Terrel, Travis Oliphant, Mark Florisson
  • Code to Cloud in under 45 minutes -- John Wetherill
  • A Gentle Introduction to Computer Vision -- Katherine Scott, Anthony Oliver
  • Documenting Your Project in Sphinx -- Brandon Rhodes
  • Contribute with me! Getting started with open source development -- Jessica McKellar
  • Intermediate Twisted: Test-Driven Networking Software -- Itamar Turner-Trauring
  • Gittip: Inspiring Generosity -- Chad Whitacre
  • The Magic of Metaprogramming -- Jeff Rush
  • You can be a speaker at PyCon! -- Anna Ravenscroft
  • sys._current_frames(): Take real-time x-rays of your software for fun and performance -- Leonardo Rochael
  • Planning and Tending the Garden: The Future of Early Childhood Python Education -- Kurt Grandis
  • powerful pyramid features -- Carlos de la Guardia
  • Python for Robotics and Hardware Control -- Jonathan Foote
  • Copyright and You -- Frank Siler
  • Chef: Automating web application infrastructure -- Kate Heddleston
  • Numba: A Dynamic Python compiler for Science -- Travis Oliphant, Siu Kwan Lam, Mark Florisson
  • Integrating Jython with Java -- Jim Baker, Shashank Bharadwaj
  • Iteration & Generators: the Python Way -- Luciano Ramalho
  • ApplePy: An Apple ][ emulator in Python -- James Tauber
  • Distributed Coordination with Python -- Ben Bangert
  • Become a logging expert in 30 minutes -- Gavin M. Roy
  • PyNES: Python programming for Nintendo 8 bits -- Guto Maia
  • Purely Python Imaging with Pymaging -- Jonas Obrist
  • Namespaces in Python -- Eric Snow

These are all set now:

  • IPython in-depth: high-productivity interactive and parallel python -- Fernando Perez, Brian Granger, Min RK
  • Pyramid for Humans -- Paul Everitt
  • Learn Python Through Public Data Hacking -- David Beazley
  • Rethinking Errors: Learning from Scala and Go -- Bruce Eckel