Will >> Will's blog

purpose: Will Kahn-Greene's blog of Miro, PyBlosxom, Python, GNU/Linux, random content, PyBlosxom, Miro, and other projects mixed in there ad hoc, half-baked, and with a twist of lemon
Page 1 of 24  >> (less recent)

Wed, 14 May 2014

Fiddling with Kibana

I just kicked off a script that's going to take around 4 hours to complete mostly because the API it's running against doesn't want me doing more than 60 requests/minute. Given I've got like 13k requests to do, that takes a while.

I'm (ab)using Elasticsearch to store the data from my script so that I can analyze it more easily--terms facet is pretty handy here.

Given that I've got some free time now, I spent 5 minutes setting up Kibana.

Steps:

  1. download the tarball
  2. untar it into a directory
  3. edit kibana-3.0.1/config.js to point to my local Elasticsearch cluster (the defaults were fine, so I could have skipped this step)
  4. cd kibana-3.0.1/ and run python -m SimpleHTTPServer 5000 (I'm using a Python-y thing here, but you can use any web-server)
  5. point my browser to http://localhost:5000

Now I'm using Kibana.

Now that I've got it working, first thing I do is click on the cog in the upper right hand corner, click on the Index tab and change the index to the one I wanted to look at. Now I'm looking at the data my script is producing.

The Kibana site says Kibana excels at timestamped data, but I think it's helpful for what I'm looking at now despite it not being timestamped. I get immediate terms facets on the fields for the doc type I'm looking at. I can run queries, pick specific columns, reorder, do graphs, save my dashboard to look at later, etc.

If you're doing Elasticsearch stuff, it's worth looking at if only to give you another tool to look at data with.

Thu, 01 May 2014

Dennis v0.4 released! Tweaks to Python 3 support, overhauled linter, string-by-string lint rules ignoring

What is it?

Dennis is a Python command line utility (and library) for working with localization. It includes:

  • a linter for finding problems in strings in .po files like invalid Python variable syntax which leads to exceptions
  • a statuser for seeing the high-level translation/error status of your .po files
  • a translator for strings in your .po files to make development easier

v0.4 released!

v0.4 sports an overhauled linter. Instead of two rules ("malformed" and whatever the other one was), it now has a bunch of much smaller and more specific rules! Also, I renamed the rules so they are all numbered!

See the table of error/warning rules and their numbers here: http://dennis.readthedocs.org/en/v0.4/linting.html#warnings-and-errors

Additionally, dennis hits false positives for a variety of reasons. If you're doing a "keep the errors out of production!" kind of thing, then false positives can prevent locale files from making it. That sucks!

To alleviate this, dennis now allows you to tell it what to ignore in the extracted comments. What's an extracted comment? It's a comment in the .po file that starts with #.. You can specify the extracted comments with "context" or similar mechanisms depending on how you're extracting strings. You can tell dennis to skip specific rules or skip all the rules on a string-by-string basis.

Ignore everything:

#. dennis-ignore: *
msgid "German makes up 10% of our visitor base"
msgstr "A német a látogatóbázisunk 10%-át teszi ki"

Ignore specific rules (comma-separated):

#. dennis-ignore: E101,E102,E103
msgid "German makes up 10% of our visitor base"
msgstr "A német a látogatóbázisunk 10%-át teszi ki"

Ignore everything, but note the beginning of the line is ignored by dennis so you can tell localizers to ignore the ignore thing:

#. localizers--ignore this comment. dennis-ignore: *
msgid "German makes up 10% of our visitor base"
msgstr "A német a látogatóbázisunk 10%-át teszi ki"

I also tweaked some of the Python 3 support code because it looked at me funny.

Also, universal wheel!

For more specifics on this release, see here: http://dennis.readthedocs.org/en/v0.4/changelog.html#version-0-4-may-1st-2014

Documentation and quickstart here: http://dennis.readthedocs.org/en/v0.4/

Source code and issue tracker here: https://github.com/willkg/dennis

2 out of 10 people saw the Pirate translation on The Web We Want (Mozilla). Arrr!

Mon, 21 Apr 2014

Dennis v0.3.11 released! Fixes and Python 3 support

What is it?

Dennis is a Python command line utility (and library) for working with localization. It includes:

  • a linter for finding problems in strings in .po files like invalid Python variable syntax which leads to exceptions
  • a statuser for seeing the high-level translation/error status of your .po files
  • a translator for strings in your .po files to make development easier

v0.3.11 released!

v0.3.11 adds Python 3 support (there might be problems, but it's working for me) and adds error detection for the case where there's a } but no {.

Definitely worth updating!

8 out of 11 people who have heard of Dennis and continue to ignore its baby mews of wonderfulness also have a severe allergy to rainbows and kittens.

Fri, 18 Apr 2014

Django Eadred v0.3 released! Django app for generating sample data.

Django Eadred gives you some scaffolding for generating sample data to make it easier for new contributors to get up and running quickly, bootstrapping required database data, and generating large amounts of random data for testing graphs and things like that.

The v0.3 release is a small one, but good:

There are no backwards-compatability problems with previous versions.

To update, do:

pip install -U eadred

Wed, 09 Apr 2014

Site development using pagekite

Problem

I have this basic problem where I do a lot of web-site work and I need to show people what I've done so far so they can review it and help me make it better or make it suit their needs better. Screenshots aren't very helpful because the site is interactive. Further, the site needs to get tested on multiple devices/platforms/browsers. Also, I need to make sure that the site is only accessed via https.

What I've been doing up to now is failing miserably: I'd push work to our staging server for people to test out, but that sucks as an answer and affects my co-workers and makes a mess of our staging server. Plus iterating on things is difficult.

So, requirements:

  1. endpoint must be https-only
  2. must be easy to set up and take down
  3. must be easy to access so people can easily test things on my local machine

Solution

I looked around and this would be pretty easy to do if I didn't have the https-only requirement. That makes things difficult without a lot of work.

Then I found pagekite. They make it really easy.

Here's how you set it up:

  1. Download and install the pagekite software: http://pagekite.net/downloads/

  2. Run your website. In my case, I'm working on Django sites, so I launch like this:

    $ ./manage.py runserver
    

    That runs the Django project I'm working on on localhost:8000.

  3. Run pagekite:

    $ pagekite.py 8000 YOUR_NAME.pagekite.me:443
    

    That creates a tunnel from your machine to the pagekite.me server. When someone accesses https://YOUR_NAME.pagekite.me/, the request goes through the tunnel to your pagekite backend and that performs the request over http to your local webserver (in my case, the Django project) bound to localhost:8000.

    Access is https-only. If anyone tries to access http://YOUR_NAME.pagekite.me/, then they get a connection error.

    The https-only requirement is satisfied by restricting the kite to only listening to port 443--the https port. That's pretty key.

This lets me run my Django project locally on http without dealing with self-signed certificates, but still require https access so data isn't floating around in clear text.

The one problem with this is that my local server thinks it's running http and so redirects that include the protocol go to http rather than https.

If you don't already have an account, I'm pretty sure step 3 will walk you through setting one up. Free accounts are limited in what they can do.

Also, they hang out on #pagekite on Freenode. I had a problem, asked a question and got a super helpful reply. The code is Open Source, so it's possible to look through it and debug it.

I'll be using this going forward.

Why write this?

This is a common use case for web developers. I figured I'd write this up because the https-only part is pretty key and it was the part that I had to ask for help with.

Thu, 03 Apr 2014

ElasticUtils v0.9 released!

What is it?

ElasticUtils is a Python library for building and executing Elasticsearch searches.

See the Quickstart for more details.

v0.9 released!

This is a big release, but there are some compromises in it that I'm not wildly excited about. Things like Elasticsearch 1.0 support didn't make the cut. I'm really sorry about that---we're working on it.

This release has a lot of changes in it. Roughly:

  • dropped pyelasticsearch for elasticsearch-py (Thank you Honza!)
  • fixed S.all() so it does what Django does which should let you use an S in the place of a QuerySet in some cases
  • new FacetResult class (Thank you James!)
  • S.facet() can take a size keyword
  • cleaned up ESTestCase
  • SearchResults now has facet data in the facets property
  • etc.

For the complete list of what's new, What's new in Version 0.9.

Many thanks to everyone who helped out: Alexey Kotlyarov, David Lundgren, Honza Král, James Reynolds, Jannis Leidel, Juan Ignacio Catalano, Kevin Stone, Mathieu Pillard, Mihnea Dobrescu-Balaur, nearlyfreeapps, Ricky Cook, Rob Hudson, William Tisäter and Will Kahn-Greene.

We're going to be sprinting on ElasticUtils 0.10 at PyCon US in Montreal mid April. If you're interested, come find me!

If you have any questions, let us know! We hang out on #elasticutils on irc.mozilla.org.

Fri, 25 Oct 2013

Dennis v0.3.10 released! Fixes, status subcommand and Zombie!

What is it?

Dennis is a Python command line utility (and library) for working with localization. It includes:

  • a linter for finding problems in strings in .po files
  • a statuser for seeing the high-level status of your .po files
  • a translator for strings .po files

v0.3.10 released!

v0.3.8 fixed mismatched errors in plural strings. Thanks Mike!

v0.3.9 fixed two false positives in error detection.

v0.3.10 adds the status subcommand and the Zombie transform which, like the dubstep transform, is silly but fun.

http://bluesock.org/~willkg/blog/static/images/thumb_sumo_zombie1.png

45 out of 47 Djangonauts use the Zombie transform to make their site accessible to those who have departed. This could open up your app to millions of new users. Truth.

Tue, 15 Oct 2013

Dennis v0.3.7 released! Dubstep and Django!

What is it?

Dennis is a Python command line utility (and library) for working with localization. It includes:

  • a linter for finding problems in strings in .po files
  • a translator for strings .po files

v0.3.7 released!

v0.3.6 fixed a goof where the linter was skipping errors. Oops.

v0.3.7 adds a dubstep translator (which is just plain silly, but awesome).

http://bluesock.org/~willkg/blog/static/images/thumb_sumo_dubstep1.png

Truth: 9 out of 10 experts agree SUMO is extra helpful in dubstep.

v0.3.7 also adds Django command shims to make it easier to use Dennis in your Django project.

Use these instructions to set up Dennis so you can use its commands with ./manage.py.

If you aren't using Dennis, yet, it's worth taking a look at. l10n tools are the best!

Wed, 18 Sep 2013

Dennis v0.3.5 released!

What is it?

Dennis is a Python command line utility (and library) for working with localization. It includes:

  • a translator for strings .po files
  • a linter for finding problems in strings in .po files

v0.3.5 released!

0.3.4 fixed an issue with the linter so it skips fuzzy strings.

0.3.5 fixes the rules default for the linter so that it includes the malformed lint rules. It also adds detection of formatting tokens like {0] where it doesn't end in a curly brace. This kicks up a ValueError in Python:

>>> '{0]'.format(1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: unmatched '{' in format
>>>

If you're using Dennis---especially to detect errors in .po files before you push them to production---you should upgrade.

Talk like a pirate day!

Tomorrow, September 19th, is Talk like a Pirate Day. Dennis can help you celebrate with its built-in Pirate translator which works on .po files, but also works on any input from command line arguments or stdin.

Translate your HTML pages:

(cat < "$1" | dennis-cmd translate --pipeline=html,pirate -) > "pirate_$1"

Translate all your git commit messages with this hooks/commit-msg:

#!/bin/bash

# Pipe the contents of the commit message file through dennis to
# a temp file, then copy it back.
(cat < $1 | dennis-cmd translate - > $1.tmp) && mv $1.tmp $1

# We always exit 0 even if the dennis-cmd fails. If the dennis-cmd
# fails, you get your original commit message. No one likes it when
# shenanigans break your stuff for realz.
exit 0;

If you forget about this blog post, these two recipes are in the recipes section of the documentation. If you have other recipes, I'd love to hear about them!

Also, the Pirate! translator can always be improved. If there are improvements you want to make, please submit a pull request!

Fri, 13 Sep 2013

ElasticUtils v0.8 and v0.8.1 released!

What is it?

ElasticUtils is a Python library for building and executing Elasticsearch searches.

v0.8 and v0.8.1 released!

I missed the announcement for v0.8, so I'll cover both v0.8 and v0.8.1 here.

Roughly:

  • ElasticUtils now requires at least pyelasticsearch 0.6
  • adds range query and filter
  • adds S.filter_raw
  • changes the Indexable.index arguments dropping force_insert and picking up overwrite_existing

For the complete list of what's new, What's new in Version 0.8.1

Many thanks to everyone who helped out: Jannis Leidel, Rob Hudson and Grégoire Vigneron.

If you have any questions, let us know! We hang out on #elasticutils on irc.mozilla.org.