Will >> Will's blog

purpose: Will Kahn-Greene's blog of Miro, PyBlosxom, Python, GNU/Linux, random content, PyBlosxom, Miro, and other projects mixed in there ad hoc, half-baked, and with a twist of lemon
Page 1 of 31  >> (less recent)

Thu, 09 Apr 2015

pyvideo status: April 9th, 2015

What is pyvideo.org

pyvideo.org is an index of Python-related conference and user-group videos on the Internet. Saw a session you liked and want to share it? It's likely you can find it, watch it, and share it with pyvideo.org.

Status

It's been about a year since my last status report. The last year was tough for a bunch of reasons I don't want to go into. On top of that, I was pretty burned out and I had a ton of other stuff I needed to do. So I didn't do much with pyvideo for a long time.

However, we had a lot of help from other people. Sheila Miguez took over adminning the site and added a bunch of conferences. Paul Collins did a ton of work fixing technical debt issues and cleaning up richard codebase. Trey Hunner got richard working with Python 3. Spencer Herzberg, Magnun Leno, Reiner Gerecke, Wes Turner, Benjamin Bertrand, and Burak Guven fixed a bunch of bugs, added new functionality and cleaned up documentation.

I started to get back to things mid-March. I started with the ansible work that Sheila did and tweaked it so we could use it to deploy pyvideo. Then I redid all the infrastructure, updated richard to Django 1.7, fixed a bunch of django-browserid related issues, nixed some code and did some other cleanup. Then I had a few days left before PyCon US 2015 and I decided to throw together a rough playlist implementation.

Playlists are something Sheila and I have talked about for a while. It's great that pyvideo is an index of videos, but until now there's been no good way of collecting a subset of videos you think are interesting to watch later. There's been no good way of curating a group of videos and then sharing that list with other people. Perhaps you want to help people learn Flask. Perhaps you want to share videos about debugging in Python. Perhaps you want to collect videos related to a class you're teaching. Perhaps you want everyone to experience Erik Rose: Man of Mystery.

I landed a rough implementation of playlists today. It's not perfect; it's missing some key things. I wrote up some issues in the richard issue tracker for features that should get implemented to make it really useful. Even without some of those things, it's useful today.

Want to try it out?

Sign in to pyvideo. You'll see a "My playlists" link in the navbar at the top. Go to that page and create a playlist.

Now you can go to any video page on the site and add that video to your playlists.

All playlists are public. You can share the url of your playlists with other people.

Next step is to implement some of the other features listed in the richard issue tracker. If there are other things you want to see or you bump into problems, toss an issue in the tracker.

I hope you find it helpful!

Thu, 19 Feb 2015

Me: 2014 retrospective

This is belated because I've been busy.

2014 was crazy. It had a lot of epically lousy parts, but also had some good parts. Like 2013, I don't know what I did with my 2014 goals.

2013 involved a lot of projects. 2014 involved fewer.

I also spent time on a bunch of other projects:

At work, I spend most of my time on fjord which powers Mozilla Input.

2014 was interesting:

Challenges for the year:

In 2015, I want to:

That's 2014 in a nutshell!

Mon, 05 Jan 2015

ElasticUtils: I'm stepping down and deprecating the project

What is^Wwas it?

ElasticUtils is^Wwas a Python library for building and executing Elasticsearch searches.

See the Quickstart for more details.

I'm taking my ball and going home

For the last few years, I've maintained ElasticUtils. It's a project I picked up after Jeff, Dave and Erik left.

Since I picked it up, we've had 13 or so releases. We switched the underlying library from pyes to pyelasticsearch and then to elasticsearch-py. We added support for a variety of Elasticsearch versions up to about 1.0. We overhauled the documentation. We overhauled the test suite. We generalized it from just a Django library into a library with a Django extension. We added MappingTypes. We worked really hard to make it backwards compatible from version to version. It's been a lot of work, but mostly in spurts.

Since the beginning of the project in May 2011 (a year before I picked up maintenance), we've had 33 people contribute patches. We've had many more point out issues, ask questions on the #elasticutils IRC channel, send me email about this and that.

That's a good run for an open source project. There are a bunch of things I wish I had done differently when I was at the helm, but it's a good run nonetheless.

The current state of things, however, is not great. ElasticUtils has a ton of issues. Lots of technical debt accrued over the years, several architectural decisions that turned out to suck and have obnoxious consequences, lack of support for new features in Elasticsearch > 1.0, etc. It'll take a lot of work to clean that up. Plus it's got a CamelCase name and that's so passé.

At PyCon 2014, Rob, Jannis and I worked with Honza on the API for the library that is now elasticsearch-dsl-py. This library has several of the things I like about ElasticUtils. It's a project that's being supported by the Elasticsearch folks. It's got momentum. It supports many of the Elasticsearch > 1.0 features. There are several libraries that sit on top of elasticsearch-dsl-py as Django shims now.

ElasticUtils is at a point where it's got a lot of problems and there are good alternatives that are better supported.

Thus, this is an excellent time for me to step down as maintainer. Going forward from today, I won't be doing any more development or maintenance.

Further, I'm deprecating the project because no one should be using an unmaintained broken project when there are better supported alternatives. That way lies madness!

So, if you're using ElasticUtils, what do you do now?

ElasticUtils 0.10.2 has the things you need to bridge from Elasticsearch 0.90 to 1.x. You could upgrade to that version and then switch to elasticsearch-dsl-py or one of the Django shims.

On that note, so long ElasticUtils and thank you to everyone who's helped on and used ElasticUtils over the years!

Tue, 16 Dec 2014

Dennis v0.6 released! Line numbers, double vowels, better cli-fu, and better output!

What is it?

Dennis is a Python command line utility (and library) for working with localization. It includes:

  • a linter for finding problems in strings in .po files like invalid Python variable syntax which leads to exceptions
  • a template linter for finding problems in strings in .pot files that make translator's lives difficult
  • a statuser for seeing the high-level translation/error status of your .po files
  • a translator for strings in your .po files to make development easier

v0.6 released!

Since v0.5, I've done the following:

  • Rewrote the command line handling using click and added an exception handler.
  • Merged the lint and linttemplate commands. Why should you care which file you're linting when the linter can figure it out for you?
  • Added the whimsical double vowel transform.
  • Added line numbers in the lint output. This will make it possible to find those pesky problematic strings in your .po/.pot files.
  • Add a line reporter to the linter.

Getting pretty close to what I want for a 1.0, so I'm pretty excited about this version.

Denise update

I've updated Denise with the latest Dennis and moved it to a better url. Lint your .po/.pot files via web service using http://denise.paas.allizom.org/.

Where to go for more

For more specifics on this release, see here: http://dennis.readthedocs.org/en/latest/changelog.html#version-0-6-december-16th-2014

Documentation and quickstart here: http://dennis.readthedocs.org/en/v0.6/

Source code and issue tracker here: https://github.com/willkg/dennis

Source code and issue tracker for Denise (Dennis-as-a-service): https://github.com/willkg/denise

6 out of 8 employees said Dennis helps them complete 1.5 more deliverables per quarter.

Sun, 24 Aug 2014

Dennis v0.5 released! New lint rules, new template linter, bunch of fixes, and now a service!

What is it?

Dennis is a Python command line utility (and library) for working with localization. It includes:

  • a linter for finding problems in strings in .po files like invalid Python variable syntax which leads to exceptions
  • a template linter for finding problems in strings in .pot files that make translator's lives difficult
  • a statuser for seeing the high-level translation/error status of your .po files
  • a translator for strings in your .po files to make development easier

v0.5 released!

Since the last release announcement, there have been a handful of new lint rules added:

  • W301: Translation consists of just white space
  • W302: The translation is the same as the original string
  • W303: There are descrepancies in the HTML between the original string and the translated string

Additionally, there's a new template linter for your .pot files which can catch things like:

  • W500: Strings with variable names like o, O, 0, l, 1 which can be hard to read and are often replaced with a similar looking letter by the translator.
  • W501: One-character variable names which don't give translators enough context about what's being translated.
  • W502: Multiple unnamed variables which can't be reordered because the order the variables are expanded is specified outside of the string.

Dennis in action

Want to see Dennis in action, but don't want to install Dennis? I threw it up as a service, though it's configured for SUMO: http://dennis-sumo.paas.allizom.org/

Note

I may change the URL and I might create a SUMO-agnostic version. If you're interested, let me know.

Where to go for more

For more specifics on this release, see here: http://dennis.readthedocs.org/en/latest/changelog.html#version-0-5-august-24th-2014

Documentation and quickstart here: http://dennis.readthedocs.org/en/v0.5/

Source code and issue tracker here: https://github.com/willkg/dennis

Source code and issue tracker for Denise (Dennis-as-a-service): https://github.com/willkg/denise

3 out of 5 summer interns use Dennis to improve their posture while pranking their mentors.

Wed, 14 May 2014

Fiddling with Kibana

I just kicked off a script that's going to take around 4 hours to complete mostly because the API it's running against doesn't want me doing more than 60 requests/minute. Given I've got like 13k requests to do, that takes a while.

I'm (ab)using Elasticsearch to store the data from my script so that I can analyze it more easily--terms facet is pretty handy here.

Given that I've got some free time now, I spent 5 minutes setting up Kibana.

Steps:

  1. download the tarball
  2. untar it into a directory
  3. edit kibana-3.0.1/config.js to point to my local Elasticsearch cluster (the defaults were fine, so I could have skipped this step)
  4. cd kibana-3.0.1/ and run python -m SimpleHTTPServer 5000 (I'm using a Python-y thing here, but you can use any web-server)
  5. point my browser to http://localhost:5000

Now I'm using Kibana.

Now that I've got it working, first thing I do is click on the cog in the upper right hand corner, click on the Index tab and change the index to the one I wanted to look at. Now I'm looking at the data my script is producing.

The Kibana site says Kibana excels at timestamped data, but I think it's helpful for what I'm looking at now despite it not being timestamped. I get immediate terms facets on the fields for the doc type I'm looking at. I can run queries, pick specific columns, reorder, do graphs, save my dashboard to look at later, etc.

If you're doing Elasticsearch stuff, it's worth looking at if only to give you another tool to look at data with.

Thu, 01 May 2014

Dennis v0.4 released! Tweaks to Python 3 support, overhauled linter, string-by-string lint rules ignoring

What is it?

Dennis is a Python command line utility (and library) for working with localization. It includes:

  • a linter for finding problems in strings in .po files like invalid Python variable syntax which leads to exceptions
  • a statuser for seeing the high-level translation/error status of your .po files
  • a translator for strings in your .po files to make development easier

v0.4 released!

v0.4 sports an overhauled linter. Instead of two rules ("malformed" and whatever the other one was), it now has a bunch of much smaller and more specific rules! Also, I renamed the rules so they are all numbered!

See the table of error/warning rules and their numbers here: http://dennis.readthedocs.org/en/v0.4/linting.html#warnings-and-errors

Additionally, dennis hits false positives for a variety of reasons. If you're doing a "keep the errors out of production!" kind of thing, then false positives can prevent locale files from making it. That sucks!

To alleviate this, dennis now allows you to tell it what to ignore in the extracted comments. What's an extracted comment? It's a comment in the .po file that starts with #.. You can specify the extracted comments with "context" or similar mechanisms depending on how you're extracting strings. You can tell dennis to skip specific rules or skip all the rules on a string-by-string basis.

Ignore everything:

#. dennis-ignore: *
msgid "German makes up 10% of our visitor base"
msgstr "A német a látogatóbázisunk 10%-át teszi ki"

Ignore specific rules (comma-separated):

#. dennis-ignore: E101,E102,E103
msgid "German makes up 10% of our visitor base"
msgstr "A német a látogatóbázisunk 10%-át teszi ki"

Ignore everything, but note the beginning of the line is ignored by dennis so you can tell localizers to ignore the ignore thing:

#. localizers--ignore this comment. dennis-ignore: *
msgid "German makes up 10% of our visitor base"
msgstr "A német a látogatóbázisunk 10%-át teszi ki"

I also tweaked some of the Python 3 support code because it looked at me funny.

Also, universal wheel!

For more specifics on this release, see here: http://dennis.readthedocs.org/en/v0.4/changelog.html#version-0-4-may-1st-2014

Documentation and quickstart here: http://dennis.readthedocs.org/en/v0.4/

Source code and issue tracker here: https://github.com/willkg/dennis

2 out of 10 people saw the Pirate translation on The Web We Want (Mozilla). Arrr!

Mon, 21 Apr 2014

Dennis v0.3.11 released! Fixes and Python 3 support

What is it?

Dennis is a Python command line utility (and library) for working with localization. It includes:

  • a linter for finding problems in strings in .po files like invalid Python variable syntax which leads to exceptions
  • a statuser for seeing the high-level translation/error status of your .po files
  • a translator for strings in your .po files to make development easier

v0.3.11 released!

v0.3.11 adds Python 3 support (there might be problems, but it's working for me) and adds error detection for the case where there's a } but no {.

Definitely worth updating!

8 out of 11 people who have heard of Dennis and continue to ignore its baby mews of wonderfulness also have a severe allergy to rainbows and kittens.

Fri, 18 Apr 2014

Django Eadred v0.3 released! Django app for generating sample data.

Django Eadred gives you some scaffolding for generating sample data to make it easier for new contributors to get up and running quickly, bootstrapping required database data, and generating large amounts of random data for testing graphs and things like that.

The v0.3 release is a small one, but good:

There are no backwards-compatability problems with previous versions.

To update, do:

pip install -U eadred

Wed, 09 Apr 2014

pyvideo status: April 9th, 2014

What is pyvideo.org

pyvideo.org is an index of Python-related conference and user-group videos on the Internet. Saw a session you liked and want to share it? It's likely you can find it, watch it, and share it with pyvideo.org.

Status

I fixed a few issues and finally (finally) pushed out major site updates. Some of them are implemented in the worst possible way (e.g. facet filters for the search page), but some of them are great (e.g. Amara subtitle support).

I'm still struggling with a lot of technical debt on the site and a lack of time to really focus on it. That's mostly what's been making fixing the issues, improving the site and adding conferences take so long.

Sheila and I will be at PyCon US and hanging around for sprint days. If anyone is interested in sprinting, we'll be there. Even if we don't get any coding done, figuring out how to solve some of the bigger problems and planning what should be done in the next year would be a huge accomplishment.

If you're at PyCon and see either of us, feel free to give us a piece of your mind in regards to how you use PyVideo and what could be better.