pyvideo status: June 21st, 2012

What is pyvideo.org

pyvideo.org is an index of Python-related videos on the Internet. For the most part, it's a collection of videos from Python-related conferences. Saw a session you liked and want to share it? It's likely you can find it, watch it, and share it with pyvideo.org.

Status

It's been a few months since my last status report. Two things have happened and one thing hasn't.

  1. richard development progresses and it's coming along nicely. There are a few more things I really want before I think it's got a critical mass. Then I think we should start doing releases.

  2. steve development started and I'm adding functionality as I need it for adding conferences in the queue.

    Yes! We have a conferences-to-add queue in pyvideo now. And yes, the queue is long. steve makes it possible to add a conference of videos without doing them one-by-one.

    Also, if you have a conference you don't see on the site, please submit a suggestion to add it. There's a "Suggestion" link in the navbar.

So now for the thing that hasn't happened.

  1. Universal Subtitles (now renamed Amara) stopped working with blip.tv videos a couple of months ago. I had hardcoded the embed code for all the blip.tv videos on pyvideo.org. Because of this, I have to go un-hardcode the embed code in order to get the videos to work on the site at all, and it's possible we'll lose all the work we did subtitling these videos.

This really bums me out, but I'm not really sure what to do about it.

That's where things are at. I essentially took a hiatus from curating pyvideo.org while we implemented and fixed a bunch of stuff in richard and while I set up the groundwork for steve. Those two are in a better position now, so I'll be able to more easily go back to curating pyvideo.org and fixing the various issues.

Pyblosxom has moved

I'm in the process of passing off maintenance to a few people who expressed interest on the pyblosxom-devel mailing list. Towards that goal, I've moved pyblosxom to a new github repository under the pyblosxom organization. Thus it's now at:

https://github.com/pyblosxom/pyblosxom

I then forked it, so you can continue looking at my fork if you so desire, but I probably won't be doing anything with it going forward.

Me and Pyblosxom

I've been working on Pyblosxom since January 9th, 2003. The initial commit from Wari is on November 24th, 2002. I discovered that (according to the commits), I was the second person to commit to the codebase. I didn't know that.

In January of 2012, I started a hiatus from Pyblosxom. We had done a ton of overhaul work for 1.4 and 1.5 and I was tired and wanted to work on other things.

I've decided it's time to end my reign as maintainer of Pyblosxom. I sent an email to the pyblosxom-devel list as such. Further, I contended that maybe it's time for the project to end altogether.

Yes, I suggested maybe it's time to end the project. The reasons are two-fold:

  1. The code has a ton of technical debt. There are a lot of plugins that need a lot of help. There's a lot of squirrely code. I've done a poor job of fixing the "Where can I find plugins?" and "Where can I find pretty flavours?" problems. Those are big problems and potentially require a lot of maintenance.

  2. It's really hard to fix architectural problems with Pyblosxom without changing the scope of the project. I contend it's way easier at this point to just start a new one.

Anyhow, I'm a little sad and I'll have to figure out what to do with my blog, but I think it's been a long time coming and it feels good to put it to rest.

So, two things:

  1. If you're interested in maintaining Pyblosxom, hop on the pyblosxom-devel mailing list and say hi!

  2. I cut a lot of my teeth on this project. Most of the current problems are my fault. If you're a current user of Pyblosxom, thank you for using something I spent a ton of time on and cared very much about and thank you for your patience as I figured a lot of things out.

On to new horizons!

Updates:

April 7th, 2021: updated the Pyblosxom url

elasticutils status -- May 18th, 2012

A few months ago, I "took over" maintenance of elasticutils. We use it in SUMO as the API for building search queries with elasticsearch.

One of the first things I did was spend some time figuring out whether we should keep working on elasticutils at all. django-haystack also provides a django-ish API for working with elasticsearch. Why have two libraries that at a high level do the same thing?

The thing is that they're not exactly the same. django-haystack is really great and supports a variety of backends for search, elasticsearch being one of them. Right now, it only has support for elasticsearch in 2.0 which is in either an alpha or beta state now (their web-site could use some updates). However, because it supports a bunch of backends, it only supports functionality that works across all of them.

elasticutils, on the other hand, is elasticsearch-specific. As elasticsearch adds functionality, we can, too. That's the compelling reason to keep working on this library. However, django-haystack has some awesome ideas that we'd like to implement in elasticutils, too. This will fix some sharp edges in elasticutils, but also make it much easier for projects to switch from one to the other.

Currently, elasticutils only handles the query side of things. django-haystack handles that, but also has an API for defining mappings, indexing, and all the other things you need with a search system.

Thus, Rob Hudson and I are going to embrace and extend elasticutils to:

  1. fix the current situation where it seems every elasticutils user is actually using their own branch with additional functionality in it (ew!)

  2. implement the rest of the things you need with a search system

  3. document the things we've learned while working with elasticutils because at a minimum, it seems most of the Mozilla projects that use elasticutils bumped into, spent time on, and solved the same problems---that's a huge waste of time and a failure on my part

One of the things users of a library need is for the library to be a mature project with releases, tagged version, documentation, tests, stability, reliability, reproduceability, communication, community and all that. Thus, I'm also going to spend some time to turn this into a real project. Towards that end, I created #elasticutils on irc.mozilla.org where we'll talk dirty elasticutils stuff. If we end up with more people pitching in, we'll create a mailing list. But for now, IRC will do.

My next step is to spend a little time cleaning up what's in the master branch, then tag and release a baseline version.

After that, I'm going to spend time identifying, thinking about and merging in the divergent functionality in the various branches while Rob works on continuing his imperative mapping work.

I think in a couple of months, we'll be in a better place and that'll make it easier for Mozilla projects and anyone else who wants to use elasticutils to use and contribute to it.

If you're a user of elasticutils, please come hang out with us! Let us know how we can better help you.

Letting it run out

A couple of years ago I applied and was granted membership into the GNOME Foundation. At the time, I was helping out with GNOME Journal and running GNOME Miro Community.

Since then, GNOME Miro Community fizzled out and stopped because it didn't find a compelling reason to exist (wasn't interesting for marketing, didn't do a good job of indexing conference video, ...). GNOME Journal fizzled too and I stopped working on it.

Also, in September, I left PCF where I was working on Miro which uses a lot of GNOME technologies.

Given that, I decided not to renew my membership. I'm a little bummed, but I'm not doing anything. I do wish them well. I will miss the periodic anti-FSF fireworks.

webdev workweek thoughts

I'm at the webdev work week in Santa Cruz, CA, USA this week. It's great to meet people I've been talking to for the last 6 months. It's also kind of nice to take a break from the SUMO sprints. I've been spending the time lifting my head and seeing what's been happening while I wasn't paying attention.

List of three things on my mind:

  1. I wish I had more time. There are so many people doing exciting things that I'd love to work on, too, but I just don't have enough time. It's hard to suppress the, "Wow---that looks awesome! I'll help you on that project!" reflex. I just don't have time. It's kind of a bummer. On the flip side, if I do discover myself with free time, I have a long (long) list of things I'm interested in.

  2. It's nice to talk about technical things. It's nice to make a (bad) joke about mime types and get a response. (An HTTP response! ha ha!)

  3. Everyone is really super and friendly. I don't think I've been in a group this big that's lacking a token jerk. This has caused me to idly wonder if maybe I'm the token jerk. If so, I hope someone leaves me a note so I can adjust accordingly.

Things I'm taking away (so far):

  1. I should dig up my compilers book and re-read it because I really liked studying compilers way back when.

  2. I screwed up with spreading Elasticsearch knowledge. I keep having conversations about problems we've seen and solved in SUMO that other projects are having. Plus it seems like everyone has their own branch of ElasticUtils with differing features. I'm going to try to spend some time working on making the situation better both by documenting things and also improving ElasticUtils so we're not all maintaining our own versions.

  3. Need to spend more time learning things.

Support The Tube Open Movie

I'm very excited about this. It's an open movie being built by friends of mine. The trailer is gorgeous. It's worth watching just to see what's possible with Free Software today.

mozilla status: April 15th, 2012

I haven't had time to blog much in the last few months. At work, I've been spending all my time with elasticsearch, elasticutils, and SUMO bug fixing. I've been working on the conversion from Sphinx search to elasticsearch for SUMO since I started at Mozilla, but I've only recently felt like I'm really getting the hang of it. There are a bunch of elasticutils-related things I want to blog about, but those will come in fugure entries.

In my spare time, I've been working on richard. This project has nothing to do with Richard of air mozilla fame, but rather is a video indexing web application. It's the software that runs pyvideo.org.

pyvideo.org has the distinction of being the first Django application I've built from the ground up. That distinction is both a virtue (yay for first apps!) and a vice (boo for silly things I did when doing it!).

The one thing I did that I'm really proud of is that when building the software, I knew I needed help if it was to succeed and thus I worked to make it easy and inviting for contributors to get involved:

  • I wrote documentation: license file, README, documentation covering how to install it for hacking, how to contribute, where to find me, ...

  • I parked the code on GitHub to make it easier for people to access. [1]

  • I made sure there were a series of issues in the issue tracker that showed the next round of things that needed to be done.

  • I made sure I had an IRC channel and that people knew where to find me to ask questions.

  • I quickly got the documentation built on ReadTheDocs.

  • I had a test infrastructure set up. [2]

  • I respond to everyone who sends an email, creates a pull request, writes an issue, says hi on IRC, ...

The end result of that is that there are 4 contributors to richard including myself and one of them is very active.

Asheesh did a talk at LibrePlanet 2012 that mentioned Mako's power law of contributions to open source projects. The gist of it is that most open source projects only ever have one contributor. [3]

Well, I've got 5 on my video index web application software that I "launched" a month ago. I'm feeling good about that.

pyvideo status: April 15th, 2012

What is pyvideo.org

pyvideo.org is an index of Python-related videos on the Internet. For the most part, it's a collection of videos from Python-related conferences. Saw a session you liked and want to share it? It's likely you can find it, watch it, and share it with pyvideo.org.

Status

I started richard in order to build pyvideo.org. I threw it together in like 3 weeks and it had a lot of issues. Since then, I've had a lot of help from Reiner Gerecke and the project is moving along. There are a few more things I want before I consider richard to have a critical mass sufficient to do a version release, but we're pretty close. I'm pretty excited about that.

I updated pyvideo.org with the latest richard code today. I also spent some time fixing the speaker data---it's still kind of screwy. Amongst other things we had 3 Wesley Chuns! Now there is only one.

M.-A. Lemburg submitted two more conferences to get added for a grand total of three in my queue now. I upstreamed some fixes to vidscraper a couple of weeks ago. Once we land the API changes in richard, I'll be able to add new conferences much more easily. Right now I have to do each video by hand... it's kind of a drag.

Anyhow, things are going well, though the site looks the same and many of the issues are still there. You'll just have to take my word for it for now.

If you're interested in helping out, I sure could use you! Testing, documenting, fixing the layout of some of the pages, implementing new features, fixing project infrastructure---there are a lot of different kinds of things you could help out with. Even looking through the code and pointing out egregious issues---that's also very helpful and very welcome.

Issue trackers are here:

Code is here:

pyvideo status: March 20th, 2012

What is pyvideo.org

pyvideo.org is an index of Python-related videos on the Internet. For the most part, it's a collection of videos from Python-related conferences. Saw a session you liked and want to share it? It's likely you can find it, watch it, and share it with pyvideo.org.

Status

I launched pyvideo.org on March 10th. That was 10 days ago. In those 10 days, I fixed a bunch of issues, screwed up the site a couple of times, got dozens of emails many with encouragement and feature requests, talked to Craig with Universal Subtitles, and watched Google Analytics with some amazement. It's been a really wild 10 days.

I think overall the experience was a successful one, but there were a series of failures most of which were because I wasn't ready to go live when I went live, but I had really run out of time. Lack of feeds, media urls, easy ways to fix data, crummy experience on iPads and Android tablets, ... The list is longish. So it goes.

Of all the successes, I'm the happiest for the following:

  • The site had 100,000 or so visitors. That roughly means that PyCon 2012 reached 98,000 more people than it otherwise would have. That means that 98,000 more people heard about programming practices, robots, outreach, libraries, applications, case studies and all the other things we learned at PyCon. That's a big win.

    Thank you to the PyCon conference crew! You run a damn good conference.

  • I got dozens of emails. Many with encouragement, some with feature requests, some with data fixing requests, and some pointing out issues with the site. I think it's awesome that so many people took the time to email me feedback to make this resource better.

  • There are 8 forks on Github of richard. Further, I've had the pleasure to add two new people to the AUTHORS file and look forward to adding a third very soon. Thank you to Reiner, David and Alejandro!

    I work on richard and pyvideo in my spare time. I think this software is useful for other conferences and other groups which is why I made it Free Software and kept it separate from pyvideo. I can't possibly do everything I want to do with it, so I spent time writing documentation to ease the barriers to entry for new contributors. I feel really good about the work I did.

I've spent the last few days writing bug reports for things that need to be fixed, fixing some, fixing some data, and working on feeds. I haven't spent a ton of time on feeds, yet. I really hope to finish it soon, though.

If you're interested in helping out, I sure could use you! Testing, documenting, fixing the layout of some of the pages, implementing new features, fixing project infrastructure---there are a lot of different kinds of things you could help out with.

Issue trackers are here: