Will >> Will's blog

purpose: Will Kahn-Greene's blog of Miro, PyBlosxom, Python, GNU/Linux, random content, PyBlosxom, Miro, and other projects mixed in there ad hoc, half-baked, and with a twist of lemon
Page 1 of 2  >> (less recent)

Wed, 09 Apr 2014

pyvideo status: April 9th, 2014

What is pyvideo.org

pyvideo.org is an index of Python-related conference and user-group videos on the Internet. Saw a session you liked and want to share it? It's likely you can find it, watch it, and share it with pyvideo.org.

Status

I fixed a few issues and finally (finally) pushed out major site updates. Some of them are implemented in the worst possible way (e.g. facet filters for the search page), but some of them are great (e.g. Amara subtitle support).

I'm still struggling with a lot of technical debt on the site and a lack of time to really focus on it. That's mostly what's been making fixing the issues, improving the site and adding conferences take so long.

Sheila and I will be at PyCon US and hanging around for sprint days. If anyone is interested in sprinting, we'll be there. Even if we don't get any coding done, figuring out how to solve some of the bigger problems and planning what should be done in the next year would be a huge accomplishment.

If you're at PyCon and see either of us, feel free to give us a piece of your mind in regards to how you use PyVideo and what could be better.

Sat, 15 Feb 2014

pyvideo status: February 15th, 2014

What is pyvideo.org

pyvideo.org is an index of Python-related conference and user-group videos on the Internet. Saw a session you liked and want to share it? It's likely you can find it, watch it, and share it with pyvideo.org.

Status

Over the last year, a number of things have led to a tangled mess of tasks that need to be done that were blocked on other tasks that were complicated by the fact that I had half-done a bunch of things. I've been chipping away at various aspects of things, but most of them were blocked on me finishing infrastructure changes I started in November when we moved everything to Rackspace.

I finally got my local pyvideo environment working and a staging environment working. I finally sorted out my postgres issues, so I've got backups and restores working (yes--I test restores). I finally fixed all the problems with my deploy script so I can deploy when I want to and can do it reliably.

Now that I've got all that working, I pushed changes to the footer recognizing that Sheila and I are co-adminning (and have been for some time) and that Rackspace is graciously hosting pyvideo.

In the queue of things to do:

  • finish up some changes to richard and then update pyvideo to the latest richard
  • re-encode all the .flv files I have from blip.tv into something more HTML5-palatable (I could use help with this--my encoding-fu sucks)
  • fix other blip.tv metadata fallout--for example most of the PyGotham videos have terrible metadata (my fault)
  • continue working on process and tools to make pyvideo easier to contribute to

That about covers it for this status report.

Questions, comments, thoughts, etc--send me email or twart me at @PyvideoOrg or or @willcage.

Sun, 24 Nov 2013

pyvideo status: November 24th, 2013

What is pyvideo.org

pyvideo.org is an index of Python-related conference and user-group videos on the Internet. Saw a session you liked and want to share it? It's likely you can find it, watch it, and share it with pyvideo.org.

Status

Lot of stuff has happened since the last status report, but there are four things of note:

  1. Sheila is now a co-admin of pyvideo.org. She has been for a couple of months. I need to update the site to reflect this.

    I'm really psyched about this. It's a ton of work and I'm just not managing it well. Splitting the work should make it more manageable.

  2. Back in July, Sheila poked me about a tweet Jesse wrote suggesting Rackspace was interested in sponsoring Open Source projects. She contacted Jesse and set everything up.

    I'm psyched that Rackspace agreed to sponsor pyvideo.org by providing free hosting. Several months later, I moved pyvideo.org from where it was before to a vm at Rackspace.

    I'm really excited about this! It makes a bunch of problems that I was trying to figure out what to do about go away.

    Thank you, Rackspace!

    I need to update the site to reflect this.

  3. Sheila discovered that blip.tv was expiring a bunch of accounts that held conference videos and that those videos would go away. She and I scrambled to download all the files from blip and move them to Rackspace cloudfiles. It's about 600 videos and around 250gb of data.

    In the process of doing that, we saved videos for DjangoCon EU 2010, DjangoCon EU 2011 and PyGotham 2012. I added these to pyvideo.org today. These videos have pages that are stubs with no metadata. I've got that in my queue of things to fix.

    Also, the thumbnails for all the videos on blip.tv are on my laptop which isn't very helpful. I need to move those and update the videos in pyvideo.org.

    As a side note, if we didn't have hosting from Rackspace, we'd have been totally screwed. Thank you, Jesse Noller and Rackspace!

  4. I've been working on the richard codebase fixing architectural problems, reducing the complexities and trying to clean it up so it's in a better state. That work is almost done. When it is, I'll update pyvideo.org with the new site. At this rate, I think I can finish the work this year, but that assumes there aren't any more emergencies.

  5. I've been thinking about how to build a better communication channel for pyvideo.org so people can more easily follow what's going on so they can act on things they're interested in.

    pyvideo.org has a "site news" section. It's a pain in the ass to use and it's not syndicated anywhere and it's likely no one sees it.

    Blogging status reports like this on my blog is better, but I don't think my blog is very widely read. Making my blog more widely-read seems like a lot of work and I'm not sure I can do it effectively anyhow.

    So I've decided to ditch the "site news" section of pyvideo.org and switch to Twitter. I started a @PyvideoOrg account.

    I'll tweet site updates, calls for help and newly posted conferences. I'm tossing around tweeting new videos when they get posted, but videos tend to get posted in huge batches and getting > 40 tweets all at once is a total drag. I'll have to think about that some more.

    Follow @PyvideoOrg if you're interested! Also, feel free to tweet at that account.

    I need to update the site to reflect this.

Also, in my life things are pretty crazy. I have a new kid and juggling everything was impossible for a while. I think that should easy up now and I can spend more time on pyvideo.org going forward.

That's the state of things!

Also, thank you thank you thank you thank you Rackspace!

Wed, 03 Apr 2013

pyvideo status: April 3rd, 2013

What is pyvideo.org

pyvideo.org is an index of Python-related conference and user-group videos on the Internet. Saw a session you liked and want to share it? It's likely you can find it, watch it, and share it with pyvideo.org.

Status

  • Videos for PyCon US 2013 are still going up. There are 115 posted and live now. There are around 30 that are waiting for presenters to look at the metadata and tell Carl whether the metadata is good or not. More on that later.

  • Several new people submitted patches to richard! Several of the patches were fixes to broken things they saw on pyvideo.org. I've applied the fixes to the site directly, but have been waiting on making any non-critical updates to the site until after things have cooled off. I think I'll do a site update in the next week or so.

  • PyData 2013 was recorded. When videos are posted, they'll be in the PyData category. I don't know what the posting schedule is.

  • I was contacted a couple of times by the inimitable Montréal Python to post their videos. They're going to test out steve which is the tool I've been writing for the last 6 months to make it possible for other folks to generate the video metadata needed by pyvideo.org.

    I eagerly look forward to their progress and to their videos getting on the site.

    If it works out well, I'll blog more about steve and look for volunteers to use steve to generate the video metadata for the ever increasing backlog.

  • Several people are gittip'ing me. It's not a lot of money, but that and the many emails I've gotten over the last few weeks about the site have been really great. I work on pyvideo.org in my free time of which I don't have a lot. It's nice to know that prioritizing pyvideo.org work over other things helps you.

That's the gist of things!

Most of the PyCon US 2013 videos that aren't live are waiting for presenters to tell Carl at NextDayVideo (carl at nextdayvideo dot com) whether the metadata is good.

  • If you see your name on this list and you've told Carl the metadata is fine already, please send him a friendly reminder.
  • If you see your name on this list and you haven't told Carl anything, please send him a "yes, this is great!" or the list of things you need corrected.
  • If you see a friend on this list, tell your friend to do one of the above.

I'll update this list as I'm aware of changes. However, I don't work for NextDayVideo, so it's entirely possible my list is not current and/or there are errors. If so, please let me know.

Here's the list (last updated 2013-04-12 7:13am -0400):

  • Digital signal processing through speech, hearing, and Python -- Mel Chua
  • Faster Python Programs through Optimization -- Mike Müller
  • Python beyond the CPU -- Andy Terrel, Travis Oliphant, Mark Florisson
  • Code to Cloud in under 45 minutes -- John Wetherill
  • A Gentle Introduction to Computer Vision -- Katherine Scott, Anthony Oliver
  • Documenting Your Project in Sphinx -- Brandon Rhodes
  • Contribute with me! Getting started with open source development -- Jessica McKellar
  • Intermediate Twisted: Test-Driven Networking Software -- Itamar Turner-Trauring
  • Gittip: Inspiring Generosity -- Chad Whitacre
  • The Magic of Metaprogramming -- Jeff Rush
  • You can be a speaker at PyCon! -- Anna Ravenscroft
  • sys._current_frames(): Take real-time x-rays of your software for fun and performance -- Leonardo Rochael
  • Planning and Tending the Garden: The Future of Early Childhood Python Education -- Kurt Grandis
  • powerful pyramid features -- Carlos de la Guardia
  • Python for Robotics and Hardware Control -- Jonathan Foote
  • Copyright and You -- Frank Siler
  • Chef: Automating web application infrastructure -- Kate Heddleston
  • Numba: A Dynamic Python compiler for Science -- Travis Oliphant, Siu Kwan Lam, Mark Florisson
  • Integrating Jython with Java -- Jim Baker, Shashank Bharadwaj
  • Iteration & Generators: the Python Way -- Luciano Ramalho
  • ApplePy: An Apple ][ emulator in Python -- James Tauber
  • Distributed Coordination with Python -- Ben Bangert
  • Become a logging expert in 30 minutes -- Gavin M. Roy
  • PyNES: Python programming for Nintendo 8 bits -- Guto Maia
  • Purely Python Imaging with Pymaging -- Jonas Obrist
  • Namespaces in Python -- Eric Snow

These are all set now:

  • IPython in-depth: high-productivity interactive and parallel python -- Fernando Perez, Brian Granger, Min RK
  • Pyramid for Humans -- Paul Everitt
  • Learn Python Through Public Data Hacking -- David Beazley
  • Rethinking Errors: Learning from Scala and Go -- Bruce Eckel

Tue, 19 Mar 2013

Adding Persona authentication to richard

tl;dr

This is a post covering my first time experience with integrating Persona authentication into my Django project named richard. I briefly cover why I did it, what I used, and list the commits I did the work in as an example of how it can be done. I hope this helps others implement it on their sites..

why

A month ago, I added Persona authentication support to richard. This allowed me to use Persona authentication for pyvideo.org. I did this for several reasons:

  1. I wanted to try it out and see how well it worked on a small Django site (tl;dr works great---I'll use this on all my sites)
  2. I wanted people to authenticate with an email-based identity rather than a social network based identity
  3. I wanted to allow people to create accounts on pyvideo.org, but didn't want to deal with the responsibility of protecting things like passwords

So that's where I'm coming from.

how

I used django-browserid which gives you some JavaScript and a few template tags that make it easy to incorporate Persona authentication into a Django app.

It took about 15 minutes to get it working. I've made some minor edits to the code since then and updated to v0.8 of django-browserid. All told, I think I've spent a couple of hours on Persona implementation.

In the process of doing that work, I hit a few minor issues, created some pull requests, helped with other pull requests and became one of the maintainers. Yay!

Here are the commits I did the work in. I figured the diffs might help you implement similar things on your sites:

That last commit updates to django-browserid master tip to pick up a fix to login failures if BROWSERID_CREATE_USER is False. That fix will be released in v0.8.1 soon.

further reading

The Mozilla Persona site helps understand why it exists and has a Developer FAQ.

The django-browserid docs are pretty good and walk through setting it up, advanced usage, and troubleshooting. I encourage you to read through them in full---it'll give you a better understanding of the pieces.

Dan Callahan did a talk at PyCon US 2013 on Persona. That's worth watching. It covers why Mozilla built it, how it works, and why it's important that it works that way. He also demos integrating it into sites and talks about using Persona authentication alongside other authentication methods.

If you're interested in adding Persona authentication to your Django site and need help, let me know.

Sun, 03 Feb 2013

pyvideo status: February 3rd, 2013

What is pyvideo.org

pyvideo.org is an index of Python-related conference and user-group videos on the Internet. Saw a session you liked and want to share it? It's likely you can find it, watch it, and share it with pyvideo.org.

Status

  • Videos for PyCon AU 2012 are posted.

    That's probably the last conference I'm going to do on my own. More about that later.

  • I've made some big changes to richard. For one, formatted fields use Markdown instead of HTML now (yay!). I've improved the API. I've made a lot of layout tweaks and user interface improvements.

  • I pushed out steve v0.1 and then promptly made a bunch of fixes, tweaks and changes. So I need to do a new release soon. steve is the utility people can use to generate conference data for pyvideo.org. See the commandline chapter for details.

I've been working on getting steve and richard to the point where I'm neither doing all the work nor am I the bottleneck for work being done.

I still need to write up a blog post on how to use steve to generate JSON files for pyvideo.org. That will make it possible for anyone to add conference video.

I'm working on changing richard to allow for other people to edit video metadata. It'll continue to be curated, but this will make it possible for other people to help because there are like 1600 videos and the repository continues to grow and I'm just one man. I have some of this worked out on paper, but it needs to be implemented.

That's the current push. I'm hoping to have a lot of this done for PyCon 2013.

Fri, 14 Sep 2012

pyvideo status: September 14th, 2012

What is pyvideo.org

pyvideo.org is an index of Python-related conference and user-group videos on the Internet. Saw a session you liked and want to share it? It's likely you can find it, watch it, and share it with pyvideo.org.

Status

I posted the videos for SciPy US 2012 earlier today. They're missing summaries and Carl is forwarding me a bunch of data correction requests. I'll work through that over the next few days.

SciPy 2012 had ok metadata. I spent about 3 hours on SciPy 2012 over the last few weeks.

Next in the hopper are PyCon AU 2012, DjangoCon US 2012 and PyCon DE 2011. You can see the queue of conferences here.

Thinking about a tip jar

pyvideo.org takes a lot of time. Plus the software it runs on is pretty cool and could/should be used for other domains. This is less of a hobby and more of a part-time job.

I've been (slowly) working on collaboration features in richard that make it easier to delegate the work to other people. Even with that, I'll be spending a lot of time on this.

One thing I was thinking about doing was adding a tip jar sort of thing to pyvideo.org. My questions to you are:

  1. is that offensive?
  2. is this site valuable enough to you that you would tip me?
  3. what systems are good for this sort of thing? PayPal? gittip?

Thu, 06 Sep 2012

pyvideo status: September 6th, 2012

What is pyvideo.org

pyvideo.org is an index of Python-related videos on the Internet. For the most part, it's a collection of videos from Python-related conferences. Saw a session you liked and want to share it? It's likely you can find it, watch it, and share it with pyvideo.org.

Status

I posted the videos for EuroPython 2012 last night. Many thanks to Omar who pulled together metadata for the conference.

If you look at the videos on the site, the data is kind of a mess. I spent a bunch of time reconciling issues with the data from the YouTube feed with data from the EuroPython 2012 site and fixed a lot of issues, but there's still a lot left to do.

I spent about 10 hours working on the data for EuroPython 2012.

My current plan is to leave it like this for now and forge ahead to catch up with other conferences from 2012. Then I'll go back and continue working on a system for crowd-sourcing metadata fixes. That will make it easier for anyone to fix data they see is wrong and also remove me as a bottleneck to a better index of Python video.

I'm working on SciPy 2012, PyCon AU 2012, and PyCon DE 2011. You can see the queue of conferences here.

Fri, 31 Aug 2012

pyvideo status: August 31st, 2012

What is pyvideo.org

pyvideo.org is an index of Python-related videos on the Internet. For the most part, it's a collection of videos from Python-related conferences. Saw a session you liked and want to share it? It's likely you can find it, watch it, and share it with pyvideo.org.

Status

I posted the videos for EuroPython 2011 last night. Many thanks to N who pulled together metadata for the conference. That saved me gobs of time.

I want to work on EuroPython 2012 next. I've pulled all the data in the YouTube channel, however, the description and speaker data isn't easily available. Best I can find is https://ep2012.europython.eu/p3/schedule/ep2012/ which isn't in a form I can do much with.

I could really use some help! I need someone to either find a conference organizer and ask them for the data in some easy-to-parse format or scrape it. If possible, a JSON format would be great, but I can do any format that has a parser in the Python stdlib, database dumps, and probably other formats as well.

The key pieces of information I need are these:

  • title (this lets me match it up to the YouTube data I already have)
  • list of speakers
  • summary (short summary about the talk---sometimes called abstract)
  • description (longer-form description of the talk)
  • language the talk is in (English? Italian? ...)

Bonus points:

  • everything is in utf-8
  • summary and description are marked up in HTML

If you can help, please email me at willkg at bluesock dot org.

Thank you!

Update: Omar sent me an XML file with all the EuroPython 2012 metadata. I'm pretty sure I'm all set now. Thank you Omar!

Sun, 19 Aug 2012

pyvideo status: August 19th, 2012

What is pyvideo.org

pyvideo.org is an index of Python-related videos on the Internet. For the most part, it's a collection of videos from Python-related conferences. Saw a session you liked and want to share it? It's likely you can find it, watch it, and share it with pyvideo.org.

Status

I've been working with Carl from NextDayVideo to get the API working so that he can push straight from his system to pyvideo.org after he's done his post-processing for a conference. That'll mean that conferences his company videos will make it to pyvideo a lot faster. That's good because they video a lot of Python-related conferences.

In the process of doing that, I made a lot of headway on fixing the richard API and also steve.

Wait, what? Who are all these people?

So, richard is the video index website software that runs pyvideo.org. It has an admin that allows you to add videos one by one, but there's no way to add a collection of videos and no way to batch-process videos. Each conference is pretty different. I decided it would be far too time-consuming to write one web ui that could do everything I need if only because I don't really know what I need because each conference is different. Instead, I decided to write a command line utility and library of utility functions that make it easy to script something for a specific set of videos. That's steve.

Today I finished up enough of steve to do PyCon AU 2011. I'm also in the middle of a couple of other conferences, but since PyCon AU 2012 is happening right now, I figured I'd switch gears and finish that one first. It took about 3 hours for 30 videos. That's not bad considering I spent some of that time fixing bugs in steve.

Anyhow, this is a milestone in the whole richard/steve/pyvideo.org thing.

The future

The future is that I don't want to be doing all this work. Going forward, I want other people to use steve to build a bunch of JSON files that they send to me. Then I'll curate that and add it to the site. In this way, it spreads the work around and I don't have to do it all.

The future also allows anyone to suggest fixes to the data for videos that are already on pyvideo.org in a way that I can go through a queue of these fixes and approve/deny them quickly. In this way, we continue to have a curated index of videos, but it's easy to suggest fixes and thus more likely they'll happen.

Both of those are down the road, but definitely before the end of this year.

Both of those are really important because a single person can't run an index of videos this size.

That's where things are at! I'll be working through the conference backlog slowly over the next few months.