Dealing with comment spam

Someone spammed my blog with 400+ comments for some shoes site. Took me less than 2 minutes to skim the emails, see the pattern, and then remove all the spam from my comments moderator queue. This sort of thing is the big strength of PyBlosxom.

I have my blog set up to store comments as individual text files in a comments/ directory. All comments have to be approved before they make it to the site. Approved comments end in .cmt and comments in the moderator queue end in .cmt-. Additionally, comments that contain one of a series of blacklisted words are rejected automatically. Any time someone posts a comment, I get an email.

It took me a minute to skim my 400+ emails and notice they're all kind of the same, 10 seconds to update the blacklist so that I won't get any additional comments like this in the future, and 10 seconds to remove all the spam from the queue with:

for mem in `grep -rl spamwordhere *`; do rm $mem; done

That was it--spam gone in less than 2 minutes. Took longer to write the blog post about it.

Dev call 12/16/2009 minutes

minutes

2.6 status (roadmap)

  • talked about release planning

  • subtitle support almost done

  • we want to continue working on performance issues

Janet

  • acquiring eggplant for testing

  • regression testing on miro master

  • tested miro community 0.8.5

Luc

  • working on 2.6 bugs

Will

  • worked on and submitted the PSF grant

  • worked on 2.6 bugs

  • doing a pylint pass on the codebase and working on code quality issues

  • almost done the subtitle menu changes for linux and windows

Ben

  • working on performance issues that happen when doing things with multiple items

  • other 2.6 work

Paul

bugzilla

  • 9 bugs/feature-requests created

  • 3 bugs marked DUPLICATE

  • 12 bugs marked FIXED

  • 1 bugs marked WORKSFORME

  • 1 bugs marked INVALID

  • 2 bugs marked INCOMPLETE

notes

We're working on Miro 2.6 targeted bugs. If there are issues that should be targeted for 2.6, let us know. If there's anything you can help with either by adding information or contributing patches to any of the bugs targeted for Miro 2.6, please do!

Subtitle support status

We've been working on subtitle support for Miro in the subtitle branch. Yesterday, Ben merged the changes into the master branch. As of this morning, nightly builds have subtitle support. Yay!

There are still a few things to implement and the code hasn't had much of a chance to settle. Also, there are still some things I think should get refactored and it's likely we'll be tweaking the interface going forward as we use it more.

Still, it's good to have landed the bulk of this feature. It's probably not wildly useful yet, but it gives us some ground work towards building better subtitle distribution/creation infrastructure going forward.

Dev call 12/09/2009 minutes

minutes

2.6 status (roadmap)

  • subtitle work is almost done

  • lots of bugs to work on, but some might have been fixed in 2.5.4--Janet is going to retest

  • lots of little bugs to work on left--we could use the help if anyone is interested in contributing!

Paul

  • Worked on Miro Community 0.8.5. It's pretty close to done--there are two outstanding bugs.

  • Hoping to push out Miro Community 0.8.5 out in the next few days. Lots of bug fixes, cool new administration stuff.

  • After that, there are a few big bugs for 0.9 to work on.

Janet

  • Worked on getting 2.5.4 out. Looks like it's been a good release with few complaints.

  • Working on figuring out the shutdown crash for Miro on Windows. This has been difficult and very trial-and-error.

  • Worked on Miro Community testing.

Ben

  • Worked with Janet on that shutdown crash.

Luc

  • Worked on Miro 2.6 issues from bugzilla. Just kind of going through and cherry-picking easy issues and nasty issues.

Will

  • Pushed out 2.5.4.

  • Worked on menu refactoring.

  • Worked on subtitles stuff.

bugzilla

  • 10 bugs/feature-requests created

  • 3 bugs marked WONTFIX

  • 3 bugs marked FIXED

  • 1 bugs marked INVALID

Team Dragon: The Book

Between the end of 2003 and mid-2007, I played in a D&D campaign that was really fantastic. The campaign ran its course and our stalwart crew of awesomeness saved the world and then we went our separate ways.

During that period of time, I kept copious notes in a MoinMoin wiki of our adventures. It was always a hope that I'd take these notes and do something with them.

I played in another campaign in 2007 and used InkScape to do a comic of the first session or two of that campaign in the style of Order of the Stick. It was a lot of fun, but took forever to do each panel. I decided it'd take me a long time to do 4 years worth of sessions in comic form.

So I started a book version. I wrote a Python script (which I've since lost) that converted MoinMoin format into restructured text. Then I threw the whole thing together with Sphinx. This allowed me to edit in restructured text, compile a LaTeX document, and then generate a PDF from that. Plus I got to spend some quality time with Sphinx to see how well it generates manuals.

That worked really well except for some minor issues.

First, I needed to set the paper size in the resulting PDF. To do that, I set the latex_preamble in the conf.py file to:

latex_preamble = '\\usepackage[papersize={6in,9in}]{geometry}\n' \
                 '\\setcounter{tocdepth}{1}'

That creates the PDF in the size I needed: 6in x 9in.

Second, I needed to fix some images so they were in a table with text. I ended up writing the LaTeX for that by hand.

Third, I didn't think the chapter headings really fit with what I wanted to build, so I changed the fncychap style to Lenny.

While I was editing the LaTeX directly, I ended up changing some of the front matter and removed the index (didn't need an index to a novel).

It took me a year to put the book together. It's around 240 pages or so. Today I finished it up, created a Lulu project for the book and had a bunch of copies printed for the others in the group. Feels good to have that done. I'm looking forward to getting a copy in the mail.

Dev call 12/02/2009 minutes

minutes

2.5.4 status (roadmap)

  • Will fixed an additional bug on Monday that Janet tested and it looks fine.

  • Will is going to build builds today. Then we'll test them and probably do a release Thursday morning and update the appcast file on Friday.

2.6 status (roadmap)

  • subtitle work is almost done

  • menus need to be refactored to finish up subtitle work

  • lots of little bugs to work on left--we could use the help if anyone is interested in contributing!

Paul

  • spent the week working on bugs in Miro Community

  • started working on custom templates allowing admins to change the way their site works

  • ten or so bugs left on the 0.9 roadmap to do

  • going to triage the bugs in order to push out a 0.9 sooner

Janet

  • working on finding an automated ui tester

  • worked on Miro Community testing, too

Will

  • did a bunch of fixes

  • getting ready to push out a 2.5.4

  • plans to work on refactoring the menus next

Ben

  • implemented subtitle support on linux

  • worked on osx bug fixes

  • started looking into VLC freeze issues

bugzilla

  • 7 bugs/feature-requests created

  • 2 bugs marked DUPLICATE

  • 6 bugs marked FIXED

  • 1 bugs marked WORKSFORME

  • 2 bugs marked INVALID

notes

I wasn't on last week's devcall. Thus, no notes.

Farewell, xine

I've been talking about removing the xine renderer for 6 months now. I don't do this lightly--generally we want to add features, not remove them. However, we're a really small team of developers working on Miro and one of the things I'm focusing on for the Miro 2.6 development cycle is to reduce and simplify wherever we can so that we can focus on the work ahead of us. Also, the gstreamer folks are doing a fantastic job and that project is really coming along. I feel very confident that switching to gstreamer is a good move.

Today is the day I did the deed. As of c1c6f33, the gtk-x11 platform of Miro no longer uses the xine renderer. This checkin removes all the xine code from the codebase.

A while back, I threw together a VLC renderer that mostly works. Other renderers can be built in the same way--the infrastructure is there for anyone to create new renderers and ship them as separate packages.

Even though this is a good move, I do realize it leaves some people out in the cold. If you are interested in championing the xine renderer and starting a project to build and maintain it, let me know and I'll do what I can to help out.

Reducing complexity for 2.6

We've been working on reducing the complexity of the code for Miro 2.6. We've done this in a few different ways and I want to summarize them here.

moved binary kit stuff to separate repositories

This dropped the size of the git repository for miro a lot. Cloning the repository is much faster now. Plus it's easier to build Miro on Windows and OSX from the source tarball.

moving libtorrent out of portable

On Linux, this allows us to rely upon Linux distributions to have packages for libtorrent (the Rasterbar version) and the Python bindings. We don't need to compile libtorrent as part of the Miro build process anymore. That dropped the build time like a rock, reduced the tarball size, and removed a bunch of issues from configuring and compiling Miro on Linux.

removing sorts.pyx and fasttypes.pyx

Removing these allowed us to remove the build dependency on Boost. That removes a bunch of assy code we had in the setup.py file. This also reduced the time it takes to build Miro on Linux.

adjustments to setup.py to be more whiny when things are missing

I added some code to the gtk-x11 setup.py to make it clearer when it's missing build dependencies. I then tested this on Kubuntu Karmic, Fedora 12, and OpenSUSE 11.2. Miro will still try to run if it's missing things--I'll look into this soon.

updated gtkx11 build docs

I went through and updated the build recipes in the gtk-x11 build docs page. I updated the recipes for Ubuntu Karmic, added recipes for Fedora 12 and OpenSUSE 11.2 (though it won't work because I couldn't find a libtorrent rasterbar package), and removed a bunch of old recipes that would never work. I'm planning to do Gentoo and some other distributions next.

I updated the build and runtime requirements lists, too.

removing xine

This hasn't been done yet, but it'll happen soon. This will remove some build requirements and it'll make our lives easier since we'll only have to support one renderer on Linux. Supporting two takes a time and effort and we're only doing a so-so job of it. Better to cut xine loose and focus on gstreamer. I'm sorry that this will affect some people. I'm hoping to rework the code so that additional renderers can be released as separate packages like I did with frontends.

conclusion

We're focusing on reducing the complexity of the codebase and build requirements to make it easier for new people to pick it up, build and contribute. If there's anything else we can do on this front--or better if there's anything YOU can help us do--let us know.

Dev call 11/18/2009 minutes

minutes

2.5.4 status

  • Ben made some fixes for 12301, those will get merged in.

  • Need to sync translations

  • A new set of branch builds today and a release at the end of this week or early next week

subtitle support status

  • Luc has it implemented for OSX. Still need implementations for Linux and Windows.

  • Will's working on the Windows implementation.

  • Ben's going to look into moving the menu/submenu code to portable.

  • Either Ben or Will will work on the Linux implementation when their plates clear.

Ben:

  • spent a while working on bug 12301. bunch of fixes that'll go into 2.5.4.

  • researched "pumping up the volume"; looks possible so he's going to implement it.

Luc:

  • frustrating week working on gstreamer subtitles support

  • had problems getting his test files to play correctly with Miro on Ubuntu Karmic

  • wants to hand off gstreamer subtitle support to someone else

  • worked on scrollbar problem with osx

Janet:

  • tested 2.5.4, tested 12301 fixes, and did a bunch of Miro Community testing

Paul:

  • worked on bugs for Miro Community 0.9

Will:

  • will merge in ben's 12301 work, sync translations, and make a set of 2.5 branch builds

  • will work on subtitle support for windows

  • will work on subtitle support for linux

bugzilla

  • 9 bugs/feature-requests created

  • 3 bugs marked WORKSFORME

  • 1 bugs marked INVALID

  • 2 bugs marked DUPLICATE

  • 1 bugs marked WONTFIX

  • 1 bugs marked FIXED

notes

Several people said they read the reports weekly and find them very useful, so I'll continue writing them.

One person mentioned how there's no public information about Miro development. That's not correct. Here's a list of urls of public information:

git repository is now http

We switched the miro repository url from https://git.participatoryculture.org/miro/ to http://git.participatoryculture.org/miro/. If you were using https, then you should switch the origin url in the .git/config file. It's unlikely that's the case, though, since it's been broken for over a month now.