Will >> Will's blog

purpose: Will Kahn-Greene's blog of Miro, PyBlosxom, Python, GNU/Linux, random content, PyBlosxom, Miro, and other projects mixed in there ad hoc, half-baked, and with a twist of lemon

Thu, 03 Apr 2014

ElasticUtils v0.9 released!

What is it?

ElasticUtils is a Python library for building and executing Elasticsearch searches.

See the Quickstart for more details.

v0.9 released!

This is a big release, but there are some compromises in it that I'm not wildly excited about. Things like Elasticsearch 1.0 support didn't make the cut. I'm really sorry about that---we're working on it.

This release has a lot of changes in it. Roughly:

  • dropped pyelasticsearch for elasticsearch-py (Thank you Honza!)
  • fixed S.all() so it does what Django does which should let you use an S in the place of a QuerySet in some cases
  • new FacetResult class (Thank you James!)
  • S.facet() can take a size keyword
  • cleaned up ESTestCase
  • SearchResults now has facet data in the facets property
  • etc.

For the complete list of what's new, What's new in Version 0.9.

Many thanks to everyone who helped out: Alexey Kotlyarov, David Lundgren, Honza Král, James Reynolds, Jannis Leidel, Juan Ignacio Catalano, Kevin Stone, Mathieu Pillard, Mihnea Dobrescu-Balaur, nearlyfreeapps, Ricky Cook, Rob Hudson, William Tisäter and Will Kahn-Greene.

We're going to be sprinting on ElasticUtils 0.10 at PyCon US in Montreal mid April. If you're interested, come find me!

If you have any questions, let us know! We hang out on #elasticutils on irc.mozilla.org.

Fri, 13 Sep 2013

ElasticUtils v0.8 and v0.8.1 released!

What is it?

ElasticUtils is a Python library for building and executing Elasticsearch searches.

v0.8 and v0.8.1 released!

I missed the announcement for v0.8, so I'll cover both v0.8 and v0.8.1 here.

Roughly:

  • ElasticUtils now requires at least pyelasticsearch 0.6
  • adds range query and filter
  • adds S.filter_raw
  • changes the Indexable.index arguments dropping force_insert and picking up overwrite_existing

For the complete list of what's new, What's new in Version 0.8.1

Many thanks to everyone who helped out: Jannis Leidel, Rob Hudson and Grégoire Vigneron.

If you have any questions, let us know! We hang out on #elasticutils on irc.mozilla.org.

Wed, 12 Jun 2013

ElasticUtils v0.7 released!

What is it?

ElasticUtils is a Python library for building and executing Elasticsearch searches.

v0.7 released!

Turns out I haven't announced an ElasticUtils release since August 2012. Why? Partially because up until now, I always had deep-seated problems with ElasticUtils and wasn't excited about announcing yet another version with things I disliked in it.

I feel really good about v0.7 for a variety of reasons. Let me tell you some of them:

  1. We switched from pyes to pyelasticsearch. I'm really happy with this.

  2. There was a monumental effort to fix sharp edges in the API, generalize bits that needed generalizing, improve the quality of the software, improve the test suite, improve the docs, ...

    Doing a git diff --stat tells me:

    65 files changed, 6164 insertions(+), 2716 deletions(-)
    

    That's a lot of change for a small project like this.

If you're using ElasticUtils, I highly encourage you to update to v0.7. We're using it on Input and Support already.

For the complete list of what's new, What's new in Version 0.7.

Many thanks to everyone who helped out: Erik Rose, Jannis Leidel, Rob Hudson, Steve Ivy, Will Kahn-Greene (oh, that's me!), Chris McDonald, Ricky Rosario, James Socol, Giorgos Logiotatidis, Mike Cooper, Grégoire Vigneron, Chris Sinchok and Brandon Adams.

If you have any questions, let us know! We hang out on #elasticutils on irc.mozilla.org.

Fri, 10 May 2013

My thoughts on Elasticsearch: Part 1: indexing

Summary

I just finished up an overhaul of ElasticUtils and then an overhaul of the search infrastructure for support.mozilla.org. During that period of time, I thought about extending the ElasticUtils documentation to include things I discovered while working on these projects. Then I decided that this information is temporal---it's probably good now, but might not be in a year. Maintaining it in the ElasticUtils docs seemed like more work than it was worth.

Thus I decided to write a series of blog posts.

This one covers indexing. Later ones will cover mappings, searching and other things.

It's also long, rambling and contains code. The rest is after the break.

Wed, 20 Mar 2013

ElasticUtils sprint at PyCon US 2013

What is it?

ElasticUtils is a Python library for building and executing ElasticSearch searches.

PyCon US 2013 sprint

I was only at the sprints for a single day. Rob and I spent some time working on elasticutils. Several good things came out of that:

  1. Rob wrote up an elasticutils Django middleware which throws a 501 or 503 page if an unhandled pyelasticsearch or requests exception is raised
  2. I fixed the Django tasks, added a test, and updated the documentation
  3. I cleaned up the Django ElasticSearchTestCase class
  4. I spent a bunch of time thinking about queries, syntax and functionality

Someone on IRC asked whe the next version of elasticutils will go out. I have no schedule right now, but I think it's important to let the code get used by projects that don't mind being bleeding edge and bake for a bit. The code in master tip right now is 0.7.dev and the big change since 0.6 is that we switched from pyes to pyelasticsearch. That's a big change---the more baking it does, the better.

Having said that, a release depends mostly on how much free time I have in the near future. I'm about to lose all free time for a bit, so my guess is that we won't see a 0.7 release until this summer unless there's a compelling reason to push one out.

In the meantime, I'm actively maintaining the v0.5 and v0.6 branches. I'd like to stop maintaining the v0.5 branch, but need to get Mozillians and AMO off of it first.

If you have any questions, let us know! We hang out on #elasticutils on irc.mozilla.org.

Thu, 02 Aug 2012

ElasticUtils v0.4 released!

What is it?

ElasticUtils is a Python library for building and executing ElasticSearch searches.

v0.4 released!

I released v0.4 a couple of days ago. This release adds new functionality, fixes some issues, adds more tests, and includes improved documentation.

On top of that, we removed the requirement for Django and moved the Django-aiding components into elasticutils.contrib.django. I personally like this because it makes it much easier to write test scripts to see how things react.

For the complete list of what's new, What's new in Version 0.4.

If you have any questions, let us know! We hang out on #elasticutils on irc.mozilla.org.

Fri, 18 May 2012

elasticutils status -- May 18th, 2012

A few months ago, I "took over" maintenance of elasticutils. We use it in SUMO as the API for building search queries with elasticsearch.

One of the first things I did was spend some time figuring out whether we should keep working on elasticutils at all. django-haystack also provides a django-ish API for working with elasticsearch. Why have two libraries that at a high level do the same thing?

The thing is that they're not exactly the same. django-haystack is really great and supports a variety of backends for search, elasticsearch being one of them. Right now, it only has support for elasticsearch in 2.0 which is in either an alpha or beta state now (their web-site could use some updates). However, because it supports a bunch of backends, it only supports functionality that works across all of them.

elasticutils, on the other hand, is elasticsearch-specific. As elasticsearch adds functionality, we can, too. That's the compelling reason to keep working on this library. However, django-haystack has some awesome ideas that we'd like to implement in elasticutils, too. This will fix some sharp edges in elasticutils, but also make it much easier for projects to switch from one to the other.

Currently, elasticutils only handles the query side of things. django-haystack handles that, but also has an API for defining mappings, indexing, and all the other things you need with a search system.

Thus, Rob Hudson and I are going to embrace and extend elasticutils to:

  1. fix the current situation where it seems every elasticutils user is actually using their own branch with additional functionality in it (ew!)
  2. implement the rest of the things you need with a search system
  3. document the things we've learned while working with elasticutils because at a minimum, it seems most of the Mozilla projects that use elasticutils bumped into, spent time on, and solved the same problems---that's a huge waste of time and a failure on my part

One of the things users of a library need is for the library to be a mature project with releases, tagged version, documentation, tests, stability, reliability, reproduceability, communication, community and all that. Thus, I'm also going to spend some time to turn this into a real project. Towards that end, I created #elasticutils on irc.mozilla.org where we'll talk dirty elasticutils stuff. If we end up with more people pitching in, we'll create a mailing list. But for now, IRC will do.

My next step is to spend a little time cleaning up what's in the master branch, then tag and release a baseline version.

After that, I'm going to spend time identifying, thinking about and merging in the divergent functionality in the various branches while Rob works on continuing his imperative mapping work.

I think in a couple of months, we'll be in a better place and that'll make it easier for Mozilla projects and anyone else who wants to use elasticutils to use and contribute to it.

If you're a user of elasticutils, please come hang out with us! Let us know how we can better help you.