Lessons learned with RabbitMQ & Celery

Posted on December 17, 2010 by David

A couple months ago, I posted about using asynchronous jobs with RabbitMQ and Celery. This is a follow-up with some lessons I learned the hard way.

Celery settings for good performance

Do not run millions of jobs with DEBUG = True. You will run out of memory — even if you have 48GB of it. On top of that, you might want to consider the celeryd option –maxtasksperchild.

Be extra careful with CELERY_SEND_TASK_ERROR_EMAILS = True. I sent 9000 emails to myself in a couple minutes. My phone which syncs my email really didn’t like it. I’m running with CELERY_STORE_ERRORS_EVEN_IF_IGNORED = True and I’m looking to get a dashboard view of it with django-sentry. I think I’m almost there.

Persistence & disk space

RabbitMQ stores messages intelligently so you don’t have to keep track of them. It’s very good at this. However, problems can arise when you’re queuing tasks faster than you’re processing them. Use rabbitmqctl which ships with RabbitMQ. If you see things like this:

% /usr/sbin/rabbitmqctl list_queues
Listing queues ...
celery	9958124
celeryevent	6841
...done.

% /usr/sbin/rabbitmqctl list_queues

Listing queues ...

celery 9958124

celeryevent 6841

...done.

There’s probably going to be some issues. Ten million messages have to be stored somewhere. By default on CentOS, they’re stored in /var. RabbitMQ really doesn’t like it when you run out of disk space for it to write persistent messages so be careful.

The new persistence engine in RabbitMQ 2.x handles this much better than before. In 1.x, the persistence log has to copy itself over every so often and copying multi-GB files all the time really slows the queue to a halt and adds to the problem of not processing tasks fast enough. On top of this, RabbitMQ writes a ton of logs, which is a good thing, but can backfire when disk runs out.

Task sets

Celery’s task sets work like magic. Instead of this:

from tasks import process_item

for item in items:
    process_item.delay(item)

from tasks import process_item

for item in items:

process_item.delay(item)

Use this:

from celery.task.sets import TaskSet
from tasks import process_item

job = TaskSet(tasks=[process_item.subtask((item,)) for item in items])
job.apply_async()

from celery.task.sets import TaskSet

from tasks import process_item

job = TaskSet(tasks=[process_item.subtask((item,)) for item in items])

job.apply_async()

Note: the first parameter to subtask is a tuple of arguments to process_item.

General tips

If you can make your tasks re-entrant — meaning they can be run with the same parameters multiple times without any side effects — your life will be a lot easier. Django’s get_or_create works wonders.
Try to break tasks into smaller subtasks. Instead of one 45 minute task, break it into 2,000 tasks that take a second or two.
If you are clever with your logging, debugging things will be a lot easier. This is generally always true, but it becomes much more apparent with celery’s concurrency.

RPC4Django v0.1.8 is Available

Posted on October 26, 2010 by David

Go get it!

Changes

Added cross referenced Sphinx based documentation
Fixed bug #570852 which caused incompatibilities with MongoDB because of the name class with the variable is_rpcmethod.
Fixed bug #658788 which caused CSRF issues with serve_rpc_request.
Added out of the box authentication as per the blueprint on Launchpad.

Django-taggit versus Django-tagging

Posted on September 26, 2010 by David

For some time, there was one re-usable Django tagging application — django-tagging — and that was it. If you didn’t like it, you rolled your own. It was certainly a decent application. You could pretty easily tag anything and provided some decent features out of the box. However, very recently, another app showed up: django-taggit. This post is going to compare and contrast the two and why I decided to switch my work project to django-taggit.

Old school tagging

Back in March when we were just starting out, I went with django-tagging because that was the only tagging app around. Back then I didn’t think about it, but now 6 months later there have been no new updates or releases to django-tagging. I think this really led to the creation of django-taggit. Django-tagging had some nice features that were pretty useful like a template tag for a tag cloud. While django-tagging got me up and running quickly, it wasn’t without its annoyances. Deleting an object left dangling references to its tags since those would not be deleted because it used generic foreign keys. There was an issue filed for this back in 2008 that was never resolved. It resulted in me having to override the delete method of every object that got tagged. If I had 3 objects that got tagged, I repeated the same snippet 3 times!

# Every tagged model with django-tagging needs this code to properly clean up

def delete(self):
    # Deleting all asociated tags.
    Tag.objects.update_tags(self, None)
    super(MyModel, self).delete()

# Every tagged model with django-tagging needs this code to properly clean up

def delete(self):

# Deleting all asociated tags.

Tag.objects.update_tags(self, None)

super(MyModel, self).delete()

The fact that such seemingly important functionality had not been added since 2008 pointed to the fact that django-tagging had been left fallow for a while.

New school tagging

At Djangocon, I first heard about django-taggit. Immediately, I liked the fact that the docs were a little bit more fleshed out than django-tagging’s docs. In addition, I found the search API for taggit to be much more intuitive.

## Django-tagging: requires a level of indirection through the TaggedItem model
from tagging.models import Tag, TaggedItem
from myapp.blog.models import Post
hacking_tag = Tag.objects.get(name='hacking')
TaggedItem.objects.get_by_model(Post, hacking_tag)


## Django-taggit: look directly through the model you are searching
from myapp.blog.models import Post
Post.objects.filter(tags__name__in=["hacking"])

## Django-tagging: requires a level of indirection through the TaggedItem model

from tagging.models import Tag, TaggedItem

from myapp.blog.models import Post

hacking_tag = Tag.objects.get(name='hacking')

TaggedItem.objects.get_by_model(Post, hacking_tag)

## Django-taggit: look directly through the model you are searching

from myapp.blog.models import Post

Post.objects.filter(tags__name__in=["hacking"])

The little things

The nicest part about django-taggit is that it integrates much better with the admin. To tag an object in the admin with django-tagging, I would need to figure out the primary key id of the object I want to tag and then go to the TaggedItem admin and then tag it by id. It was unintuitive and error prone. With django-taggit, it’s as easy as editing an object the normal way. A “tags” field shows up and it explains that it simply accepts a comma separated list.

The one feature I liked from django-tagging that taggit doesn’t implement is the tag cloud. I can understand that different folks want clouds done slightly differently and that it’s not a feature that has one right way to do it. However, it was pretty damn convenient.

All told, django-taggit seems to do the job that I want it to do and it stays out of the way otherwise. It’s much more intuitive to setup and use. It’s actively maintained and the docs are better. There’s nothing not to like. At the same time, I don’t want to say it’s the end all. I’d love to see the django-tagging guys come back with some great new features because apps can always be better.

Djangocon 2010 Day Two

Posted on September 8, 2010 by David

More live-ish blogging…

The keynote

Eric Florenzano’s talk Why Django Sucks, and How We Can Fix It (slides) builds on top of the 2008 keynote by Cal Henderson entitled Why I Hate Django by pointing out instances where Django can improve. Like Cal, Eric complained about some things — some of which may not be solvable — and hopefully like some of the things Cal complained about they’ll get fixed. The note about needing more contributors again came up. Becoming a core developer is pretty much impossible. He complained about the reusability of apps citing django-avatar as an example. By rigidly defining a model, a “reusable app” becomes somewhat locked and cannot easily store new metadata. I really liked the concept of lazy foreign keys.

I’m somewhat torn on the idea of just switching Django’s source to github. I don’t fully buy Russell’s argument that you can just checkin to github and it will trigger to subversion. While that is a true statement, by having mirrors in bitbucket, launchpad and github, the Django core has fractioned the social aspects of those services. User comments are going to be split between between those services rather than being concentrated. However, similar to how Django used to allow comments in the documentation, these comments may not be that useful. I also think removing the Django admin from Django would be a travesty.

Django Security

Adam Baldwin’s Pony Pwning (notes, slides) was a decent hundred foot overview of Django security and web security in general. I would have really liked to see more details although it looked like some security vulnerability he found was redacted from his slides to give the Django developers time to fix it. Other than that, he said that the Django community as a whole “gets” security (I generally agree) and that while Django is fairly secure by default developers still manage to make mistakes. He did point out that there is no clickjacking protection for the admin and using X-FRAME-OPTIONS would be a good addition. Also, it seems that Django’s escaping could be improved. I liked that he pushed pen testing with w3af and running a web app firewall like mod_security. While frameworks can buy a certain level of security cheaply since it’s unlikely that a single developer or group will get everything right and it’s more likely that a well thought through framework will be more secure. However, then problems in the framework are discovered and basically all sites using it are suddenly vulnerable. I think some security researcher really just needs to spend some time with Django and really push the limits of its security model. I’ve talked about doing it at work, but buyin will be tough.

Other Stuff

Andrew Godwin’s (the developer of South) talk Step Away From That Database (slides – pdf) was the hundred foot overview of the various data stores for Django. One interesting trend that I’m picking up on is that Django developers seem to dislike MySQL a lot and Postgres is preferred by far. This might have something to do with MySQL development stalling and getting forked into Drizzle.

I enjoyed, but don’t entirely agree with Malcolm Tredinnick’s Modeling challenges (data) which seemed to be more about how to model complex data into Django’s models but in reality could have very well left out Django entirely and focused on databases and data modeling. I really liked the part about modeling dates that have different precisions. However, I am less sold on his implementation of modeling sports teams and players from data retrieved from retrosheet.

class Membership(models.Model):
    """
    A specification of belonging to something for a period of time. Concrete
    base classes with supply the "somethings".
    """
    joined = models.DateField()
    departed = models.DateField(null=True, blank=True)

    class Meta:
        abstract = True

    def _to_string(self, lhs, rhs):
        pairing = u"%s - %s" % (lhs, rhs)
        if self.departed:
            return u"%s (%s - %s)" % (pairing,
                    self.joined.strftime("%d %b %Y"),
                    self.departed.strftime("%d %b %Y"))
        return u"%s (%s - )" % (pairing, self.joined.strftime("%d %b %Y"))

class TeamMember(Membership):
    team = models.ForeignKey(Team, related_name="members")
    person = models.ForeignKey(Person)
    role = models.CharField(max_length=2,
            choices=((COACH, "coach"), (PLAYER, "player")))

    def __unicode__(self):
        return self._to_string(self.team, self.person)

class Membership(models.Model):

"""

A specification of belonging to something for a period of time. Concrete

base classes with supply the "somethings".

"""

joined = models.DateField()

departed = models.DateField(null=True, blank=True)

class Meta:

abstract = True

def _to_string(self, lhs, rhs):

pairing = u"%s - %s" % (lhs, rhs)

if self.departed:

return u"%s (%s - %s)" % (pairing,

self.joined.strftime("%d %b %Y"),

self.departed.strftime("%d %b %Y"))

return u"%s (%s - )" % (pairing, self.joined.strftime("%d %b %Y"))

class TeamMember(Membership):

team = models.ForeignKey(Team, related_name="members")

person = models.ForeignKey(Person)

role = models.CharField(max_length=2,

choices=((COACH, "coach"), (PLAYER, "player")))

def __unicode__(self):

return self._to_string(self.team, self.person)

While this will result in fairly normalized data form data that looks like “keara001″,”Austin Kearns” where 001 is Kearns’ first season and gets incremented yearly. Basically, this easily allows you to join data easily and find all the history of where a player was and is, but it doesn’t take into account seasons very well. I think retrosheet does it the way they do so that you can easily get a player’s stats for a given year. It’s a complicated problem but I’ve seen retrosheet’s solution in many places.

Django add-ons

I think the best talk of the day so far (it JUST finished) was Eric Holscher’s Large Problems in Django, Mostly Solved (slides) which basically gives the hundred foot overview of the add-ons available and the external packages you should be using. Some stuff is pretty clear like pip, south, celery, fabric and sphinx, but on top of that there were packages that I hadn’t heard of or knew relatively little about like Haystack for search and Gunicorn for easy or simple deployments (it also might be the right solution for an easy Pythonic Hudson replacement). I was interested that Eric sees that Piston is in competition with Tastypie and that django-tagging is being overtaken by django-taggit. At work, we setup django-tagging but since then it seems that django-taggit has emerged since then. I also loved Eric’s metric of lines of docs and lines of tests as a metric of how good a project is.

Themes

djangopackages.com is the new place to go for Django add-ons.
1.3 will probably have some good logging stuff built-in

Djangocon Day One

Posted on September 7, 2010September 7, 2010 by David

As I promised, here’s some (semi-live) blogging from Djangocon.

The first talk of the day was Scaling the World’s Largest Django Application given by the guys at Disqus (slides). The basic gist is how they scaled up Django to handle a bajillion requests per month. One thing that I noticed is that to reach this scale, they are effectively throwing away referential integrity in their database. When asked about it, one of the guy mentioned that they have some scripts to verify that things have some level of integrity which I thought was a little scary. I was happy to see that they are basically measuring everything. On top of the usual continuous integration stuff like unit tests, lint (Pyflakes, actually) they are logging every query and every traceback using a package they opensourced named Sentry. In addition, the brief mention of the save() method’s concurrency issues was interesting.

Jeff Balogh’s talk on Switching addons.mozilla.org [AMO] from CakePHP to Django (slides – pdf) was on the similar subject of switching the high traffic site AMO to Django from PHP. My favorite point from this talk is how they handle the classic stale DB replication problem of a user submitting new data (to the master) and then not seeing their data (from the slave). Basically, they use a custom Django router that detects a POST and switches that user’s session to always read and write from the master which is pretty damn clever. Mozilla also de-normalized their database in order store a reference to the latest add-on (a classic problem). However, Jeff did mention that they might switch that to storing that info in cache. The main gist of this talk was cache everything.

Russell Keith-Magee’s So you want to be a core developer? and James Bennett’s Topics of Interest (going on now — see live blogging) are both on the future of Django and how to get involved and what needs to change in the Django community. The Django people need more people to get involved. James said there are only 14 committing developers and Russell said that to get Django 1.2 out the door he had to review tickets for 5 hours a night for 2 weeks because nobody else was doing it. As James said, their bus numbers — the number of people who get hit by a bus and then you’re screwed — is frighteningly small. There’s only 2 people who know Django Oracle support.

Themes from the con

Git seems to be winning in the DVCS space. It really seems like git and github are taking over and Bazaar and Mercurial are being left at the wayside. I’ve seen tons of links to github and zero to bitbucket or launchpad. James Bennett just now briefly mentioned bitbucket and launchpad.
Django developers think that database referential integrity is overrated. Multiple people mentioned that integrity is a farce when things scale insanely.
Deploy early and often. The Mozilla guys deploy at least weekly. The Disqus guys deploy daily or more often. Although Jeff Balogh from Mozilla didn’t say it, I wonder if this means that like Disqus they are running out of trunk rather than branching and releasing.
Celery (see my previous post) is awesome and everybody seems to be adopting it.
If you aren’t using Pip and Virtualenv, you should (previous post). However, you probably shouldn’t deploy a production box from Pip and Pypi.

David Fischer dot Name

Some Things to Some People

Development