Djangocon 2011 Day Two

I enjoyed some brief time traveling when Jacob showed what Django looked like in 2005 or so. It has come a long way.

OMG APIs

I attended two talks on APIs today: Isaac Kelly’s talk on Tastypie and Tareque Hossain’s talk on the Promises & Lies in REST. Tareque’s talk involved PBS’ use of Piston and the changes that they had to make (presumably because the core has not been updated). It seems like a number of new projects in the Django/REST space have cropped up (on top of Tastypie) such as Django REST framework and dj-webmachine. At last year’s Djangocon, Eric Holscher (I think) mentioned that it seemed like there was agreement on Piston for Django REST interfaces and now the REST community is fragmenting a little and using a variety of different tools and methodologies.

Tareque recommended a number of methodologies in his talk that I would say are not very RESTful such as including the status code in the data (as opposed to just using the HTTP status code), putting the API version in the URL (a good idea but maybe that should be in the header) or putting the desired output format in the URL (.xml, .json, etc) as opposed to in the HTTP header. Perhaps thinking about “not very RESTful” though is not the right way to think about it. In his talk, Isaac said that “Restish is enough” and maybe that is the answer. If you’re doing most of the RESTful things, you’re Doing Things Right. On the other hand, once you say “Restish is enough” you’re basically admitting that everybody does REST differently and that divergence in REST interfaces is going to continue for at least the foreseeable future.

Updates on Piston

About a year ago, I wrote a little about why I’m not using Piston. Piston appears to be dead! There hasn’t been a commit since September which is almost a full year ago. This project was touted as “the way” to do REST APIs in Django and I’m sad that it doesn’t seem to be maintained. I saw some other forks of the project on Github, but there still doesn’t seem to be much work on it lately. Does anybody know what happened?

Piston Looks Good, But I’m Not Using It

Edit (November 2, 2012): This is horribly outdated. Use class-based views or tastypie.

Firstly, I’ve been missing in action for a few months and I apologize to you, my loyal reader, for that. Without making excuses (here comes the excuses), work has been picking up, my girlfriend moved from about 15 miles away to only about 8 blocks away and Starcraft II is in beta. Regardless, I’m back in the Python action. WoooHooo!

REST interfaces & Django

This post is somewhat of a follow-up on my post on RESTful Django web services because I didn’t really talk in my previous post about Piston. Piston (sometimes django-piston) is a library for creating RESTful services in Django and it supports some of the features that I spoke about in my previous post such as good caching support with Django’s cache framework, different output formats (eg. XML & JSON) via what Piston calls emitters, and the ability but not the requirement to use Django models as REST resources. I don’t know how I missed Piston before, but people blog (*) about it and it has made the rounds on the Django User’s list. However, even after looking closely at it, I decided not to go with it. In this post I’m going to talk about what I did and did not like and why I rolled my own REST micro-framework. That almost sounds like I’m giving myself too much credit given that my micro-framework is only ~30 lines.

(*) BTW, Despite the fact that Eric updates his blog somewhat infrequently (sounds familiar) it is well worth a read.

Piston: the good

Piston ships with quite a bit of good documentation and allegedly is used to power some of BitBucket’s services — lending to its credibility. Specifically, I liked the fact that it plugged directly into Django models. You simply write a short Handler for your model explaining what fields to expose and you’re mostly done.

It effectively wraps up your handler and does all the JSON/XML/YAML serialization for you while still giving you the ability to customize it. On top of this, it plugs in nicely with Django’s form validation and allows you to do some other nice features like throttling requests based on which user does it.

Piston: the bad & the ugly

I started to look at Piston, but because I wasn’t using throttling, using OAuth, outputting anything other than JSON and I wasn’t tying to models I didn’t think that Piston bought me anything. In reality, it wasn’t doing anything my me other than properly returning HttpResponseNotAllowed. My other issue is that this project involved different outputs based on HTTP headers. For example, a GET on a certain URL would return JSON formatted data (a read in the CRUD world) if an HTTP header was present and an HTML page presenting that data if it wasn’t. Piston uses different emitters based on a request parameter format (eg. /path/resource/?format=JSON). Piston gets you up and running quickly, but it didn’t fit my use case.

Also, this is a little nitpicky, but when I see something like:

I cringe a little bit considering that status code 403 is the correct status code for Forbidden. There’s a ticket for this already. Why did Piston define constants for returning various status codes anyway when that functionality is already built into Django. Is rc.DELETED so much easier than HttpResponse(status_code=204)? Perhaps it’s a little clearer and Django really should have HttpResponse subclasses for even the less common responses, but I think this definitely involves repeating yourself (and Django’s mantra is don’t repeat yourself).

The solution

I always wondered why Django didn’t allow for routing URLs based on the HTTP method: It seems like such a common use case. The developers discussed it back in 2006, but in the end it was decided that building only the simple case was best as it yielded a relatively clean urls.py. Building off of that thread, the example in the Django book (search for “method_splitter”) and another blog post, I rolled a little framework to meet my needs instead of using something like Piston.

I found this to be a much simpler and easily extensible. The argument against this is that urls.py becomes bigger, but in a lot of ways I found this to be clearer. From reading the urlpatterns, I can quickly tell exactly what gets called in each case. In addition, routing differently based on HTTP headers, cookies, the source or anything else becomes as simple as adding a parameter and a little code to service_dispatcher.

In the end, it’s wasn’t that I didn’t like Piston, it’s just that I didn’t need it.

RESTful Django Powered Web Services

What is REST

REST is an alternative to RPC based web services such as JSONRPC, XMLRPC and SOAP. Instead of simply using HTTP POST for all of its requests (with JSONRPC’s proposed GET implementation excepted) like RPC services do, it uses all the HTTP methods. It usually includes GET, POST, DELETE, PUT and other methods to achieve different results and thereby uses relatively few URIs.

Some people think any web service that makes various services available at URIs is REST. It isn’t. Some people make a service at one URI for getting an object, another for saving the object, another for getting a list of ojects, another for a list of objects matching a certain criteria. This is just RPC outisde of the realm of a specific protocol (like XMLRPC). If people are going to use simple HTTP RPC requests to get all their data but not follow any specific pattern, they’d be better off with a real RPC implementation.

How is it Better (or Worse)

REST has a lot going for it. Because it is a little more “native” to the HTTP protocol, caching can work very efficiently. Depending on language support, it may be be easier to work with a REST interface than working with a more complex RPC specific protocol. Its simplicity can be very beautiful. The RESTful idea of making your data available as a “resource” that links via hypertext to more resources can make REST very powerful.

Instead of

GET http://example.com/testcase/56

<testcase>
<results>
<result>1</result>
<result>2</result>
</results>
</testcase>

You have

GET http://example.com/testcase/56

<testcase>
<results>
<result>http://example.com/result/1</result>
<result>http://example.com/result/2</result>
</results>
</testcase>

In the 2nd method a full test case object can be generated by off of a request to the testcase object and later requests for the result objects. The client would not need to know anything special about testcases or the specific domain as it would in the first example.

RPC also has a lot going for it and there are some cases where I would pick it over REST. Caching is not always very important and when it isn’t, the benefits REST are not as apparent. Most RPC protocols already have the capability out of the box to construct objects (for SOAP — very complex objects) from web service calls. They also usually have introspection methods or WSDL to figure out what services are available. These would need to be built by a crafty REST service developer. RPC, however, doesn’t take much advantage of the HTTP protocol in that most requests are just POST requests with an RPC payload. At the same time, every HTTP implementation supports POST and not all of them support PUT or DELETE.

Next Steps

Django has a few libraries to help with REST interfaces, but nothing I’ve seen is that great. I am going to look into creating one or contributing to an existing project. Here are some things I’d like to see in a REST API:

  • In the Django 1.1 development version, PUT, DELETE, OPTIONS and HEAD are available in django.test.client.Client. A REST interface should use them by default and have another mechanism for clients that do not support these lesser used HTTP methods.
  • caching and ETags
  • different output formats (eg. XML and JSON)
  • service/resource discovery or introspection (similar to WSDL or system.listMethods)
  • a client library than can generate complex native python objects given a URI
  • models and other sources of data as REST resources
  • integration at some point with the Django trunk!
What’s Already Out There

There’s a few Django projects for making data available via REST. These efforts seem to have stalled or be in infant stages.

  • Django model views — A GSoC project to make Django model data available via REST.This project never seemed to get far off of the ground. I don’t think it has been updated much since 2007.
  • Django REST interface — Another GSoC project to create RESTful interfaces.There seem to be some active users of this one and it seems to be more fully featured than the above model views project. However, it has stalled and there hasn’t been much work on it in the past few years.
  • Django RESTAPI — Another project to make models available via REST.This project seems to have been more recently updated and it seems ok, but it still isn’t ready for prime time or in Pypi.
  • RESTinPy — A sourceforge project that makes data available in REST.
    This project seems somewhat advanced but it hasn’t been updated since the first cut was put onto sourceforge and Pypi.
  • DAPI — Another model to REST mapping module
Reading

Edit (July 15, 2010): I wrote an update involving Piston, a popular REST framework for Django.
Update (September 7, 2011): There were some updates on what folks in the community were using at Djangocon.