First Version of RPC4Django

Posted on July 6, 2009 by David

The first version of my unified xmlrpc and jsonrpc server is available. Go get it!

Unified XML and JSON RPC Dispatcher for Django

Posted on June 26, 2009 by David

After looking around at the rpc support available in Django, I think I will create and distribute my own application. Here are the goals:

Complete xml and json rpc support
Easy identification of rpc methods by a decorator
Customizable documentation (which is absent from DocXMLRPCServer)
Support rpc introspection
Support for method signatures (which is absent from SimpleXMLRPCServer)
Easy installation and integration into Django projects
Licensed properly for open source or commercial software

Updates to come. I’m off to San Francisco for a weekend of fun.

Deploying Django Powered Web Applications

Posted on June 11, 2009June 11, 2009 by David

Terminology

Firstly, there’s a little problem of terminology to take care of. What many people call a web application, Django calls a “project”. Instead, the Django team uses the term “application” to describe a web application component that can be deployed into one of many projects. To describe it in the wordpress paradigm, wordpress could be a Django project (if it weren’t written in php) but the blogging component, the tagging component and the themes manager might all be separate Django applications.

This distinction really pushes the concept of re-using components. For example, once some one were to write a tagging Django application, the same app could be deployed for photo tagging, blog tagging and other types of content management. These applications are supposed to be completely contained and include their seed data (fixtures), database models, and templates.

The issue and what brings me to the main part of my post is what to do with the media? Should an application include its own images, css, javascript? What’s the best way to deploy them in a convenient way for being served?

Deployment

Packaging python modules is a relatively trivial task and there is a well defined approach for it: distutils. This creates a more or less standard installer that allows anyone to install your package with a single command. From there, it can be put into the python package index and easy_install can install your python package (and dependency packages) with a single command. This works great for python packages like Django and BeautifulSoup, but how would it work for a whole web application? It made me wonder how Ellington, the flagship Django product, does it.

When distributing a python module, it makes sense to support as many platforms and configurations as possible. However, when deploying a full web application into a production environment, it makes sense to restrict your platform to what is tried and true. Ellington’s website, for example states:

Ellington takes advantage of the most secure and flexible open-source technology available: Apache for web serving, Python for programming, PostgreSQL for data, all optimized to work together on a stable, high-performance Linux platform.

Ellington isn’t intended to run on a wide variety of platforms even though it probably could. It is meant to run a production grade newspaper and therefore they specify its exact dependencies — probably the specific versions of apache, postgres, python and even linux!

So what have I learned about deployment and what do I do with all the media? After browsing django-users and the blogosphere, sticking with distutils is a great idea. I think that packaging each separate Django application is good idea. Each package is completely self contained and includes its media, templates, and code. In addition, the project settings should be minimal and possibly contained in another easily deployed python module controlled by distutils. In terms of fitting the whole thing together and deploying end to end, this is where the native package manager comes into play. This is the best way to manage both python and external dependencies. Rpm, msi or deb installers could fetch all the appropriate python modules (your Django applications), install the right version of your database and web server, sync your database, create the symbolic links to your media and even fill out your basic settings. For larger installations that require the database to be split from the python code and from the static media, this process still makes sense with few changes.

Django with JsonRPC and XMLRPC

Posted on June 4, 2009July 14, 2009 by David

[Edit: take a look at RPC4Django for a JSONRPC and XMLRPC server for Django]

I corresponded recently with a developer working on a Django-powered jsonrpc library. In the past, I have done some work on web applications that require good external interfaces. In some cases, however, it makes sense to make the same methods available via both jsonrpc and xmlrpc.

For javascript and flash, json makes a lot of sense. For communication between client side and server side, jsonrpc works very well since json is natively supported and speed can be more of a factor in presentation. However, if external interfaces are also going to interact with your API, jsonrpc is not as well supported as xmlrpc. Virtually every language has good libraries for xmlrpc. For this reason, it makes good sense to combine the two and make the same methods available to both.

import logging
from SimpleXMLRPCServer import SimpleXMLRPCDispatcher
from django.http import HttpResponse, Http404
from jsonrpc import JsonRpc

xmlrpcdispatcher = SimpleXMLRPCDispatcher(allow_none=False, encoding=None)
jsonrpcdispatcher = JsonRpc()
# ... register methods with xmlrpc and jsonrpc

def rpc_handler(request):
    if request.META['CONTENT_TYPE'] == 'application/json':
        response = jsonrpcdispatcher.handle_request(request)
    elif request.META['CONTENT_TYPE'] == 'text/xml':
        response = HttpResponse()
        response.write(xmlrpcdispatcher._marshaled_dispatch(request.raw_post_data))
        response['Content-length'] = str(len(response.content))
    else:
        # display documentation, or just raise a 404
        logging.debug('rpc request type: %s' %request.META['CONTENT_TYPE'])
        raise Http404

    return response

import logging

from SimpleXMLRPCServer import SimpleXMLRPCDispatcher

from django.http import HttpResponse, Http404

from jsonrpc import JsonRpc

xmlrpcdispatcher = SimpleXMLRPCDispatcher(allow_none=False, encoding=None)

jsonrpcdispatcher = JsonRpc()

# ... register methods with xmlrpc and jsonrpc

def rpc_handler(request):

if request.META['CONTENT_TYPE'] == 'application/json':

response = jsonrpcdispatcher.handle_request(request)

elif request.META['CONTENT_TYPE'] == 'text/xml':

response = HttpResponse()

response.write(xmlrpcdispatcher._marshaled_dispatch(request.raw_post_data))

response['Content-length'] = str(len(response.content))

else:

# display documentation, or just raise a 404

logging.debug('rpc request type: %s' %request.META['CONTENT_TYPE'])

raise Http404

return response

Subversion Backups via Email

Posted on May 30, 2009May 30, 2009 by David

I wrote a simple subversion backup script that runs on my hosting provider, webfaction, and backs up my subversion repository. I have this script running in cron and sending periodic backups to gmail. However, it should work on any unix based system with python, gzip, and subversion. Simply set the SMTP settings (configured in the webfaction panel if you have webfaction), your email information and the path to subversion and run the script.

This script requires python 2.5 or 2.6. It will require modification under python 2.4.


!/usr/bin/env python2.6

import os
import smtplib
import time
import mimetypes
from datetime import datetime

# python 2.5/2.6 is required because the email library changed after 2.4
from email import encoders
from email.mime.multipart import MIMEMultipart
from email.mime.base import MIMEBase

# SMTP settings
SMTPUSER = 'YOURSMTPUSERNAME'
SMTPPASS = 'YOURSMTPPASSOWRD'
SMTPSERVER = 'smtp.webfaction.com'
SMTPPORT = 25

# who to address the email to and from
TO = 'YOUREMAIL@EXAMPLE.com'
FROM = 'backups@webfaction.com'

# svn location
SVN_LOCATION = 'PATH_TO_SVN_DIR'

# these probably do not need to be changed
SVN_BACKUP_FILE = 'svn.dump'
GZIP_BACKUP_FILE = SVN_BACKUP_FILE + '.gz'
BACKUP_LOCATION = '/tmp/'+SVN_BACKUP_FILE
ZIPPED_BACKUP = BACKUP_LOCATION+'.gz'
BACKUP_CMD = 'svnadmin dump '+SVN_LOCATION+' &gt; ' + BACKUP_LOCATION
GZIP_CMD = 'gzip -f '+BACKUP_LOCATION

print '*******************************************************'
print '** Backing up repository'
print '*******************************************************'
os.system(BACKUP_CMD)

print '*******************************************************'
print '** Zipping backup'
print '*******************************************************'
os.system(GZIP_CMD)

print '*******************************************************'
print '** Emailing backup'
print '*******************************************************'
msg = MIMEMultipart()
msg['Subject'] = 'Subversion Backup '+str(datetime.now())
msg['From'] = FROM
msg['To'] = list().append(TO)
msg.preample = 'Should not see this in a MIME-aware mail reader.\n'

# add the gzipped attachment
fp = open(ZIPPED_BACKUP, 'rb')
att = MIMEBase('application', 'gzip')
att.set_payload(fp.read())
encoders.encode_base64(att)
att.add_header('Content-Disposition', 'attachment', filename=GZIP_BACKUP_FILE)
fp.close()
msg.attach(att)

# Send the email via our own SMTP server.
s = smtplib.SMTP(SMTPSERVER, SMTPPORT)
s.ehlo()
s.starttls()
s.ehlo()
s.login(SMTPUSER, SMTPPASS)
s.sendmail(FROM, TO, msg.as_string())
s.quit()

# removing backup
os.remove(ZIPPED_BACKUP)

!/usr/bin/env python2.6

import os

import smtplib

import time

import mimetypes

from datetime import datetime

# python 2.5/2.6 is required because the email library changed after 2.4

from email import encoders

from email.mime.multipart import MIMEMultipart

from email.mime.base import MIMEBase

# SMTP settings

SMTPUSER = 'YOURSMTPUSERNAME'

SMTPPASS = 'YOURSMTPPASSOWRD'

SMTPSERVER = 'smtp.webfaction.com'

SMTPPORT = 25

# who to address the email to and from

TO = 'YOUREMAIL@EXAMPLE.com'

FROM = 'backups@webfaction.com'

# svn location

SVN_LOCATION = 'PATH_TO_SVN_DIR'

# these probably do not need to be changed

SVN_BACKUP_FILE = 'svn.dump'

GZIP_BACKUP_FILE = SVN_BACKUP_FILE + '.gz'

BACKUP_LOCATION = '/tmp/'+SVN_BACKUP_FILE

ZIPPED_BACKUP = BACKUP_LOCATION+'.gz'

BACKUP_CMD = 'svnadmin dump '+SVN_LOCATION+' > ' + BACKUP_LOCATION

GZIP_CMD = 'gzip -f '+BACKUP_LOCATION