Take this trivial example:

from django.db import models
 
class Sport(object):
    name = models.CharField(max_length=50)
 
class Season(models.Model):
    starts = models.DateField()
    ends = models.DateField()
    sport = models.ForeignKey(Sport)
 
class Team(object):
    name = models.CharField(max_length=100)
    sport = models.ForeignKey(Sport)
 
class Game(object):
    season = models.ForeignKey(Season)
    home_team = models.ForeignKey(Team, related_name="home_games")
    away_team = modesl.ForeignKey(Team, related_name="away_games")

In the admin change form for Game, it is obviously desirable to only permit teams to be selected that match the Season‘s Sport. Unfortunately, because fields are defined on the class rather than the instance (such as inside of __init__), there is no obvious way to create a relationship based on the values in the instance.

Inside the ModelAdmin class is the method get_formset(self, request, obj=None, **kwargs). The parameter obj stores the current instance, if any. The significance of this is that this method is a hook with access to the instance data and is called for every form as it is built.

That makes it possible to filter the Teams based on the current form’s instance.

from django.contrib import admin
from django import forms
from myapp.models import Team, Game
 
def game_form_factory(sport):
    class RuntimeGameForm(forms.ModelForm):
        home_team = forms.ModelChoiceField(label="Home",
                queryset=Team.objects.filter(sport=sport))
        away_team = forms.ModelChoiceField(label="Away",
                queryset=Team.objects.filter(sport=sport))
 
        class Meta:
            model = Game
 
    return RuntimeGameForm
 
class GameAdmin(admin.modelAdmin):
    model = Game
 
    def get_formset(self, request, obj=None, **kwargs):
        if obj is not None:
            self.form = game_form_factory(obj.season.sport)
        return super(GameAdmin, self).get_formset(request, obj, **kwargs)

Here is how it works. When the GameAdmin form is built, get_formset is called. If this is an edit form (add form’s will not have instance data) the Game instance is passed as the obj parameter. In this case, the instance sets the form attribute to be the result of calling game_form_factory, which is a class factory function.

What if we want the Game form to be an inline form for the Season form? The major difference with inline form sets is that the instance passed to get_formset is now that of the parent form, rather than the form set model (in this case, Season instead of Game.)

The class factory function remains essentially unchanged. The Game admin model requires only a small change.

class GameAdminInline(admin.TabularInline):
    model = Game
 
    def get_formset(self, request, obj=None, **kwargs):
        if obj is not None:
            self.form = game_form_factory(obj.sport) # obj is a Season
        return super(GameAdminInline, self).get_formset(request, obj,
                **kwargs)

Submit article

class MultiFormDict(dict):
    def __init__(self, separator, *args, **kwargs):
        """Identical to a regular dict, apart from requiring a `separator` token."""
        self.separator = separator
        self.separator_length = len(separator)
        super(MultiFormDict, self).__init__(*args, **kwargs)
 
    def group_iter(self, key):
        """Iterator over the values associated with a key."""
        key = str(key)
        for item in self.keys():
            if item.startswith(key):
                item_key = item.replace('%s%s' % (key, self.separator), '')
                yield item_key, self[item]
 
    def group_keys(self):
        """Iterates over the list of unique group keys in this MultiFormDict."""
        seen = set([])
        for key in self.keys():
            found = key.find(self.separator)
            if found != -1:
                k = key[0:found]
                if k not in seen:
                    seen.add(k)
                    yield k
 
    def group_dict(self, key):
        """Returns a dictionary of keys prepended with key and their values."""
        d = {}
        for k, v in self.group_iter(key):
            d[k] = v
        return d

Its usage is pretty basic, but having it factored out in its own class certainly makes it faster to write your own formsets.

 post_data = MultiFormDict('-', {
    '0-title': 'Google',
    '0-url': 'http://www.google.com',
    '1-title': 'Yahoo',
    '1-url': 'http://www.yahoo.com',
    '2-title': 'Python',
    '2-url': 'http://www.python.org'
})
 
for key in post_data.group_keys():
    print 'Form #%s' % key
    print 'Data: %r' % post_data.group_dict(key)
    for k, v in post_data.group_iter(key):
        print '\t', k, '=', v

Again, nothing spectacular, but a useful bit of code for working with large forms.

Submit article

One answer is to keep a copy in the cache, and a second copy on disk (or in the database) as a backup. The datastore module works a lot like the cache module, except that there is no explicit set() operation. Instead, get is called with the key, a function that will generate a new object for the cache, and the object’s time to live.

import datastore
data = datastore.get('foo', get_more_foo, 5*60)

What happens here? First, the cache is checked for an object keyed ‘foo’. If one is found, it is returned. If not, get_more_foo is called to create a new object, which is stored in the cache for 5*60 seconds, after which the updated object is pickled and backed up to the file system.

If get_more_foo raises an exception, the result of the last successful call to get_more_foo (the backup copy) is returned instead. The backup value is also put back into Django’s cache to keep access quick until the next attempt at updating ‘foo’, but for only half the regular time ((5*60)/2 seconds).

The only issue is when there is nothing in the cache and there is no backup. This can happen on the first call to datastore.get. In this case, the exception raised by get_more_foo is propagated to the caller.

Therefore, it is useful to wrap the call to datastore.get in a try/except block:

import datastore
 
try:
    data = datastore.get('foo', get_more_foo, 5*60)
except FooNotAvailable:
    ...

The datastore is controlled by three required settings (in settings.py). DATASTORE_DIR is the directory in which to place the backup file. /tmp is a good place for it. DATASTORE_CULL_AFTER is the number of calls to datastore.get before the DATASTORE_DIR is cleaned of old backups that have not been accessed recently. DATASTORE_CULL_TIME is the number of seconds since the last access, after which the backup becomes free for deletion (checked against file atime). This should be proportionately large to the reliability of the data source. Each successful call to get_more_foo will regenerate the backup, updating the atime. If the source is likely to be unavailable for hours at a time, a DATASTORE_CULL_TIME in the neighborhood of twelve hours may be in order.

Note – this file uses the with statement, so a recent version of Python is required.

download datastore

Submit article

from models import Person
person = Person.objects.get_or_create(first_name="John", last_name="Johnson")

It adds syntactic sugar to the common case of checking if an entry exists, creating it if it does not, then updating (or creating with new) values. The problem is that its slow.

I offload our most expensive database work on our server to a Lisp-based XML RPC server using CLSQL. I wrote an analog of get_or_create for CLSQL. It does not have the advantage of Django’s keyword searches; all searches use = and pull the first row that matches the value.

The first function, get-instance is a helper to get a row in a view-class using a field/value pair.
create/update takes the first three arguments and passes them to get-instance to get any existing match.

(defun get-instance (cls pk val)
  #.(clsql:locally-enable-sql-reader-syntax)
  (let ((inst (clsql:select cls :flatp t :refresh t :limit 1
                            :where [= (clsql:sql-expression :attribute pk) val])))
    #.(clsql:restore-sql-reader-syntax-state)
    (if inst (car inst) nil)))
 
(defun create/update (cls pk val &rest args)
  (let ((inst (get-instance cls pk val)))
        (if inst
            (apply #'reinitialize-instance inst args)
            (apply #'make-instance cls pk val args))))

Usage:

(clsql:def-view-class person ()
    ((id
        :reader person-id
        :db-kind :key
        :db-constraints :not-null
        :type integer
        :initarg :id)
    (last-name
        :accessor person-last-name
        :db-constraints :not-null
        :type (clsql:varchar 25)
        :initarg :last-name)
    (first-name
        :accessor person-first-name
        :db-constraints :not-null
        :type (clsql:varchar 25)
        :initarg :first-name)))
 
(defvar john (create/update 'person :id 3))

If there is row with id 3 in the database, it will grab that (or, if the key is not unique, it will grab the first matching row.) If there is no entry with that id, it will create a new one, that can then be updated the normal way:

(setf (person-last-name john) "John")
(setf (person-first-name john) "Johnson")

The other alternative is to get and set all in one go:

(defvar john (create/update 'person
                            :id 3
                            :last-name "Johnson"
                            :first-name "John"))

It’s pretty simple but it is very useful.

Submit article

The global interpreter lock (GIL) is the method that Python uses to maintain internal thread-safety when not all Python objects are thread-safe. The GIL ensures that no Python object may be modified by multiple threads at the same time. This is transparent to the Python programmer; the GIL is locking internal structures. A list that is modified by multiple threads without explicit locking will still behave erratically.

This makes threading less useful in web development. Because Python can not directly maintain state between reqeusts (indirectly, via the client, state may be maintained using session cookies or other similar strategies), each request is a new thread:

 1. Determine what data is being requested
 2. Get the data
 3. Format the data
 4. Send the data

Each of these steps depends on the state information from the previous step. There is not a lot of room for asynchronicity. The usefulness of threading is pushed back to the web server. That is where integrated server/framework solutions, like Ocsigen, have an advantage.

Many web-based applications, however, are not simple database front ends. There are a few common situations where threading becomes useful; specifically, when there are side effects of a request.

On a side note, experts will tell you that only POST request should result in side effects. This is misleading. POST should certainly be used for user-intended side effects. However, what happends when there is an error loading a page?

Error messages

In the event of a coding or environment error, Django sends me an email. This is a wonderful (and an occassionally annoying) feature. But what if the problem is caused by the data we input, rather than the code? I can detect this in my code and present the user with a nice error page apologizing for the inconvenience, but now I want to notify the content team that an entry in the database is invalid in some way.

Sending a message, especially if Django is connecting to a remote mail server, can take a while. This is a good use for threading. The thread should deal with as little data processing as possible, concentrating instead on the side effect.

import threading
from django.core.mail import send_mail
 
def foo(request, identifier):
    data = get_some_data(identifier)
    try:
        error_check(data) # raise exception if data invalid
    except InvalidData, e:
        # Format our information here, in the main thread
        subject = "Invalid data in entry %d" % identifier
        message = """
            There is an error in entry %d.  Please check this
            data at http://path/to/django/admin/app/table/%d.
        """ % (identifier, identifier)
        recipients = ['someone@somewere.com', 'someoneelse@somewhere.com']
        from = 'root@server.com'
 
        # Create a new thread in Daemon mode to send message
        t = threading.Thread(target=send_mail,
                             args=[subject, message, from, recipients],
                             kwargs={'fail_silently': True})
        t.setDaemon(True)
        t.start()
    return HttpResponseServerError(some_error_page)

Note particularly that we use t.setDaemon(True) on the thread before starting it. This tells Python not to wait for the thread to exit before returning data to the client.

Writing files

Many of our web-based programs write data to static files. Some applications maintain logs. Others publish static HTML files to the enterprise server. These are great uses for Python threads, since the GIL is released during file IO operations.

Caching remote data

We are a large business and applications are developed by various (often isolated) units. I often find myself in a situation where my application depends on mutable data extracted from another system but does not have direct access to the data’s database.

Rather than depend on low internal network latencies and download the data extract on each page load, I cache the data and keep a local copy, using a script to synchronize the data. For simple data, this can be stored using Django’s low level cache. But if some state needs to be maintained for that data, a database table is a better method.

When there are a large number of such objects, a scheduled update process becomes burdensome, especially if the number of objects increases regularly. Therefore, the solution is to trigger the event when the data is accessed (via a page load).

Each cached database entry gets a timestamp field to note its last update. The model defines an update method and the model manager’s get() method checks the entry’s timestamp to see if the data needs to be freshened. The update routine is called in a separate thread.

Warning: in Django, when a model instance is found via a relationship, the manager’s get() method is not called! This is because these objects are accessed via a django.db.models.fields.related.ManyRelatedManager, rather than the model’s own manager. If you wish to solve this without duplicating code, look into Django signals.

The problem is that this is not a simple side effect and the GIL will get in the way. The solution is to do all of the necessary logic to determine if the object needs to be updated in the main thread. Just before returning the response to the user, the update thread is started in Daemon mode. That minimizes competition for the GIL. The user will not get the updated version of the object on this page load, but the next user will.

The other alternative is to use os.fork or the subprocess module to launch the update routine in a separate process (with its own interpreter instance and its own GIL). This sort of solution gets used a lot in PHP because of its lack of threads. In Python, threads tend to be more useful and much less resource hungry.

Submit article

My experiences with this has been less than wonderful. I wrote a simple middleware class to import psyco as suggested here:

# Be sure to only load on the proper architecture
from platform import architecture
if not settings.DEBUG and architecture()[0] == '32bit':
    if architecture()[0] == '32bit':
        try:
            import psyco
        except ImportError:
            pass
 
class PsycoMiddleware(object):
    def process_request(self, request):
        # Do not waste time trying to optimize the re module
        psyco.cannotcompile(re.compile)
        # Limit memory usage
        psyco.profile(memory=2048)
        return None

In development this appeared to work fine (we excluded the if not settings.DEBUG portion while testing). Once we were up and running on the production server, however, mysterious exceptions began to surface.

In particular, we seemed to be “missing” on some page hits. Specifically, we would get TemplateDoesNotExist exceptions when the templates did, in fact, exist. We were never able to sort that out. They immediately stopped when we commented out the psyco-related stuff. We tried changing the memory usage, excluding more items from compilation (including the Django loader classes), all to no avail.

My recommendation is avoidance of psyco in a Django setting. If someone has an idea what might have caused this, I’d love to hear an explanation.

Submit article

1. The CACHE_MIDDLEWARE_ANONYMOUS_ONLY setting
2. The vary_on_cookie decorator
3. Template fragment caching
4. The low-level caching API

The CACHE_MIDDLEWARE_ANONYMOUS_ONLY setting

The CACHE_MIDDLEWARE_ANONYMOUS_ONLY setting causes Django to ignore the cache if the user is not anonymous. This is less helpful than it seems. At the Dayton Daily News, we require a trivial registration to access many areas of the site. Using this setting means that the entire page cannot be cached because of a simple “Welcome, username” line in the rendered view.

Another obstacle to using the site-level cache is that many demographic-tracking packages require setting client-specific Javascript variables in the rendered view and then accessing a script on another server. The per-site cache will cache these as well, distorting your analytics.

The vary_on_cookie decorator

The vary_on_cookie decorator (found in django.views.decorators.vary) is a simple way to tell upstream caches to cache a view based on the content of the user’s cookie. This means that each user will get their own page cached.

This is a useful decorator and a part of any caching setup for user-based sites. On its own, however, it still means that if a user visits a page only once, your server must perform all the work of rendering a page. The server gains no benefit when another user visits, since the page must be generated anew and then cached for this user as well.

Template fragment caching

This is a new feature in the development version of Django. It consists of a simple template tag that signals the framework to cache a portion of the rendered template. For example:

{% load cache %} {# thanks to AdamG for noticing the typo here #}
...stuff you don't want to cache
{ cache 300 "some" "section" user.id }
...stuff you do want to cache
{% endcache %}
...more stuff you don't want to cache

The cache tag accepts the number of seconds for which the cache should remain valid and a series of keys used to uniquely identify the cache. You may use any number of keys. Addng user.id to the mix will make this portion of the template cached on a per-user basis. Using something static will make it a standard, all-user, cache.

I experimented with template fragment caching while developing an application for which we expected extremely high traffic. In the end, we decided that the overhead did not justify the savings.

This is a very new feature and naturally not ready for production use. In the next stable release of Django I imagine that it will be considerably more efficient.

The low-level caching API

The low-level caching API is the solution for serious fine-tuning of your cache. It is located in django.core.cache. It is laughably simple to use:

from django.core.cache
 
CACHE_EXPIRES = 5 * 60 # 5 minutes
 
def some_view(request, object_id):
    cache_key = "someobjectcache%s" % object_id
    object_list = cache.get(cache_key)
    #if not object_list:
    #AdamG noted that this check avoids empty lists
    #evaluating to False, as "if not object_list" did
    if object_list is None:
        object_list = expensive_lookup()
        cache.set(cache_key, object_list, CACHE_EXPIRES)
    ...

The cache is accessed via a unique key. You can cache anything that can be safely picked in Python, including query sets from Django’s ORM. If the cache has expired, cache.get(key) returns None. Setting a key in the cache requires the unique key, the object to cache, and the time in seconds for which the cache is valid.

Since Django’s template engine is quite fast, we use the low-level API to cache the most expensive portions of each page: large database lookups, search results, the result of filtering large sets of data, ad infinitum.

This has given us the biggest savings in terms of memory, database hits, and CPU usage.

One final trick

A couple of our applications are real database hogs. They have a wide range of queries that get pulled over and over. Clearly, dropping into raw SQL and pulling lists rather than objects is the best way to streamline this type of demand, but then we lose the benefit of our custom model manager and model methods. Another neat trick is to add caching to your model manager itself:

from django.contrib.sites.models import Site
from django.core.cache
 
CACHE_EXPIRES = 5 * 60 # 10 minutes
 
class ObjectManager(models.Manager):
    def get_query_set(self, *args, **kwargs):
        cache_key = 'objectlist%d%s%s' % (
            Site.objects.get_current().id, # unique for site
            ''.join([str(a) for a in args]), # unique for arguments
            ''.join('%s%s' % (str(k), str(v)) for k, v in kwargs.iteritems())
        )
 
        object_list = cache.get(cache_key)
        #if not object_list:
        #AdamG noted that this check avoids empty lists
        #evaluating to False, as "if not object_list" did
        if object_list is None:
            object_list = super(ObjectManager, self).get_query_set(*args, **kwargs)
            cache.set(cache_key, object_list, CACHE_EXPIRES)
        return object_list

This custom model manager caches query sets using the arguments passed to get_query_set(). If they are fewer than 10 minutes old, they returned from the cache; otherwise, they are returned as a fresh query set and added to the cache. This technique can be used for busy databases to cache all possible queries performed by your application.

Submit article

The only feature of Code Igniter that I really miss in Django is the ability to add a page without first registering the url, formulating a regular expression to describe the parameters and variations of the view function, and then creating the view and template.

For a production website, this is a better method than Code Igniter’s strategy of http://domain.com/class/method/arg/arg/…/, but CI’s is much easier during development. My urls may change half a dozen (or more) times in the course of the project, especially if marketing or advertising is involved.

Fortunately, there is an easy fix. A generic, introspective function and a carefully crafted pattern in urls.py can save the day. The url pattern is simple enough:

(r'^/path/to/app/root/(?P.+?)/(?:(?P.+?)/)?$', free_pattern)

Placing this at the end of your urlpatterns tuple will allow any undefined urls to go through this as a last-chance catch-all. It captures the view as first segment (after the application’s root path) as well as any other text afterward (leaving out leading and trailing slashes).

Here is the function:

def free_pattern(request, view, segments=""):
    segments = segments.split(r'/')
    try:
        view_fn = getattr(PATH_TO_APP, view)
    except AttributeError:
        raise Http404
    return view_fn(request, *segments)

The function is simple enough. It accepts the request, the view’s name, and the rest of the segments. The segments are split by the forward slash to turn them into a list.

If the view is found in the appropriate module (defined using PATH_TO_APP, which should point to whichever application we are talking about), it is run with *segments expanded as its arguments. If the view is not found, we raise an Http404 error.

This could easily be expanded to introspect the path to the application, but since Django supports a number of different layouts for applications and projects, that will most likely be very customized to your setup.

For example, I typically store my applications in an apps module, each with its own urls.py file that is included at the project level. That way, an update to an application does not require synchronizing changes across multiple projects.

Submit article