<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Artful Code &#187; python</title>
	<atom:link href="http://www.artfulcode.net/tags/python/feed/" rel="self" type="application/rss+xml" />
	<link>http://www.artfulcode.net</link>
	<description>Resources and tips for dynamic, interactive languages.</description>
	<lastBuildDate>Fri, 09 Sep 2011 02:15:45 +0000</lastBuildDate>
	<language>en</language>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
	<generator>http://wordpress.org/?v=3.3.1</generator>
		<item>
		<title>Runtime ChoiceField filtering in Django&#8217;s admin</title>
		<link>http://www.artfulcode.net/articles/runtime-choicefield-filtering-in-djangos-admin/</link>
		<comments>http://www.artfulcode.net/articles/runtime-choicefield-filtering-in-djangos-admin/#comments</comments>
		<pubDate>Fri, 31 Jul 2009 13:34:03 +0000</pubDate>
		<dc:creator>Jeff</dc:creator>
				<category><![CDATA[Programming]]></category>
		<category><![CDATA[django]]></category>
		<category><![CDATA[python]]></category>

		<guid isPermaLink="false">http://www.artfulcode.net/?p=741</guid>
		<description><![CDATA[Django 1.x brought with it much finer grained control over the admin application with admin forms and inline form sets. However, I still keep running into the same problem that I have since I started using Django &#8211; you cannot provide a limited queryset for a select field that depends on other instance variables. Take [...]]]></description>
			<content:encoded><![CDATA[<p>Django 1.x brought with it much finer grained control over the admin application with admin forms and inline form sets. However, I still keep running into the same problem that I have since I started using Django &#8211; you cannot provide a limited queryset for a select field that depends on other instance variables.<span id="more-741"></span></p>
<p>Take this trivial example:</p>

<div class="wp_syntax"><div class="code"><pre class="python" style="font-family:monospace;"><span style="color: #ff7700;font-weight:bold;">from</span> django.<span style="color: black;">db</span> <span style="color: #ff7700;font-weight:bold;">import</span> models
&nbsp;
<span style="color: #ff7700;font-weight:bold;">class</span> Sport<span style="color: black;">&#40;</span><span style="color: #008000;">object</span><span style="color: black;">&#41;</span>:
    name = models.<span style="color: black;">CharField</span><span style="color: black;">&#40;</span>max_length=<span style="color: #ff4500;">50</span><span style="color: black;">&#41;</span>
&nbsp;
<span style="color: #ff7700;font-weight:bold;">class</span> Season<span style="color: black;">&#40;</span>models.<span style="color: black;">Model</span><span style="color: black;">&#41;</span>:
    starts = models.<span style="color: black;">DateField</span><span style="color: black;">&#40;</span><span style="color: black;">&#41;</span>
    ends = models.<span style="color: black;">DateField</span><span style="color: black;">&#40;</span><span style="color: black;">&#41;</span>
    sport = models.<span style="color: black;">ForeignKey</span><span style="color: black;">&#40;</span>Sport<span style="color: black;">&#41;</span>
&nbsp;
<span style="color: #ff7700;font-weight:bold;">class</span> Team<span style="color: black;">&#40;</span><span style="color: #008000;">object</span><span style="color: black;">&#41;</span>:
    name = models.<span style="color: black;">CharField</span><span style="color: black;">&#40;</span>max_length=<span style="color: #ff4500;">100</span><span style="color: black;">&#41;</span>
    sport = models.<span style="color: black;">ForeignKey</span><span style="color: black;">&#40;</span>Sport<span style="color: black;">&#41;</span>
&nbsp;
<span style="color: #ff7700;font-weight:bold;">class</span> Game<span style="color: black;">&#40;</span><span style="color: #008000;">object</span><span style="color: black;">&#41;</span>:
    season = models.<span style="color: black;">ForeignKey</span><span style="color: black;">&#40;</span>Season<span style="color: black;">&#41;</span>
    home_team = models.<span style="color: black;">ForeignKey</span><span style="color: black;">&#40;</span>Team, related_name=<span style="color: #483d8b;">&quot;home_games&quot;</span><span style="color: black;">&#41;</span>
    away_team = modesl.<span style="color: black;">ForeignKey</span><span style="color: black;">&#40;</span>Team, related_name=<span style="color: #483d8b;">&quot;away_games&quot;</span><span style="color: black;">&#41;</span></pre></div></div>

<p>In the admin change form for <code>Game</code>, it is obviously desirable to only permit teams to be selected that match the <code>Season</code>&#8216;s <code>Sport</code>. Unfortunately, because fields are defined on the <em>class</em> rather than the <em>instance</em> (such as inside of <code>__init__</code>), there is no obvious way to create a relationship based on the values in the instance.</p>
<p>Inside the <code>ModelAdmin</code> class is the method <code>get_formset(self, request, obj=None, **kwargs)</code>. The parameter <code>obj</code> stores the current instance, if any. The significance of this is that this method is a hook with access to the instance data and is called for every form as it is built.</p>
<p>That makes it possible to filter the <code>Teams</code> based on the current form&#8217;s instance.</p>

<div class="wp_syntax"><div class="code"><pre class="python" style="font-family:monospace;"><span style="color: #ff7700;font-weight:bold;">from</span> django.<span style="color: black;">contrib</span> <span style="color: #ff7700;font-weight:bold;">import</span> admin
<span style="color: #ff7700;font-weight:bold;">from</span> django <span style="color: #ff7700;font-weight:bold;">import</span> forms
<span style="color: #ff7700;font-weight:bold;">from</span> myapp.<span style="color: black;">models</span> <span style="color: #ff7700;font-weight:bold;">import</span> Team, Game
&nbsp;
<span style="color: #ff7700;font-weight:bold;">def</span> game_form_factory<span style="color: black;">&#40;</span>sport<span style="color: black;">&#41;</span>:
    <span style="color: #ff7700;font-weight:bold;">class</span> RuntimeGameForm<span style="color: black;">&#40;</span>forms.<span style="color: black;">ModelForm</span><span style="color: black;">&#41;</span>:
        home_team = forms.<span style="color: black;">ModelChoiceField</span><span style="color: black;">&#40;</span>label=<span style="color: #483d8b;">&quot;Home&quot;</span>,
                queryset=Team.<span style="color: black;">objects</span>.<span style="color: #008000;">filter</span><span style="color: black;">&#40;</span>sport=sport<span style="color: black;">&#41;</span><span style="color: black;">&#41;</span>
        away_team = forms.<span style="color: black;">ModelChoiceField</span><span style="color: black;">&#40;</span>label=<span style="color: #483d8b;">&quot;Away&quot;</span>,
                queryset=Team.<span style="color: black;">objects</span>.<span style="color: #008000;">filter</span><span style="color: black;">&#40;</span>sport=sport<span style="color: black;">&#41;</span><span style="color: black;">&#41;</span>
&nbsp;
        <span style="color: #ff7700;font-weight:bold;">class</span> Meta:
            model = Game
&nbsp;
    <span style="color: #ff7700;font-weight:bold;">return</span> RuntimeGameForm
&nbsp;
<span style="color: #ff7700;font-weight:bold;">class</span> GameAdmin<span style="color: black;">&#40;</span>admin.<span style="color: black;">modelAdmin</span><span style="color: black;">&#41;</span>:
    model = Game
&nbsp;
    <span style="color: #ff7700;font-weight:bold;">def</span> get_formset<span style="color: black;">&#40;</span><span style="color: #008000;">self</span>, request, obj=<span style="color: #008000;">None</span>, <span style="color: #66cc66;">**</span>kwargs<span style="color: black;">&#41;</span>:
        <span style="color: #ff7700;font-weight:bold;">if</span> obj <span style="color: #ff7700;font-weight:bold;">is</span> <span style="color: #ff7700;font-weight:bold;">not</span> <span style="color: #008000;">None</span>:
            <span style="color: #008000;">self</span>.<span style="color: black;">form</span> = game_form_factory<span style="color: black;">&#40;</span>obj.<span style="color: black;">season</span>.<span style="color: black;">sport</span><span style="color: black;">&#41;</span>
        <span style="color: #ff7700;font-weight:bold;">return</span> <span style="color: #008000;">super</span><span style="color: black;">&#40;</span>GameAdmin, <span style="color: #008000;">self</span><span style="color: black;">&#41;</span>.<span style="color: black;">get_formset</span><span style="color: black;">&#40;</span>request, obj, <span style="color: #66cc66;">**</span>kwargs<span style="color: black;">&#41;</span></pre></div></div>

<p>Here is how it works. When the <code>GameAdmin</code> form is built, <code>get_formset</code> is called. If this is an edit form (add form&#8217;s will not have instance data) the <code>Game</code> instance is passed as the <code>obj</code> parameter. In this case, the instance sets the form attribute to be the result of calling <code>game_form_factory</code>, which is a class factory function.</p>
<p>What if we want the <code>Game</code> form to be an inline form for the <code>Season</code> form? The major difference with inline form sets is that the instance passed to <code>get_formset</code> is now that of the parent form, rather than the form set model (in this case, <code>Season</code> instead of <code>Game</code>.)</p>
<p>The class factory function remains essentially unchanged. The <code>Game</code> admin model requires only a small change.</p>

<div class="wp_syntax"><div class="code"><pre class="python" style="font-family:monospace;"><span style="color: #ff7700;font-weight:bold;">class</span> GameAdminInline<span style="color: black;">&#40;</span>admin.<span style="color: black;">TabularInline</span><span style="color: black;">&#41;</span>:
    model = Game
&nbsp;
    <span style="color: #ff7700;font-weight:bold;">def</span> get_formset<span style="color: black;">&#40;</span><span style="color: #008000;">self</span>, request, obj=<span style="color: #008000;">None</span>, <span style="color: #66cc66;">**</span>kwargs<span style="color: black;">&#41;</span>:
        <span style="color: #ff7700;font-weight:bold;">if</span> obj <span style="color: #ff7700;font-weight:bold;">is</span> <span style="color: #ff7700;font-weight:bold;">not</span> <span style="color: #008000;">None</span>:
            <span style="color: #008000;">self</span>.<span style="color: black;">form</span> = game_form_factory<span style="color: black;">&#40;</span>obj.<span style="color: black;">sport</span><span style="color: black;">&#41;</span> <span style="color: #808080; font-style: italic;"># obj is a Season</span>
        <span style="color: #ff7700;font-weight:bold;">return</span> <span style="color: #008000;">super</span><span style="color: black;">&#40;</span>GameAdminInline, <span style="color: #008000;">self</span><span style="color: black;">&#41;</span>.<span style="color: black;">get_formset</span><span style="color: black;">&#40;</span>request, obj,
                <span style="color: #66cc66;">**</span>kwargs<span style="color: black;">&#41;</span></pre></div></div>

<!-- Social Bookmarks BEGIN -->
<div class="social_bookmark">
<a><strong><em>Submit article</em></strong></a>
<br />
<div class="d">
<br />
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://del.icio.us/post?url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fruntime-choicefield-filtering-in-djangos-admin%2F&amp;title=Runtime+ChoiceField+filtering+in+Django%26%238217%3Bs+admin" rel="nofollow" title="Add to&nbsp;Del.icio.us"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/delicious.png" title="Add to&nbsp;Del.icio.us" alt="Add to&nbsp;Del.icio.us" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://digg.com/submit?phase=2&amp;url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fruntime-choicefield-filtering-in-djangos-admin%2F&amp;title=Runtime+ChoiceField+filtering+in+Django%26%238217%3Bs+admin" rel="nofollow" title="Add to&nbsp;digg"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/digg.png" title="Add to&nbsp;digg" alt="Add to&nbsp;digg" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://www.dzone.com/links/add.html?description=Runtime+ChoiceField+filtering+in+Django%26%238217%3Bs+admin&amp;url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fruntime-choicefield-filtering-in-djangos-admin%2F&amp;title=Runtime+ChoiceField+filtering+in+Django%26%238217%3Bs+admin" rel="nofollow" title="Add to&nbsp;DZone"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/dzone.png" title="Add to&nbsp;DZone" alt="Add to&nbsp;DZone" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://www.facebook.com/sharer.php?u=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fruntime-choicefield-filtering-in-djangos-admin%2F" rel="nofollow" title="Add to&nbsp;Facebook"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/facebook.png" title="Add to&nbsp;Facebook" alt="Add to&nbsp;Facebook" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://reddit.com/submit?url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fruntime-choicefield-filtering-in-djangos-admin%2F&amp;title=Runtime+ChoiceField+filtering+in+Django%26%238217%3Bs+admin" rel="nofollow" title="Add to&nbsp;reddit"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/reddit.png" title="Add to&nbsp;reddit" alt="Add to&nbsp;reddit" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://www.stumbleupon.com/submit?url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fruntime-choicefield-filtering-in-djangos-admin%2F&amp;title=Runtime+ChoiceField+filtering+in+Django%26%238217%3Bs+admin" rel="nofollow" title="Add to&nbsp;Stumble Upon"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/stumbleupon.png" title="Add to&nbsp;Stumble Upon" alt="Add to&nbsp;Stumble Upon" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://www.technorati.com/faves?add=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fruntime-choicefield-filtering-in-djangos-admin%2F" rel="nofollow" title="Add to&nbsp;Technorati"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/technorati.png" title="Add to&nbsp;Technorati" alt="Add to&nbsp;Technorati" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://twitter.com/home/?status=Check+out+Runtime+ChoiceField+filtering+in+Django%26%238217%3Bs+admin+@+http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fruntime-choicefield-filtering-in-djangos-admin%2F" rel="nofollow" title="Add to&nbsp;Twitter"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/twitter.png" title="Add to&nbsp;Twitter" alt="Add to&nbsp;Twitter" /></a>
<br />
</div>
</div>
<!-- Social Bookmarks END -->
]]></content:encoded>
			<wfw:commentRss>http://www.artfulcode.net/articles/runtime-choicefield-filtering-in-djangos-admin/feed/</wfw:commentRss>
		<slash:comments>10</slash:comments>
		</item>
		<item>
		<title>MultiFormDict</title>
		<link>http://www.artfulcode.net/articles/multiformdict/</link>
		<comments>http://www.artfulcode.net/articles/multiformdict/#comments</comments>
		<pubDate>Sun, 11 Jan 2009 18:08:14 +0000</pubDate>
		<dc:creator>Jeff</dc:creator>
				<category><![CDATA[Programming]]></category>
		<category><![CDATA[django]]></category>
		<category><![CDATA[python]]></category>

		<guid isPermaLink="false">http://www.artfulcode.net/?p=454</guid>
		<description><![CDATA[For the moment, I am stuck on Django 0.96 on one of our servers at work, but needed the formset functionality from the newer forms module.  It is a common problem and easily dealt with by prepending keys with a common token to group forms&#8217; values together.  I ended up writing a little derived class [...]]]></description>
			<content:encoded><![CDATA[<p>For the moment, I am stuck on Django 0.96 on one of our servers at work, but needed the formset functionality from the newer forms module.  It is a common problem and easily dealt with by prepending keys with a common token to group forms&#8217; values together.  I ended up writing a little derived class that makes things simple.<span id="more-454"></span></p>

<div class="wp_syntax"><div class="code"><pre class="python" style="font-family:monospace;"><span style="color: #ff7700;font-weight:bold;">class</span> MultiFormDict<span style="color: black;">&#40;</span><span style="color: #008000;">dict</span><span style="color: black;">&#41;</span>:
    <span style="color: #ff7700;font-weight:bold;">def</span> <span style="color: #0000cd;">__init__</span><span style="color: black;">&#40;</span><span style="color: #008000;">self</span>, separator, <span style="color: #66cc66;">*</span>args, <span style="color: #66cc66;">**</span>kwargs<span style="color: black;">&#41;</span>:
        <span style="color: #483d8b;">&quot;&quot;&quot;Identical to a regular dict, apart from requiring a `separator` token.&quot;&quot;&quot;</span>
        <span style="color: #008000;">self</span>.<span style="color: black;">separator</span> = separator
        <span style="color: #008000;">self</span>.<span style="color: black;">separator_length</span> = <span style="color: #008000;">len</span><span style="color: black;">&#40;</span>separator<span style="color: black;">&#41;</span>
        <span style="color: #008000;">super</span><span style="color: black;">&#40;</span>MultiFormDict, <span style="color: #008000;">self</span><span style="color: black;">&#41;</span>.<span style="color: #0000cd;">__init__</span><span style="color: black;">&#40;</span><span style="color: #66cc66;">*</span>args, <span style="color: #66cc66;">**</span>kwargs<span style="color: black;">&#41;</span>
&nbsp;
    <span style="color: #ff7700;font-weight:bold;">def</span> group_iter<span style="color: black;">&#40;</span><span style="color: #008000;">self</span>, key<span style="color: black;">&#41;</span>:
        <span style="color: #483d8b;">&quot;&quot;&quot;Iterator over the values associated with a key.&quot;&quot;&quot;</span>
        key = <span style="color: #008000;">str</span><span style="color: black;">&#40;</span>key<span style="color: black;">&#41;</span>
        <span style="color: #ff7700;font-weight:bold;">for</span> item <span style="color: #ff7700;font-weight:bold;">in</span> <span style="color: #008000;">self</span>.<span style="color: black;">keys</span><span style="color: black;">&#40;</span><span style="color: black;">&#41;</span>:
            <span style="color: #ff7700;font-weight:bold;">if</span> item.<span style="color: black;">startswith</span><span style="color: black;">&#40;</span>key<span style="color: black;">&#41;</span>:
                item_key = item.<span style="color: black;">replace</span><span style="color: black;">&#40;</span><span style="color: #483d8b;">'%s%s'</span> <span style="color: #66cc66;">%</span> <span style="color: black;">&#40;</span>key, <span style="color: #008000;">self</span>.<span style="color: black;">separator</span><span style="color: black;">&#41;</span>, <span style="color: #483d8b;">''</span><span style="color: black;">&#41;</span>
                <span style="color: #ff7700;font-weight:bold;">yield</span> item_key, <span style="color: #008000;">self</span><span style="color: black;">&#91;</span>item<span style="color: black;">&#93;</span>
&nbsp;
    <span style="color: #ff7700;font-weight:bold;">def</span> group_keys<span style="color: black;">&#40;</span><span style="color: #008000;">self</span><span style="color: black;">&#41;</span>:
        <span style="color: #483d8b;">&quot;&quot;&quot;Iterates over the list of unique group keys in this MultiFormDict.&quot;&quot;&quot;</span>
        seen = <span style="color: #008000;">set</span><span style="color: black;">&#40;</span><span style="color: black;">&#91;</span><span style="color: black;">&#93;</span><span style="color: black;">&#41;</span>
        <span style="color: #ff7700;font-weight:bold;">for</span> key <span style="color: #ff7700;font-weight:bold;">in</span> <span style="color: #008000;">self</span>.<span style="color: black;">keys</span><span style="color: black;">&#40;</span><span style="color: black;">&#41;</span>:
            found = key.<span style="color: black;">find</span><span style="color: black;">&#40;</span><span style="color: #008000;">self</span>.<span style="color: black;">separator</span><span style="color: black;">&#41;</span>
            <span style="color: #ff7700;font-weight:bold;">if</span> found <span style="color: #66cc66;">!</span>= -<span style="color: #ff4500;">1</span>:
                k = key<span style="color: black;">&#91;</span><span style="color: #ff4500;">0</span>:found<span style="color: black;">&#93;</span>
                <span style="color: #ff7700;font-weight:bold;">if</span> k <span style="color: #ff7700;font-weight:bold;">not</span> <span style="color: #ff7700;font-weight:bold;">in</span> seen:
                    seen.<span style="color: black;">add</span><span style="color: black;">&#40;</span>k<span style="color: black;">&#41;</span>
                    <span style="color: #ff7700;font-weight:bold;">yield</span> k
&nbsp;
    <span style="color: #ff7700;font-weight:bold;">def</span> group_dict<span style="color: black;">&#40;</span><span style="color: #008000;">self</span>, key<span style="color: black;">&#41;</span>:
        <span style="color: #483d8b;">&quot;&quot;&quot;Returns a dictionary of keys prepended with key and their values.&quot;&quot;&quot;</span>
        d = <span style="color: black;">&#123;</span><span style="color: black;">&#125;</span>
        <span style="color: #ff7700;font-weight:bold;">for</span> k, v <span style="color: #ff7700;font-weight:bold;">in</span> <span style="color: #008000;">self</span>.<span style="color: black;">group_iter</span><span style="color: black;">&#40;</span>key<span style="color: black;">&#41;</span>:
            d<span style="color: black;">&#91;</span>k<span style="color: black;">&#93;</span> = v
        <span style="color: #ff7700;font-weight:bold;">return</span> d</pre></div></div>

<p>Its usage is pretty basic, but having it factored out in its own class certainly makes it faster to write your own formsets.</p>

<div class="wp_syntax"><div class="code"><pre class="python" style="font-family:monospace;"> post_data = MultiFormDict<span style="color: black;">&#40;</span><span style="color: #483d8b;">'-'</span>, <span style="color: black;">&#123;</span>
    <span style="color: #483d8b;">'0-title'</span>: <span style="color: #483d8b;">'Google'</span>,
    <span style="color: #483d8b;">'0-url'</span>: <span style="color: #483d8b;">'http://www.google.com'</span>,
    <span style="color: #483d8b;">'1-title'</span>: <span style="color: #483d8b;">'Yahoo'</span>,
    <span style="color: #483d8b;">'1-url'</span>: <span style="color: #483d8b;">'http://www.yahoo.com'</span>,
    <span style="color: #483d8b;">'2-title'</span>: <span style="color: #483d8b;">'Python'</span>,
    <span style="color: #483d8b;">'2-url'</span>: <span style="color: #483d8b;">'http://www.python.org'</span>
<span style="color: black;">&#125;</span><span style="color: black;">&#41;</span>
&nbsp;
<span style="color: #ff7700;font-weight:bold;">for</span> key <span style="color: #ff7700;font-weight:bold;">in</span> post_data.<span style="color: black;">group_keys</span><span style="color: black;">&#40;</span><span style="color: black;">&#41;</span>:
    <span style="color: #ff7700;font-weight:bold;">print</span> <span style="color: #483d8b;">'Form #%s'</span> <span style="color: #66cc66;">%</span> key
    <span style="color: #ff7700;font-weight:bold;">print</span> <span style="color: #483d8b;">'Data: %r'</span> <span style="color: #66cc66;">%</span> post_data.<span style="color: black;">group_dict</span><span style="color: black;">&#40;</span>key<span style="color: black;">&#41;</span>
    <span style="color: #ff7700;font-weight:bold;">for</span> k, v <span style="color: #ff7700;font-weight:bold;">in</span> post_data.<span style="color: black;">group_iter</span><span style="color: black;">&#40;</span>key<span style="color: black;">&#41;</span>:
        <span style="color: #ff7700;font-weight:bold;">print</span> <span style="color: #483d8b;">'<span style="color: #000099; font-weight: bold;">\t</span>'</span>, k, <span style="color: #483d8b;">'='</span>, v</pre></div></div>

<p>Again, nothing spectacular, but a useful bit of code for working with large forms.</p>
<!-- Social Bookmarks BEGIN -->
<div class="social_bookmark">
<a><strong><em>Submit article</em></strong></a>
<br />
<div class="d">
<br />
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://del.icio.us/post?url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fmultiformdict%2F&amp;title=MultiFormDict" rel="nofollow" title="Add to&nbsp;Del.icio.us"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/delicious.png" title="Add to&nbsp;Del.icio.us" alt="Add to&nbsp;Del.icio.us" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://digg.com/submit?phase=2&amp;url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fmultiformdict%2F&amp;title=MultiFormDict" rel="nofollow" title="Add to&nbsp;digg"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/digg.png" title="Add to&nbsp;digg" alt="Add to&nbsp;digg" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://www.dzone.com/links/add.html?description=MultiFormDict&amp;url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fmultiformdict%2F&amp;title=MultiFormDict" rel="nofollow" title="Add to&nbsp;DZone"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/dzone.png" title="Add to&nbsp;DZone" alt="Add to&nbsp;DZone" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://www.facebook.com/sharer.php?u=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fmultiformdict%2F" rel="nofollow" title="Add to&nbsp;Facebook"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/facebook.png" title="Add to&nbsp;Facebook" alt="Add to&nbsp;Facebook" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://reddit.com/submit?url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fmultiformdict%2F&amp;title=MultiFormDict" rel="nofollow" title="Add to&nbsp;reddit"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/reddit.png" title="Add to&nbsp;reddit" alt="Add to&nbsp;reddit" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://www.stumbleupon.com/submit?url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fmultiformdict%2F&amp;title=MultiFormDict" rel="nofollow" title="Add to&nbsp;Stumble Upon"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/stumbleupon.png" title="Add to&nbsp;Stumble Upon" alt="Add to&nbsp;Stumble Upon" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://www.technorati.com/faves?add=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fmultiformdict%2F" rel="nofollow" title="Add to&nbsp;Technorati"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/technorati.png" title="Add to&nbsp;Technorati" alt="Add to&nbsp;Technorati" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://twitter.com/home/?status=Check+out+MultiFormDict+@+http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fmultiformdict%2F" rel="nofollow" title="Add to&nbsp;Twitter"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/twitter.png" title="Add to&nbsp;Twitter" alt="Add to&nbsp;Twitter" /></a>
<br />
</div>
</div>
<!-- Social Bookmarks END -->
]]></content:encoded>
			<wfw:commentRss>http://www.artfulcode.net/articles/multiformdict/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Python in NetBeans</title>
		<link>http://www.artfulcode.net/articles/python-in-netbeans/</link>
		<comments>http://www.artfulcode.net/articles/python-in-netbeans/#comments</comments>
		<pubDate>Sun, 14 Dec 2008 00:28:53 +0000</pubDate>
		<dc:creator>Jeff</dc:creator>
				<category><![CDATA[Programming]]></category>
		<category><![CDATA[java]]></category>
		<category><![CDATA[python]]></category>

		<guid isPermaLink="false">http://www.artfulcode.net/?p=377</guid>
		<description><![CDATA[For the past few weeks I've been using the NetBeans (with the new Python module) as my primary IDE.  I generally use TextMate, which is more than an editor and less than the kitchen sink, lying somewhere between BBEdit and Emacs.  TextMate is fast and a pleasure to use, but lately I have found myself in need of something a little more robust.]]></description>
			<content:encoded><![CDATA[<p>For the past few weeks I&#8217;ve been using the NetBeans (<a title="Python for NetBeans" href="http://wiki.netbeans.org/Python">with the new Python module</a>) as my primary IDE.  I generally use <a href="http://macromates.com">TextMate</a>, which is more than an editor and less than the kitchen sink, lying somewhere between BBEdit and Emacs.  TextMate is fast and a pleasure to use, but lately I have found myself in need of something a little more robust.<span id="more-377"></span></p>
<p>Back in October of 2007, I wrote an article on <a title="Article: Java from a functional perspective" href="http://www.artfulcode.net/articles/java-functional-perspective/">Java</a>.  I used NetBeans exclusively during the project about which the article is written and found it to be a solid, well-thought out application.  It is reasonably fast and &#8220;feels good&#8221; to use, which is something I feel most Java applications lack.  It has its problems; in particular, it is not nearly so easy to add language support to NetBeans as it is to TextMate or JEdit.  However, it is beginning to build a decent set of language support plugins and supports most of the languages I use in development: Python, PHP, and Javascript.  It even has support for several of the more popular JS frameworks, including Prototype/Scriptaculous and JQuery (both of which I regularly use).</p>
<p>My first impression is good.  The Python support is very nice.  It has a few problems guessing at types, which is annoying if you leave auto-hinting enabled, and it does not know about the with_statement (which it believes to be an unused import).  Aside from that, it does a good job with hinting, although that is not a feature I particularly need or want, and it manages imports well, removing unused ones and reordering them.</p>
<p>The versioning system integration is good, too, and sheds a few clicks from my life.  It <em>is</em> annoying to need a wizard to create a blank file, though.  I do get a lot of use out of NetBeans&#8217; code checking.  In a mature product, it is easy to end up with a lot of unused variables as code gets added and removed.  The refactoring features are simply a blessing, as anyone who has had to go through several dozen files to make a critical API update can tell you.</p>
<p>One other nice feature that is not immediately apparent is intelligent undo.  I know it&#8217;s not a big deal, but it&#8217;s the little things that give a product its luster.</p>
<p>Most of these features are, in fact, available in a nice little editor written in Python named <a title="Editra editor" href="http://editra.org/">Editra</a>.  But Editra&#8217;s graphics (using wxwidgets) are slow and I don&#8217;t have the patience for an editor that does visibly jerks while updating the screen.  I think that I will use NetBeans for now.</p>
<p><strong>Update:</strong> Shannon -jj Behrens wrote another <a href="http://jjinux.blogspot.com/2009/01/ide-netbeans.html">good review of Python in Netbeans on his blog</a> (<a href="http://jjinux.blogspot.com">jjinux.blogspot.com</a>).</p>
<!-- Social Bookmarks BEGIN -->
<div class="social_bookmark">
<a><strong><em>Submit article</em></strong></a>
<br />
<div class="d">
<br />
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://del.icio.us/post?url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fpython-in-netbeans%2F&amp;title=Python+in+NetBeans" rel="nofollow" title="Add to&nbsp;Del.icio.us"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/delicious.png" title="Add to&nbsp;Del.icio.us" alt="Add to&nbsp;Del.icio.us" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://digg.com/submit?phase=2&amp;url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fpython-in-netbeans%2F&amp;title=Python+in+NetBeans" rel="nofollow" title="Add to&nbsp;digg"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/digg.png" title="Add to&nbsp;digg" alt="Add to&nbsp;digg" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://www.dzone.com/links/add.html?description=Python+in+NetBeans&amp;url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fpython-in-netbeans%2F&amp;title=Python+in+NetBeans" rel="nofollow" title="Add to&nbsp;DZone"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/dzone.png" title="Add to&nbsp;DZone" alt="Add to&nbsp;DZone" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://www.facebook.com/sharer.php?u=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fpython-in-netbeans%2F" rel="nofollow" title="Add to&nbsp;Facebook"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/facebook.png" title="Add to&nbsp;Facebook" alt="Add to&nbsp;Facebook" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://reddit.com/submit?url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fpython-in-netbeans%2F&amp;title=Python+in+NetBeans" rel="nofollow" title="Add to&nbsp;reddit"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/reddit.png" title="Add to&nbsp;reddit" alt="Add to&nbsp;reddit" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://www.stumbleupon.com/submit?url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fpython-in-netbeans%2F&amp;title=Python+in+NetBeans" rel="nofollow" title="Add to&nbsp;Stumble Upon"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/stumbleupon.png" title="Add to&nbsp;Stumble Upon" alt="Add to&nbsp;Stumble Upon" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://www.technorati.com/faves?add=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fpython-in-netbeans%2F" rel="nofollow" title="Add to&nbsp;Technorati"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/technorati.png" title="Add to&nbsp;Technorati" alt="Add to&nbsp;Technorati" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://twitter.com/home/?status=Check+out+Python+in+NetBeans+@+http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fpython-in-netbeans%2F" rel="nofollow" title="Add to&nbsp;Twitter"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/twitter.png" title="Add to&nbsp;Twitter" alt="Add to&nbsp;Twitter" /></a>
<br />
</div>
</div>
<!-- Social Bookmarks END -->
]]></content:encoded>
			<wfw:commentRss>http://www.artfulcode.net/articles/python-in-netbeans/feed/</wfw:commentRss>
		<slash:comments>14</slash:comments>
		</item>
		<item>
		<title>Django Caching with Backups</title>
		<link>http://www.artfulcode.net/articles/django-caching-backups/</link>
		<comments>http://www.artfulcode.net/articles/django-caching-backups/#comments</comments>
		<pubDate>Wed, 12 Nov 2008 17:16:46 +0000</pubDate>
		<dc:creator>Jeff</dc:creator>
				<category><![CDATA[Programming]]></category>
		<category><![CDATA[django]]></category>
		<category><![CDATA[python]]></category>

		<guid isPermaLink="false">http://www.artfulcode.net/articles/django-caching-backups/</guid>
		<description><![CDATA[Django&#8217;s cache is useful to speed up access to data from slow sources, such as a remote RSS feed. But if the remote source becomes inaccessible, the data disappears after the cache expires it. One solution is to store the data longer, but that can lead to stale data. Another solution is to keep two [...]]]></description>
			<content:encoded><![CDATA[<p>Django&#8217;s cache is useful to speed up access to data from slow sources, such as a remote RSS feed. But if the remote source becomes inaccessible, the data disappears after the cache expires it. One solution is to store the data longer, but that can lead to stale data. Another solution is to keep two cached copies for differing lengths of time, but that can quickly eat up a lot of memory.<span id="more-10"></span></p>
<p>One answer is to keep a copy in the cache, and a second copy on disk (or in the database) as a backup. The <code>datastore</code> module works a lot like the <code>cache</code> module, except that there is no explicit <code>set()</code> operation. Instead, <code>get</code> is called with the key, a function that will generate a new object for the cache, and the object&#8217;s time to live.</p>

<div class="wp_syntax"><div class="code"><pre class="python" style="font-family:monospace;"><span style="color: #ff7700;font-weight:bold;">import</span> datastore
data = datastore.<span style="color: black;">get</span><span style="color: black;">&#40;</span><span style="color: #483d8b;">'foo'</span>, get_more_foo, <span style="color: #ff4500;">5</span><span style="color: #66cc66;">*</span><span style="color: #ff4500;">60</span><span style="color: black;">&#41;</span></pre></div></div>

<p>What happens here?  First, the cache is checked for an object keyed &#8216;foo&#8217;. If one is found, it is returned. If not, <code>get_more_foo</code> is called to create a new object, which is stored in the cache for 5*60 seconds, after which the updated object is pickled and backed up to the file system.</p>
<p>If <code>get_more_foo</code> raises an exception, the result of the last successful call to <code>get_more_foo</code> (the backup copy) is returned instead. The backup value is also put back into Django&#8217;s cache to keep access quick until the next attempt at updating &#8216;foo&#8217;, but for only half the regular time ((5*60)/2 seconds).</p>
<p>The only issue is when there is nothing in the cache and there is no backup. This can happen on the first call to <code>datastore.get</code>. In this case, the exception raised by <code>get_more_foo</code> is propagated to the caller.</p>
<p>Therefore, it is useful to wrap the call to <code>datastore.get</code> in a try/except block:</p>

<div class="wp_syntax"><div class="code"><pre class="python" style="font-family:monospace;"><span style="color: #ff7700;font-weight:bold;">import</span> datastore
&nbsp;
<span style="color: #ff7700;font-weight:bold;">try</span>:
    data = datastore.<span style="color: black;">get</span><span style="color: black;">&#40;</span><span style="color: #483d8b;">'foo'</span>, get_more_foo, <span style="color: #ff4500;">5</span><span style="color: #66cc66;">*</span><span style="color: #ff4500;">60</span><span style="color: black;">&#41;</span>
<span style="color: #ff7700;font-weight:bold;">except</span> FooNotAvailable:
    ...</pre></div></div>

<p>The datastore is controlled by three required settings (in settings.py). <code>DATASTORE_DIR</code> is the directory in which to place the backup file. /tmp is a good place for it. <code>DATASTORE_CULL_AFTER</code> is the number of calls to <code>datastore.get</code> before the <code>DATASTORE_DIR</code> is cleaned of old backups that have not been accessed recently. <code>DATASTORE_CULL_TIME</code> is the number of seconds since the last access, after which the backup becomes free for deletion (checked against file atime). This should be proportionately large to the reliability of the data source. Each successful call to <code>get_more_foo</code> will regenerate the backup, updating the atime. If the source is likely to be unavailable for hours at a time, a <code>DATASTORE_CULL_TIME</code> in the neighborhood of twelve hours may be in order.</p>
<p>Note &#8211; this file uses the with statement, so a recent version of Python is required.</p>
<p><a href="/wp-content/uploads/2008/12/datastorepy.gz">download datastore</a></p>
<!-- Social Bookmarks BEGIN -->
<div class="social_bookmark">
<a><strong><em>Submit article</em></strong></a>
<br />
<div class="d">
<br />
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://del.icio.us/post?url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fdjango-caching-backups%2F&amp;title=Django+Caching+with+Backups" rel="nofollow" title="Add to&nbsp;Del.icio.us"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/delicious.png" title="Add to&nbsp;Del.icio.us" alt="Add to&nbsp;Del.icio.us" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://digg.com/submit?phase=2&amp;url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fdjango-caching-backups%2F&amp;title=Django+Caching+with+Backups" rel="nofollow" title="Add to&nbsp;digg"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/digg.png" title="Add to&nbsp;digg" alt="Add to&nbsp;digg" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://www.dzone.com/links/add.html?description=Django+Caching+with+Backups&amp;url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fdjango-caching-backups%2F&amp;title=Django+Caching+with+Backups" rel="nofollow" title="Add to&nbsp;DZone"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/dzone.png" title="Add to&nbsp;DZone" alt="Add to&nbsp;DZone" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://www.facebook.com/sharer.php?u=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fdjango-caching-backups%2F" rel="nofollow" title="Add to&nbsp;Facebook"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/facebook.png" title="Add to&nbsp;Facebook" alt="Add to&nbsp;Facebook" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://reddit.com/submit?url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fdjango-caching-backups%2F&amp;title=Django+Caching+with+Backups" rel="nofollow" title="Add to&nbsp;reddit"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/reddit.png" title="Add to&nbsp;reddit" alt="Add to&nbsp;reddit" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://www.stumbleupon.com/submit?url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fdjango-caching-backups%2F&amp;title=Django+Caching+with+Backups" rel="nofollow" title="Add to&nbsp;Stumble Upon"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/stumbleupon.png" title="Add to&nbsp;Stumble Upon" alt="Add to&nbsp;Stumble Upon" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://www.technorati.com/faves?add=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fdjango-caching-backups%2F" rel="nofollow" title="Add to&nbsp;Technorati"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/technorati.png" title="Add to&nbsp;Technorati" alt="Add to&nbsp;Technorati" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://twitter.com/home/?status=Check+out+Django+Caching+with+Backups+@+http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fdjango-caching-backups%2F" rel="nofollow" title="Add to&nbsp;Twitter"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/twitter.png" title="Add to&nbsp;Twitter" alt="Add to&nbsp;Twitter" /></a>
<br />
</div>
</div>
<!-- Social Bookmarks END -->
]]></content:encoded>
			<wfw:commentRss>http://www.artfulcode.net/articles/django-caching-backups/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Mutable default parameter values in Python</title>
		<link>http://www.artfulcode.net/articles/mutable-default-parameter-values-python/</link>
		<comments>http://www.artfulcode.net/articles/mutable-default-parameter-values-python/#comments</comments>
		<pubDate>Tue, 22 Jul 2008 15:09:51 +0000</pubDate>
		<dc:creator>Jeff</dc:creator>
				<category><![CDATA[Programming]]></category>
		<category><![CDATA[python]]></category>

		<guid isPermaLink="false">http://www.artfulcode.net/articles/mutable-default-parameter-values-python/</guid>
		<description><![CDATA[In Python, everything is an object. Variables are technically names for references to objects. Therefore, when passing an argument to a function, what is in fact being passed is the value of the reference. This leads to some useful techniques. Mutable and immutable objects as parameters The issue is muddied further by mutable and immutable [...]]]></description>
			<content:encoded><![CDATA[<p>In Python, everything is an object.  Variables are technically names for references to objects.  Therefore, when passing an argument to a function, what is in fact being passed is the value of the reference.  This leads to some useful techniques.<span id="more-17"></span></p>
<h4>Mutable and immutable objects as parameters</h4>
<p>The issue is muddied further by mutable and immutable objects.  When an immutable object (such as an integer or string) is passed to a function, it is <em>in effect</em> passed by value, since any change to the object creates a new object.  However, when the object is mutable, as are dictionaries and lists, any changes made to the object are destructive and reflected in the original object.  Operating on a list, for example, can be made safe by copying the list:</p>

<div class="wp_syntax"><div class="code"><pre class="python" style="font-family:monospace;"><span style="color: #ff7700;font-weight:bold;">def</span> nondestructive_modify<span style="color: black;">&#40;</span>mutable_list<span style="color: black;">&#41;</span>:
    copied = mutable_list<span style="color: black;">&#91;</span>:<span style="color: black;">&#93;</span>
    copied.<span style="color: black;">reverse</span><span style="color: black;">&#40;</span><span style="color: black;">&#41;</span>
    <span style="color: #ff7700;font-weight:bold;">return</span> copied</pre></div></div>

<p>Without copying the list, modifying <code>mutable_list</code> in this example would destructively modify the original list.  Moreover, strings are immutable in Python.  This is uncommon, and is a source of confusion for many users.  Strings need not be copied for safe manipulation in a function, but operations on large strings are slower because they involve allocating a new string.</p>
<h4>Immutable default parameter values</h4>
<p>Function definitions are executable statements.  This has an interesting side effect when combined with named parameters.  Take the following function, which uses an accumulator to recursively collect only even number from a list:</p>

<div class="wp_syntax"><div class="code"><pre class="python" style="font-family:monospace;"><span style="color: #ff7700;font-weight:bold;">def</span> only_evens<span style="color: black;">&#40;</span>items, acc=<span style="color: black;">&#91;</span><span style="color: black;">&#93;</span><span style="color: black;">&#41;</span>:
    <span style="color: #ff7700;font-weight:bold;">if</span> <span style="color: #008000;">len</span><span style="color: black;">&#40;</span>items<span style="color: black;">&#41;</span> <span style="color: #66cc66;">&amp;</span>gt<span style="color: #66cc66;">;</span> <span style="color: #ff4500;">0</span>:
        <span style="color: #ff7700;font-weight:bold;">if</span> <span style="color: #008000;">type</span><span style="color: black;">&#40;</span>items<span style="color: black;">&#91;</span><span style="color: #ff4500;">0</span><span style="color: black;">&#93;</span><span style="color: black;">&#41;</span> == <span style="color: #008000;">int</span> <span style="color: #ff7700;font-weight:bold;">and</span> items<span style="color: black;">&#91;</span><span style="color: #ff4500;">0</span><span style="color: black;">&#93;</span><span style="color: #66cc66;">%</span>2 == <span style="color: #ff4500;">0</span>:
            acc.<span style="color: black;">append</span><span style="color: black;">&#40;</span>items<span style="color: black;">&#91;</span><span style="color: #ff4500;">0</span><span style="color: black;">&#93;</span><span style="color: black;">&#41;</span>
        <span style="color: #ff7700;font-weight:bold;">return</span> only_evens<span style="color: black;">&#40;</span>items<span style="color: black;">&#91;</span><span style="color: #ff4500;">1</span>:<span style="color: black;">&#93;</span>, acc<span style="color: black;">&#41;</span>
    <span style="color: #ff7700;font-weight:bold;">else</span>:
        <span style="color: #ff7700;font-weight:bold;">return</span> acc</pre></div></div>

<p>The accumulator is defined in the parameter list, and seems to have a default value of <code>[]</code>.  In fact, <code>[]</code> is evaluated when the function definition is executed and <code>acc</code> is bound to a reference to that established list object, which starts out empty.  Each time the function is called, <code>acc</code> is <em>not</em> rebound to an empty list.  It remains bound to the same reference from call to call.  Therefore,</p>

<div class="wp_syntax"><div class="code"><pre class="python" style="font-family:monospace;">only_evens<span style="color: black;">&#40;</span><span style="color: black;">&#91;</span><span style="color: #ff4500;">0</span>, <span style="color: #ff4500;">1</span>, <span style="color: #ff4500;">2</span>, <span style="color: #ff4500;">3</span>, <span style="color: #ff4500;">4</span><span style="color: black;">&#93;</span><span style="color: black;">&#41;</span>
<span style="color: #808080; font-style: italic;"># =&amp;gt; [0, 2, 4]</span>
&nbsp;
only_evens<span style="color: black;">&#40;</span><span style="color: black;">&#91;</span><span style="color: #ff4500;">10</span>, <span style="color: #ff4500;">11</span>, <span style="color: #ff4500;">12</span>, <span style="color: #ff4500;">13</span>, <span style="color: #ff4500;">14</span><span style="color: black;">&#93;</span><span style="color: black;">&#41;</span>
<span style="color: #808080; font-style: italic;"># =&amp;gt; [0, 2, 4, 10, 12, 14]</span></pre></div></div>

<p>The convention to avoid this is to set the default value to <code>None</code>, and then test against that:</p>

<div class="wp_syntax"><div class="code"><pre class="python" style="font-family:monospace;"><span style="color: #ff7700;font-weight:bold;">def</span> only_evens<span style="color: black;">&#40;</span>items, acc=<span style="color: #008000;">None</span><span style="color: black;">&#41;</span>:
    <span style="color: #ff7700;font-weight:bold;">if</span> acc <span style="color: #ff7700;font-weight:bold;">is</span> <span style="color: #008000;">None</span>:
        acc = <span style="color: black;">&#91;</span><span style="color: black;">&#93;</span>
    <span style="color: #ff7700;font-weight:bold;">if</span> <span style="color: #008000;">len</span><span style="color: black;">&#40;</span>items<span style="color: black;">&#41;</span> <span style="color: #66cc66;">&amp;</span>gt<span style="color: #66cc66;">;</span> <span style="color: #ff4500;">0</span>:
        <span style="color: #ff7700;font-weight:bold;">if</span> <span style="color: #008000;">type</span><span style="color: black;">&#40;</span>items<span style="color: black;">&#91;</span><span style="color: #ff4500;">0</span><span style="color: black;">&#93;</span><span style="color: black;">&#41;</span> == <span style="color: #008000;">int</span> <span style="color: #ff7700;font-weight:bold;">and</span> items<span style="color: black;">&#91;</span><span style="color: #ff4500;">0</span><span style="color: black;">&#93;</span><span style="color: #66cc66;">%</span>2 == <span style="color: #ff4500;">0</span>:
            acc.<span style="color: black;">append</span><span style="color: black;">&#40;</span>items<span style="color: black;">&#91;</span><span style="color: #ff4500;">0</span><span style="color: black;">&#93;</span><span style="color: black;">&#41;</span>
        <span style="color: #ff7700;font-weight:bold;">return</span> only_evens<span style="color: black;">&#40;</span>items<span style="color: black;">&#91;</span><span style="color: #ff4500;">1</span>:<span style="color: black;">&#93;</span>, acc<span style="color: black;">&#41;</span>
    <span style="color: #ff7700;font-weight:bold;">else</span>:
        <span style="color: #ff7700;font-weight:bold;">return</span> acc</pre></div></div>

<p>One advantage of this is that mutable default parameters can be used to create a <a href="http://en.wikipedia.org/wiki/Closure_(computer_science)">closure</a>) (granted, this example is inefficient and contrived only as an illustration; the better solution here is to use a generator):</p>

<div class="wp_syntax"><div class="code"><pre class="python" style="font-family:monospace;"><span style="color: #ff7700;font-weight:bold;">def</span> counter<span style="color: black;">&#40;</span>n=<span style="color: black;">&#91;</span><span style="color: #ff4500;">0</span><span style="color: black;">&#93;</span><span style="color: black;">&#41;</span>:
    n<span style="color: black;">&#91;</span><span style="color: #ff4500;">0</span><span style="color: black;">&#93;</span> = n<span style="color: black;">&#91;</span><span style="color: #ff4500;">0</span><span style="color: black;">&#93;</span> + <span style="color: #ff4500;">1</span>
    <span style="color: #ff7700;font-weight:bold;">return</span> n<span style="color: black;">&#91;</span><span style="color: #ff4500;">0</span><span style="color: black;">&#93;</span>
&nbsp;
counter<span style="color: black;">&#40;</span><span style="color: black;">&#41;</span> <span style="color: #808080; font-style: italic;"># =&amp;gt; 1</span>
counter<span style="color: black;">&#40;</span><span style="color: black;">&#41;</span> <span style="color: #808080; font-style: italic;"># =&amp;gt; 2</span>
counter<span style="color: black;">&#40;</span><span style="color: black;">&#41;</span> <span style="color: #808080; font-style: italic;"># =&amp;gt; 3</span></pre></div></div>

<p>Another use is to <a href="http://en.wikipedia.org/wiki/Memoization">memoize</a> a function:</p>

<div class="wp_syntax"><div class="code"><pre class="python" style="font-family:monospace;"><span style="color: #ff7700;font-weight:bold;">def</span> factorial<span style="color: black;">&#40;</span>n, seen=<span style="color: black;">&#123;</span><span style="color: black;">&#125;</span><span style="color: black;">&#41;</span>:
    <span style="color: #ff7700;font-weight:bold;">if</span> n <span style="color: #66cc66;">&amp;</span>gt<span style="color: #66cc66;">;</span>= <span style="color: #ff4500;">0</span>:
        <span style="color: #ff7700;font-weight:bold;">return</span> <span style="color: #ff4500;">1</span>
    <span style="color: #ff7700;font-weight:bold;">if</span> seen.<span style="color: black;">has_key</span><span style="color: black;">&#40;</span>n<span style="color: black;">&#41;</span>:
        <span style="color: #ff7700;font-weight:bold;">return</span> seen<span style="color: black;">&#91;</span>n<span style="color: black;">&#93;</span>
    <span style="color: #ff7700;font-weight:bold;">else</span>:
        seen<span style="color: black;">&#91;</span>n<span style="color: black;">&#93;</span> = n <span style="color: #66cc66;">*</span> factorial<span style="color: black;">&#40;</span>n-<span style="color: #ff4500;">1</span><span style="color: black;">&#41;</span>
        <span style="color: #ff7700;font-weight:bold;">return</span> seen<span style="color: black;">&#91;</span>n<span style="color: black;">&#93;</span></pre></div></div>

<p>This has implications in threaded code.  The closed over variable is shared among function calls; access to it in threaded code should be guarded as if it were a global variable.</p>
<p>More information:</p>
<ul>
<li> <a href="http://docs.python.org/ref/function.html">Python function documentation</a></li>
<li> <a href="http://www.testingreflections.com/node/view/5126">Reference/value passing in Python</a></li>
<li> <a href="http://groups.google.com/group/comp.lang.python/browse_thread/thread/46680e91e2d04d9a/c23bc6f95d0668b3#c23bc6f95d0668b3">Discussion of default parameters on C.L.P.</a></li>
<li> <a href="http://effbot.org/zone/default-values.htm">Default Parameter Values in Python (effbot.org)</a></li>
</ul>
<!-- Social Bookmarks BEGIN -->
<div class="social_bookmark">
<a><strong><em>Submit article</em></strong></a>
<br />
<div class="d">
<br />
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://del.icio.us/post?url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fmutable-default-parameter-values-python%2F&amp;title=Mutable+default+parameter+values+in+Python" rel="nofollow" title="Add to&nbsp;Del.icio.us"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/delicious.png" title="Add to&nbsp;Del.icio.us" alt="Add to&nbsp;Del.icio.us" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://digg.com/submit?phase=2&amp;url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fmutable-default-parameter-values-python%2F&amp;title=Mutable+default+parameter+values+in+Python" rel="nofollow" title="Add to&nbsp;digg"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/digg.png" title="Add to&nbsp;digg" alt="Add to&nbsp;digg" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://www.dzone.com/links/add.html?description=Mutable+default+parameter+values+in+Python&amp;url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fmutable-default-parameter-values-python%2F&amp;title=Mutable+default+parameter+values+in+Python" rel="nofollow" title="Add to&nbsp;DZone"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/dzone.png" title="Add to&nbsp;DZone" alt="Add to&nbsp;DZone" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://www.facebook.com/sharer.php?u=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fmutable-default-parameter-values-python%2F" rel="nofollow" title="Add to&nbsp;Facebook"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/facebook.png" title="Add to&nbsp;Facebook" alt="Add to&nbsp;Facebook" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://reddit.com/submit?url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fmutable-default-parameter-values-python%2F&amp;title=Mutable+default+parameter+values+in+Python" rel="nofollow" title="Add to&nbsp;reddit"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/reddit.png" title="Add to&nbsp;reddit" alt="Add to&nbsp;reddit" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://www.stumbleupon.com/submit?url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fmutable-default-parameter-values-python%2F&amp;title=Mutable+default+parameter+values+in+Python" rel="nofollow" title="Add to&nbsp;Stumble Upon"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/stumbleupon.png" title="Add to&nbsp;Stumble Upon" alt="Add to&nbsp;Stumble Upon" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://www.technorati.com/faves?add=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fmutable-default-parameter-values-python%2F" rel="nofollow" title="Add to&nbsp;Technorati"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/technorati.png" title="Add to&nbsp;Technorati" alt="Add to&nbsp;Technorati" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://twitter.com/home/?status=Check+out+Mutable+default+parameter+values+in+Python+@+http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fmutable-default-parameter-values-python%2F" rel="nofollow" title="Add to&nbsp;Twitter"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/twitter.png" title="Add to&nbsp;Twitter" alt="Add to&nbsp;Twitter" /></a>
<br />
</div>
</div>
<!-- Social Bookmarks END -->
]]></content:encoded>
			<wfw:commentRss>http://www.artfulcode.net/articles/mutable-default-parameter-values-python/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Extending Python with (almost) anything</title>
		<link>http://www.artfulcode.net/articles/extending-python-almost-anything/</link>
		<comments>http://www.artfulcode.net/articles/extending-python-almost-anything/#comments</comments>
		<pubDate>Mon, 24 Mar 2008 19:42:00 +0000</pubDate>
		<dc:creator>Jeff</dc:creator>
				<category><![CDATA[Software]]></category>
		<category><![CDATA[c]]></category>
		<category><![CDATA[newlisp]]></category>
		<category><![CDATA[python]]></category>

		<guid isPermaLink="false">http://www.artfulcode.net/articles/extending-python-almost-anything/</guid>
		<description><![CDATA[Among the various methods of extending Python about which I have written, Python 2.5&#8242;s ctypes module is possibly the easiest. Libraries written in many languages beyond C, including ECL, may be compiled into a shared library or DLL. Many other languages, like Lua and newLISP, include simple DLLs of their own to embed their interpreters [...]]]></description>
			<content:encoded><![CDATA[<p>Among the <a href="http://www.artfulcode.net/articles/extending-python-pyrex/">various methods</a> of <a href="/articles/writing-python-modules-c-swig/">extending Python</a> about which I have written, Python 2.5&#8242;s <code>ctypes</code> module is possibly the easiest.<span id="more-28"></span></p>
<p>Libraries written in many languages beyond C, including <a href="http://ecls.sourceforge.net/">ECL</a>, may be compiled into a shared library or DLL.  Many other languages, like <a href="http://www.lua.org/">Lua</a> and <a href="http://www.newlisp.org">newLISP</a>, include simple DLLs of their own to embed their interpreters in an application.</p>
<p><code>ctypes</code> allows you to wrap a shared library in pure Python.  Here is a quick example to get you started.  newLISP exports a simple function, <code>newlispEvalStr</code>, which accepts a string of newLISP code and returns the result of the code&#8217;s evaluation as a string.  To compile newLISP as a shared library, use the <code>PLATFORM_lib</code> variant of your system&#8217;s <code>make</code> target (quick note: <code>make install</code> does not install the shared library; it must be manually copied to your path).  The helper function, <code>find_library</code> (located in <code>ctypes.util</code>), gives you a platform-independent way of finding the needed path string for a library.</p>

<div class="wp_syntax"><div class="code"><pre class="python" style="font-family:monospace;"><span style="color: #ff7700;font-weight:bold;">from</span> ctypes <span style="color: #ff7700;font-weight:bold;">import</span> <span style="color: #66cc66;">*</span>
<span style="color: #ff7700;font-weight:bold;">from</span> ctypes.<span style="color: black;">util</span> <span style="color: #ff7700;font-weight:bold;">import</span> find_library
&nbsp;
lib_path = find_library<span style="color: black;">&#40;</span><span style="color: #483d8b;">'newlisp'</span><span style="color: black;">&#41;</span>
newlisp = CDLL<span style="color: black;">&#40;</span>lib_path<span style="color: black;">&#41;</span></pre></div></div>

<p><code>CDLL</code> extends <code>LoadLibrary</code>, the primary utility class in <code>ctypes</code>.  In Windows, there are also <code>OleDLL</code> and <code>WinDll</code>.  All of these release the global interpreter lock when a foreign function is entered and reclaim it when the function call completes.  There is also a <code>PyDLL</code> class that does not release the GIL; its main purpose is the provide direct access to the Python C API from within your Python application.</p>
<p><code>newlispEvalStr</code> is accessed as a method of the CDLL instance.  As with any low level interface, it is important to know the library you are using.  Accessing invalid function names can lead to unexpected results or system instability.</p>

<div class="wp_syntax"><div class="code"><pre class="python" style="font-family:monospace;">nl_eval = newlisp.<span style="color: black;">newlispEvalStr</span></pre></div></div>

<p><code>nl_eval</code> is now an instance of the class <code>_FuncPtr</code>.  To be safe, we want to make sure that only strings get passed to the function and to access the returned data as a string.<br />
<code>ctypes</code> provides the convenience properties <code>argtypes</code> and <code>restype</code> for this purpose.<br />
<code>argtypes</code> is set to lists of argument types (using the c-mapped types <a href="http://docs.python.org/lib/node453.html">here</a>).<br />
<code>restype</code> is a single c type:</p>

<div class="wp_syntax"><div class="code"><pre class="python" style="font-family:monospace;">nl_eval.<span style="color: black;">argtypes</span> = <span style="color: black;">&#91;</span>c_char_p<span style="color: black;">&#93;</span>
nl_eval.<span style="color: black;">restype</span> = c_char_p</pre></div></div>

<p>We are now ready to evaluate newLISP code:</p>

<div class="wp_syntax"><div class="code"><pre class="python" style="font-family:monospace;">nl_eval<span style="color: black;">&#40;</span><span style="color: #483d8b;">'(+ 2 2)'</span><span style="color: black;">&#41;</span> <span style="color: #808080; font-style: italic;"># =&amp;gt; '4'</span>
nl_eval<span style="color: black;">&#40;</span><span style="color: #483d8b;">&quot;(apply + '(4 3 2 5 4))&quot;</span><span style="color: black;">&#41;</span> <span style="color: #808080; font-style: italic;"># =&amp;gt; '18'</span></pre></div></div>

<p>A more general function can be defined to coerce the desired result type after <code>nl_eval</code> returns:<br />
&lt;</p>

<div class="wp_syntax"><div class="code"><pre class="python" style="font-family:monospace;"><span style="color: #ff7700;font-weight:bold;">def</span> newlisp_eval<span style="color: black;">&#40;</span><span style="color: #dc143c;">code</span>, return_type=<span style="color: #008000;">str</span><span style="color: black;">&#41;</span>:
    <span style="color: #ff7700;font-weight:bold;">return</span> return_type<span style="color: black;">&#40;</span>nl_eval<span style="color: black;">&#40;</span><span style="color: #dc143c;">code</span><span style="color: black;">&#41;</span><span style="color: black;">&#41;</span>
&nbsp;
newlisp_eval<span style="color: black;">&#40;</span><span style="color: #483d8b;">'(+ 2 2)'</span>, <span style="color: #008000;">int</span><span style="color: black;">&#41;</span> <span style="color: #808080; font-style: italic;"># =&amp;gt; 4</span></pre></div></div>

<p>What&#8217;s even better is that because the GIL is released and the library (in this instance) is not directly modifying any Python data, we can make effective use of threading.</p>
<p>Finally, here is a simple module putting all this together:</p>

<div class="wp_syntax"><div class="code"><pre class="python" style="font-family:monospace;"><span style="color: #ff7700;font-weight:bold;">from</span> ctypes <span style="color: #ff7700;font-weight:bold;">import</span> <span style="color: #66cc66;">*</span>
<span style="color: #ff7700;font-weight:bold;">from</span> ctypes.<span style="color: black;">util</span> <span style="color: #ff7700;font-weight:bold;">import</span> find_library
&nbsp;
<span style="color: #ff7700;font-weight:bold;">class</span> LibraryNotFound<span style="color: black;">&#40;</span><span style="color: #008000;">Exception</span><span style="color: black;">&#41;</span>:
    <span style="color: #ff7700;font-weight:bold;">pass</span>
&nbsp;
<span style="color: #ff7700;font-weight:bold;">class</span> newLISP<span style="color: black;">&#40;</span><span style="color: #008000;">object</span><span style="color: black;">&#41;</span>:
    <span style="color: #ff7700;font-weight:bold;">def</span> <span style="color: #0000cd;">__init__</span><span style="color: black;">&#40;</span><span style="color: #008000;">self</span><span style="color: black;">&#41;</span>:
        found = find_library<span style="color: black;">&#40;</span><span style="color: #483d8b;">'newlisp'</span><span style="color: black;">&#41;</span>
        <span style="color: #ff7700;font-weight:bold;">if</span> found <span style="color: #ff7700;font-weight:bold;">is</span> <span style="color: #008000;">None</span>:
            <span style="color: #ff7700;font-weight:bold;">raise</span> LibraryNotFound<span style="color: black;">&#40;</span><span style="color: black;">&#41;</span>
        <span style="color: #ff7700;font-weight:bold;">else</span>:
            <span style="color: #008000;">self</span>.<span style="color: black;">lib</span> = CDLL<span style="color: black;">&#40;</span>found<span style="color: black;">&#41;</span>
            <span style="color: #008000;">self</span>._eval = <span style="color: #008000;">self</span>.<span style="color: black;">lib</span>.<span style="color: black;">newlispEvalStr</span>
            <span style="color: #008000;">self</span>._eval.<span style="color: black;">argtypes</span> = <span style="color: black;">&#91;</span>c_char_p<span style="color: black;">&#93;</span>
            <span style="color: #008000;">self</span>._eval.<span style="color: black;">restype</span> = c_char_p
&nbsp;
    <span style="color: #ff7700;font-weight:bold;">def</span> <span style="color: #008000;">eval</span><span style="color: black;">&#40;</span><span style="color: #008000;">self</span>, <span style="color: #dc143c;">code</span>, return_type=<span style="color: #008000;">str</span><span style="color: black;">&#41;</span>:
        <span style="color: #dc143c;">code</span> = c_char_p<span style="color: black;">&#40;</span><span style="color: #dc143c;">code</span><span style="color: black;">&#41;</span>
        result = <span style="color: #008000;">self</span>._eval<span style="color: black;">&#40;</span><span style="color: #dc143c;">code</span><span style="color: black;">&#41;</span>
        <span style="color: #ff7700;font-weight:bold;">if</span> result:
            result = result.<span style="color: black;">strip</span><span style="color: black;">&#40;</span><span style="color: black;">&#41;</span>
            <span style="color: #ff7700;font-weight:bold;">return</span> return_type<span style="color: black;">&#40;</span>result<span style="color: black;">&#41;</span>
        <span style="color: #ff7700;font-weight:bold;">else</span>:
            <span style="color: #ff7700;font-weight:bold;">return</span> <span style="color: #008000;">None</span></pre></div></div>

<p>Links:</p>
<ul>
<li> <a href="http://docs.python.org/lib/module-ctypes.html">Python ctypes documentation</a></li>
</ul>
<!-- Social Bookmarks BEGIN -->
<div class="social_bookmark">
<a><strong><em>Submit article</em></strong></a>
<br />
<div class="d">
<br />
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://del.icio.us/post?url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fextending-python-almost-anything%2F&amp;title=Extending+Python+with+%28almost%29+anything" rel="nofollow" title="Add to&nbsp;Del.icio.us"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/delicious.png" title="Add to&nbsp;Del.icio.us" alt="Add to&nbsp;Del.icio.us" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://digg.com/submit?phase=2&amp;url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fextending-python-almost-anything%2F&amp;title=Extending+Python+with+%28almost%29+anything" rel="nofollow" title="Add to&nbsp;digg"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/digg.png" title="Add to&nbsp;digg" alt="Add to&nbsp;digg" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://www.dzone.com/links/add.html?description=Extending+Python+with+%28almost%29+anything&amp;url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fextending-python-almost-anything%2F&amp;title=Extending+Python+with+%28almost%29+anything" rel="nofollow" title="Add to&nbsp;DZone"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/dzone.png" title="Add to&nbsp;DZone" alt="Add to&nbsp;DZone" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://www.facebook.com/sharer.php?u=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fextending-python-almost-anything%2F" rel="nofollow" title="Add to&nbsp;Facebook"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/facebook.png" title="Add to&nbsp;Facebook" alt="Add to&nbsp;Facebook" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://reddit.com/submit?url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fextending-python-almost-anything%2F&amp;title=Extending+Python+with+%28almost%29+anything" rel="nofollow" title="Add to&nbsp;reddit"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/reddit.png" title="Add to&nbsp;reddit" alt="Add to&nbsp;reddit" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://www.stumbleupon.com/submit?url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fextending-python-almost-anything%2F&amp;title=Extending+Python+with+%28almost%29+anything" rel="nofollow" title="Add to&nbsp;Stumble Upon"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/stumbleupon.png" title="Add to&nbsp;Stumble Upon" alt="Add to&nbsp;Stumble Upon" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://www.technorati.com/faves?add=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fextending-python-almost-anything%2F" rel="nofollow" title="Add to&nbsp;Technorati"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/technorati.png" title="Add to&nbsp;Technorati" alt="Add to&nbsp;Technorati" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://twitter.com/home/?status=Check+out+Extending+Python+with+%28almost%29+anything+@+http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fextending-python-almost-anything%2F" rel="nofollow" title="Add to&nbsp;Twitter"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/twitter.png" title="Add to&nbsp;Twitter" alt="Add to&nbsp;Twitter" /></a>
<br />
</div>
</div>
<!-- Social Bookmarks END -->
]]></content:encoded>
			<wfw:commentRss>http://www.artfulcode.net/articles/extending-python-almost-anything/feed/</wfw:commentRss>
		<slash:comments>3</slash:comments>
		</item>
		<item>
		<title>Threading in Django</title>
		<link>http://www.artfulcode.net/articles/threading-django/</link>
		<comments>http://www.artfulcode.net/articles/threading-django/#comments</comments>
		<pubDate>Fri, 07 Mar 2008 16:46:00 +0000</pubDate>
		<dc:creator>Jeff</dc:creator>
				<category><![CDATA[Software]]></category>
		<category><![CDATA[django]]></category>
		<category><![CDATA[python]]></category>
		<category><![CDATA[threads]]></category>

		<guid isPermaLink="false">http://www.artfulcode.net/articles/threading-django/</guid>
		<description><![CDATA[Profitable use of threading in web development is rare, particularly when contending with Python&#8217;s global interpreter lock. There are a few notable exceptions to this. The global interpreter lock (GIL) is the method that Python uses to maintain internal thread-safety when not all Python objects are thread-safe. The GIL ensures that no Python object may [...]]]></description>
			<content:encoded><![CDATA[<p>Profitable use of threading in web development is rare, particularly when contending with Python&#8217;s global interpreter lock.  There are a few notable exceptions to this.<span id="more-30"></span></p>
<p>The global interpreter lock (GIL) is the method that Python uses to maintain internal thread-safety when not all Python objects are thread-safe.  The GIL ensures that no Python object may be modified by multiple threads at the same time.  This is transparent to the Python programmer; the GIL is locking internal structures.  A list that is modified by multiple threads without explicit locking will still behave erratically.</p>
<p>This makes threading less useful in web development.  Because Python can not directly maintain state between reqeusts (indirectly, via the client, state may be maintained using session cookies or other similar strategies), each request is a new thread:</p>
<pre><code> 1. Determine what data is being requested
 2. Get the data
 3. Format the data
 4. Send the data
</code></pre>
<p>Each of these steps depends on the state information from the previous step.  There is not a lot of room for asynchronicity.  The usefulness of threading is pushed back to the web server.  That is where integrated server/framework solutions, like <a href="http://www.ocsigen.org/">Ocsigen</a>, have an advantage.</p>
<p>Many web-based applications, however, are not simple database front ends.  There are a few common situations where threading becomes useful; specifically, when there are side effects of a request.</p>
<p>On a side note, experts will tell you that only POST request should result in side effects.  This is misleading.  POST should certainly be used for user-intended side effects.  However, what happends when there is an error loading a page?</p>
<h4>Error messages</h4>
<p>In the event of a coding or environment error, Django sends me an email.  This is a wonderful (and an occassionally annoying) feature.  But what if the problem is caused by the data we input, rather than the code?  I can detect this in my code and present the user with a nice error page apologizing for the inconvenience, but now I want to notify the content team that an entry in the database is invalid in some way.</p>
<p>Sending a message, especially if Django is connecting to a remote mail server, can take a while.  This is a good use for threading.  The thread should deal with as little data processing as possible, concentrating instead on the side effect.</p>

<div class="wp_syntax"><div class="code"><pre class="python" style="font-family:monospace;"><span style="color: #ff7700;font-weight:bold;">import</span> <span style="color: #dc143c;">threading</span>
<span style="color: #ff7700;font-weight:bold;">from</span> django.<span style="color: black;">core</span>.<span style="color: black;">mail</span> <span style="color: #ff7700;font-weight:bold;">import</span> send_mail
&nbsp;
<span style="color: #ff7700;font-weight:bold;">def</span> foo<span style="color: black;">&#40;</span>request, identifier<span style="color: black;">&#41;</span>:
    data = get_some_data<span style="color: black;">&#40;</span>identifier<span style="color: black;">&#41;</span>
    <span style="color: #ff7700;font-weight:bold;">try</span>:
        error_check<span style="color: black;">&#40;</span>data<span style="color: black;">&#41;</span> <span style="color: #808080; font-style: italic;"># raise exception if data invalid</span>
    <span style="color: #ff7700;font-weight:bold;">except</span> InvalidData, e:
        <span style="color: #808080; font-style: italic;"># Format our information here, in the main thread</span>
        subject = <span style="color: #483d8b;">&quot;Invalid data in entry %d&quot;</span> <span style="color: #66cc66;">%</span> identifier
        message = <span style="color: #483d8b;">&quot;&quot;&quot;
            There is an error in entry %d.  Please check this
            data at http://path/to/django/admin/app/table/%d.
        &quot;&quot;&quot;</span> <span style="color: #66cc66;">%</span> <span style="color: black;">&#40;</span>identifier, identifier<span style="color: black;">&#41;</span>
        recipients = <span style="color: black;">&#91;</span><span style="color: #483d8b;">'someone@somewere.com'</span>, <span style="color: #483d8b;">'someoneelse@somewhere.com'</span><span style="color: black;">&#93;</span>
        <span style="color: #ff7700;font-weight:bold;">from</span> = <span style="color: #483d8b;">'root@server.com'</span>
&nbsp;
        <span style="color: #808080; font-style: italic;"># Create a new thread in Daemon mode to send message</span>
        t = <span style="color: #dc143c;">threading</span>.<span style="color: black;">Thread</span><span style="color: black;">&#40;</span>target=send_mail,
                             args=<span style="color: black;">&#91;</span>subject, message, <span style="color: #ff7700;font-weight:bold;">from</span>, recipients<span style="color: black;">&#93;</span>,
                             kwargs=<span style="color: black;">&#123;</span><span style="color: #483d8b;">'fail_silently'</span>: <span style="color: #008000;">True</span><span style="color: black;">&#125;</span><span style="color: black;">&#41;</span>
        t.<span style="color: black;">setDaemon</span><span style="color: black;">&#40;</span><span style="color: #008000;">True</span><span style="color: black;">&#41;</span>
        t.<span style="color: black;">start</span><span style="color: black;">&#40;</span><span style="color: black;">&#41;</span>
    <span style="color: #ff7700;font-weight:bold;">return</span> HttpResponseServerError<span style="color: black;">&#40;</span>some_error_page<span style="color: black;">&#41;</span></pre></div></div>

<p>Note particularly that we use <code>t.setDaemon(True)</code> on the thread before starting it.  This tells Python not to wait for the thread to exit before returning data to the client.</p>
<h4>Writing files</h4>
<p>Many of our web-based programs write data to static files.  Some applications maintain logs.  Others publish static HTML files to the enterprise server.  These are great uses for Python threads, since the GIL is released during file IO operations.</p>
<h4>Caching remote data</h4>
<p>We are a large business and applications are developed by various (often isolated) units.  I often find myself in a situation where my application depends on mutable data extracted from another system but does not have direct access to the data&#8217;s database.</p>
<p>Rather than depend on low internal network latencies and download the data extract on each page load, I cache the data and keep a local copy, using a script to synchronize the data.  For simple data, this can be stored using Django&#8217;s low level cache.  But if some state needs to be maintained for that data, a database table is a better method.</p>
<p>When there are a large number of such objects, a scheduled update process becomes burdensome, especially if the number of objects increases regularly.  Therefore, the solution is to trigger the event when the data is accessed (via a page load).</p>
<p>Each cached database entry gets a timestamp field to note its last update.  The model defines an update method and the model manager&#8217;s <code>get()</code> method checks the entry&#8217;s timestamp to see if the data needs to be freshened.  The update routine is called in a separate thread.</p>
<p>Warning: in Django, when a model instance is found via a relationship, the manager&#8217;s <code>get()</code> method is not called!  This is because these objects are accessed via a <code>django.db.models.fields.related.ManyRelatedManager</code>, rather than the model&#8217;s own manager.  If you wish to solve this without duplicating code, look into Django <code>signals</code>.</p>
<p>The problem is that this is not a simple side effect and the GIL will get in the way.  The solution is to do all of the necessary logic to determine if the object needs to be updated in the main thread.  Just before returning the response to the user, the update thread is started in Daemon mode.  That minimizes competition for the GIL.  The user will not get the updated version of the object on this page load, but the next user will.</p>
<p>The other alternative is to use <code>os.fork</code> or the <code>subprocess</code> module to launch the update routine in a separate process (with its own interpreter instance and its own GIL).  This sort of solution gets used a lot in PHP because of its lack of threads.  In Python, threads tend to be more useful and much less resource hungry.</p>
<!-- Social Bookmarks BEGIN -->
<div class="social_bookmark">
<a><strong><em>Submit article</em></strong></a>
<br />
<div class="d">
<br />
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://del.icio.us/post?url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fthreading-django%2F&amp;title=Threading+in+Django" rel="nofollow" title="Add to&nbsp;Del.icio.us"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/delicious.png" title="Add to&nbsp;Del.icio.us" alt="Add to&nbsp;Del.icio.us" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://digg.com/submit?phase=2&amp;url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fthreading-django%2F&amp;title=Threading+in+Django" rel="nofollow" title="Add to&nbsp;digg"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/digg.png" title="Add to&nbsp;digg" alt="Add to&nbsp;digg" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://www.dzone.com/links/add.html?description=Threading+in+Django&amp;url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fthreading-django%2F&amp;title=Threading+in+Django" rel="nofollow" title="Add to&nbsp;DZone"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/dzone.png" title="Add to&nbsp;DZone" alt="Add to&nbsp;DZone" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://www.facebook.com/sharer.php?u=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fthreading-django%2F" rel="nofollow" title="Add to&nbsp;Facebook"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/facebook.png" title="Add to&nbsp;Facebook" alt="Add to&nbsp;Facebook" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://reddit.com/submit?url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fthreading-django%2F&amp;title=Threading+in+Django" rel="nofollow" title="Add to&nbsp;reddit"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/reddit.png" title="Add to&nbsp;reddit" alt="Add to&nbsp;reddit" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://www.stumbleupon.com/submit?url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fthreading-django%2F&amp;title=Threading+in+Django" rel="nofollow" title="Add to&nbsp;Stumble Upon"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/stumbleupon.png" title="Add to&nbsp;Stumble Upon" alt="Add to&nbsp;Stumble Upon" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://www.technorati.com/faves?add=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fthreading-django%2F" rel="nofollow" title="Add to&nbsp;Technorati"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/technorati.png" title="Add to&nbsp;Technorati" alt="Add to&nbsp;Technorati" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://twitter.com/home/?status=Check+out+Threading+in+Django+@+http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fthreading-django%2F" rel="nofollow" title="Add to&nbsp;Twitter"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/twitter.png" title="Add to&nbsp;Twitter" alt="Add to&nbsp;Twitter" /></a>
<br />
</div>
</div>
<!-- Social Bookmarks END -->
]]></content:encoded>
			<wfw:commentRss>http://www.artfulcode.net/articles/threading-django/feed/</wfw:commentRss>
		<slash:comments>7</slash:comments>
		</item>
		<item>
		<title>Psyco and Django</title>
		<link>http://www.artfulcode.net/articles/psyco-and-django/</link>
		<comments>http://www.artfulcode.net/articles/psyco-and-django/#comments</comments>
		<pubDate>Fri, 15 Feb 2008 22:25:06 +0000</pubDate>
		<dc:creator>Jeff</dc:creator>
				<category><![CDATA[Programming]]></category>
		<category><![CDATA[Tips]]></category>
		<category><![CDATA[django]]></category>
		<category><![CDATA[python]]></category>

		<guid isPermaLink="false">http://www.artfulcode.net/articles/psyco-and-django/</guid>
		<description><![CDATA[Psyco is a module that optimizes Python applications on the fly. Numerous resources online describe how to use psyco in a Django-powered application to speed it up. My experiences with this has been less than wonderful. I wrote a simple middleware class to import psyco as suggested here: # Be sure to only load on [...]]]></description>
			<content:encoded><![CDATA[<p>Psyco is a module that optimizes Python applications on the fly.  Numerous resources online describe how to use psyco in a Django-powered application to speed it up.<span id="more-34"></span></p>
<p>My experiences with this has been less than wonderful.  I wrote a simple middleware class to import psyco as suggested <a href="http://code.djangoproject.com/wiki/PsycoMiddleware">here</a>:</p>

<div class="wp_syntax"><div class="code"><pre class="python" style="font-family:monospace;"><span style="color: #808080; font-style: italic;"># Be sure to only load on the proper architecture</span>
<span style="color: #ff7700;font-weight:bold;">from</span> <span style="color: #dc143c;">platform</span> <span style="color: #ff7700;font-weight:bold;">import</span> architecture
<span style="color: #ff7700;font-weight:bold;">if</span> <span style="color: #ff7700;font-weight:bold;">not</span> settings.<span style="color: black;">DEBUG</span> <span style="color: #ff7700;font-weight:bold;">and</span> architecture<span style="color: black;">&#40;</span><span style="color: black;">&#41;</span><span style="color: black;">&#91;</span><span style="color: #ff4500;">0</span><span style="color: black;">&#93;</span> == <span style="color: #483d8b;">'32bit'</span>:
    <span style="color: #ff7700;font-weight:bold;">if</span> architecture<span style="color: black;">&#40;</span><span style="color: black;">&#41;</span><span style="color: black;">&#91;</span><span style="color: #ff4500;">0</span><span style="color: black;">&#93;</span> == <span style="color: #483d8b;">'32bit'</span>:
        <span style="color: #ff7700;font-weight:bold;">try</span>:
            <span style="color: #ff7700;font-weight:bold;">import</span> psyco
        <span style="color: #ff7700;font-weight:bold;">except</span> <span style="color: #008000;">ImportError</span>:
            <span style="color: #ff7700;font-weight:bold;">pass</span>
&nbsp;
<span style="color: #ff7700;font-weight:bold;">class</span> PsycoMiddleware<span style="color: black;">&#40;</span><span style="color: #008000;">object</span><span style="color: black;">&#41;</span>:
    <span style="color: #ff7700;font-weight:bold;">def</span> process_request<span style="color: black;">&#40;</span><span style="color: #008000;">self</span>, request<span style="color: black;">&#41;</span>:
        <span style="color: #808080; font-style: italic;"># Do not waste time trying to optimize the re module</span>
        psyco.<span style="color: black;">cannotcompile</span><span style="color: black;">&#40;</span><span style="color: #dc143c;">re</span>.<span style="color: #008000;">compile</span><span style="color: black;">&#41;</span>
        <span style="color: #808080; font-style: italic;"># Limit memory usage</span>
        psyco.<span style="color: #dc143c;">profile</span><span style="color: black;">&#40;</span>memory=<span style="color: #ff4500;">2048</span><span style="color: black;">&#41;</span>
        <span style="color: #ff7700;font-weight:bold;">return</span> <span style="color: #008000;">None</span></pre></div></div>

<p>In development this appeared to work fine (we excluded the <code>if not settings.DEBUG</code> portion while testing).  Once we were up and running on the production server, however, mysterious exceptions began to surface.</p>
<p>In particular, we seemed to be &#8220;missing&#8221; on some page hits.  Specifically, we would get TemplateDoesNotExist exceptions when the templates did, in fact, exist.  We were never able to sort that out.  They immediately stopped when we commented out the psyco-related stuff.  We tried changing the memory usage, excluding more items from compilation (including the Django loader classes), all to no avail.</p>
<p>My recommendation is avoidance of psyco in a Django setting.  If someone has an idea what might have caused this, I&#8217;d love to hear an explanation.</p>
<!-- Social Bookmarks BEGIN -->
<div class="social_bookmark">
<a><strong><em>Submit article</em></strong></a>
<br />
<div class="d">
<br />
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://del.icio.us/post?url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fpsyco-and-django%2F&amp;title=Psyco+and+Django" rel="nofollow" title="Add to&nbsp;Del.icio.us"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/delicious.png" title="Add to&nbsp;Del.icio.us" alt="Add to&nbsp;Del.icio.us" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://digg.com/submit?phase=2&amp;url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fpsyco-and-django%2F&amp;title=Psyco+and+Django" rel="nofollow" title="Add to&nbsp;digg"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/digg.png" title="Add to&nbsp;digg" alt="Add to&nbsp;digg" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://www.dzone.com/links/add.html?description=Psyco+and+Django&amp;url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fpsyco-and-django%2F&amp;title=Psyco+and+Django" rel="nofollow" title="Add to&nbsp;DZone"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/dzone.png" title="Add to&nbsp;DZone" alt="Add to&nbsp;DZone" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://www.facebook.com/sharer.php?u=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fpsyco-and-django%2F" rel="nofollow" title="Add to&nbsp;Facebook"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/facebook.png" title="Add to&nbsp;Facebook" alt="Add to&nbsp;Facebook" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://reddit.com/submit?url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fpsyco-and-django%2F&amp;title=Psyco+and+Django" rel="nofollow" title="Add to&nbsp;reddit"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/reddit.png" title="Add to&nbsp;reddit" alt="Add to&nbsp;reddit" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://www.stumbleupon.com/submit?url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fpsyco-and-django%2F&amp;title=Psyco+and+Django" rel="nofollow" title="Add to&nbsp;Stumble Upon"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/stumbleupon.png" title="Add to&nbsp;Stumble Upon" alt="Add to&nbsp;Stumble Upon" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://www.technorati.com/faves?add=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fpsyco-and-django%2F" rel="nofollow" title="Add to&nbsp;Technorati"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/technorati.png" title="Add to&nbsp;Technorati" alt="Add to&nbsp;Technorati" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://twitter.com/home/?status=Check+out+Psyco+and+Django+@+http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fpsyco-and-django%2F" rel="nofollow" title="Add to&nbsp;Twitter"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/twitter.png" title="Add to&nbsp;Twitter" alt="Add to&nbsp;Twitter" /></a>
<br />
</div>
</div>
<!-- Social Bookmarks END -->
]]></content:encoded>
			<wfw:commentRss>http://www.artfulcode.net/articles/psyco-and-django/feed/</wfw:commentRss>
		<slash:comments>5</slash:comments>
		</item>
		<item>
		<title>Per-user caching in Django</title>
		<link>http://www.artfulcode.net/articles/per-user-caching-django/</link>
		<comments>http://www.artfulcode.net/articles/per-user-caching-django/#comments</comments>
		<pubDate>Tue, 12 Feb 2008 21:22:55 +0000</pubDate>
		<dc:creator>Jeff</dc:creator>
				<category><![CDATA[Programming]]></category>
		<category><![CDATA[Tips]]></category>
		<category><![CDATA[django]]></category>
		<category><![CDATA[python]]></category>

		<guid isPermaLink="false">http://www.artfulcode.net/articles/per-user-caching-django/</guid>
		<description><![CDATA[Django comes with an easy-to-use caching framework. With a few simple decorators, an application&#8217;s views are cached. Decorators can even be used to control upstream caches, such as those maintained by ISPs. Nevertheless, if a rendered view is customized with information individual to a user, these caching options cease to be useful. Django has several [...]]]></description>
			<content:encoded><![CDATA[<p>Django comes with an easy-to-use caching framework.  With a few simple decorators, an application&#8217;s views are cached.  Decorators can even be used to control upstream caches, such as those maintained by ISPs.  Nevertheless, if a rendered view is customized with information individual to a user, these caching options cease to be useful.  <span id="more-35"></span>Django has several solutions for this scenario:</p>
<pre><code>1. The CACHE_MIDDLEWARE_ANONYMOUS_ONLY setting
2. The vary_on_cookie decorator
3. Template fragment caching
4. The low-level caching API
</code></pre>
<h4>The CACHE_MIDDLEWARE_ANONYMOUS_ONLY setting</h4>
<p>The <code>CACHE_MIDDLEWARE_ANONYMOUS_ONLY</code> setting causes Django to ignore the cache if the user is not anonymous.  This is less helpful than it seems.  At the Dayton Daily News, we require a trivial registration to access many areas of the site.  Using this setting means that the entire page cannot be cached because of a simple &#8220;Welcome, username&#8221; line in the rendered view.</p>
<p>Another obstacle to using the site-level cache is that many demographic-tracking packages require setting client-specific Javascript variables in the rendered view and then accessing a script on another server.  The per-site cache will cache these as well, distorting your analytics.</p>
<h4>The vary_on_cookie decorator</h4>
<p>The <code>vary_on_cookie</code> decorator (found in <code>django.views.decorators.vary</code>) is a simple way to tell upstream caches to cache a view based on the content of the user&#8217;s cookie.  This means that each user will get their own page cached.</p>
<p>This is a useful decorator and a part of any caching setup for user-based sites.  On its own, however, it still means that if a user visits a page only once, your server must perform all the work of rendering a page.  The server gains no benefit when another user visits, since the page must be generated anew and then cached for this user as well.</p>
<h4>Template fragment caching</h4>
<p>This is a new feature in the development version of Django.  It consists of a simple template tag that signals the framework to cache a portion of the rendered template.  For example:</p>
<pre><code>{% load cache %} {# thanks to AdamG for noticing the typo here #}
...stuff you don't want to cache
{ cache 300 "some" "section" user.id }
...stuff you do want to cache
{% endcache %}
...more stuff you don't want to cache
</code></pre>
<p>The cache tag accepts the number of seconds for which the cache should remain valid and a series of keys used to uniquely identify the cache.  You may use any number of keys.  Addng <code>user.id</code> to the mix will make this portion of the template cached on a per-user basis.  Using something static will make it a standard, all-user, cache.</p>
<p>I experimented with template fragment caching while developing an application for which we expected extremely high traffic.  In the end, we decided that the overhead did not justify the savings.</p>
<p>This is a very new feature and naturally not ready for production use.  In the next stable release of Django I imagine that it will be considerably more efficient.</p>
<h4>The low-level caching API</h4>
<p>The low-level caching API is <strong>the</strong> solution for serious fine-tuning of your cache.  It is located in <code>django.core.cache</code>.  It is laughably simple to use:</p>

<div class="wp_syntax"><div class="code"><pre class="python" style="font-family:monospace;"><span style="color: #ff7700;font-weight:bold;">from</span> django.<span style="color: black;">core</span>.<span style="color: black;">cache</span>
&nbsp;
CACHE_EXPIRES = <span style="color: #ff4500;">5</span> <span style="color: #66cc66;">*</span> <span style="color: #ff4500;">60</span> <span style="color: #808080; font-style: italic;"># 5 minutes</span>
&nbsp;
<span style="color: #ff7700;font-weight:bold;">def</span> some_view<span style="color: black;">&#40;</span>request, object_id<span style="color: black;">&#41;</span>:
    cache_key = <span style="color: #483d8b;">&quot;someobjectcache%s&quot;</span> <span style="color: #66cc66;">%</span> object_id
    object_list = cache.<span style="color: black;">get</span><span style="color: black;">&#40;</span>cache_key<span style="color: black;">&#41;</span>
    <span style="color: #808080; font-style: italic;">#if not object_list:</span>
    <span style="color: #808080; font-style: italic;">#AdamG noted that this check avoids empty lists</span>
    <span style="color: #808080; font-style: italic;">#evaluating to False, as &quot;if not object_list&quot; did</span>
    <span style="color: #ff7700;font-weight:bold;">if</span> object_list <span style="color: #ff7700;font-weight:bold;">is</span> <span style="color: #008000;">None</span>:
        object_list = expensive_lookup<span style="color: black;">&#40;</span><span style="color: black;">&#41;</span>
        cache.<span style="color: #008000;">set</span><span style="color: black;">&#40;</span>cache_key, object_list, CACHE_EXPIRES<span style="color: black;">&#41;</span>
    ...</pre></div></div>

<p>The cache is accessed via a unique key.  You can cache anything that can be safely picked in Python, including query sets from Django&#8217;s ORM.  If the cache has expired, <code>cache.get(key)</code> returns <code>None</code>.  Setting a key in the cache requires the unique key, the object to cache, and the time in seconds for which the cache is valid.</p>
<p>Since Django&#8217;s template engine is <a href="http://www.codeirony.com/?p=9">quite fast</a>, we use the low-level API to cache the most expensive portions of each page: large database lookups, search results, the result of filtering large sets of data, ad infinitum.</p>
<p>This has given us the biggest savings in terms of memory, database hits, and CPU usage.</p>
<h4>One final trick</h4>
<p>A couple of our applications are real database hogs.  They have a wide range of queries that get pulled over and over.  Clearly, dropping into raw SQL and pulling lists rather than objects is the best way to streamline this type of demand, but then we lose the benefit of our custom model manager and model methods.  Another neat trick is to add caching to your model manager itself:</p>

<div class="wp_syntax"><div class="code"><pre class="python" style="font-family:monospace;"><span style="color: #ff7700;font-weight:bold;">from</span> django.<span style="color: black;">contrib</span>.<span style="color: black;">sites</span>.<span style="color: black;">models</span> <span style="color: #ff7700;font-weight:bold;">import</span> Site
<span style="color: #ff7700;font-weight:bold;">from</span> django.<span style="color: black;">core</span>.<span style="color: black;">cache</span>
&nbsp;
CACHE_EXPIRES = <span style="color: #ff4500;">5</span> <span style="color: #66cc66;">*</span> <span style="color: #ff4500;">60</span> <span style="color: #808080; font-style: italic;"># 10 minutes</span>
&nbsp;
<span style="color: #ff7700;font-weight:bold;">class</span> ObjectManager<span style="color: black;">&#40;</span>models.<span style="color: black;">Manager</span><span style="color: black;">&#41;</span>:
    <span style="color: #ff7700;font-weight:bold;">def</span> get_query_set<span style="color: black;">&#40;</span><span style="color: #008000;">self</span>, <span style="color: #66cc66;">*</span>args, <span style="color: #66cc66;">**</span>kwargs<span style="color: black;">&#41;</span>:
        cache_key = <span style="color: #483d8b;">'objectlist%d%s%s'</span> <span style="color: #66cc66;">%</span> <span style="color: black;">&#40;</span>
            Site.<span style="color: black;">objects</span>.<span style="color: black;">get_current</span><span style="color: black;">&#40;</span><span style="color: black;">&#41;</span>.<span style="color: #008000;">id</span>, <span style="color: #808080; font-style: italic;"># unique for site</span>
            <span style="color: #483d8b;">''</span>.<span style="color: black;">join</span><span style="color: black;">&#40;</span><span style="color: black;">&#91;</span><span style="color: #008000;">str</span><span style="color: black;">&#40;</span>a<span style="color: black;">&#41;</span> <span style="color: #ff7700;font-weight:bold;">for</span> a <span style="color: #ff7700;font-weight:bold;">in</span> args<span style="color: black;">&#93;</span><span style="color: black;">&#41;</span>, <span style="color: #808080; font-style: italic;"># unique for arguments</span>
            <span style="color: #483d8b;">''</span>.<span style="color: black;">join</span><span style="color: black;">&#40;</span><span style="color: #483d8b;">'%s%s'</span> <span style="color: #66cc66;">%</span> <span style="color: black;">&#40;</span><span style="color: #008000;">str</span><span style="color: black;">&#40;</span>k<span style="color: black;">&#41;</span>, <span style="color: #008000;">str</span><span style="color: black;">&#40;</span>v<span style="color: black;">&#41;</span><span style="color: black;">&#41;</span> <span style="color: #ff7700;font-weight:bold;">for</span> k, v <span style="color: #ff7700;font-weight:bold;">in</span> kwargs.<span style="color: black;">iteritems</span><span style="color: black;">&#40;</span><span style="color: black;">&#41;</span><span style="color: black;">&#41;</span>
        <span style="color: black;">&#41;</span>
&nbsp;
        object_list = cache.<span style="color: black;">get</span><span style="color: black;">&#40;</span>cache_key<span style="color: black;">&#41;</span>
        <span style="color: #808080; font-style: italic;">#if not object_list:</span>
        <span style="color: #808080; font-style: italic;">#AdamG noted that this check avoids empty lists</span>
        <span style="color: #808080; font-style: italic;">#evaluating to False, as &quot;if not object_list&quot; did</span>
        <span style="color: #ff7700;font-weight:bold;">if</span> object_list <span style="color: #ff7700;font-weight:bold;">is</span> <span style="color: #008000;">None</span>:
            object_list = <span style="color: #008000;">super</span><span style="color: black;">&#40;</span>ObjectManager, <span style="color: #008000;">self</span><span style="color: black;">&#41;</span>.<span style="color: black;">get_query_set</span><span style="color: black;">&#40;</span><span style="color: #66cc66;">*</span>args, <span style="color: #66cc66;">**</span>kwargs<span style="color: black;">&#41;</span>
            cache.<span style="color: #008000;">set</span><span style="color: black;">&#40;</span>cache_key, object_list, CACHE_EXPIRES<span style="color: black;">&#41;</span>
        <span style="color: #ff7700;font-weight:bold;">return</span> object_list</pre></div></div>

<p>This custom model manager caches query sets using the arguments passed to <code>get_query_set()</code>.  If they are fewer than 10 minutes old, they returned from the cache; otherwise, they are returned as a fresh query set and added to the cache.  This technique can be used for busy databases to cache all possible queries performed by your application.</p>
<!-- Social Bookmarks BEGIN -->
<div class="social_bookmark">
<a><strong><em>Submit article</em></strong></a>
<br />
<div class="d">
<br />
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://del.icio.us/post?url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fper-user-caching-django%2F&amp;title=Per-user+caching+in+Django" rel="nofollow" title="Add to&nbsp;Del.icio.us"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/delicious.png" title="Add to&nbsp;Del.icio.us" alt="Add to&nbsp;Del.icio.us" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://digg.com/submit?phase=2&amp;url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fper-user-caching-django%2F&amp;title=Per-user+caching+in+Django" rel="nofollow" title="Add to&nbsp;digg"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/digg.png" title="Add to&nbsp;digg" alt="Add to&nbsp;digg" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://www.dzone.com/links/add.html?description=Per-user+caching+in+Django&amp;url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fper-user-caching-django%2F&amp;title=Per-user+caching+in+Django" rel="nofollow" title="Add to&nbsp;DZone"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/dzone.png" title="Add to&nbsp;DZone" alt="Add to&nbsp;DZone" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://www.facebook.com/sharer.php?u=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fper-user-caching-django%2F" rel="nofollow" title="Add to&nbsp;Facebook"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/facebook.png" title="Add to&nbsp;Facebook" alt="Add to&nbsp;Facebook" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://reddit.com/submit?url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fper-user-caching-django%2F&amp;title=Per-user+caching+in+Django" rel="nofollow" title="Add to&nbsp;reddit"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/reddit.png" title="Add to&nbsp;reddit" alt="Add to&nbsp;reddit" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://www.stumbleupon.com/submit?url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fper-user-caching-django%2F&amp;title=Per-user+caching+in+Django" rel="nofollow" title="Add to&nbsp;Stumble Upon"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/stumbleupon.png" title="Add to&nbsp;Stumble Upon" alt="Add to&nbsp;Stumble Upon" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://www.technorati.com/faves?add=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fper-user-caching-django%2F" rel="nofollow" title="Add to&nbsp;Technorati"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/technorati.png" title="Add to&nbsp;Technorati" alt="Add to&nbsp;Technorati" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://twitter.com/home/?status=Check+out+Per-user+caching+in+Django+@+http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fper-user-caching-django%2F" rel="nofollow" title="Add to&nbsp;Twitter"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/twitter.png" title="Add to&nbsp;Twitter" alt="Add to&nbsp;Twitter" /></a>
<br />
</div>
</div>
<!-- Social Bookmarks END -->
]]></content:encoded>
			<wfw:commentRss>http://www.artfulcode.net/articles/per-user-caching-django/feed/</wfw:commentRss>
		<slash:comments>4</slash:comments>
		</item>
		<item>
		<title>Partial application and currying</title>
		<link>http://www.artfulcode.net/articles/partial-application-and-currying/</link>
		<comments>http://www.artfulcode.net/articles/partial-application-and-currying/#comments</comments>
		<pubDate>Tue, 04 Dec 2007 18:33:00 +0000</pubDate>
		<dc:creator>Jeff</dc:creator>
				<category><![CDATA[Programming]]></category>
		<category><![CDATA[functional]]></category>
		<category><![CDATA[javascript]]></category>
		<category><![CDATA[python]]></category>

		<guid isPermaLink="false">http://www.artfulcode.net/articles/partial-application-and-currying/</guid>
		<description><![CDATA[Currying, known in Python land as partial application, is a technique in which a function taking multiple arguments composes a function that takes fewer arguments (in most languages, reducing to one, although this is not the case in Python) by partially applying it to given parameters. For example, a function, sum, might be used to [...]]]></description>
			<content:encoded><![CDATA[<p>Currying, known in Python land as partial application, is a technique in which a function taking multiple arguments composes a function that takes fewer arguments (in most languages, reducing to one, although this is not the case in Python) by partially applying it to given parameters.  For example, a function, sum, might be used to compose a new function called &#8220;plus_one&#8221; by currying it with the value of one.  The composed function is not evaluated; it is returned as a function object which may then be applied to other parameters.<span id="more-41"></span></p>
<p>Python&#8217;s partial() is contained in the <a href="http://docs.python.org/lib/module-functools.html">functools module</a> (included since Python 2.5, I believe).  The first argument passed must be the function to be curried, and the rest are positional or keyword arguments that will be used to curry the passed function.</p>
<p>A common case where partials are useful is in defining a compare function for a list.  It often happens that the list must be sorted according to rules defined at runtime.  A partial application can simplify the process, especially if there is a complex sort algorithm.  Assume a list of Items, items.  The list will be sorted according to get_sort_attribute(), which returns the name of the attribute of Item which will be used to perform the sort, and get_sort_direction, which returns either &#8220;desc&#8221; or &#8220;asc.&#8221;  Rather than using a long series of if/else statements and calling the sort() method in various ways, partial() can be used to progressively modify the sort.</p>
<p>To sort a list this way, we might have something like:</p>

<div class="wp_syntax"><div class="code"><pre class="python" style="font-family:monospace;"><span style="color: black;">&#40;</span><span style="color: #ff7700;font-weight:bold;">def</span> compare<span style="color: black;">&#40;</span>a, b<span style="color: black;">&#41;</span>:
    attr = get_sort_attribute<span style="color: black;">&#40;</span><span style="color: black;">&#41;</span>
    <span style="color: #008000;">dir</span> = get_sort_direction<span style="color: black;">&#40;</span><span style="color: black;">&#41;</span>
    value_a = <span style="color: #008000;">getattr</span><span style="color: black;">&#40;</span>a, attr<span style="color: black;">&#41;</span>
    value_b = <span style="color: #008000;">getattr</span><span style="color: black;">&#40;</span>b, attr<span style="color: black;">&#41;</span>
    <span style="color: #ff7700;font-weight:bold;">if</span> <span style="color: #008000;">dir</span> == <span style="color: #483d8b;">'desc'</span>:
        <span style="color: #ff7700;font-weight:bold;">return</span> <span style="color: #008000;">cmp</span><span style="color: black;">&#40;</span>value_b, value_a<span style="color: black;">&#41;</span>
    <span style="color: #ff7700;font-weight:bold;">else</span>:
        <span style="color: #ff7700;font-weight:bold;">return</span> <span style="color: #008000;">cmp</span><span style="color: black;">&#40;</span>value_a, value_b<span style="color: black;">&#41;</span>
items.<span style="color: black;">sort</span><span style="color: black;">&#40;</span>compare<span style="color: black;">&#41;</span></pre></div></div>

<p>This can get pretty long-winded, especially if our function, compare, performs complex operations before performing the comparison.  Here is the same thing using partial applications to compose our function instead:</p>

<div class="wp_syntax"><div class="code"><pre class="python" style="font-family:monospace;"><span style="color: #ff7700;font-weight:bold;">from</span> functools <span style="color: #ff7700;font-weight:bold;">import</span> partial
&nbsp;
<span style="color: #808080; font-style: italic;"># Wrap getattr in a lambda so that can accept keyword arguments</span>
attribute_getter = partial<span style="color: black;">&#40;</span><span style="color: #ff7700;font-weight:bold;">lambda</span> <span style="color: #008000;">object</span>, name:
    <span style="color: #008000;">getattr</span><span style="color: black;">&#40;</span><span style="color: #008000;">object</span>, name<span style="color: black;">&#41;</span>, name=get_sort_attribute<span style="color: black;">&#40;</span><span style="color: black;">&#41;</span><span style="color: black;">&#41;</span>
sort_fn = partial<span style="color: black;">&#40;</span>items.<span style="color: black;">sort</span>, key=attribute_getter<span style="color: black;">&#41;</span>
<span style="color: #ff7700;font-weight:bold;">if</span> get_sort_direction<span style="color: black;">&#40;</span><span style="color: black;">&#41;</span> == <span style="color: #483d8b;">'desc'</span>:
    sort_fn = partial<span style="color: black;">&#40;</span>sort_fn, <span style="color: #008000;">reversed</span>=<span style="color: #008000;">True</span><span style="color: black;">&#41;</span>
sort_fn<span style="color: black;">&#40;</span><span style="color: black;">&#41;</span></pre></div></div>

<p>This is certainly more concise.  I found a nice little currying function for Javascript <a href="http://www.dustindiaz.com/javascript-curry/">here</a> (<strong>edit:</strong> it was pointed out in a reader&#8217;s comment (here and at <a href="http://www.dzone.com/links/partial_application_and_currying.html">dzone</a>) that this version of curry does not work on previously curried functions; below it is a modified version which will function as expected):</p>

<div class="wp_syntax"><div class="code"><pre class="javascript" style="font-family:monospace;"><span style="color: #006600; font-style: italic;">/*
function curry(fn, scope) {
    var scope = scope || window;
    var args = [];
    for (var i=2, len = arguments.length; i &amp;lt; len; ++i) {
        args.push(arguments[i]);
    };
    return function() {
        /* one big problem here is that the following statement
        is not returning the applied function. */</span>
        fn.<span style="color: #660066;">apply</span><span style="color: #009900;">&#40;</span>scope<span style="color: #339933;">,</span> args<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span><span style="color: #339933;">;</span>
<span style="color: #009900;">&#125;</span>
<span style="color: #339933;">*/</span>
&nbsp;
<span style="color: #003366; font-weight: bold;">function</span> curry<span style="color: #009900;">&#40;</span>fn<span style="color: #339933;">,</span> scope<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #003366; font-weight: bold;">var</span> scope <span style="color: #339933;">=</span> scope <span style="color: #339933;">||</span> window<span style="color: #339933;">;</span>
    <span style="color: #003366; font-weight: bold;">var</span> args <span style="color: #339933;">=</span> <span style="color: #009900;">&#91;</span><span style="color: #009900;">&#93;</span><span style="color: #339933;">;</span>
    <span style="color: #000066; font-weight: bold;">for</span> <span style="color: #009900;">&#40;</span><span style="color: #003366; font-weight: bold;">var</span> i <span style="color: #339933;">=</span> <span style="color: #CC0000;">2</span><span style="color: #339933;">;</span> i <span style="color: #339933;">&amp;</span>lt<span style="color: #339933;">;</span> arguments.<span style="color: #660066;">length</span><span style="color: #339933;">;</span> i<span style="color: #339933;">++</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
        args.<span style="color: #660066;">push</span><span style="color: #009900;">&#40;</span>arguments<span style="color: #009900;">&#91;</span>i<span style="color: #009900;">&#93;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span>
    <span style="color: #000066; font-weight: bold;">return</span> <span style="color: #003366; font-weight: bold;">function</span><span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
        <span style="color: #006600; font-style: italic;">// this takes care of the arguments problem</span>
        <span style="color: #003366; font-weight: bold;">var</span> fn_args <span style="color: #339933;">=</span> <span style="color: #009900;">&#91;</span><span style="color: #009900;">&#93;</span><span style="color: #339933;">;</span>
        <span style="color: #000066; font-weight: bold;">for</span> <span style="color: #009900;">&#40;</span><span style="color: #003366; font-weight: bold;">var</span> i <span style="color: #339933;">=</span> <span style="color: #CC0000;">0</span><span style="color: #339933;">;</span> i <span style="color: #339933;">&amp;</span>lt<span style="color: #339933;">;</span> args.<span style="color: #660066;">length</span><span style="color: #339933;">;</span> i<span style="color: #339933;">++</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
            fn_args.<span style="color: #660066;">push</span><span style="color: #009900;">&#40;</span>args<span style="color: #009900;">&#91;</span>i<span style="color: #009900;">&#93;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        <span style="color: #009900;">&#125;</span>
        <span style="color: #000066; font-weight: bold;">for</span> <span style="color: #009900;">&#40;</span><span style="color: #003366; font-weight: bold;">var</span> i <span style="color: #339933;">=</span> <span style="color: #CC0000;">0</span><span style="color: #339933;">;</span> i <span style="color: #339933;">&amp;</span>lt<span style="color: #339933;">;</span> arguments.<span style="color: #660066;">length</span><span style="color: #339933;">;</span> i<span style="color: #339933;">++</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
            fn_args.<span style="color: #660066;">push</span><span style="color: #009900;">&#40;</span>arguments<span style="color: #009900;">&#91;</span>i<span style="color: #009900;">&#93;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        <span style="color: #009900;">&#125;</span>
        <span style="color: #006600; font-style: italic;">// this takes care of the null return problem</span>
        <span style="color: #000066; font-weight: bold;">return</span> fn.<span style="color: #660066;">apply</span><span style="color: #009900;">&#40;</span>scope<span style="color: #339933;">,</span> fn_args<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
    <span style="color: #009900;">&#125;</span><span style="color: #339933;">;</span>
<span style="color: #009900;">&#125;</span></pre></div></div>

<p>I often use this in my Django web projects.  I will include a basic error function which inserts an error message into an element with a particular id on the base template.  Note that I use jQuery in the following examples.</p>

<div class="wp_syntax"><div class="code"><pre class="javascript" style="font-family:monospace;"><span style="color: #003366; font-weight: bold;">function</span> err<span style="color: #009900;">&#40;</span>target<span style="color: #339933;">,</span> msg<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    $<span style="color: #009900;">&#40;</span><span style="color: #3366CC;">'#'</span> <span style="color: #339933;">+</span> target<span style="color: #009900;">&#41;</span>.<span style="color: #660066;">html</span><span style="color: #009900;">&#40;</span>msg<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
<span style="color: #009900;">&#125;</span></pre></div></div>

<p>In templates that extend that template, I can then use currying to modify that for a particular location defined in this template:</p>

<div class="wp_syntax"><div class="code"><pre class="javascript" style="font-family:monospace;"><span style="color: #003366; font-weight: bold;">var</span> err <span style="color: #339933;">=</span> curry<span style="color: #009900;">&#40;</span>err<span style="color: #339933;">,</span> window<span style="color: #339933;">,</span> <span style="color: #3366CC;">'err_div_in_this_template'</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span></pre></div></div>

<p>I can further use this to create custom error callbacks for ajax functions:</p>

<div class="wp_syntax"><div class="code"><pre class="javascript" style="font-family:monospace;"><span style="color: #003366; font-weight: bold;">var</span> err404 <span style="color: #339933;">=</span> curry<span style="color: #009900;">&#40;</span>err<span style="color: #339933;">,</span> window<span style="color: #339933;">,</span> <span style="color: #3366CC;">&quot;The server could not be contacted.&quot;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span></pre></div></div>

<p>Now, err404 is the equivalent of:</p>

<div class="wp_syntax"><div class="code"><pre class="javascript" style="font-family:monospace;"><span style="color: #003366; font-weight: bold;">function</span> err404<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    $<span style="color: #009900;">&#40;</span><span style="color: #3366CC;">'#'</span> <span style="color: #339933;">+</span> <span style="color: #3366CC;">'err_div_in_this_template'</span><span style="color: #009900;">&#41;</span>.<span style="color: #660066;">html</span><span style="color: #009900;">&#40;</span><span style="color: #3366CC;">&quot;The server could not be contacted.&quot;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
<span style="color: #009900;">&#125;</span></pre></div></div>

<p>Obviously, this is a pretty trivial example, but it does a good job of showing a real-world use for currying.</p>
<!-- Social Bookmarks BEGIN -->
<div class="social_bookmark">
<a><strong><em>Submit article</em></strong></a>
<br />
<div class="d">
<br />
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://del.icio.us/post?url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fpartial-application-and-currying%2F&amp;title=Partial+application+and+currying" rel="nofollow" title="Add to&nbsp;Del.icio.us"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/delicious.png" title="Add to&nbsp;Del.icio.us" alt="Add to&nbsp;Del.icio.us" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://digg.com/submit?phase=2&amp;url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fpartial-application-and-currying%2F&amp;title=Partial+application+and+currying" rel="nofollow" title="Add to&nbsp;digg"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/digg.png" title="Add to&nbsp;digg" alt="Add to&nbsp;digg" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://www.dzone.com/links/add.html?description=Partial+application+and+currying&amp;url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fpartial-application-and-currying%2F&amp;title=Partial+application+and+currying" rel="nofollow" title="Add to&nbsp;DZone"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/dzone.png" title="Add to&nbsp;DZone" alt="Add to&nbsp;DZone" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://www.facebook.com/sharer.php?u=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fpartial-application-and-currying%2F" rel="nofollow" title="Add to&nbsp;Facebook"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/facebook.png" title="Add to&nbsp;Facebook" alt="Add to&nbsp;Facebook" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://reddit.com/submit?url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fpartial-application-and-currying%2F&amp;title=Partial+application+and+currying" rel="nofollow" title="Add to&nbsp;reddit"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/reddit.png" title="Add to&nbsp;reddit" alt="Add to&nbsp;reddit" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://www.stumbleupon.com/submit?url=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fpartial-application-and-currying%2F&amp;title=Partial+application+and+currying" rel="nofollow" title="Add to&nbsp;Stumble Upon"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/stumbleupon.png" title="Add to&nbsp;Stumble Upon" alt="Add to&nbsp;Stumble Upon" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://www.technorati.com/faves?add=http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fpartial-application-and-currying%2F" rel="nofollow" title="Add to&nbsp;Technorati"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/technorati.png" title="Add to&nbsp;Technorati" alt="Add to&nbsp;Technorati" /></a>
<a onclick="window.open(this.href, '_blank', 'scrollbars=yes,menubar=no,height=600,width=750,resizable=yes,toolbar=no,location=no,status=no'); return false;" href="http://twitter.com/home/?status=Check+out+Partial+application+and+currying+@+http%3A%2F%2Fwww.artfulcode.net%2Farticles%2Fpartial-application-and-currying%2F" rel="nofollow" title="Add to&nbsp;Twitter"><img class="social_img" src="http://www.artfulcode.net/wp-content/plugins/social-bookmarks/images/twitter.png" title="Add to&nbsp;Twitter" alt="Add to&nbsp;Twitter" /></a>
<br />
</div>
</div>
<!-- Social Bookmarks END -->
]]></content:encoded>
			<wfw:commentRss>http://www.artfulcode.net/articles/partial-application-and-currying/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
	</channel>
</rss>

