The problem

I recently worked on a project with a backend using the Django admin library. It is a wonderful tool, but the filter interface is not equipped to deal with the thousands of options that were being populated in this particular form entry (a necessary parameter of the project.)

The Django filter interface would take 30 seconds or more to load the multi-select’s data on each form load, even on speedy computers. Moreover, the search filter breaks up queries at spaces, then performs a substring search for each token for every option in the select. Suffice it to say that this locked up more than a few browsers in our testing.

The solution

My first thought was to create my own search using a ternary tree or trie to index the data, but building these structures turned out to be extremely slow implemented in Javascript, taking even more time than the Django filter to initialize.

I ended up trying what seemed like a foolish idea. Create one large string to search by joining all of the options’ labels together (and delimiting with pipes). I assumed this would take a huge amount of memory, but in fact it took less than Javascript trees and was much faster than operating directly on DOM nodes. By delimiting the strings, a bounded regular expression could be used to perform reasonably fast searches of the data. There would be the need to identify the options to which the strings originated, for which I created a simple hash mapping option label to value.

This solution has two main problems – it assumes that all entries in the select box have both a unique label and a unique value, and no label can have a pipe in it. However, assuming that these conditions are met, this solution seems to work pretty speedily, and without overtaxing the browser.

The code

This code is implemented as a jQuery plugin. It hides the select and changes made to the widget automatically update the select. A search box and two lists are inserted into the document; the user enters a partial match in the search box and hits return. The first list is populated with the search results. Double-clicking on a result copies it into the second list.

Items that are already selected are automatically put into the second list. Regular expressions are permitted, and must be surrounded by forward slashes.

This module has been lightly tested in Firefox 2 and 3, Safari (latest), Opera (latest), and IE 7. A few concessions in favor of efficiency have been made at the expense of clarity (mainly for IE, which seems to have the slowest Javascript engine), primarily in the SelectMap constructor. jQuery’s each method and string concatenation proved to be too much for IE, so they were replaced with a for loop over the select node’s legacy options array. Iterative string building was replaced with Array.join, which is faster (but more expensive in memory) in IE.

To use it:

$(function() {
    var select = new BigSelect('target_id');
    // or...
    $('#target_id').bigSelect();
});

The code is available here. You can see a live example here.

Submit article

Much of the module uses the small convenience classes to “serialize” SQL expressions. Most of the module’s classes have :serialize methods that render the encapsulated data as a string. For example, the Field class:

(let ((f (Field "table" "fieldname")))
    (:serialize f))
=> "table.fieldname"

…or the Condition class:

(let ((c (Condition ">" "salary" 65000)))
    (:serialize c))
=> "salary > '65000'"

Note that values are enclosed in single quotes for ANSI compliance. The most interesting function in the module, however, is sql:expr, which uses the match-based primitives in the functional module to generate various types of expressions:

(sql:expr "employees" "first_name")
=> "employees.first_name"
 
(sql:expr "LIKE" "first_name" "Stev%")
=> "first_name LIKE 'Stev%'"
 
(sql:expr "OR" (sql:expr "LIKE" "first_name" "Stev%")
               (sql:expr "LIKE" "last_name" "John%"))
=> "((first_name LIKE 'Stev%') OR (last_name LIKE 'John%'))"
 
(sql:expr 6)
=>"6"
 
(sql:expr 'myapp:employees)
=> "employees"
 
(sql:expr "CONV" "AF" 16 10)
=> "CONV('AF',16,10)"
 
(sql:expr "LIKE" (sql:expr "employees" "first_name") "Stev%")
=> "employees.first_name LIKE 'Stev%'"

Additional functions directly express select, update, insert, and delete statements or handle data type conversions (such as parsing SQL datetimes).

Functional

Another addition to Artful Code’s module list is the functional module. This library provides some basic conditionals that make use of match to process data. These macros express program logic by associating blocks of code with the structure of data.

Here is an example using match-case. match-case accepts a single expression and a series of forms that describe what to do based on its structure. Each case form consists of a match expression, a list of variables to be locally bound to the result of the match, and an individual form to evaluate in a local scope with the matched variables bound:

(let ((x '(1 2 3 4 5)))
    (match-case x
        ((? ? ?) (a b c)
         (println "a b and c do not get bound here, because they do not match x"))
        ((? ? ? *) (a b c d)
         (println "a = 1, b = 2, c = 3, and d = '(4 5)"))))

Here, the second block would match because (? ? ? *) matches against (1 2 3 4 5) and creates the binding association, ((a 1) (b 2) (c 3) (d (4 5))).

Also included in the module is match-cond, which is more powerful than match-case. match-cond works like cond, except that instead of a user-defined conditional, the first form in each case is a list of pattern, symbol-list, and target. See the documentation for more details.

Edit (2009-02-16): the functional module in mentioned in this post has been replaced with the matching module.

Submit article

1. CSV – a module for parsing CSV files
2. Mysql – a wrapper for the standard MySQL module, with some helpful additions
3. Http – a simple, bare-bones HTTP 1.0 library
4. SocketServer – an extensible socket server framework
5. Xmlrpc-server – an XML-RPC server using SocketServer and Http
6. generics – generic functions for newLISP in the style of CLOS
7. locks – FOOP classes for process synchronization

There is also a utilities library, which contains functions and classes required by the other libraries.

newLISP 9.4.2 is required for several of these (util.lsp requires 9.4.2 due to a dictionary syntax bug in 9.4.1).

Update (2008-07-09): I’ve moved a few modules from the projects page to the modules page, most notably the request, response, and xml modules.

Update (2009-02-10): Several of these modules have disappeared (SocketServer and locks) and have been replaced. The new modules are sockets, which is more generalized and powerful than SocketServer was, and MP,  which is a full-featured multi-processing module.

Submit article

The Request module parses GET, POST, and COOKIE data and provides a few simple functions to retrieve these values. Values may be looked up by key using:

(Request:get "foo")
(Request:post "foo")
(Request:cookie "foo")

POST data is restricted to 4096 bytes. This can be altered by editing the *max-post-length* constant. I have included a few useful predicates:

(Request:get?)
(Request:post?)
(Request:cookie? "foo")

The request method can be directly determined using (Request:method). The path and domain are stored in Request:path and Request:domain, respectively. I included one other feature of interest for those who use mod_rewrite to create pretty urls.
(Request:segment) allows access to the individual url path segments. If called with an integer argument, it will attempt to return the segment at that index. Calling a higher index than exists results in an out of bounds error. If (Request:segment) is called sequentially, it will return the next segment in the list until there are no more segments left, after which it will return nil. Sample usage:

(while (set 's (Request:segment))
    (do-something-with s))

Additionally, the entire set of segments can be accessed as a list using (Request:segments).

The Response module contains functions to control the output of HTTP headers. Generic headers may be set using (Response:header "foo" "bar") and the current set of headers may be queried with (Response:header? "foo"). Content-type is handled automatically, although it may be overridden, e.g. (Response:header "Content-type" "text/plain"). The default content type is text/html; charset=utf-8.

Adding display content to the response in simple.
(Response:write "foo") will modify the content in place. This may be overridden when outputting the content using the default functor, e.g. (Response "foo"). The result of this is a string; it must be printed to the browser. Other types of output may be achieved with various functions:

(print (Response:redirect "/some/other/path/"))
(print (Response:error "
 
Some error has occurred
 
"))
(print (Response:not-found "
 
Page not found
 
"))

There are usage examples at the top of the module. Custom handlers need only override the internal function, (Response:_response), which simply outputs the entire page, including headers.

Cookies are dealt with using generic syntax and are automatically added to response output.

(Response:set-cookie "foo" "bar")
(Response:delete-cookie "foo")

Finally, one thing that users of the CGI module may miss is the put-page routine. At the moment, there is no equivalent to this in the Response module. I am currently working on a Template module that will provide similar functionality. Until then, there is no reason that put-page could not be copied and pasted into Response (or elsewhere) as required.

One note of warning:
Request and the newLISP CGI module will not work together. Once the POST data has been read from stdin, it is consumed. Whichever module is called first will have the POST data, which will be unavailable to the other module.

Submit article