The Mysql module differs from the distribution standard module in several important ways. Most obviously, it uses FOOP wrappers for MySQL types. It also requires clients to free results instances; in the standard module, only the base MYSQL instance itself must be freed (using MySQL:close-db).

The significance of this is that it is much simpler to create multiple connections (without having to duplicate the entire context at compile time). Result sets are completely independent of each other, and several may be maintained in any state at once. This also means that a spawned process may be given its own Mysql instance to use without having to worry about other processes’ instances interfering. Using the standard module, the entire context would need to be cloned at compile time and given a static symbol reference (e.g., (new 'MySQL 'db)) in order to run multiple instances or connections to a server.

Moreover, because this module uses unpack and MySQL C API accessor functions, there is no need for the client to calculate member offsets in MySQL compound types. So long as newLisp was compiled for the same target as the libmysqlclient library (both are 32 bit or both are 64 bit), everything should work out of the box. Additionally, MySQL errors are now checked in the connect and query functions and re-thrown as interpreter errors. Instead of checking for nil returns and a using MySQL:error to get the error message, standard error  handling with the catch function may be used.

Several convenience functions and macros have been defined in the mysql context for common operations, including connecting to the database and iterating over results without having to worry about catching errors and managing memory. SQL statements may be passed as strings or a list containing a format string and its parameters, which will type-checked and automatically be escaped as needed.

(mysql:on-connect '("localhost" "user" "secret" "my_database")
  (lambda (db err)
    (if err
      (println "Error! " err)
      (mysql:row-iter db "SELECT * FROM some_table" true
        (lambda (row)
          (println row))))))

This module has been tested with MySQL version 5 and 5.1 and newLisp version 10.0.1. It requires newLisp 10.0 or later. You can download it from the Artful Code newLisp module repository.

Submit article

The basics

The basics of the MySQL full-text search functions are well-documented in the MySQL online documentation. For those lacking patience, here is a quick rundown.

Full-text searching is somewhat akin to a LIKE condition, but is much faster, requiring a FULLTEXT index to be created for the table columns targeted in the search. To search the title and description columns of a table, entries, the following statement would create the proper index:

ALTER TABLE entries ADD FULLTEXT(title, description)

To search these columns for the text, “python threading,” the MATCH...AGAINST functions are used:

SELECT id, MATCH(title, description) AGAINST ('python threading') AS score
FROM entries
ORDER BY score DESC

Notice that we keep the result of the match. The value returned is a float representing the relevance of the match. The higher the number, the more relevant the match.

There are several caveats to the full-text search. In particular, any words that are common between many entries are treated as noise and their relevance in any search is diminished. This means that were every article in entries to be about threading in Python, searching for “python threading” may not return extremely relevant results. Refer to the MySQL docs for more information.

The hard part

If the content to be searched is not conveniently located in one table, things get more complex. In this case, a method must be devised to create an intermediary table to contain the search target.

This might be accomplished with a cron script that aggregates the information nightly or using stored procedures to keep the target table updated.

Refining results

A common case is to weight the search to favor more recent results. Assuming that each entry has a DATETIME field named timestamp, this is easily accomplished by using the entry’s age to modify the score.

For an even reduction to the score based on the article’s age, divide the score by the age, which is determined with DATEDIFF(NOW(), timestamp).

(MATCH(title, description) AGAINST ('python threading'))/GREATEST(1, DATEDIFF(NOW(), timestamp))

Since DATEDIFF returns the difference in days, an entry written today could cause division by zero.
GREATEST means that entries written today and yesterday have equal weight, but prevents results from omitting today’s articles.

A quick test of this will show that results become wildly incorrect after a few days as the text match score begins to diminish further with age. This effect can be reduced by taking the LOG of the age, making the divisor increase less and less the greater the age.

LOG(GREATEST(1, DATEDIFF(NOW(), timestamp)))

The use of LOG causes a steep drop initially, smothing over time. For a less dramatic effect, substituting the square root causes a similar drop in the weight of the entry’s age over time, but diminishing less starkly over time and without the initial steep drop.

SQRT(GREATEST(1, DATEDIFF(NOW(), timestamp)))

The complete SQL statement is now:

SELECT id,
  (MATCH(title, description) AGAINST ('python threading'))/SQRT(GREATEST(1, DATEDIFF(NOW(), timestamp)))
  AS score
FROM entries
ORDER BY score DESC

Submit article

Updating paths

The file paths in the MySQL module only include the most common libmysqlclient paths. Update the files list to contain /usr/lib64 as well:

(set 'files '(
    "/usr/lib64/libmysqlclient.so" ; 64 bit Linux, UNIX
    "/usr/lib/libmysqlclient.so" ; Linux, UNIX
    "/usr/local/mysql/lib/libmysqlclient.dylib" ; MacOS X
))

If you are not sure where you libmysqlclient library is, you can find it with this command (assuming it is somewhere in /usr):

find /usr -name "libmysqlclient.so"

Updating alignments

The offsets for most structure fields are set at the beginning of mysql5.lsp. They need to be updated:

(constant 'NUM_ROWS_OFFSET (if big-endian 4 0))
(constant 'NUM_FIELDS_OFFSET 96)
(constant 'ERROR_OFFSET 141)
(constant 'INSERT_ID_OFFSET (if big-endian 836 832))
(constant 'AFFECTED_ROWS_OFFSET (if big-endian 828 824))

This is not necessarily the same on all systems. In the newLISP source tree is a folder named util. Compile sql.c and execute it to get the proper offsets. Note that the numbers generated are only one of the two used in this; you need to first determine the endianness of your system. The module itself uses this newLISP form:

(= (pack ">ld" 1) (pack "ld" 1))

If that returns true, update the first number and offset the other number by -4. Otherwise, update the second number and offset the first by 4 (except for NUM_FIELDS_OFFSET and ERROR_OFFSET).

The offsets for the MYSQL_FIELD structure fields are hard-coded in the functions fetch-row and keep-type.

In keep-type, update the line:

(set 'data (get-int (int (+ type_ptr (* 19 4)))))

…to:

(set 'data (get-int (int (+ type_ptr (+ (* 9 8) (* 10 4))))))

That reflects that there are 9 8-byte fields (all char* or ulong) and 10 4-byte fields (all uint) in the MYSQL_FIELD struct. That will keep the coerced types returned from queries straight.

Next, in fetch-row, the pointer to the field structure is found using:

(set 'field_addr (get-int (int (+ rdata (* field 4)))))

Change the 4 to 8:

(set 'field_addr (get-int (int (+ rdata (* field 8)))))

…or you will only get half the fields’ values :).

Data types

On the newLISP forum, Lutz pointed out that in fetch-row, the call to get-int must be changed to get-long for the 64-bit library. After a quick look at the types returned in the MySQL C api docs, I also updated num-rows, affected-rows and inserted-id.

Test it out

Load the updated module in the newLISP shell and run:

(test-mysql)

Submit article

Much of the module uses the small convenience classes to “serialize” SQL expressions. Most of the module’s classes have :serialize methods that render the encapsulated data as a string. For example, the Field class:

(let ((f (Field "table" "fieldname")))
    (:serialize f))
=> "table.fieldname"

…or the Condition class:

(let ((c (Condition ">" "salary" 65000)))
    (:serialize c))
=> "salary > '65000'"

Note that values are enclosed in single quotes for ANSI compliance. The most interesting function in the module, however, is sql:expr, which uses the match-based primitives in the functional module to generate various types of expressions:

(sql:expr "employees" "first_name")
=> "employees.first_name"
 
(sql:expr "LIKE" "first_name" "Stev%")
=> "first_name LIKE 'Stev%'"
 
(sql:expr "OR" (sql:expr "LIKE" "first_name" "Stev%")
               (sql:expr "LIKE" "last_name" "John%"))
=> "((first_name LIKE 'Stev%') OR (last_name LIKE 'John%'))"
 
(sql:expr 6)
=>"6"
 
(sql:expr 'myapp:employees)
=> "employees"
 
(sql:expr "CONV" "AF" 16 10)
=> "CONV('AF',16,10)"
 
(sql:expr "LIKE" (sql:expr "employees" "first_name") "Stev%")
=> "employees.first_name LIKE 'Stev%'"

Additional functions directly express select, update, insert, and delete statements or handle data type conversions (such as parsing SQL datetimes).

Functional

Another addition to Artful Code’s module list is the functional module. This library provides some basic conditionals that make use of match to process data. These macros express program logic by associating blocks of code with the structure of data.

Here is an example using match-case. match-case accepts a single expression and a series of forms that describe what to do based on its structure. Each case form consists of a match expression, a list of variables to be locally bound to the result of the match, and an individual form to evaluate in a local scope with the matched variables bound:

(let ((x '(1 2 3 4 5)))
    (match-case x
        ((? ? ?) (a b c)
         (println "a b and c do not get bound here, because they do not match x"))
        ((? ? ? *) (a b c d)
         (println "a = 1, b = 2, c = 3, and d = '(4 5)"))))

Here, the second block would match because (? ? ? *) matches against (1 2 3 4 5) and creates the binding association, ((a 1) (b 2) (c 3) (d (4 5))).

Also included in the module is match-cond, which is more powerful than match-case. match-cond works like cond, except that instead of a user-defined conditional, the first form in each case is a list of pattern, symbol-list, and target. See the documentation for more details.

Edit (2009-02-16): the functional module in mentioned in this post has been replaced with the matching module.

Submit article

from models import Person
person = Person.objects.get_or_create(first_name="John", last_name="Johnson")

It adds syntactic sugar to the common case of checking if an entry exists, creating it if it does not, then updating (or creating with new) values. The problem is that its slow.

I offload our most expensive database work on our server to a Lisp-based XML RPC server using CLSQL. I wrote an analog of get_or_create for CLSQL. It does not have the advantage of Django’s keyword searches; all searches use = and pull the first row that matches the value.

The first function, get-instance is a helper to get a row in a view-class using a field/value pair.
create/update takes the first three arguments and passes them to get-instance to get any existing match.

(defun get-instance (cls pk val)
  #.(clsql:locally-enable-sql-reader-syntax)
  (let ((inst (clsql:select cls :flatp t :refresh t :limit 1
                            :where [= (clsql:sql-expression :attribute pk) val])))
    #.(clsql:restore-sql-reader-syntax-state)
    (if inst (car inst) nil)))
 
(defun create/update (cls pk val &rest args)
  (let ((inst (get-instance cls pk val)))
        (if inst
            (apply #'reinitialize-instance inst args)
            (apply #'make-instance cls pk val args))))

Usage:

(clsql:def-view-class person ()
    ((id
        :reader person-id
        :db-kind :key
        :db-constraints :not-null
        :type integer
        :initarg :id)
    (last-name
        :accessor person-last-name
        :db-constraints :not-null
        :type (clsql:varchar 25)
        :initarg :last-name)
    (first-name
        :accessor person-first-name
        :db-constraints :not-null
        :type (clsql:varchar 25)
        :initarg :first-name)))
 
(defvar john (create/update 'person :id 3))

If there is row with id 3 in the database, it will grab that (or, if the key is not unique, it will grab the first matching row.) If there is no entry with that id, it will create a new one, that can then be updated the normal way:

(setf (person-last-name john) "John")
(setf (person-first-name john) "Johnson")

The other alternative is to get and set all in one go:

(defvar john (create/update 'person
                            :id 3
                            :last-name "Johnson"
                            :first-name "John"))

It’s pretty simple but it is very useful.

Submit article