void strtoupper(char *str)
{
    if (str) {  // null ptr check, courtesy of Michael
        while (*str != '\0') {
            // destructively modify the contents at the current pointer location
            // using the dereference operator to access the value at the current
            // pointer address.
            *str = toupper(*str);
            ++str;
        }
    }
}
 
int main()
{
    char my_str[] = "hello world";
    strtoupper(my_str);
    printf("%s", my_str);
    return 0;
}

my_str is actually a pointer to a block of memory holding chars. This allows us to use address math to access indices of the array and modify them using the dereference operator. In fact, an array index such as my_str[3] is identical to the expression *(my_str + 3).

char my_str[] = "hello world";
*my_str = toupper(*my_str);
*(my_str + 6) = toupper(*(my_str + 6));
printf("%s", my_str); // prints, "Hello World"

However, if my_str is declared as a char pointer to the string literal “hello world” rather than a char array, these operations fail:

char *my_str = "hello world";
*my_str = toupper(*my_str); // fails
*(my_str + 6) = toupper(*(my_str + 6)); // fails
printf("%s", my_str);

Let’s explore the difference between the two declarations.

char *a = "hello world";
char b[] = "hello world";

In the compiled program, it is likely that “hello world” is stored literally inside the executable. It is effectively an immutable, constant value. Pointing char *a to it provides the scope with read-only access to an immutable block of memory. Therefore, attempting to assign a value might cause other code that points to the same memory to behave erratically (read this response to the above post on Stack Overflow for an excellent explanation of this behavior.)

The declaration of char b[] instead declares a locally allocated block of memory that is then filled with the chars, “hello world”. b is now a pointer to the first address of that array. The complete statement, combining the declaration and assignment, is shorthand. Dispensing with the array size (e.g., char instead of char[12]) is permitted as the compiler is able to ascertain its size from the string literal it was assigned.

In both cases the pointer is used to access array indices:

int i;
for (i = 0; a[i] != '\0'; ++i)
    printf("%c", toupper(a[i]));

However, only with b is the program able to modify the values in memory, since it is explicitly copied to a mutable location on the stack in its declaration:

int i;
for (i = 0; b[i] != '\0'; ++i)
    b[i] = toupper(b[i]);
printf("%s", b);

Submit article

Importing foreign functions

Loading a function from a shared library is done with the ‘import’ function. Once imported, the function now exists as a normal function in the namespace in which is was imported. From the newLisp documentation:

;; import in Linux
(import "libc.so.6" "printf")
;; import in Mac OS X
(import "libc.dylib" "printf")
;; import in CYGWIN
(import "cygwin1.dll" "printf")

printf is now a function that may be called as any other. Because functions evaluate to themselves and import returns the function value, it is simple to bind the function to a different name:

(setf print-f (import "libc.dylib" "printf"))

Calling foreign functions

Foreign functions are called in the same was any other function.  They can be applied, curried, or mapped.

(printf "%g %s %d %c\n" 1.23 "hello" 999 65)
1.23 hello 999 A

The values that the foreign function receives are pointers to the values that were passed to the function in newLisp. In most cases, this is not problematic, but it is something that must be kept in mind.

Return values

Often, a library function will not return a value directly. Often, functions return a pointer to a value or struct in memory. newLisp cannot guess at the size of the return value or its composition, so it is up to the programmer to know what kind of value is being returned.

For functions that return a pointer to a single value, the functions get-int, get-char, and get-float may be used to find the value to which the pointer points. It is important to check the return value of such a function, as passing an invalid value or null pointer to these functions may crash the interpreter.

(setf ptr (get-pointer-from-some-c-func))
(if-not (zero? ptr)
  (setf value (get-int ptr)))

Strings may be accessed using get-string, although functions that return string pointers often leave it to the consuming function to free the memory. To do so, the function ‘free’ (which frees memory) must first be imported from libc:

(import "libc.dylib" "free")
(setf ptr (get-string-pointer))
(if-not (zero? ptr)
  (setf str (get-string ptr)) ; copy the c string to a newLisp string
  (free ptr)) ; free the string allocated by the foreign function

Deconstructing structs

Functions in 3rd party libraries often return custom types that newLisp cannot introspect. In these cases, the programmer must tell newLisp how to unpack the values.

To that end, newLisp provides the pack and unpack functions. These can be used to pack newLisp values into a structure and vice versa. pack and unpack use a specially formated string that corresponds to the members of the struct. Take the following struct as an example:

struct foo {
  int int_value;
  double float_value;
}

It could be unpacked using the format, “lu lf” (see the documentation for pack and unpack for a full explanation of the format used). Unpacking with this format will return a list of the struct’s values.

(unpack "lu lf" foo-instance) ; returns (42 84.5)

A function that accepts such a struct can be passed the return value of the pack function:

(foofunction (pack "lu lf" 42 84.5))

This makes it simple to unpack complex types into newLisp values.

Putting it all together

It is helpful to create a base FOOP class to work with compound types. Specific types can be prototyped from this class, changing only the pack format to match.

(setf Pointer:pack-format nil)
 
(define (Pointer:Pointer addr)
  (list (context) addr))
 
(define (Pointer:pointer inst)
  (inst 1))
 
(define (Pointer:member inst n , unpacked)
  (setf unpacked (unpack Pointer:pack-format (:pointer inst)))
  (unpacked n))

The Pointer class gives us a few useful methods. :pointer returns the pointer address and :member returns the nth member of the struct. So, the foo class would look like this:

(new 'Pointer 'Foo)
(setf Foo:pack-format "lu lf")

It might also be helpful to add a couple of accessor functions:

(define (Foo:int-val inst , ptr)
  (setf ptr (:member inst 0))
  (if-not (zero? ptr)
    (get-int ptr)))
 
(define (Foo:float-val inst , ptr)
  (setf ptr (:member inst 1))
  (if-not (zero? ptr)
    (get-float ptr)))

Functions from a foreign library that work on ‘foo’ types can now be added directly to the Foo class as methods, and passed the return value of (:pointer my-foo). This makes working with foreign types reasonably trivial.

Submit article

Updating paths

The file paths in the MySQL module only include the most common libmysqlclient paths. Update the files list to contain /usr/lib64 as well:

(set 'files '(
    "/usr/lib64/libmysqlclient.so" ; 64 bit Linux, UNIX
    "/usr/lib/libmysqlclient.so" ; Linux, UNIX
    "/usr/local/mysql/lib/libmysqlclient.dylib" ; MacOS X
))

If you are not sure where you libmysqlclient library is, you can find it with this command (assuming it is somewhere in /usr):

find /usr -name "libmysqlclient.so"

Updating alignments

The offsets for most structure fields are set at the beginning of mysql5.lsp. They need to be updated:

(constant 'NUM_ROWS_OFFSET (if big-endian 4 0))
(constant 'NUM_FIELDS_OFFSET 96)
(constant 'ERROR_OFFSET 141)
(constant 'INSERT_ID_OFFSET (if big-endian 836 832))
(constant 'AFFECTED_ROWS_OFFSET (if big-endian 828 824))

This is not necessarily the same on all systems. In the newLISP source tree is a folder named util. Compile sql.c and execute it to get the proper offsets. Note that the numbers generated are only one of the two used in this; you need to first determine the endianness of your system. The module itself uses this newLISP form:

(= (pack ">ld" 1) (pack "ld" 1))

If that returns true, update the first number and offset the other number by -4. Otherwise, update the second number and offset the first by 4 (except for NUM_FIELDS_OFFSET and ERROR_OFFSET).

The offsets for the MYSQL_FIELD structure fields are hard-coded in the functions fetch-row and keep-type.

In keep-type, update the line:

(set 'data (get-int (int (+ type_ptr (* 19 4)))))

…to:

(set 'data (get-int (int (+ type_ptr (+ (* 9 8) (* 10 4))))))

That reflects that there are 9 8-byte fields (all char* or ulong) and 10 4-byte fields (all uint) in the MYSQL_FIELD struct. That will keep the coerced types returned from queries straight.

Next, in fetch-row, the pointer to the field structure is found using:

(set 'field_addr (get-int (int (+ rdata (* field 4)))))

Change the 4 to 8:

(set 'field_addr (get-int (int (+ rdata (* field 8)))))

…or you will only get half the fields’ values :).

Data types

On the newLISP forum, Lutz pointed out that in fetch-row, the call to get-int must be changed to get-long for the 64-bit library. After a quick look at the types returned in the MySQL C api docs, I also updated num-rows, affected-rows and inserted-id.

Test it out

Load the updated module in the newLISP shell and run:

(test-mysql)

Submit article

Libraries written in many languages beyond C, including ECL, may be compiled into a shared library or DLL. Many other languages, like Lua and newLISP, include simple DLLs of their own to embed their interpreters in an application.

ctypes allows you to wrap a shared library in pure Python. Here is a quick example to get you started. newLISP exports a simple function, newlispEvalStr, which accepts a string of newLISP code and returns the result of the code’s evaluation as a string. To compile newLISP as a shared library, use the PLATFORM_lib variant of your system’s make target (quick note: make install does not install the shared library; it must be manually copied to your path). The helper function, find_library (located in ctypes.util), gives you a platform-independent way of finding the needed path string for a library.

from ctypes import *
from ctypes.util import find_library
 
lib_path = find_library('newlisp')
newlisp = CDLL(lib_path)

CDLL extends LoadLibrary, the primary utility class in ctypes. In Windows, there are also OleDLL and WinDll. All of these release the global interpreter lock when a foreign function is entered and reclaim it when the function call completes. There is also a PyDLL class that does not release the GIL; its main purpose is the provide direct access to the Python C API from within your Python application.

newlispEvalStr is accessed as a method of the CDLL instance. As with any low level interface, it is important to know the library you are using. Accessing invalid function names can lead to unexpected results or system instability.

nl_eval = newlisp.newlispEvalStr

nl_eval is now an instance of the class _FuncPtr. To be safe, we want to make sure that only strings get passed to the function and to access the returned data as a string.
ctypes provides the convenience properties argtypes and restype for this purpose.
argtypes is set to lists of argument types (using the c-mapped types here).
restype is a single c type:

nl_eval.argtypes = [c_char_p]
nl_eval.restype = c_char_p

We are now ready to evaluate newLISP code:

nl_eval('(+ 2 2)') # => '4'
nl_eval("(apply + '(4 3 2 5 4))") # => '18'

A more general function can be defined to coerce the desired result type after nl_eval returns:
<

def newlisp_eval(code, return_type=str):
    return return_type(nl_eval(code))
 
newlisp_eval('(+ 2 2)', int) # => 4

What’s even better is that because the GIL is released and the library (in this instance) is not directly modifying any Python data, we can make effective use of threading.

Finally, here is a simple module putting all this together:

from ctypes import *
from ctypes.util import find_library
 
class LibraryNotFound(Exception):
    pass
 
class newLISP(object):
    def __init__(self):
        found = find_library('newlisp')
        if found is None:
            raise LibraryNotFound()
        else:
            self.lib = CDLL(found)
            self._eval = self.lib.newlispEvalStr
            self._eval.argtypes = [c_char_p]
            self._eval.restype = c_char_p
 
    def eval(self, code, return_type=str):
        code = c_char_p(code)
        result = self._eval(code)
        if result:
            result = result.strip()
            return return_type(result)
        else:
            return None

Links:

• Python ctypes documentation

Submit article

Pyrex has several advantages over brute force solutions like SWIG. Pyrex makes the creation of new built-in data types a walk in the park. There are no cryptic functions for dealing with structs or unions. Perhaps most significantly, though, is the fact that modules written for Pyrex are written in a syntax very close to Python. In fact, according to the Pyrex website, “the fundamental nature of Pyrex can be summed up as follows: Pyrex is Python with C data types.”

Python is simple, elegant, and productive. That is why Pythonistas use Python instead of C. The one place where C is required is when Python lacks a particular functionality that would be too expensive if implemented in pure Python. Pyrex solves this by allowing the implementation to be written in Python, but then compiled to C (which can then be compiled to machine code.)

In the previous article on SWIG and Python, I wrote a simple C extension that counted the occurrences of each ASCII character in a string. The reason for this is that iteration over Python’s strings is slow. So let’s create a new data type using Pyrex, the CharIterator. Our type will be constructed from a Python string and be maintained internally as a character array. The type will implement the Python iterator and return the ASCII code for the current index’s character. Here is our simple Pyrex implementation, which we will call mymod.pyx:

cdef class CharIterator:
    cdef char *value
    cdef int index
    cdef int last_char
 
    def __init__(self, char *value):
        self.value = value
        self.index = -1
        self.last_char = len(value)-1
 
    def __iter__(self):
        return self
 
    def __next__(self):
        if self.index <= self.last_char:
            raise StopIteration
        self.index = self.index + 1
        return self.value[self.index]

CharIterator maintains state in self.index, which is one below the current 0-base index. The reason for this is so that the character being returned from __next__ does not need to be stored in a temporary variable while index is incremented. The other possibility would be to return self.value[self.index-1], but that would be inelegant and inefficient (although inc/dec operations have minimal overhead). More information about the specifics of Pyrex syntax (such as cdef and when to use static type declarations) may be found in the Pyrex documentation.

Now, compile the module. Write a simple setup.py:

from distutils.core import setup
from Pyrex.Distutils.extension import Extension
from Pyrex.Distutils import build_ext
 
setup(
  name = 'mymod',
  ext_modules=[Extension("mymod", ["mymod.pyx"])],
  cmdclass = {'build_ext': build_ext}
)

…and run “python setup.py build”, then “python setup.py install” to install to the Python site-packages directory. You can also skip the last command and manually copy mymod.o from the build/lib.path_based_on_os_and_version/ directory. Now test the module:

from mymod import CharIterator
 
text = "Hello world."
chars = CharIterator(text)
for c in chars:
    print c, "=", chr(c)
 
# Output:
# 72 = H
# 101 = e
# 108 = l
# 108 = l
# 111 = o
# 32 =
# 119 = w
# 111 = o
# 114 = r
# 108 = l
# 100 = d
# 46 = .

A little testing shows that our CharIterator is a little under twice as fast (roughly 40% on my machine) than iterating over a string array, counting occurrences of each character. The code used for the test was:

from time import clock
from mymod import CharIterator
 
def string_of_file(file_name):
    file_handle = open(file_name)
    file_text = file_handle.read()
    file_handle.close()
    return file_text
 
def string_of_int(i):
    if i > 32:
        return chr(i)
    elif i == 9:
        return " Tab"
    elif i == 10:
        return " Newline"
    elif i == 13:
        return " Carriage return"
    elif i == 32:
        return " Space"
    else:
        return " Not counted"
 
def count_chars_pyrex(text):
    chars = CharIterator(file_text)
    ascii = range(0, 128)
    counts = [0 for i in ascii]
    for c in chars:
        counts[c] += 1
    return convert_list(counts)
 
def count_chars_native(text):
    ascii = range(0, 128)
    counts =  [0 for i in ascii]
    for c in text:
        i = ord(c)
        counts[i] += 1
    return convert_list(counts)
 
def convert_list(lst):
    ascii = range(0, 128)
    counts = {}
    for i in ascii:
        char = string_of_int(i)
        if char in counts.keys():
            counts[char] += lst[i]
        else:
            counts[char] = lst[i]
    return counts
 
def print_counts(counts):
    chars = counts.keys()
    chars.sort()
    for k in chars:
        v = counts[k]
        if v > 0:
            print "%s: %d" % (k.lstrip(), v)
 
if __name__ == '__main__':
    file_name = "/Users/jober/Desktop/Projects/war_and_peace.txt"
    file_text = string_of_file(file_name)
 
    start = clock()
    native_counts = count_chars_native(file_text)
    end = clock()
    native_time = end-start
    print "Native implementation ran in %f seconds." % native_time
 
    start = clock()
    pyrex_counts = count_chars_pyrex(file_text)
    end = clock()
    pyrex_time = end-start
    print "Pyrex implementation ran in %f seconds." % pyrex_time
 
    print "----------"
    print "Subject text is %d characters long." % len(file_text)
    if pyrex_time < native_time:
        print "Pyrex implementation was faster by %f seconds." % (native_time - pyrex_time)
    else:
        print "Native implementation was faster by %f seconds." % (pyrex_time - native_time)
 
    print "----------"
    print_counts(pyrex_counts)

More information about Pyrex:

• Pyrex homepage
• Documentation
• Tutorial (http://ldots.org/pyrex-guide/ is apparently gone for good)
• Writing an executable with Pyrex

Submit article

I recently began a seemingly simple project that involves counting the number of each character in a potentially large string. The program only needs to consider ascii characters, with a few significant exceptions- smart quotes and ticks in Word documents must be counted as ascii single and double quotes, respectively. Happily, Python’s re is UTF-aware and so can make these substitutions for us. However, iteration in Python, especially over a potentially large string, is expensive, and using regular expressions for this sort of thing is not to be thought of. Regular expressions are expensive to begin with, especially if we are performing 128 of them (one for each ascii character that could potentially appear in a string), each over the entire string. The best performance will be had by iterating over each character in the string and incrementing a dictionary/list/array/whatever containing counts of each character.

I tried this in pure Python. The performance was average. I used the The Project Gutenberg free online copy of War and Peace. On a quad-core G5 PowerMac, it took several seconds just to iterate through the string, let alone access a dictionary to store counts. Using a Python iterator did not increase performance significantly.

So I wrote a simple C module with a function that accepts a string pointer, iterates through each character, incrementing an array (using the int cast of the character as the character’s index in the array). It was as fast as can be expected from C, processing the entire file in under one tenth of a second. Here is the code (without the test function that iterated over the file itself- that will not be necessary from Python). Included is a helper function, count_of_char, which is only there to make access to the TALLY struct more simple from Python (tally.c):

#include 
#include 
#include 
#define CHARS 128
 
typedef struct
{
    char *text;
    int counts[CHARS];
} TALLY;
 
TALLY init(char *str);
void count_chars(TALLY *t);
int count_of_char(TALLY *t, char c);
 
TALLY init(char *str)
{
    int i;
    TALLY t;
    t.text = str;
    for (i = 0; i < CHARS; i++)
    {
        t.counts[i] = 0;
    }
    count_chars(&t);
    return t;
}
 
void count_chars(TALLY *t)
{
    int i;
    for (i = 0; i < strlen(t->text); i++)
    {
        t->counts[(int) t->text[i]]++;
    }
}
 
int count_of_char(TALLY *t, char c)
{
    return t->counts[(int) c];
}

Now, how to make this accessible to Python? It took me a matter of an hour or so to write the C library. Writing the Python API should not take longer than the library itself, so I decided to use SWIG to generate the necessary code. SWIG uses an interface file rather than parsing the actual C library. Here is the simple one I used for the library above (tally.i):

%module tally

%{
    #define CHARS 128

    typedef struct
    {
        char *text;
        int counts[CHARS];
    } TALLY;

    extern TALLY init(char *str);
    extern int count_of_char(TALLY *t, char c);
    extern TALLY init_from_file(char *file_path);
%}

typedef struct
{
    char *text;
    int counts[CHARS];
} TALLY;

extern TALLY init(char *str);
extern int count_of_char(TALLY *t, char c);
extern TALLY init_from_file(char *file_path);

That was straight out of the example in the SWIG docs for Python. It doesn’t take much. Often, an interface file is not even required. I then ran swig -python tally.i. This created a few files (tally_wrapper.c and tally.py). Now, since we are using Python and have access to distutils, we can write a simple setup.py file to do the work of compiling and linking for us:

import distutils
from distutils.core import setup, Extension
 
setup(name = "Tally",
      version = "1.0",
      ext_modules = [Extension("_tally", ["tally.i","tally.c"])])

Note the naming conventions here. The extension library must end up named _<name&rt;.so, with the leading underscore, for SWIG. Now, we can just run python setup.py build, and then python setup.py install to compile and install our module. From there, we can use a simple Python CGI (or FastCGI or mod_python app, depending on what your host supports) that reads in the string from a web form and uses our new library, tally, to count the characters.

from operator import itemgetter
import tally
 
def tally_string(str, min=0, sort=False):
    t = tally.init(str)
    chars = {}
    for i in xrange(0, 127):
        x = tally.count_of_char(t, i)
        if x >= min:
            chars[i] = x
    if sort:
        chars = sorted(chars.items(), key=itemgetter(1), reverse=True)
    return chars

Note the use of itemgetter to sort the dictionary we created by value. The reason that I decided to create the Python storage from Python rather than C was that I was trying to avoid using the Python API in my library at all to maintain its usefulness in other applications. Now I don’t have to change the source code at all to use this library in another app. Additionally, I had never used SWIG before and did not want to overcomplicate matters by messing with lower-level stuff, especially since the big performance problem (the string iteration) had been solved, and I now had cycles to spare.

To see the code in action, here is a working version. This is a work in progress, obviously, but it demonstrates the speed gain.

Submit article

newLisp uses the ‘import function to define a function from a foreign library. The syntax is (import file-path func-name). For example, if we wished to use libmysqlclient, as in the previous tutorials,

(setq libmysqlclient "/path/to/libmysqlclient.15.dylib")
(import libmysqlclient "mysql_init")
(import libmysqlclient "mysql_close")

Now, both mysql_init and mysql_close have been wrapped for us and can be used as regular newLisp functions. From the MySQL C API docs, we know that mysql_init takes a pointer as a parameter and returns a pointer to a MYSQL type struct. In newLisp, calling the function with a NULL pointer and assigning the new MYSQL pointer is as easy as,

(setq *mysql* (mysql_init 0))

That’s it. The symbols mysql now returns a foreign address. Since we don’t ever need to access it, we can simply pass mysql to our other functions, such as mysql_close, directly:
<

(mysql_close *mysql*)
The MySQL docs describe mysql_real_connect as:
<pre lang="c">MYSQL *mysql_real_connect(MYSQL *mysql, const char *host,
    const char *user, const char *passwd, const char *db, unsigned int port,
    const char *unix_socket, unsigned long client_flag)

That’s a long definition, but it accepts NULL for the last four arguments. So, in newLisp:

(mysql_real_connect *mysql* "someuser" "secret" "mydatabase" 0 0 0)

Easy! This will reassign mysql if successful or return NULL if it fails (it is a destructive function). To test for this, we can define our own simple wrapper for convenience:

(define (connect host user pass (db 0) (port 0) (socket 0))
    (not (zero? (mysql_real_connect *mysql* host user pass db
        port socket 0))))

This accepts our parameters and defaults db, port, and socket to C NULL. It then calls the function and verifies evaluates to true or nil, depending on whether or not the function returned 0, which is the C NULL.

Next, we will discuss how to run a query and access results stored in a struct. Note: I never wrote the second part to this article. Sorry.

Submit article

Before that can be done, however, a note for those who do not know C. Functions in C have a declared type, as do all of their arguments. When reading the docs for a particular library, you will want to know description conventions for C routines. Let’s use the example of mysql_init() to begin with.

MYSQL *mysql_init(MYSQL *mysql)

The first part, MYSQL *mysql_init, defines the function’s return type and name. MYSQL is a type created by the library that stores a struct. This is actually completely irrelevant at this point, because of the asterisk. The asterisk means that the function is actually allocating a pointer. Although we can pass an already existing MYSQL object pointer (that’s the second part), generally we will just use this function to create the internal struct needed to run the program. We will need to store the pointer that is returned.

The second part, MYSQL *mysql, states this- namely, that a pointer (named mysql, and we know it’s a pointer because it’s labelled with an asterisk- *mysql) will be returned. We store it because this will be passed to most of the other routines in the library.

Technically, we do not really need to define the routine in Lisp. The function, foreign-funcall, allows us to call the foreign function directly.

(foreign-funcall "mysql_init" :pointer (null-pointer) :pointer)

In this, we apply mysql_init to a null pointer (an empty pointer argument), created using the CFFI function (null-pointer). The last symbol states we will get a pointer back.

In order to wrap this as a pretty Lisp function, we do:

(defcfun ("mysql_init" libmysql-init) :pointer
  (mysql :pointer))

This points the Lisp symbol libmysql-init to “mysql_init” in the C library. The next argument, :pointer, states that this returns a pointer. Next, we can have an &rest list of (parameter-name :type), to define the parameters to pass to the function, in this case, (mysql :pointer). Available types are listed in the CFFI documentation.

However, we don’t want to have to run (null-pointer) every time we use this function. We also need to store the return value for future use. So let’s wrap it in another function to make it more expressive:

(defun mysql-init ()
  (setq *mysql* (libmysql-init (null-pointer)))
  (or *mysql* (setq *mysql* nil)))

Now, we can simply run (mysql-init) and we will have the symbol *mysql* pointing to the returned pointer or nil in the event of a failure.

If we do have a failure, we can use the following definitions to get at the error text:

(defcfun ("mysql_error" libmysql-error) :string
    (mysql :pointer))
 
(defun mysql-error ()
    (libmysql-error *mysql*))

Of course, we will want to add error handling and some other niceties, but having these declared makes writing and testing those completed functions that much easier.

Submit article

My database of choice (and the one available to me on my host) is MySQL. Unfortunately, this is the piece of CLSQL that I can’t seem to get running on OSX. In every dialect I’ve tried (sbcl, cmucl, clisp, openmcl, ad infinitum), CFFI/UFFI cannot find the libmysqlclient library- even when it has been pushed onto the library search path… but that is a rant for a different blog (or Usenet).

CFFI is generally available via ASDF and plays well with GNU CLISP. It has a fairly simple, intuitive syntax for making C routines available to lisp (as a side note, I have not encountered anything as easy as newLisp’s C library importer, although this is made easier in that newLisp is implemented in C).

Using CFFI, we can fairly easily create mappings for the most important MySQL C api functions. This entry will walk through the process of importing the library; future entries will go one to defining a few of the most basic routines for our lisp image.

Let’s begin by defining our mysql wrapper package.

(defpackage :mysql-api
  (:use :common-lisp :cffi))
(in-package :mysql-api)

To load the library itself, we use the function, define-foreign-library, in the CFFI package. The first argument to define-foreign-library is the symbol we wish to point to the library. Next, we create list similar in syntax to cond, where we define the names of our libraries based on, for example, the host OS. Since I happen to know that CFFI does find the library path even when explicitly specified, we can conveniently specify a list of full paths to try for each OS. The full list of platform symbols available here.

(define-foreign-library libmysqlclient
  (:darwin (:or "/usr/local/mysql/lib/libmysqlclient.dylib"
                "/sw/lib/libmysqlclient.dylib"
                "/usr/local/lib/libmysqlclient.dylib"))
  (t (:default "libmysqlclient")))

The last line is a catch-all, and will format the filename based on the OS (i.e., libmysqlclient.so for unix, .dll for Windows). Although I know my library is located in /usr/local/mysql/lib, I have included various other commonly used paths for MySQL libraries in OSX.

Then, to load the library:

(use-foreign-library libmysqlclient)

This will return a pointer to a foreign address (for the C-challenged, a pointer is a variable to points to a value’s address in memory; it is used to access a value by reference, rather than by value) or, in the event that it did not work, it will throw up an error. Generally, the error will be caused by CFFI not being able to find the library. If this is the case, verify that your account has access to the file and that the file is included in one of the paths above.

Next, we will begin defining our C routines in lisp.

Submit article