Basic syntax

Like many newLISP sequence functions, find-all is defined for both strings and lists. The basic syntax is:

Strings: (find-all pattern target [expression [option]])
Lists:   (find-all pattern target [expression [comparison-function]])

String searches and regular expressions

The most basic string search finds occurrences of one string inside of another.

(set 'str "Now is the time for all good men to come to the aid of their country.")
(find-all "the" str)
; ("the" "the" "the")

There are three occurrences (including one within the word, ‘their’). To find only occurrences of ‘the’ as a complete word, a regular expression is used:

(find-all {\bthe\b} str $0 0)
; ("the" "the")

The first parameter is a regular expression, defined in curly braces to eliminate the need to double-escape entities (like b). The final parameter, 0, is the PCRE option (a full list of options is available in the newLISP documents for regex.

The third parameter is key to one of the more powerful features of find-all – substitution. The expression parameter is applied to each element found. Inside of this expression, the global variable $0 represents the entire matched element. In a regular expression search, captured matches are available via subsequently enumerated variables: $1, $2, etc.

For example, to convert all occurrences of ‘the’ to upper case in the resulting list:

(find-all {\bthe\b} str (upper-case $0) 0)
;("THE" "THE")

Here is a short program to count the number of occurrences of some common words in the text of War and Peace:

(set 'text (read-file "war_and_peace.txt"))
(set 'words (find-all {\b\S+\b} text $0 1)) ; split into words
 
(println (format "%12s: %6d" "Total words" (length words)))
(letn ((common-words '("and" "or" "the"))
       (counts (map 'list common-words (count common-words words))))
  (dolist (word counts)
    (println (format "%12s: %6d" (word 0) (word 1)))))

With the results:

 Total words: 564345
         and:  21023
          or:   1542
         the:  31702

This isn’t necessarily efficient; finding all words and then counting occurrences of three specific words in a list of more than half a million elements results in a lot of wasted time. The regular expression used also includes opening and closing punctuation in the word (i.e. a quote that begins “The…” would result in a missed occurrence of ‘the’, because the list would contain “The instead. It also does not account for case when counting occurrences.

Here is a more efficient version:

(set 'common-words '("and" "or" "the"))
(set 're (format {\b(%s)\b} (join common-words "|")))
(set 'occurrences (find-all re text (lower-case $0) 1))
(println (map 'list common-words (count common-words occurrences)))
; (("and" 22267) ("or" 1582) ("the" 34619))

Note that find-all is recursive and newLISP has a limited stack size, which can be set during execution with the -s switch:

newlisp -s 500000 common_counts.lsp

The previous example would hit the default stack limit of 2,048. This can be solved by using map instead of find-all‘s substitution expression:

(set 'common-words '("and" "or" "the"))
(set 're (format {\b(%s)\b} (join common-words "|")))
(set 'occurrences (map 'lower-case (find-all re text $0 1)))
(println (map 'list common-words (count common-words occurrences)))
; (("and" 22267) ("or" 1582) ("the" 34619))

The substitution expression is also useful to cause side effects, squeezing even more efficiency out of the algorithm. Here, find-all increments a count of each word and stores it in a dictionary.

(define counts:counts)
 
(set 'common-words '("and" "or" "the"))
(set 're (format {\b(%s)\b} (join common-words "|")))
(find-all re text (counts (lower-case $0) (+ 1 (or (counts (lower-case $0)) 0))) 1)
 
(dolist (w common-words)
  (println (format "%4s: %6d" w (counts w))))
 
 ; and:  22267
 ;  or:   1582
 ; the:  34619

This algorithm is quite fast, although still somewhat weighty in RAM, since it stores the entire text before doing its work.

List searches

find-all list searches are (arguably) nearly as powerful as regular expressions. By default, find-all compares elements of target with pattern using match, and the default expression is the entire matched element ($0):

(set 'target '((a 1) (b 2) (c 3) (d 4 5)))
 
(find-all '(? ?) target)
; ((a 1) (b 2) (c 3))
 
(find-all '(d *) target)
; ((d 4 5))

Using the substitution expression, lists may be unified or matched further:

(find-all '(? ?) target (unify '(Letter Number) $0))
; (((Letter a) (Number 1)) ((Letter b) (Number 2)) ((Letter c) (Number 3)))

Simple list searches using other comparators than match are straight-forward:

(find-all 4 '(1 2 3 4 5) $0 >)
; (1 2 3)
 
(find-all 4 '(1 2 3 4 5) (+ 1 $0) >)
; (2 3 4)

find-all is convenient for searching XML. newLISP parses XML into a tree. For example:

(xml-type-tags nil nil nil nil)
(set 'xml (xml-parse (get-url "http://www.weather.gov/xml/current_obs/index.xml")
    (+ 1 2 4 16)))

For more info on parsing XML in newLISP, see this article and the newLISP documentation.

The XML at the url above lists weather stations. Here is an example station entry, under the root element, “wx_station_index”:

<station>
	<station_id>TAPA</station_id>
	<state>AG</state>
	<station_name>Vc Bird Intl Airport Antigua</station_name>
	<latitude>17.117</latitude>
	<longitude>-61.783</longitude>
	<html_url>http://weather.noaa.gov/weather/current/TAPA.html</html_url>
	<rss_url>http://weather.gov/xml/current_obs/TAPA.rss</rss_url>
	<xml_url>http://weather.gov/xml/current_obs/TAPA.xml</xml_url>
</station>

assoc is useful for finding a path in a parsed XML list:

(set 'stations-xml (assoc (xml "wx_station_index")))

However, assoc only returns the first element of the list that matches. In the XML article linked above, pop-assoc was used to collect elements iteratively:

(set 'stations '())
(while (assoc (xml "wx_station_index" "station"))

That is a handy way of collecting all elements, especially in an irregular document. With find-all, elements may be found just as easily and without modifying the original list:

(find-all '("station" *) (assoc (xml "wx_station_index")))

From there, it is just as simple to aggregate the values from each element:

(set 'station-pattern
    '("station"
        ("station_id" ?)
        ("state" ?)
        ("station_name" ?)
        ("latitude" ?)
        ("longitude" ?)
        ("html_url" ?)
        ("rss_url" ?)
        ("xml_url" ?)))
 
(find-all '("station" *) (assoc (xml "wx_station_index"))
    (match station-pattern $0))

This returns a list of the node values for each station. The data can be applied to a different associative structure easily using unify:

(find-all '("station" *) (assoc (xml "wx_station_index"))
    (unify '(Id State Name Lat Lon Html Rss Xml)
        (match station-pattern $0)))

Each element in this list this creates looks like:

((Id "TAPA")
 (State "AG")
 (Name "Vc Bird Intl Airport Antigua")
 (Lat "17.117")
 (Lon "-61.783")
 (Html "http://weather.noaa.gov/weather/current/TAPA.html")
 (Rss "http://weather.gov/xml/current_obs/TAPA.rss")
 (Xml "http://weather.gov/xml/current_obs/TAPA.xml"))

A completely impractical example

There are further implications of find-all‘s ability to cause side effects in the substitution expression. To download the XML for each individual station, something like this could be done (don’t really do this – it will attempt to spawn more than 2,000 processes):

(set 'processes
    (find-all '("station" *) (assoc (xml "wx_station_index"))
      (spawn (sym (lookup "station_name" $0) 'WEATHER)
             (get-url (lookup "xml_url" $0)))))
 
(sync 30000) ; wait 30 seconds for all downloads to finish

Assuming that the application is able to download the XML file for more than 2,000 stations in 30 seconds (and that there is no hard limit to forked processes, as there is in OSX), the context WEATHER will contain the XML for all stations.

The infeasability of this example aside, it demonstrates, as an example, how easily a user-supplied XML file could be used to script a newLISP application.

Further documentation

• find-all
• unify
• match
• regex
• xml-parse

Submit article

What makes newLISP’s XML handling great?

The majority of modern languages take an object oriented approach to XML. They parse the document and store it in an object. The ugliest use the functions defined in the DOM specification. A couple have more expressive interfaces, such as PHP’s simplexml, or merely novel, such as Python’s event-based handling.

Each of these languages have list structures and an excellent, established array of functions for dealing with them. So why use a complex, arcane interface? Why not simply translate the XML into a list?

The fundamental structure in lisp is the linked list. Lists can be of arbitrary length and depth and can be used to create infinitely complex structures. newLISP parses an XML string and converts it into a list.

Lisp’s strength is list processing. Working with XML as a list is therefore a simple task. Elements can be matched and located using match and unify or by use of a recursive function.

What doesn’t work?

newLISP is lacking in a function to convert a list back to XML. However, this is not too difficult to achieve on our own.

A concrete example

I recently built an application to locally cache information from another program which stores photos and publishes photos’ metadata via XML. The application that uses the cached data is written in Python, but I have never liked the way that Python (or most languages) deal with XML, so I decided to write this module in newLISP.

To download the remote document:

(define (get-xml-data url)
    (let ((xml-data (get-url url 3000)))
        (if (regex "ERR: (.+?)$" xml-data)
            (throw-error (format "download error: %s" $1)))
        xml-data))

This function downloads a url and returns it as a string. If the host times out after 3000 milliseconds or there is another error (which get-url stores in its return value preceded by “ERR:”), we throw an error. We match this using regex so that any error text can be captured in the system variable $1. Otherwise the XML is returned as a string.

We can then define a function to parse the result and verify its validity:

(define (parse-xml-data str)
    (xml-type-tags nil "CDATA" nil nil)
    (let ((xml (xml-parse str 5)))
        (or xml (throw-error (xml-error)))))

Let’s walk through this function. First, xml-type-tags defines how we want xml-parse to deal with TEXT, CDATA, COMMENT, and ELEMENT type tags. By default, these are included as strings in each node. I only want to know if something is CDATA and may contain oddball characters.

The xml-parse function’s third parameter is formed by adding the value of various options together. Granted this is a very C-like approach and not extremely expressive, but hey, life isn’t fair. Here is a list of the options and values (from the newLISP documentation):

1    suppress whitespace text tags
2    suppress empty attribute lists
4    suppress comment tags
8    translate string tags into symbols
16   add SXML (S-expression XML) attribute tags

From this we can see that we have remove whitespace tags and comments. I do not recommend suppressing empty attribute lists; there is little way to intuit then whether or not the first item in an element is its attribute list or a child element. We then evaluate to either the XML data or, if XML is nil (meaning there was an error parsing the XML), we throw an error.

We now have our data. Here is a fragment of the XML document and its resulting parsed list:

 
        9374
        /some/path/on/the/server
        something.jpg
something_thumb.jpg
        January 9, 2008 4:55 PM EST
        January 9, 2008 5:20 PM EST
 
        9375
        /some/path/on/the/server
        something2.jpg
something_thumb2.jpg
        January 9, 2008 4:56 PM EST
        January 9, 2008 5:21 PM EST

'(("album" ()
    ("media"
      (("type" "photo"))
       ("id" () "9374")
       ("media_root" () "/some/path/on/the/server")
       ("fullsize" () "something.jpg")
       ("thumbnail" () "something_thumb.jpg")
       ("authored" () "January 9, 2008 4:55 PM EST")
       ("submitted" () "January 9, 2008 5:20 PM EST"))
    ("media"
      (("type" "photo"))
        ("id" () "9375")
        ("media_root" () "/some/path/on/the/server")
        ("fullsize" () "something2.jpg")
        ("thumbnail" () "something_thumb2.jpg")
        ("authored" () "January 9, 2008 4:56 PM EST")
        ("submitted" () "January 9, 2008 5:21 PM EST"))))

The similarities between XML and lisp lists are obvious (as is the wordiness of XML.)

Accessing the first media element is simple: (assoc (xml "album" "media")). Note that the full path is used to find the nested association. However, that will return only the first association found for the key “media”. What if we want all media elements?

(define (media xml , lst)
(while (assoc (xml "album" "media"))
  (push (pop-assoc (xml "album" "media")) lst -1))
lst))

This function shows how pop-assoc can be used to reduce a list.
pop-assoc returns the popped association, which is then pushed onto the list that gets returned. The result of this function:

'((("media"
    ("id" "9374")
    ("media_root" "/some/path/on/the/server")
    ("fullsize" "something.jpg")
    ("thumbnail" "something_thumb.jpg")
    ("authored" "January 9, 2008 4:55 PM EST")
    ("submitted" "January 9, 2008 5:20 PM EST"))
   ("media"
    ("id" "9375")
    ("media_root" "/some/path/on/the/server")
    ("fullsize" "something2.jpg")
    ("thumbnail" "something_thumb2.jpg")
    ("authored" "January 9, 2008 4:56 PM EST")
    ("submitted" "January 9, 2008 5:21 PM EST"))))

Exporting a list to XML

This is an area where newLISP is lacking. There is no function to return a list to XML. To that end I’ve written a basic XML module to automate parsing XML in a defined manner and then writing the list back to XML. One neat thing I have done in this module is to make the xml-type-tag symbols evaluate to macros that each know how to render a node as a string.

So long as the sxml conventions are maintained (a full element is (element "tag-name" (@ attribute-list) child-node-list), i.e. (XML:element "a" (@ ("href" "http://www.artfulcode.net")) ((XML:text "An excellent website"))) => <a href="http://www.artfulcode.net">An excellent website</a>) and empty attributes lists are provided (as (@)), conversion back and forth is straigh-forward.

If the XML string that is parsed has a declaration, it will be maintained. Otherwise, it uses a generic UTF-8 declaration. Here is the code:

(context 'XML)
 
(set 'declaration {< ?xml version="1.0" encoding="UTF-8" ?>})
 
(define-macro (text)
  "Formats plain text."
  (string (args 0)))
 
(define-macro (comment)
  "Formats comments."
  (format {<!-- %s -->} (string (args 0))))
 
(define-macro (cdata)
  "Formats CDATA nodes."
  (format "&lt; ![CDATA[%s]]&gt;" (string (args 0))))
 
(define-macro (@ attr)
  "Turns an s-xml attribute list into an XML attribute list."
  (cond
    ((null? attr) "")
    ((list? attr)
      (format { %s="%s"%s}
        (string (attr 0))
        (string (attr 1))
        (eval (cons @ (args)))))))
 
(define-macro (element tag-name attributes children)
  "Recursively evaluates the contents of the s-xml list."
  (let ((string-format {&lt; %s%s&lt;%s&gt;/%s&gt;})
        (items (list (string tag-name)
                     (eval attributes)
                     (join (map eval children) "")
                     (string tag-name))))
       (format string-format items)))
 
(define (xml-&gt;list xml-string)
  "Parses XML string xml-string. Returns a list in s-xml format."
  (if (starts-with xml-string "&lt; ?xml")
    (begin
      (regex {^(&lt;\?xml .+? \?&gt;)$} xml-string 16)
      (set 'declaration $1)))
  (xml-type-tags 'text 'cdata 'comment 'element)
  (first (xml-parse xml-string 17)))
 
(define (list-&gt;xml xml-lst)
  "Evaluates an s-xml-formatted list to XML."
  (format "%s\n%s" declaration (eval xml-lst)))
 
(context MAIN)

The declaration can be manually set via the symbol XML:declaration. The XML node symbols element, text, comment, cdata, text, and @ are all expected to be XML context symbols. The main forward-facing functions are xml->list and list->xml. They are used, simply enough, like this:

(set 'xml-string {
 
        9374
        /some/path/on/the/server
        something.jpg
something_thumb.jpg
        January 9, 2008 4:55 PM EST
        January 9, 2008 5:20 PM EST
 
        9375
        /some/path/on/the/server
        something2.jpg
something_thumb2.jpg
        January 9, 2008 4:56 PM EST
        January 9, 2008 5:21 PM EST
 
})

(set 'xml-list (XML:xml->list xml-string))

Now, xml-list is:

(XML:element "album" (@)
    ((XML:element "media" (@)
        ((XML:element "id" (@) ((XML:text "9374")))
        (XML:element "media_root" (@)
            ((XML:text "/some/path/on/the/server")))
        (XML:element "fullsize" (@) ((XML:text "something.jpg")))
        (XML:element "thumbnail" (@) ((XML:text "something_thumb.jpg")))
        (XML:element "authored" (@)
            ((XML:text "January 9, 2008 4:55 PM EST")))
        (XML:element "submitted" (@)
            ((XML:text "January 9, 2008 5:20 PM EST")))))
    (XML:element "media" (@)
        ((XML:element "id" (@) ((XML:text "9375")))
        (XML:element "media_root" (@)
            ((XML:text "/some/path/on/the/server")))
        (XML:element "fullsize" (@) ((XML:text "something2.jpg")))
        (XML:element "thumbnail" (@) ((XML:text "something_thumb2.jpg")))
        (XML:element "authored" (@)
            ((XML:text "January 9, 2008 4:56 PM EST")))
        (XML:element "submitted" (@)
            ((XML:text "January 9, 2008 5:21 PM EST")))))))

We can convert back with XML:list->xml:

(set 'new-xml-string (XML:list->xml xml-list))

Which makes new-xml-string:

< ?xml version="1.0" encoding="UTF-8" ?>
 
        9374
        /some/path/on/the/server
        something.jpg
something_thumb.jpg
        January 9, 2008 4:55 PM EST
        January 9, 2008 5:20 PM EST
 
        9375
        /some/path/on/the/server
        something2.jpg
something_thumb2.jpg
        January 9, 2008 4:56 PM EST
        January 9, 2008 5:21 PM EST

The only difference is that the new string has an XML declaration.

download

Submit article