It feels pretty top heavy, though. I think they tried to bolt the QNX UI on top of BB OS, but without the power and efficiency underneath the hood to make it perform adequately on my handset. The 9650 seems high end for OS 5 but low end for OS 6. Some of the rendering, smooth scrolling, and transparency effects really bog down the system, at least on the 9650’s hardware.

The only other real problem I have with it is that it is missing a feature I really like from OS 5 – the ability to send a picture via MMS directly from the browser.

For a while, I used an OS 5 theme which eliminated many of the more expensive UI elements, like app tabs, but at some point I lost the ability to use themes, even after a clean install.

So for now, I’m back to OS 5, and my phone is much snappier again. Oh well. Better to be a big fish in a little pond, I suppose.

Submit article

The Playbook has suffered from a rash of poor reviews, largely due to the lack of apps and RIM’s decision to release without  the native PIM suite ready. These things obviously detract from the experience, especially the absence of email and messaging software.

I try to take the long view, though. This is a brand new tablet platform, and software environment will eventually get a complete host of apps, some useful, some not, just as is the case with the Apple app store as well as the Android marketplace. For the moment, though, it is mostly filled with what appear to be very mediocre experiments with the platform. Very few are worth the money at the moment.

The system is very slick, though. The navigation becomes very natural with very little effort, and the browser is lightening fast. The bridge functionality is excellent, although it will frustrate the unenlightened (read: non-Blackberry users) that it requires a Blackberry hand set to use.

Bridge is extremely cool. It gets overlooked as a stop-gap until RIM releases the official, native PIM, but it gives Blackberry users something important that a more typical suite might easily overlook. Messages are pushed to a Blackberry, rather than synchronized, as is the case with IMAP. Bridge maintains control over those messages on the hand held, rather than the tablet. That’s the right way, because a phone is going to stay with you much more than a tablet (new toy syndrome notwithstanding).

Bridge gives me an extra piece of awesome, too. The Bridge Browser, while not running at native speed, provides a decent web experience without tethering charges. That not only saves me money, but it keeps me feeling young by letting me pretend that I am sticking it to the man, and you just can’t put a price on that.

Therefore, I really hope that bridge is not condemned once RIM releases their PIM suite. I really want to see them add to it and give it the polish that RIM is so good at. I hope it continues as a special extra for BB users, and becomes fully integrated, which is one of the major areas of polish were RIM lords it over the competition

On another note, I also very humbly ask that whoever is in charge of the keyboard make it behave exactly as it does on my Blackberry. I am tired of having to type apostrophes and capitalize with a separate button, thank you very much.

Submit article

What I like about TextMate

Let me repeat that I like TextMate. I have happily used it for years. It has some wonderful features. It is reasonably fast and its indentation is great. It has a great selection of plugins for a wide variety of languages and is easy to extend with new language modules. It has a decent macro system based on regular expressions that, while not nearly as powerful as scripted editors, provides 80% of what most users need.

Problems with TextMate

TextMate, for all its sterling qualities – and there are many – has a few problems that, while not deal-breakers in themselves, together provide a less-than-satisfactory experience.

Split-windows

This is a rather stark issue, and is a feature that has been promised in the next release. Nevertheless, its absence often means either filling my screen with open TextMate windows or firing up emacs when I need to look at many files at once – or many parts of the same file at once. C, in particular, tends toward long functions, and is especially painful to write in TextMate.

Large files

TextMate handles many open files well, but a single large file will hang the entire application. I can accept a large file taking a long time to load, but please! Load the file in the background, so that I can keep working. Even opening a large file in a second window blocks the entire application. Find and replace also choke on larger documents, sometimes taking minutes to complete.

Key-bindings

TextMate’s key-bindings can be puzzling. Many common commands (especially in Python), such as converting between spaces and tabs in a document, don’t have bindings and are several levels deep in the menu. Other commands of dubious utility are given simple bindings, such as “Execute line/selection inserting result” (control-R.) Many have extremely difficult combinations, such as control-alt-command-D to toggle to project drawer, or control-option-shift-P to toggle text mode (and then – because several bundles use this same key-binding, god knows why – you are given a drop-down to choose which command you wanted.)

Choosing a new editor

I was initiated into the cult of emacs in college and have used it for several years. I’ve played with BBEdit (which I think is horribly cluttered and completely lost in the pre-OSX world), NetBeans, and Eclipse (which is, amazingly, slower and clunkier than NetBeans.) I love emacs’ extensibility and its indentation is second to none. It suffers from the key-binding issue as well, though. Becoming efficient in emacs is a journey, not a destination. I find myself spending more time navigating its arcane and bureaucratic configuration system than actually writing code in it.

NetBeans in some ways seems very polished, and the new Python module is very good (I have written about this before.) Refactoring across files is extremely useful. It has good debugger integration. Unfortunately, tight integration with any particular language costs a great deal in load times and can lead to an unresponsive UI. And for god’s sake – why does it have to be such a chore to open a file outside of a project?

Panic’s Coda is pretty and has some neat features, but in the end, SubEthaEdit (its editor) has poor syntax support and its extension system feels very clunky. Editra is nice, but slow and buggy on OSX.

Vim

After a disappointing look around the net for better choices, I decided to give Vim another try. I had a passing familiarity with some of its basic functionality from my days as a system administrator, it is scriptable, and it has a large number of plugins for syntax support.

Lack of distractions

Overall, I am satisfied with Vim’s minimalism. Its indentation is pretty second rate, but I can live with it. Vim feels extremely comfortable. Over the years, Vim has gently washed away all of the “features” of an editor that actually get in the programmer’s way and come up with a system that drastically increases productivity and makes coding a more satisfying experience.

Load speed

While not the primary measure of an editor, load time is important. How many programmers (myself included) give up on Eclipse or NetBeans right away because it takes more than half a minute to start up? Load time is one of those things that, while really just an annoyance, can completely color the experience of using a program. Vim loads quickly. It is one less thing to get in the way of productivity.

Efficiency

Nothing beats Vim for efficiency. If you ever wonder why so many applications have modes that mimic Vim, it is because once you learn Vim’s bindings, there is nothing faster. Jumping around a file with precision is a breeze. Coming from emacs, I expected it to take a lot longer to become comfortable with Vim’s style and bindings, but it has a much milder learning curve than emacs. Search and replace is also fast and easy to use. I quickly found myself giving up MacVim in favor of the command line. Copy and paste – the biggest barrier to command-line editors for me – is fast and easy with Vim’s visual mode.

Other features

Vim 7 has auto-complete, although I rarely use it. I’m much more comfortable with an enormous stack of O’Reilly books. Vim’s code folding is also very nice. I generally turn code folding off because it slows most editors down so drastically, but it does not appear to affect Vim.

Two of Vim’s features really stand out for me: reformatting text and retabbing a document. I love being able to convert a file’s tabs to spaces just by typing :retab. It is much more efficient than having to go through two levels of menus in TextMate. And being able to reformat Python doc strings is very useful. It removes an annoyance that can make updating documentation a real chore.

Conclusion

Vim strikes the perfect balance for me. At first glance, it seems archaic, especially for those born after the mouse became standard. After a few days of using it, though, I find myself enabling Vim bindings mode in any application that has it, using the Vimperator plugin for Firefox, and wishing that OSX’s terminal was more easily scriptable. If you haven’t used Vim, give it a chance. Coming from a graphic editor like TextMate, it is a much different experience and, arguably, a more satisfying one.

Here are some other folks’ experiences with the transition in both directions:

• Coming home to Vim
• From Vim to TextMate

Edit [2009-08-03 15:50]: added links to similar stories

Submit article

The initial problem was getting it to find the MySQL headers and libraries installed by macports, which was easy enough to fix.

LDCONFIG="-L/opt/local/mysql/lib"
export LDCONFIG
CPPFLAGS="-I/opt/local/include/mysql5"
export CPPFLAGS

It found the files it needed, but it still didn’t pass the test.  It complained about not finding various API functions in the MySQL library. It turns out that it was detecting my system as 64bit (when it is in fact 32).  For that, I had to run make with CONFIGUREARGS:

make CONFIGUREARGS="--with-abi=32"

…and that did it.  Thank god for the good folks in #pike on irc.freenode.net!

Submit article

There are several slime profiling functions, but the simplest is slime-profile-package (M-x slime-profile-package), which profiles all functions in a package. You will be asked for the package name to profile. All functions in the package will be profiled. slime-profile-functions gives you more control over which functions to profile. The profiling of a single function can be controlled with slime-toggle-profile-fdefinition.

After entering the package name, you are asked whether to record the most common callers and whether to profile methods as well as typical functions. To view the results, you use slime-profile-report, which displays a table of the most commonly called functions, time spent in each, amount of consing in each, calls, total time, etc.

Before the next run, be sure to use M-x slime-profile-reset to reset the profiling statistics so you get a fresh and accurate report.

Optimizing functions

Generally, you should attempt to write a more efficient algorithm before depending on compiler optimization declarations. The slime profiler will tell you which functions are taking the majority of your application’s time, and making these more efficient will serve you better than tuning the manner in which a function is compiled. However, once your function is written to your satisfaction, you may wish to further tune it by sacrificing speed for lower memory usage (or vice versa) or trading debug information for extra speed.

Lisp compilers accept a few optimization declarations that can earn you significant speed-ups. The compiler generally optimizes code based on four variables: speed, safety, debug, and compilation-speed. These can be set between 0 and 3 with a declaration:

(defun foo ()
     (declare (optimize (speed 2)
                        (safety 3)
                        (debug 3)
                        (compilation-speed 0)))
     (format t "~&Hello world~%"))

Generally, lowering the debug value of a function makes trouble-shooting your application extremely difficult. For example, the name of the function may disappear from backtraces. Therefore, lowering this value is usually only worthwhile once the program has been suitably tested and deployed. Use the profiler to determine which portions of your code will provide the most benefit.

Macros and optimization

Macros are sometimes touted as a way to optimize code. In a compiled lisp program, they can cause some calculations and expansions to be performed at compile time, saving run-time. In situations where much of the information is known ahead of time, this can be an effective technique, although overuse can be dangerous. For example, rewriting a function as a macro will make debugging more difficult, since the macro is expanded in the compiled code. A problem in a macro may only be shown as a problem in the function that utilizes the macro.

Note that in interpreted lisps, especially those without any form of compilation (such as newLISP), macros provide no speed-up. In fact, in newLISP, macros are just functions that do not evaluate their arguments.

More info

slime profiling documentation

Submit article

Writing software almost always means processing some form of user input. When the format of that input is pre-determined and (mostly) guaranteed to be valid, regular expressions are just so much overhead. But when the validity of user input can vary widely, such as with form validation, regular expressions are much more concise than writing a custom parser.

Regular expressions are slower for simple matching

A regular expression must be compiled. Incautiously crafted patterns can take up quite a bit of memory or result in poorly performing matches. Routines that make heavy use of regular expressions can quickly become a bottlneck.

If a match operation doesn’t need features of regular expressions, such as alternating patterns or backreferences, it is faster to use string matching. It takes much less time to see if a string is terminated by a semi-colon in Python using string.endswith(';') than re.compile(r';$').search(string), especially over many iterations.

The exception is when a pattern is matched often over lengthy iterations. Once compiled, regular expression matching is O(log n). Regular string matching is typically O(n). However, it generally takes an extraordinarily large number of iterations before regular expressions become more efficient than string matching.

Optimizing for speed

Entire books have been written on this subject. Here are a few tips.

Alternating patterns (such as (abc|def)) can be expensive. Always try to put the pattern most likely to match first ((John|Rumplestiltskin), rather than (Rumplestiltskin|John)).

Avoid nesting repeating patterns when possible. They grow quickly in memory and increase the number of possible matches exponentially (thereby slowing a match down noticeably).

As the target string gets longer, matching slows drastically when using repeating patterns. Follow indefinite patterns with quantifiers ({min, max}) or with a literal or atomic group.

When possible, use anchors (^ and $) or lookahead/lookbehind to limit the scope of a pattern and make failures occur faster.

Optimizing for memory usage

Some types of expressions can grow to quite large sizes in memory. A little reading shows how apparently simple expressions can grow when compiled:

(abc|def){2,4}
is compiled as if it were
(abc|def)(abc|def)((abc|def)(abc|def)?)?

Indefinite and large quantifiers (*, +, and {min,max}) are expanded. Imagine if the example above were (abc|def){2,1000}, or if the pattern matched were more complex and contained internal quantifiers, such as (abc+|def){2,1000}. A little care is needed to prevent such exponential expansion.

One solution is to use a subroutine. Although slower, a subroutine calls back a previous match without the memory-eating expansion:

(abc+|def)(?1){1,999}

The subroutine backreferences the first grouped match and repeats it. However, subroutine matches are treated as atomic groups. When an indefinite match fails, the engine will typically backtrack and see if a smaller substring of the target matches.

For example, \w+0 matches one or more letters followed by a zero. If it is matched against “abcdefg0″, it matches all of the letters and the zero. If it is matched against “abcdefg1″, it will backtrack and attempt to match against substrings of the target (“abcdefg”, “abcdef”, “abcde”, …).

This may or may not be the desired behavior, but it will keep the pattern’s footprint down.

Links

• Regular Expression Matching Can Be Simple And Fast
• RegexKit docs
• Faster JavaScript Trim

Submit article

My experiences with this has been less than wonderful. I wrote a simple middleware class to import psyco as suggested here:

# Be sure to only load on the proper architecture
from platform import architecture
if not settings.DEBUG and architecture()[0] == '32bit':
    if architecture()[0] == '32bit':
        try:
            import psyco
        except ImportError:
            pass
 
class PsycoMiddleware(object):
    def process_request(self, request):
        # Do not waste time trying to optimize the re module
        psyco.cannotcompile(re.compile)
        # Limit memory usage
        psyco.profile(memory=2048)
        return None

In development this appeared to work fine (we excluded the if not settings.DEBUG portion while testing). Once we were up and running on the production server, however, mysterious exceptions began to surface.

In particular, we seemed to be “missing” on some page hits. Specifically, we would get TemplateDoesNotExist exceptions when the templates did, in fact, exist. We were never able to sort that out. They immediately stopped when we commented out the psyco-related stuff. We tried changing the memory usage, excluding more items from compilation (including the Django loader classes), all to no avail.

My recommendation is avoidance of psyco in a Django setting. If someone has an idea what might have caused this, I’d love to hear an explanation.

Submit article

1. The CACHE_MIDDLEWARE_ANONYMOUS_ONLY setting
2. The vary_on_cookie decorator
3. Template fragment caching
4. The low-level caching API

The CACHE_MIDDLEWARE_ANONYMOUS_ONLY setting

The CACHE_MIDDLEWARE_ANONYMOUS_ONLY setting causes Django to ignore the cache if the user is not anonymous. This is less helpful than it seems. At the Dayton Daily News, we require a trivial registration to access many areas of the site. Using this setting means that the entire page cannot be cached because of a simple “Welcome, username” line in the rendered view.

Another obstacle to using the site-level cache is that many demographic-tracking packages require setting client-specific Javascript variables in the rendered view and then accessing a script on another server. The per-site cache will cache these as well, distorting your analytics.

The vary_on_cookie decorator

The vary_on_cookie decorator (found in django.views.decorators.vary) is a simple way to tell upstream caches to cache a view based on the content of the user’s cookie. This means that each user will get their own page cached.

This is a useful decorator and a part of any caching setup for user-based sites. On its own, however, it still means that if a user visits a page only once, your server must perform all the work of rendering a page. The server gains no benefit when another user visits, since the page must be generated anew and then cached for this user as well.

Template fragment caching

This is a new feature in the development version of Django. It consists of a simple template tag that signals the framework to cache a portion of the rendered template. For example:

{% load cache %} {# thanks to AdamG for noticing the typo here #}
...stuff you don't want to cache
{ cache 300 "some" "section" user.id }
...stuff you do want to cache
{% endcache %}
...more stuff you don't want to cache

The cache tag accepts the number of seconds for which the cache should remain valid and a series of keys used to uniquely identify the cache. You may use any number of keys. Addng user.id to the mix will make this portion of the template cached on a per-user basis. Using something static will make it a standard, all-user, cache.

I experimented with template fragment caching while developing an application for which we expected extremely high traffic. In the end, we decided that the overhead did not justify the savings.

This is a very new feature and naturally not ready for production use. In the next stable release of Django I imagine that it will be considerably more efficient.

The low-level caching API

The low-level caching API is the solution for serious fine-tuning of your cache. It is located in django.core.cache. It is laughably simple to use:

from django.core.cache
 
CACHE_EXPIRES = 5 * 60 # 5 minutes
 
def some_view(request, object_id):
    cache_key = "someobjectcache%s" % object_id
    object_list = cache.get(cache_key)
    #if not object_list:
    #AdamG noted that this check avoids empty lists
    #evaluating to False, as "if not object_list" did
    if object_list is None:
        object_list = expensive_lookup()
        cache.set(cache_key, object_list, CACHE_EXPIRES)
    ...

The cache is accessed via a unique key. You can cache anything that can be safely picked in Python, including query sets from Django’s ORM. If the cache has expired, cache.get(key) returns None. Setting a key in the cache requires the unique key, the object to cache, and the time in seconds for which the cache is valid.

Since Django’s template engine is quite fast, we use the low-level API to cache the most expensive portions of each page: large database lookups, search results, the result of filtering large sets of data, ad infinitum.

This has given us the biggest savings in terms of memory, database hits, and CPU usage.

One final trick

A couple of our applications are real database hogs. They have a wide range of queries that get pulled over and over. Clearly, dropping into raw SQL and pulling lists rather than objects is the best way to streamline this type of demand, but then we lose the benefit of our custom model manager and model methods. Another neat trick is to add caching to your model manager itself:

from django.contrib.sites.models import Site
from django.core.cache
 
CACHE_EXPIRES = 5 * 60 # 10 minutes
 
class ObjectManager(models.Manager):
    def get_query_set(self, *args, **kwargs):
        cache_key = 'objectlist%d%s%s' % (
            Site.objects.get_current().id, # unique for site
            ''.join([str(a) for a in args]), # unique for arguments
            ''.join('%s%s' % (str(k), str(v)) for k, v in kwargs.iteritems())
        )
 
        object_list = cache.get(cache_key)
        #if not object_list:
        #AdamG noted that this check avoids empty lists
        #evaluating to False, as "if not object_list" did
        if object_list is None:
            object_list = super(ObjectManager, self).get_query_set(*args, **kwargs)
            cache.set(cache_key, object_list, CACHE_EXPIRES)
        return object_list

This custom model manager caches query sets using the arguments passed to get_query_set(). If they are fewer than 10 minutes old, they returned from the cache; otherwise, they are returned as a fresh query set and added to the cache. This technique can be used for busy databases to cache all possible queries performed by your application.

Submit article

The only feature of Code Igniter that I really miss in Django is the ability to add a page without first registering the url, formulating a regular expression to describe the parameters and variations of the view function, and then creating the view and template.

For a production website, this is a better method than Code Igniter’s strategy of http://domain.com/class/method/arg/arg/…/, but CI’s is much easier during development. My urls may change half a dozen (or more) times in the course of the project, especially if marketing or advertising is involved.

Fortunately, there is an easy fix. A generic, introspective function and a carefully crafted pattern in urls.py can save the day. The url pattern is simple enough:

(r'^/path/to/app/root/(?P.+?)/(?:(?P.+?)/)?$', free_pattern)

Placing this at the end of your urlpatterns tuple will allow any undefined urls to go through this as a last-chance catch-all. It captures the view as first segment (after the application’s root path) as well as any other text afterward (leaving out leading and trailing slashes).

Here is the function:

def free_pattern(request, view, segments=""):
    segments = segments.split(r'/')
    try:
        view_fn = getattr(PATH_TO_APP, view)
    except AttributeError:
        raise Http404
    return view_fn(request, *segments)

The function is simple enough. It accepts the request, the view’s name, and the rest of the segments. The segments are split by the forward slash to turn them into a list.

If the view is found in the appropriate module (defined using PATH_TO_APP, which should point to whichever application we are talking about), it is run with *segments expanded as its arguments. If the view is not found, we raise an Http404 error.

This could easily be expanded to introspect the path to the application, but since Django supports a number of different layouts for applications and projects, that will most likely be very customized to your setup.

For example, I typically store my applications in an apps module, each with its own urls.py file that is included at the project level. That way, an update to an application does not require synchronizing changes across multiple projects.

Submit article

Unfortunately, there is sometimes quite a bit of overhead associated with it. On the newLisp forums, someone asked for help replacing items from one list that matched items from another. He has two lists; list one contains about 30,000 strings. List two is a two dimensional list containing about 100,000 strings total.

The idea is to check for equality against the 30k strings in list one on each of the cells in list two, then replace that with a predefined string if equality is met.

The elegant solution is to use applicative and recursive algorithms. Naturally, the forum user used a recursive function, ref-all, which provides an index of placement of an atom in a list (i.e. (ref-all "baz" '("foo" "bar" "baz" "bat")) => 2). Then he mapped a lambda to list one that ref-all’d the item from list one in list two, replacing it with “FOO”.

The problem was that this was using a recursive function on a huge list over and over again, and newLisp is not optimized for large lists of strings as it is. It turns out that I was able to speed it up by using one of newLisp’s iterators, dolist, which simply iterates over a list. The code I ended up with was:

(set 'relacement "FOO")
(dolist (o one)
  (dolist (r (ref-all o two))
    (nth-set (two r) replacement)))

This reduced execution time by almost 10% on my machine (granted, my machine is powered by hamsters).

This just goes to show, efficiency is more important than elegance. If the code doesn’t do what it needs to do in an effective manner, what does it matter how pretty it is?

Submit article