By: Jeff

Jeff — Wed, 01 Dec 2010 02:52:48 +0000

When you get into result sets that large, you can end up spending more time instantiating objects or in I/O reading the cache entry (from a memcached socket, file system, or wherever) than you actually save from caching. A cache entry of 60,000 objects may be past the threshold where caching is beneficial. Try caching the primary key values as a list of numbers instead. That should serialize quickly, with little introspection overhead, and generate a much smaller string for the cache driver to store.

By: seb

seb — Sat, 27 Nov 2010 03:35:56 +0000

2583486 function calls (2463431 primitive calls) in 12.707 CPU seconds


   Ordered by: standard name

ncalls tottime percall cumtime percall filename:lineno(function) 1 0.162 0.162 12.707 12.707 :1() 1 0.000 0.000 0.000 0.000 __init__.py:172(field_cast_sql) 2 0.000 0.000 0.000 0.000 __init__.py:197(last_executed_query) 1 0.000 0.000 0.000 0.000 __init__.py:198(db_type) 1 0.000 0.000 0.000 0.000 __init__.py:210() 1 0.000 0.000 0.000 0.000 __init__.py:228(lookup_cast) 2 0.000 0.000 0.000 0.000 __init__.py:236(max_name_length) 1 0.000 0.000 0.000 0.000 __init__.py:263(get_db_prep_value) 2 0.000 0.000 0.000 0.000 __init__.py:273(compiler) 1 0.000 0.000 0.000 0.000 __init__.py:278(get_prep_lookup) 1 0.000 0.000 0.000 0.000 __init__.py:303(get_db_prep_lookup) 1 0.000 0.000 0.000 0.000 __init__.py:462(get_internal_type) 1 0.000 0.000 0.000 0.000 __init__.py:476(get_prep_value) 2 0.000 0.000 0.000 0.000 __init__.py:73(cursor) 2 0.000 0.000 0.000 0.000 __init__.py:80(make_debug_cursor) 2 0.000 0.000 0.000 0.000 base.py:163(_cursor) 2 0.000 0.000 0.000 0.000 base.py:197(execute) 2 0.000 0.000 0.000 0.000 base.py:215(convert_query) 60000 0.027 0.000 0.027 0.000 base.py:243(__init__) 60000 1.567 0.000 2.392 0.000 base.py:250(__init__) 59999 0.094 0.000 0.094 0.000 base.py:363(__reduce__) 23 0.000 0.000 0.000 0.000 base.py:85(quote_name) 2 0.000 0.000 0.000 0.000 compiler.py:11(__init__) 2 0.000 0.000 0.000 0.000 compiler.py:135(get_columns) 2 0.000 0.000 0.000 0.000 compiler.py:17(pre_sql_setup) 2 0.000 0.000 0.000 0.000 compiler.py:215(get_default_columns) 2 0.000 0.000 0.000 0.000 compiler.py:280(get_ordering) 24 0.000 0.000 0.000 0.000 compiler.py:31(quote_name_unless_alias) 2 0.000 0.000 0.000 0.000 compiler.py:418(get_from_clause) 2 0.000 0.000 0.000 0.000 compiler.py:462(get_grouping) 2 0.000 0.000 0.000 0.000 compiler.py:47(as_sql) 2 0.000 0.000 0.000 0.000 compiler.py:656(deferred_to_columns) 60002 0.103 0.000 1.350 0.000 compiler.py:666(results_iter) 2 0.001 0.000 1.214 0.607 compiler.py:703(execute_sql) 603 0.003 0.000 1.213 0.002 compiler.py:741() 43/8 0.000 0.000 0.000 0.000 copy.py:144(deepcopy) 13 0.000 0.000 0.000 0.000 copy.py:197(_deepcopy_atomic) 12/10 0.000 0.000 0.000 0.000 copy.py:224(_deepcopy_list) 6/3 0.000 0.000 0.000 0.000 copy.py:232(_deepcopy_tuple) 1 0.000 0.000 0.000 0.000 copy.py:251(_deepcopy_dict) 42 0.000 0.000 0.000 0.000 copy.py:261(_keep_alive) 3 0.000 0.000 0.000 0.000 copy.py:300(_reconstruct) 60005 1.999 0.000 6.518 0.000 copy_reg.py:59(_reduce_ex) 1 0.000 0.000 0.000 0.000 copy_reg.py:92(__newobj__) 2 0.000 0.000 0.000 0.000 datastructures.py:108(__deepcopy__) 4 0.000 0.000 0.000 0.000 datastructures.py:121(__iter__) 8 0.000 0.000 0.000 0.000 datastructures.py:138(items) 4 0.000 0.000 0.000 0.000 datastructures.py:141(iteritems) 9 0.000 0.000 0.000 0.000 datastructures.py:145(keys) 8 0.000 0.000 0.000 0.000 datastructures.py:151(values) 2 0.000 0.000 0.000 0.000 datastructures.py:154(itervalues) 2 0.000 0.000 0.000 0.000 datastructures.py:181(copy) 1 0.000 0.000 0.000 0.000 datastructures.py:453(__init__) 10 0.000 0.000 0.000 0.000 datastructures.py:84(__new__) 10 0.000 0.000 0.000 0.000 datastructures.py:89(__init__) 120000 0.053 0.000 0.053 0.000 dispatcher.py:139(send) 2 0.000 0.000 0.000 0.000 encoding.py:29(smart_unicode) 1 0.000 0.000 0.000 0.000 encoding.py:41(is_protected_type) 60002 0.077 0.000 0.120 0.000 encoding.py:54(force_unicode) 59999 0.090 0.000 0.210 0.000 fields.py:72(__set__) 3 0.000 0.000 0.000 0.000 functional.py:274(__getattr__) 2 0.000 0.000 0.000 0.000 loading.py:104(app_cache_ready) 2 0.000 0.000 0.000 0.000 loading.py:149(get_models) 1 0.000 0.000 0.000 0.000 locmem.py:46(get) 1 0.000 0.000 0.000 0.000 locmem.py:70(_set) 1 0.000 0.000 12.543 12.543 locmem.py:78(set) 2 0.000 0.000 0.000 0.000 manager.py:107(get_query_set) 1 0.000 0.000 0.001 0.001 manager.py:131(get) 2 0.000 0.000 0.000 0.000 manager.py:209(__get__) 1 0.000 0.000 12.544 12.544 models.py:14(get_query_set) 1 0.000 0.000 0.000 0.000 models.py:18() 1 0.000 0.000 0.001 0.001 models.py:7(get_current) 120007 0.058 0.000 0.058 0.000 options.py:196(_fields) 5 0.000 0.000 0.000 0.000 options.py:211(get_fields_with_model) 1 0.000 0.000 0.000 0.000 options.py:235(_many_to_many) 1 0.000 0.000 0.000 0.000 options.py:243(get_m2m_with_model) 1 0.000 0.000 0.000 0.000 options.py:265(get_field) 1 0.000 0.000 0.000 0.000 options.py:275(get_field_by_name) 1 0.000 0.000 0.000 0.000 options.py:314(init_name_map) 1 0.000 0.000 0.000 0.000 options.py:351(get_all_related_objects_with_model) 1 0.000 0.000 0.000 0.000 options.py:362(_fill_related_objects_cache) 1 0.000 0.000 0.000 0.000 options.py:388(get_all_related_m2m_objects_with_model) 1 0.000 0.000 0.000 0.000 options.py:399(_fill_related_many_to_many_cache) 2 0.000 0.000 0.000 0.000 options.py:436(get_parent_list) 2 0.000 0.000 0.000 0.000 options.py:480(pk_index) 2 0.000 0.000 0.000 0.000 query.py:105(__init__) 1 0.000 0.000 0.000 0.000 query.py:1102(add_q) 1 0.000 0.000 0.000 0.000 query.py:1144(setup_joins) 1 0.000 0.000 0.000 0.000 query.py:1345(trim_joins) 3 0.000 0.000 0.000 0.000 query.py:1465(can_filter) 1 0.000 0.000 0.000 0.000 query.py:1516(add_ordering) 1 0.000 0.000 0.000 0.000 query.py:1536(clear_ordering) 2 0.000 0.000 0.000 0.000 query.py:1686(get_loaded_field_names) 60006 0.032 0.000 0.032 0.000 query.py:1725(_aggregate_select) 11 0.000 0.000 0.000 0.000 query.py:1743(_extra_select) 1 0.001 0.001 0.001 0.001 query.py:176(__getstate__) 4 0.000 0.000 0.000 0.000 query.py:1830(get_proxied_model) 2 0.000 0.000 0.000 0.000 query.py:208(get_compiler) 60002 0.323 0.000 4.411 0.000 query.py:213(iterator) 1 0.000 0.000 0.000 0.000 query.py:220(get_meta) 2 0.000 0.000 0.001 0.000 query.py:228(clone) 4 0.000 0.000 0.000 0.000 query.py:30(__init__) 1 0.000 0.000 0.001 0.001 query.py:328(get) 4 0.000 0.000 0.000 0.000 query.py:512(deferred_to_data) 1 0.000 0.000 0.000 0.000 query.py:545(filter) 1 0.000 0.000 0.000 0.000 query.py:559(_filter_or_exclude) 1 0.000 0.000 4.430 4.430 query.py:56(__getstate__) 2 0.000 0.000 0.000 0.000 query.py:603(table_alias) 1 0.000 0.000 0.000 0.000 query.py:636(order_by) 60002 0.150 0.000 0.346 0.000 query.py:728(db) 2 0.020 0.010 4.431 2.215 query.py:73(__len__) 2 0.000 0.000 0.001 0.000 query.py:739(_clone) 1 0.000 0.000 0.000 0.000 query.py:769(get_initial_alias) 2 0.000 0.000 0.000 0.000 query.py:788(join) 2 0.000 0.000 0.000 0.000 query.py:866(setup_inherited_models) 1 0.000 0.000 0.000 0.000 query.py:955(add_filter) 1 0.000 0.000 0.000 0.000 query_utils.py:152(__init__) 2/1 0.000 0.000 0.000 0.000 subclassing.py:20(inner) 4/1 0.000 0.000 0.000 0.000 subclassing.py:45(inner) 1 0.000 0.000 0.000 0.000 synch.py:36(reader_enters) 1 0.000 0.000 0.000 0.000 synch.py:48(reader_leaves) 1 0.000 0.000 0.000 0.000 synch.py:59(writer_enters) 1 0.000 0.000 0.000 0.000 synch.py:71(writer_leaves) 4 0.000 0.000 0.000 0.000 threading.py:114(acquire) 4 0.000 0.000 0.000 0.000 threading.py:134(release) 2 0.000 0.000 0.000 0.000 threading.py:219(_is_owned) 2 0.000 0.000 0.000 0.000 threading.py:270(notify) 2 0.000 0.000 0.000 0.000 threading.py:308(acquire) 2 0.000 0.000 0.000 0.000 threading.py:329(release) 14 0.000 0.000 0.000 0.000 threading.py:64(_note) 8 0.000 0.000 0.000 0.000 threading.py:796(currentThread) 1 0.000 0.000 0.000 0.000 tree.py:120(start_subtree) 1 0.000 0.000 0.000 0.000 tree.py:140(end_subtree) 12 0.000 0.000 0.000 0.000 tree.py:18(__init__) 5/4 0.000 0.000 0.000 0.000 tree.py:55(__deepcopy__) 1 0.000 0.000 0.000 0.000 tree.py:71(__nonzero__) 1 0.000 0.000 0.000 0.000 tree.py:83(add) 2 0.000 0.000 0.000 0.000 utf_8.py:15(decode) 2 0.000 0.000 0.000 0.000 util.py:12(execute) 603 0.001 0.000 0.002 0.000 util.py:35(__getattr__) 59999 0.152 0.000 0.554 0.000 util.py:48(typecast_date) 2 0.000 0.000 0.000 0.000 util.py:8(__init__) 60002 0.196 0.000 0.196 0.000 utils.py:122(_route_db) 2 0.000 0.000 0.000 0.000 utils.py:85(__getitem__) 1 0.000 0.000 0.000 0.000 where.py:130(make_atom) 1 0.000 0.000 0.000 0.000 where.py:198(sql_for_columns) 1 0.000 0.000 0.000 0.000 where.py:270(__init__) 1 0.000 0.000 0.000 0.000 where.py:273(__getstate__) 1 0.000 0.000 0.000 0.000 where.py:287(__setstate__) 1 0.000 0.000 0.000 0.000 where.py:297(prepare) 1 0.000 0.000 0.000 0.000 where.py:302(process) 1 0.000 0.000 0.000 0.000 where.py:36(add) 5/4 0.000 0.000 0.000 0.000 where.py:74(as_sql) 2 0.000 0.000 0.000 0.000 {_codecs.utf_8_decode} 10 0.000 0.000 0.000 0.000 {built-in method acquire} 8 0.000 0.000 0.000 0.000 {built-in method release} 2 0.000 0.000 0.000 0.000 {built-in method sub} 1 5.931 5.931 12.543 12.543 {cPickle.dumps} 1 0.000 0.000 0.000 0.000 {callable} 60622 0.056 0.000 0.056 0.000 {getattr} 120032 0.033 0.000 0.033 0.000 {hasattr} 115 0.000 0.000 0.000 0.000 {id} 120043 0.042 0.000 0.042 0.000 {isinstance} 11 0.000 0.000 0.000 0.000 {issubclass} 60006 0.035 0.000 0.035 0.000 {iter} 120027/18 0.020 0.000 4.431 0.246 {len} 60007 0.335 0.000 0.335 0.000 {map} 3 0.000 0.000 0.000 0.000 {method '__reduce_ex__' of 'object' objects} 21 0.000 0.000 0.000 0.000 {method 'add' of 'set' objects} 84 0.000 0.000 0.000 0.000 {method 'append' of 'list' objects} 17 0.000 0.000 0.000 0.000 {method 'copy' of 'dict' objects} 2 0.000 0.000 0.000 0.000 {method 'decode' of 'str' objects} 1 0.000 0.000 0.000 0.000 {method 'disable' of '_lsprof.Profiler' objects} 11 0.000 0.000 0.000 0.000 {method 'extend' of 'list' objects} 603 0.654 0.001 1.208 0.002 {method 'fetchmany' of 'sqlite3.Cursor' objects} 94 0.000 0.000 0.000 0.000 {method 'get' of 'dict' objects} 2 0.000 0.000 0.000 0.000 {method 'index' of 'list' objects} 1 0.000 0.000 0.000 0.000 {method 'items' of 'dict' objects} 2 0.000 0.000 0.000 0.000 {method 'iteritems' of 'dict' objects} 8 0.000 0.000 0.000 0.000 {method 'join' of 'str' objects} 6 0.000 0.000 0.000 0.000 {method 'keys' of 'dict' objects} 2 0.000 0.000 0.000 0.000 {method 'pop' of 'dict' objects} 3 0.000 0.000 0.000 0.000 {method 'pop' of 'list' objects} 2 0.000 0.000 0.000 0.000 {method 'replace' of 'str' objects} 60000 0.067 0.000 0.067 0.000 {method 'split' of 'str' objects} 23 0.000 0.000 0.000 0.000 {method 'startswith' of 'str' objects} 5 0.000 0.000 0.000 0.000 {method 'update' of 'dict' objects} 3 0.000 0.000 0.000 0.000 {method 'update' of 'set' objects} 6 0.000 0.000 0.000 0.000 {range} 1019986 0.422 0.000 0.632 0.000 {setattr} 8 0.000 0.000 0.000 0.000 {thread.get_ident} 5 0.000 0.000 0.000 0.000 {time.time} 8 0.000 0.000 0.000 0.000 {zip}

Is it the creation of the cache_key?

By: seb

seb — Sat, 27 Nov 2010 03:23:44 +0000

Hi,

I used your “One final Trick”-Snippet expecting that a my query (around 60 000) items would be the second time much faster, but it wasn’t. It took more time then a normal ‘objects.all()’ each request. What happend here?

By: Sumit Chachra

Sumit Chachra — Fri, 06 Feb 2009 03:55:06 +0000

Challenge with last solutions is saves/deletes/edits.

Saves and Deletes can be detected using signals, updates/edits ?

Also even though you detect things, its difficult to know which keys to invalidate since 1 row of the db got updated?

Any workarounds to that ?

Comments on: Per-user caching in Django

By: Jeff

By: seb

By: seb

By: Sumit Chachra