Python extension for libmemcache, the C API to memcached. cmemcache API is very similar to python-memcache, except for some return codes which follow libmemcache conventions.
cmemcache is about 2 times faster then python-memcache with short key names (8 characters), faster with larger key names.
This extension was created after doing some timings on a simple session caching scheme where I noticed that python-memcache was 'only' three times as fast as PostgreSQL. I expected it to be faster than that, since PostgreSQL does a lot more than memcached. It was suggested that it was perhaps python-memcache, which for instance gets the initial part of the message from memcached byte for byte. After discovering libmemcache it seemed pretty straight forward to create an extension on top of that.
cmemcache has 2 clients: StringClient and Client. StringClient is the extension that only supports python strings for values. The api was copied from python-memcache, except that it excepts strings only. Client is a module that implements caching of arbitrary python objects using Pickle on top of the StringClient. This code is a copy/paste from python-memcache. Most of the test code in test.py is run on the cmemcache Client and the python-memcache Client to make sure that they are interchangeable. Although I have not tested this but it should be possible to mix cmemcache and python-memcache clients in a running system as well, since they use the same constants for encoding object types.
The speed difference would be less if the memcached protocol would be changed to precede the reply header with its size. A client could then read the header size, read the full header (in one read), parse header to get the data size, and read data (this is already done in one read).
cmemcache:
libmemcache:
set_servers with the wrong port number causes a segfault (in libmemcache). See commented out testing code in test.py.
mc_err_filter_add() broken. This results in warnings on add() and replace().
This is fixed by applying the patch from the download section.
libmemcache-1.4.0.rc2 is not compatible with memcached 1.2.1, this results in get_stats returning no stats.
start memcached, create Client, kill memcached, start memcached, do a get() and python process exits, with messages like:
[ERROR@1170236923.979369] mcm_buf_read():361: read(2) failed: Operation now in progress: server unexpectedly closed connection
libmemcache does an exit on severe errors. It is possible to install an error handler that could change the severity to not exit and then the python extension could throw a python exception. However, I do not know if libmemcache will recover correctly.
I do not have time at the moment to try and fix this.
Reported by Mark and Philip, 31/01/2007.
Versions:
0.95
Fixed expire time internal type (was int, must be time_t). To be on the save side the expire time is also clamped to 2**31-1 to follow the memcache protocol spec. Old code caused problems when using sys.maxint on 64-bit machines. Reported and patched by Simon Law.
Added missing debuglog() implementation, copy/paste error from memcache.py. get() now returns None on all error cases. Reported by Simon Law.
Fixed some memory leaks in get_multi().
Fixed get_multi() return values for Client. All results would be returned as strings, instead of the proper types. Reported and fixed by Alfred J Fazio.
cmemcache.py uses debuglog() (memcache.py remnant) but debuglog() is not even defined, doh! Reported by Simon Law. His patch uses python module logging, but to avoid dependencies I added the same stderr logger as used in memcache.py. One can override the cmemcache.log variable to install another log function.
Fixed memory leak caused by not Py_DECREF of key and val objects when calling PyDict_SetItem(). Reported and fixed by Alfred J Fazio.
Allocated my own context and moved from 'mc_*' api to 'mcm_*' api to use. Install error handler to change the 'cont' state of the memcache_err_ctxt object to 'y' to try not to abort on fatal errors. This is an attempt to not crash python on fatal errors from libmemcache, but libmemcache needs considerable changes to make it work. I am not too sure about my libmemcache so I have not released them yet. Code is compatible with libmemcache-1.4.0 though.
Copyright (C) 2006-2008 Gijsbert de Haan. This code is distributed under the GNU General Public License.
Gijsbert de Haan <gijsbert.de.haan@gmail.com>