27 May 2009

Ruby's memcache-client 1.7.2 does NOT support compression; patch it in with 7 lines of code

The Ruby memcache-client 1.7.2 code seems like the most popular memcached client. Alas, memcache-client 1.7.2 does not recognize :compress => true, all the demos out there notwithstanding.
Let's fix this by monkeypatching Zlib compression into Marshal.


require 'zlib'

module Marshal
@@load_uc = method :load
@@dump_uc = method :dump
def self.load(v) @@load_uc[Zlib::Inflate.inflate(v)] end
def self.dump(v) Zlib::Deflate.deflate(@@dump_uc[v]) end
end

And there we go!  Four lines of patching Marshal.load for a better memory footprint.

To dissect each phrase of that sentence:
  1. "Four lines": I wanted just to try how well zlib would work on my Marshalled ActiveRecord objects and html fragments, and it did so handily, almost 3:1.  Indeed, the only reason I poked around at the source code is because one of my largest but still highly-compressible HTML fragments was 1.2MB, over the size limit.  I've since gone back to storing large HTML fragments on disk (uncompressed), having found many more values to store in Memcached.
  2. "patching Marshal.load": monkeypatching Marshal is not as bad as String.  Chances are, you use the Marshal format as a blob, and you keep your Marshal files to yourself (and leave external serialization to friendlier fare like JSON).  So, all in all, it's much easier to change the Marshal format than mucking through the memcache-client code.
  3. "better memory footprint": instead of Zlib, try LZMA, with slightly smaller compressed sizes than BZIP and faster decompression times, good properties for cache compression.  But Zlib is already in the standard library, so it's a good first approximation.
The ersatz alias_method_chaining feels kludgy, as does Ruby's distinction between methods and lambdae.  Ah well.

Thoughts?