slightly new, but it smacks of genius.

Manual CPSing in the database

2009-08-31T17:18:00.000-07:00

I'm currently implementing text notes for NoDictionaries. You enter a really insightful comment, you need to log in to post your comment, so you log in and your comment is there.

A pretty convenient way of going about is to have a hash table of closures, and store the continuation of the commit, continuation-passing style.

However, it'd be better to persist data in the database. (What happens if you restart the server?)

Here's how committing a text note works in NoDictionaries: when you POST /add-note, you basically make the note , except the associated text word's id is the negative of the correct id, and the user id is whatever user id you have, whether a real database row or just the time in seconds of your first page view. Then you're redirected to GET /commit-note, which makes sure you're logged in (and takes you on a round-trip to the login/signup if necessary), verifies the id of the note, the id of your user id, and the negativity of the text word id. You can spoof it if you know the database id of an uncommited note, the database id of the word it's associated with, and the permanent or temporary user id that the person submitted it has, so you'd basically have to snoop on their wifi for that split second when they're filling out the login/signup form.

It still feels like a kludge.

DB indices

2009-08-31T09:05:00.000-07:00

I think it'd be pretty nifty to get a daily summary of which indices would have been much handier to have: for example, snapshot my DB at 4am, record the day's GETs/POSTs/PUTs/DELETEs, index the hell out of my tables (creating 2^n indices for each combination of columns), give me statistics based on speed of completing a day's work, the IO and RAM required, and let me tweak the parameters, and then upgrade my box or add indices or whatever.

(This is poorly thought out, but interesting nonetheless.)

Ruby's memcache-client 1.7.2 does NOT support compression; patch it in with 7 lines of code

2009-05-27T07:43:00.000-07:00

The Ruby memcache-client 1.7.2 code seems like the most popular memcached client. Alas, memcache-client 1.7.2 does not recognize :compress => true, all the demos out there notwithstanding.

Let's fix this by monkeypatching Zlib compression into Marshal.


require 'zlib'

module Marshal
  @@load_uc = method :load
  @@dump_uc = method :dump
  def self.load(v) @@load_uc[Zlib::Inflate.inflate(v)] end
  def self.dump(v) Zlib::Deflate.deflate(@@dump_uc[v]) end
end

And there we go! Four lines of patching Marshal.load for a better memory footprint.

To dissect each phrase of that sentence:

"Four lines": I wanted just to try how well zlib would work on my Marshalled ActiveRecord objects and html fragments, and it did so handily, almost 3:1. Indeed, the only reason I poked around at the source code is because one of my largest but still highly-compressible HTML fragments was 1.2MB, over the size limit. I've since gone back to storing large HTML fragments on disk (uncompressed), having found many more values to store in Memcached.
"patching Marshal.load": monkeypatching Marshal is not as bad as String. Chances are, you use the Marshal format as a blob, and you keep your Marshal files to yourself (and leave external serialization to friendlier fare like JSON). So, all in all, it's much easier to change the Marshal format than mucking through the memcache-client code.
"better memory footprint": instead of Zlib, try LZMA, with slightly smaller compressed sizes than BZIP and faster decompression times, good properties for cache compression. But Zlib is already in the standard library, so it's a good first approximation.

The ersatz alias_method_chaining feels kludgy, as does Ruby's distinction between methods and lambdae. Ah well.

Thoughts?

Reality

2009-03-29T14:21:00.000-07:00

When art critics get together they talk about Form and Structure and Meaning. When artists get together they talk about where you can buy cheap turpentine. -- Picasso

A Full Web Service with HTTP caching in 7 lines

2009-02-24T09:38:00.000-08:00

Two lovely gems, sinatra and rack-cache. Sinatra is pretty easy web-service-creation, and Rack::Cache is pretty easy http caching. Together? Jubilation.


require 'rubygems'
require 'sinatra'
require 'rack/cache'

use Rack::Cache

get('/quadruple/:n') {
  sleep 1
  response.headers['Cache-Control'] = 'max-age=1000000'
  (params[:n].to_i * 4).to_s
}

and then

ruby sinatra-add.rb -e production

and you're done.

There are, of course, many other fiddly bits to configure Rack::Cache with, like use Rack::Cache, :entitystore => 'file:/tmp/' if you don't want to keep it all in a hash in memory, and :verbose => false if you don't want that in your logs, but that's basically it.

It's pretty amazing: that's a real live web service, with HTTP caching, in 7 gentle lines. (I don't count the closing brace, nor the sleep 1, which is purely for effect.)

Anyone know an easier way in any other language? Either in number of lines, or in directness of code?

Also, how bad would it be to authbind that to port 80, and just let it go open to the world?

Update: It's interesting to me for the same reason PHP is interesting, both as a social commentary on the bits of plumbing we've agreed upon as useful, and as an aid to wrapping an HTTP interface around some Ada code with its own homegrown database.

Top five contradictory positions of Ron Paul Graham.

2007-12-01T15:18:00.000-08:00

1. Supports secure borders, but writes macros for variable capture.
2. Rejects conspiracy theories of 9/11, but funded a social news website full of 9/11 conspiracy theories.
3. Will launch an ad blimp in two weeks, but remains cynical about submarine-style PR campaigns.
4. Says he doesn't organize his supporters on the internet, but provides a Q&A internet forum.
5. Opposes inflation, but supports accumulator generators.

Leave more in the comments...

Anti-Features?, or, Never attribute to malice what can be explained by math.

2007-11-24T16:23:00.000-08:00

Apparently, if you buy a cheap camera, and you can't save an uncompressed picture, that's treacherous computing, that's an anti-feature.

It's not. The camera won't stop working if you correctly patch in a different chip set.

And assuming the worst, assuming that the chips are exactly the same in an expensive camera, it still doesn't matter. The native resolution is still small on the cheap camera--that's mostly what makes it cheap--so you'll be better off buying the high-resolution expensive camera.

Of course, if you really want a camera that you can program, you might like an XO that drives a Roomba.

cheap illustrations.

2007-06-06T19:42:00.000-07:00

I'm a cheap nerd, so I printed myself an enormous wall decoration last August 2006: it's Pi, in a textual fractal. (This is it to two levels, and the one in my room is to five.) 3'x5', $10 from Kinko's. Nice conversation piece.

I've been printing more things recently, I've been thinking about combining text and images, and an author of two books on classical themes bemoaned the expensiveness of art in an email today. (Illustrating an book on Aesop's Fables, specifically.)

Most art schools have job boards; the artists can be of varying qualities, but sketches in pen are quick to do, and hard to do badly. (Lots of approximate lines.) Twenty seconds a sketch, which is a not-too-fast speed, scan one as you're doing the next one, and you've got a hundred in an hour.

Has anyone used Flickr for (cc)by-sa images? I wonder if that's cheaper.

Amazingly bad APIs?

2007-05-21T06:46:00.000-07:00

There are some amazingly bad APIs in Java, so Paul Buchheit says. The best API is system(), or backticks.

The end result is basically an ImageMagick conversion (kudos to a useful incantation for the less than sign):

mogrify -geometry "220x133>" -antialias -quality 90 /tmp/c.jpg /tmp/c-thumb.jpg

You wouldn't write an encyclopedia in limericks (though it's been done before), so there's no need to use a general purpose programming language to manipulate images. Small Pieces Loosely Joined is pretty popular for Unix, CGI/REST, Erlang, Messenger RNA, etc.

microsoft knows drm.

2007-05-02T17:31:00.000-07:00

I've been hearing a lot about this string of hexadecimal numbers. It starts with 09, ends with c0, and I think it has f9 and 88 somewhere inside.

Let's ask Microsoft search what it is!

require('open-uri') && puts(open('http://search.msn.com/results.aspx?q=09+f9+88+c0') {|f| f.read}.downcase.gsub(/<[^>]+>/,'').tr('^0-9a-f','').scan(/09.+?c0/).inject(Hash.new(0)) {|h,nu| h[nu]+=1;h}.sort_by {|str,freq| freq}.last.first)

Whew!

For non-Rubyists: require we can open urls like files, and put this string: open the msn search page for all pages that have 09, f9, 88, and c0, read it in one gulp into lowercase, regexp out all html tags, translate out any character that's not a hex digit, scan for all substrings that start with 09 and end with c0, make a histogram* of the array, sort by most popular, and take the most popular string.
(* Inject a hash table through the array of scanned substrings; the strings are the keys, the frequencies are the values; add one to the value every time you see any string, starting at zero.)

Remember folks, this is Microsoft's suggested answer to the 128-bit programming challenge posed earlier today, so like love, and Cambridge weather, it's just temporary.

fark is grabbing copyrights?

2007-04-26T08:42:00.000-07:00

Some would think so. But if you read really closely, you're not submitting anything meaningful, you're submitting a web link. And if they want the copyright of that web link, then, ok, TinyURL it. They'll have the copyright to a TinyURL uid. And the copyright to any images on the page stay where they were.

Pretty simple.

kata 6.

2007-04-08T16:28:00.000-07:00

prag dave's anagrams resonated with me, because i'm working on hashing text down.

so follow along in irb, if you have /usr/share/dict/words:

class Symbol
def to_proc(*args) lambda {|*a| a.first.send self, *(args+a[1..-1])} end
alias [] to_proc
end # for faux currying

w = File.readlines('/usr/share/dict/words').map {|w| w.strip.downcase}.uniq ;:done
h = Hash.new([])
w.each {|word| nu = word.split('').sort.join; h[nu]+=[word]} ;:done
anas = h.values.find_all {|v| v.size > 1} ;:done
puts anas.map(&:join[',']) # all anagram n-tuples
puts "---"
puts anas.sort_by(&:size)[-30..-1].map(&:join[',']) # the top by set size
puts "---"
puts anas.sort_by {|a| a.first.size}[-30..-1].map(&:join[',']) # the top by word size

in fairness, the symbol-currying is in "sym2proc.r", so it's really just 10 lines of code, but that's the general idea.
(lots of the library functions look haskelly, but ruby just felt better for string processing)

eval is amazing.

2007-03-08T15:41:00.000-08:00

In terms of things, eval is much bigger than JSON callbacks, even bigger than Lisp itself. It's big like the ribosome --- eval is how things come alive.

For example, I'm rendering urls and associated metadata in the browser from a ruby cgi script. Instead of some big XML specification, I'm just passing an array of strings from server to client. It's big_array.inspect in Ruby, and eval(inspectedBigArray) in Javascript. No monadic parsers, no macro magic; inspect and eval, code I don't even have to write myself. (And if the inspected big array is too big, the browser can start lysis; so it goes.)

G J Sussman: Programming is a good medium for expressing poorly-understood and sloppily-formulated ideas: exactly the opposite of people who'd want to plague me with type theory.

origins

2007-02-23T06:16:00.000-08:00

Some people call it disclaimers, some call it biscuiting, some youthful naïveté. When I was 16, my poetry teacher had told us that the only unwritten preface we were allowed to give to any piece was "It's slightly new, but it smacks of genius."