Computation, Caffeination, and Conflagration

Have you ever asked a search engine to find "Pages Similar To This One?" At the core of that search engine may lie a document vector model. To understand how this works, consider that in order to index and search for each document, a search engine must first tokenize each document. Semantic sugar (stopwords) like the, and, and a are stripped from the document, and some words are normalized (crying ⇒ cry). For each document d, the number of times a particular token appears in the document is taken to represent a vector in an n-dimensional space.

We live in a three dimensional space, and points in our universe are therefore formed by linear combination of vectors in three directions. Documents live in n-dimensional space. What is n? n is the total number of tokens that have been or ever can be found in any document ("that's "all words, ever"). Every document has a special place in n-dimensional space. And just as you can find the closest supermarkets in three dimensional space with a little trigonometry, so too can you find the nearest documents in n-dimensional space using a little n-gonometry. Those are the "Pages Similar To This One."

Speaking of supermarkets... What happens when you fill up your cart with groceries? Some catfood here, two red peppers, one green pepper, three cans of soup (spaghetti with meatballs), a gallon of milk, a Hershey's chocolate bar. Your shopping cart is more unique than you think. And when you go to checkout, the items in that cart become a set of vectors in an n-dimensional space, where n is the set of all products for sale in the supermarket. Sure, one week you don't need catfood, and decide to try a different brand of toothpaste. But for every cart, its possible to ask "Find Me Shoppers Similar To This One." And chances are, those shoppers are you!

Supermarkets can use the "club card" and the vector space model simultaneously. Once your card identity is linked to your own little home in n-dimensional space, they can identify you by name next time... even if you don't swipe your card, and pay by cash.

I make this point only to give perspective to an experience I had this week. My wife and I stopped in the supermarket to pick up some popcorn before watching a DVD with a friend. And realizing that we had not yet had dinner that night, we wandered the aisles self-indulgently, grabbing any snack-food caught our eyes. We checked out with shopping bags full of absolute crap. On the back of my receipt, there was an advertisement for Cheetos. We hadn't purchased any product confusingly similar to Cheetos...

And then I realized. We were the shopper most similar to The Stoner. Not just any stoner, but that Archetypal Stoner. That "aggregate" customer, that volume in n-dimensional space carved out by 2am munchie runs since time immemorial.

And dammit, I had just swiped my club card.