Monkeys & Robots

The Cloud Mirror from eric gradman on Vimeo.

The CLOUD MIRROR is an interactive augmented reality art installation by Eric Gradman of Monkeys & Robots. Live video captured by a camera and is re-projected on the wall behind the camera, functioning like a “magic mirror.” But the CLOUD MIRROR software alters the images on the way to the screen. It runs an algorithm that tracks faces from frame to frame and also examines each frame for 2D barcodes printed on attendee badges. By pairing each face with a badge, and each badge id with a database row, the CLOUD MIRROR can identify by name whoever is standing in front of the installation.

The CLOUD MIRROR then augments each frame, adding a thought bubble to each face in the image. The contents of that thought bubble are selected from a set of “tags” associated with that person. Tags come from various sources, including Facebook, Twitter, and SMS data.

When registering for the event, attendees were asked to optionally provide their Twitter name, Facebook profile ID, and to answer the question “Where is your favorite place in LA?” In the weeks leading up to the event, the CLOUD MIRROR software sent a friend request to any attendee that provided that information. The poor trusting souls who accepted this request had their personal profile gently data-mined. Specifically, the information captured was “Facebook updates,” “Twitter updates,” and “Facebook relationship status.”

CLOUD MIRROR also capitalized on peoples’ innate desire to embarrass their friends by allowing anyone to anonymously “graffiti” in a thought bubble by sending an SMS message to a special number containing the target’s unique badge ID.

Behind the scenes, CLOUD MIRROR is divided into two parts. The computer vision component is written in C++ using OpenCV’s Viola-Jones face detector; ARToolkitPlus for fiducial glyph identification; and Pango+Cairo for text and graphics rendering. The camera is a Point Grey Flea2 with a wide-angle fixed focus lens and a big LED ring light.

Once the vision component has identified a face and correlated it with a badge ID, it consults an SQLite database for the content of the thought bubble. The SQLite database is kept up to date by a background Python process which is using the Twitter API and Facebook API to periodically grab new statuses.

Both components ran on a Shuttle PC running Ubuntu Linux.

The SMS component was written by Chris Nelson and ran on a separate machine. It fires decoded SMS messages via UDP at yet another Python process, which also updated the database.

The CLOUD MIRROR was unveiled at the April 16 Mindshare event in Los Angeles. Each month, Mindshare features new interactive installations from the members of Mindshare Labs, a collective of artists, scientists, and engineers.

UPDATE:
More pictures available here.

interactive

Internet, I was recently working on a project that required decoding incoming SMS messages from a T39 unlocked GSM phone over bluetooth. This was non-trivial. But for you, henceforth, it shall be trivial. For more information, see this place.


Copyright (c) 2009 Eric Gradman
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

import serial
import time
from cStringIO import StringIO
from math import ceil
from binascii import unhexlify, hexlify
from itertools import *

class T39(object):
  def __init__(self, file="/dev/tty.T39-SerialPort1-1"):
    self.file = file

    self.ser = serial.Serial(file, 115200)
    self.pdu = PDU()

    for s in ('ATZ', 'AT+CPMS="ME","ME","ME"', 'AT+CNMI=3,3,2,0,0'):
      self.ser.write("%s\r" % s)

      while True:
        d=self.ser.readline()
        print d
        if d.startswith("OK"):
          break

  def fetch(self):
    s = self.ser.readline()
    return self.pdu.decode(s)

class PDU(object):
  def decode(self, s):
    s = unhexlify(s)
    d = StringIO(s)

    # parse SMSC information
    p = {}
    p['smsc_len'] = d.read(1)
    p['type_of_address'] = d.read(1)
    p['sc_num'] = self.unsemi(d.read(ord(p['smsc_len'])-1))

    p['msg_type'] = d.read(1)
    p['address_len'] = d.read(1)
    p['type_of_address'] = d.read(1)

    p['sender_num'] = self.unsemi(d.read(int(ceil(ord(p['address_len'])/2.0))))
    p['pid'] = d.read(1)
    p['dcs'] = d.read(1)
    p['ts'] = d.read(7)
    p['udl'] = d.read(1)
    p['user_data'] = d.read(ord(p['udl']))
    p['user_data'] = self.decode_user_data(p['user_data'])

    return p

  def decode_user_data(self, s):
    """PDU user data is stored in a strange 7-bit packed format"""
    bytes = map(ord, s)
    strips = cycle(range(1,9))
    out = ""
    c = 0    # carry
    clen = 0 # carry length in bits
    while len(bytes):
      strip = strips.next()
      if strip == 8:
        byte = 0
        ms = 0
        ls = 0
      else:
        byte = bytes.pop(0)
        # take strip bytes off the top
        ms = byte >> (8-strip)
        ls = byte & (0xff >> strip)
      #print "%d byte %x ms %x ls %x" % (strip, byte, ms, ls)

      # append the previous
      byte = ((ls << clen) | c) & 0xff
      out += chr(byte)

      c = ms
      clen = strip % 8

    if strip == 7:  out += chr(ls) # changed 6/11/09 to incorporate Carl's suggestion in comments
    return out

  def unsemi(self, s):
    """turn PDU semi-octets into a string"""
    l = list(hexlify(s))
    out = ""
    while len(l):
      out += l.pop(1)
      out += l.pop(0)
    return out

interactive, robots

90 degree mirrorshades. Available long ago from Sharper Image, and now the best way to watch a movie on a laptop in bed.

Uncategorized

After months of developing art that encourages others to interact, I decided to build a project that is all about me. The NARCISYSTEM is an attempt to strap to my body as many biometric sensors as I could reasonably acquire, and use their output to drive various visualizations in a venue. Its first and only presentation was at the March Mindshare event in Los Angeles.

At the heart of the belt is a Funnel IO Arduino clone from Sparkfun Electronics. This excellent piece of hardware has a built-in XBee socket, USB power, and a LiPO charge circuit built in. Its my board of choice for new designs. This Arduino sucked in all the sensor readings, and transmitted them via serial XBee to my Powerbook. I used an XBee Adapter Kit from Ladyada to interface XBee to the computer. Note, if you’re trying this at home, do not follow Ladyada’s instructions to program a Funnel IO over XBee (at least on OSX.) Instead, use the configurator tool that comes with Funnel IO.

The belt itself is a made of latex and was custom made for me by Devan and Isa of PodBelt and Psymbiote.

What sensors did I use?

• Heartrate monitor. In order to detect heartbeats, I used a Polar heart rate monitor strapped to my chest. I read this data with a Polar Heart Rate Monitor Interrface available from Sparkfun Electronics. This board has some fancy software to compute heartrates, but since I wanted an “heart beat edge trigger,” I soldered a wire to a probe hole near the LED and hooked it up to a digital input.
• EEG. To achieve something that looked like an EEG, I ripped apart a Ramsey Electronics Electrocardiogram Heart Monitor Kit. For EEG probes, I used the business end of my OCZ Neural Impulse Actuator. I then proceeded to add bias resistors and voltage dividers at random to the ECG kit (a process reminiscent of circuit bending) until the output signal looked compatible with my Arduino A/D converter. Technically this functioned as an EMG—not an EEG—because I didn’t get a clean enough signal to do frequency analysis. OCZ: if you are reading this article, take note! Because you have steadfastly refused to provide even the merest sliver of developer support I have resorted to chopping the custom plug off your probes and using them in a $40.00 kit. Next, I will use your interface hardware to prop up my desk. Your software I have no use for.
• Breathalyzer. I measured my breath alcohol concentration using a MQ-3 Alcohol Gas Sensor from SFE. This cute little sensor was strapped to my arm, so I had to remember to occasionally breathe on it. I was surprised at how well this sensor performed. Over the course of the NARCISYSTEM development, I tried on numerous occasions to quantify this sensor’s accuracy. But I was consistently foiled by a progressive breakdown of my scientific methods, followed by unconsciousness.
• Compass.I measured my bearing using a HMC 6352 compass module from SFE (are you detecting the pattern yet?) This I2C sensor is incredibly easy to interface and provides accurate data. This is the same sensor I used to build my Haptic Compass. In fact, it is the exact same sensor, delicately desoldered and haphardly replaced.
• Accelerometer.The accelerometer is a ADXL202 on a SFE breakout board. This is a tried-and-true analog accelerometer.

How did I visualize the data?

On the day of the event, I found myself with fewer “output modalities” than sensors. I showed up with eight LED Parcans (available from AmericanDJ at Guitar Center.) I had a subwoofer. I had a fog machine. I had an iPhone. I had a few missing cables.

I erected a lighting tree with the Parcans illuminating a “ring” around the venue. These lights would all flash red briefly every time my heart beat. To illuminate them, I used an ENTEC USB Pro adapter, using my Python DMX modules.

With the bearing information from the compass, I projected my bearing vector on the lights, such that they turned blue only in the direction in which I was facing. As I stood in the middle of the venue and turned in a circle, it appeared to me as though all the lights were blue. But an observer would see that the illumination was in fact tracking my direction.

I used the differentiated the accelerometer data (to get jerk) and used that to directly drive a subwoofer. I was actually driving the subwoofer constantly below the resonant frequency. Only when I took a step would the frequency jump into an audible range. The end effect was that the venue shook with my every step, as though a giant had stepped. It was barely perceptible (so I wouldn’t drive anyone crazy) but very effective. To drive the subwoofer, I sent OSC messages to the ChucK audio programming environment.

The breathalyzer was set to trigger the fog machine when my intoxication reached a threshold. However, the fog machine I borrowed was, um, excessively volcanic and was deemed unsuitable for indoor use. So at the last minute I set it up outside, and made it triggerable from my iPhone.

I was using the EEG/EMG data to alter the tempo of a synthetic music composition (also running in ChucK.) This worked unpredictably, even with tremendous filtering, so I eventually turned it off and removed the headband, so I looked like a complete dork only from the waist down.

Finally, since no contemporary project is complete without gratuitous use of an iPhone, I wrote a visualizer for all this data using the “custom control” feature of OSCemote and transmitted data to it via OSC. OSCemote is far and away my favorite app store application, for the simple reason that it has saved me from actually having to learn any ObjectiveC. I developed my custom visualizer in Dashcode, which was surprisingly fun and easy to use.

interactive

I have now used the Twitter 1000 times. In honor of this meaningless base-10 milestone, I have written a short python script to compute summary statistics on my timewasting. I release this into the public domain, in hopes that in time the blame is shared among us all.

You’ll need python twitter module and numpy.

import twitter
import numpy
import datetime

username = 'egradman'
password = 'xxx'

api = twitter.Api(username=username, password=password)
statuses = api.GetUserTimeline(username, count=999)

average = numpy.average([len(s.GetText()) for s in statuses])
stddev = numpy.std([len(s.GetText()) for s in statuses])

stamps = [s.GetCreatedAtInSeconds() for s in statuses]
stamps.reverse()

diff = numpy.ediff1d(stamps)
davg = numpy.average(diff)
average_interval = datetime.datetime.fromtimestamp(davg) - datetime.datetime.fromtimestamp(0)
longest_interval = datetime.datetime.fromtimestamp(max(diff)) - datetime.datetime.fromtimestamp(0)

print "tweets:", len(statuses)
print "average len:", average
print "stddev len:", stddev

print "average interval:", average_interval
print "longest interval:", longest_interval

interactive

I’m presenting ShadowSmoke, ArtFall, and the Haptic Compass at TEDxUSC’s reception tomorrow afternoon.

Hope to see you there!

And in the meantime, check out these videos of the Haptic Compass + NARSCISYSTEM visualizer I cooked up tonight for TEDxUSC:

Uncategorized

I was inspired by Dorkbot this afternoon, so once I could extract no more sympathy for my hangover I decided to get some work done. There is no better cure.

interactive

This is a great article about Mindshare, featuring lots of pictures of ShadowSmoke:

Uncategorized

To everyone who suggested I add the word “iPhight” to the Urban Dictionary, I think we can now put it to rest.

I am strangely unaffected by this turn of events.

Thanks for your definition of iPhight!

Editors reviewed your entry and have decided to not publish it.

To get a better idea of what editors publish and reject, sign up as an Urban Dictionary Editor here: http://editor.urbandictionary.com/

Urban Dictionary

—–

iPhight

The culmination of a bar-dispute in which all participants whip out smartphones and compete to see who can Google the answer fastest.

The debate over who starred in that shitty movie was ultimately ended by an iPhight.

Uncategorized

Uncategorized