relief projection

augmentedsculpture by pablo valbuena, originally uploaded by hc gilje.

I found the title of this post in one of Michael Naimarks essays, I guess it also could have been called augmented reality, projection of a virtual object onto a physical object, projecting a virtual layer ontop of a physical geometry, masking of projections, etc.

I have been researching different ways of projecting on other things than flat surfaces: projections that project on objects, follow the shape of the room, and projections of virtual 3D shapes onto physical 3D shapes.

In my own work I have used projections as advanced light sources, masking as a way to fit flat projections on objects and surfaces, but also to create the illusion of multiple screens from a single source. Some examples here.

My goal has been to create tools which make it easy to start working with a physical space immediately, being able to make changes in realtime. I have mainly done this by using multiple opengl videoplane layers in max/msp jitter, with one of the layers having a drawing mode so you are able to draw the shape of a particular object after you have placed a opengl layer over it. I made a crude 3 layer tool for the workshop I did at KHIO this summer to enable the participants to immediately start relating to the physical space.
The master of multiple opengl videoplanes in Norway is Piotr Pajchel in his work with Verdensteatret.

I have done some experiments with projecting a 3D shape onto physical objects, but still have a long way to go in terms of having a simple setup for this.
Obviously I have been looking at what other people have been doing, but none of systems I have found seems to be available to the public, and few of them seem to have been used beyond the developing-period of the system, which might be a sign of them not being as flexible as wanted, and maybe also quite timeconsuming to prepare.

Most systems uses a method to track the shape/space they want to project onto in combination with custommade software, to be able to map the projected image correctly onto the physical object, which is related to the lens specifications of the projector, the placement of the projector in relation to the objects to be projected on, etc.

The LightTwist system developed at the University of Montreal (not much seems to have happened after 2004) use “a panoramic (catadioptric) camera to get correspondances between each projector pixel with the camera pixel. This camera represents the viewpoint of our futur observers. Then, from what the observer should see, we can build the projector images from their respective mapping.”

The videobjects from Whitevoid design in Germany is a software for realtime distortion of video to fit physical objects, but using predistorted video, and you calibrate it either with a helpgrid or by importing a model of the realworld setup. So you would need to first create the 3D shapes to project onto, and then decide how the video will map onto the 3D objects, and finally doing the calibration to match up the virtual objects with the physical ones.

I think the most spectacular callibration solution so far is the “automatic projector calibration with embedded light sensors” (pdf), a collaboration between people from Carnegie-Mellon, Mitsubishi Electric Research Lab and Stanford. They use fiberoptics and light sensors built into the objects/surfaces to be projected on, and by projecting a series of grey coded binary patterns, a custom-made software is able to adjust the image in less than a second to perfectly fit the projectionsurface, with a much higher resolution than a camerabased solution. Take a look at the impressive video:

The pdf and video seems to be from 2004, but I found some more information here. They are hoping to make the system fast enough for live tracking of moving objects, and also to make the calibration pattern invisible using infrared light.

If you have a big budget you could always invite Circus of Now to do the video for you (”We build skyscrapers of light”).

At Ars Electronica this year I had the pleasure to see Palbo Valbuena´s Augmented Sculpture (image at top of this post) which consists of a physical structure in the corner of the room, with the exact same virtual shape projected onto it using one projector. By then animating the color and lighting of this virtual shape, some very interesting light/shadowplays happen. Valbuena collaborates with some game developers in Spain who constructed the virtual model and animation in a standard 3D software.
This work shows the potential in augmented reality using videoprojection, and I hope to see more of his work soon (He has a big outdoor installation in Madrid at the moment, hopefully there will be some documentation soon.)

update feb 5th 2008: Valbuena has updated his website with documentation of several projects: different versions of the augmented sculpture and the public square installation in Madrid.

 

Verdensteatret installations

Louder_k11, originally uploaded by hc gilje.

This weekend I got the chance to see the two installations “Fortellerorkesteret” and “Louder” by Verdensteatret at Kunstnernes Hus in Oslo. Both come out of two theater performances by Verdensteatret. I had the opportunity to be a little bit involved in the production of Fortellerorkesteret so it was nice to see it in this huge beautiful space.

The installations are a mix of sculpture,sound,video,kinetic objects,light and shadows, and are inspiring examples of compositions in space. Fortellerorkesteret has a more theater structure, resemblant of old mechanical puppet theater, while Louder is more of a spatial experience dominated by the huge mechanical spider and the numerous speakers.

more images of louder and fortellerorkesteret

Results of video for stage workshop

khio workshop 09, originally uploaded by hc gilje.

I taught a one week workshop for the director-,choreography-, and scenography students at the National Academy of the Arts in Oslo last week.
My approach was to develop tools in the form of small software applications which made it very easy for the students to immediately start exploring ways of using video in a space.
We also tested out many different projection surfaces, privalite, dmx-controlled shutters, physical masks.

You can see a slideshow of some of the results here.

And even better, you can check out two of the applications I made (built using maxmsp and jitter), for both osX and windows. One is a simple capture program for recording either live stream, animation or recorded footage already on the camera, to your harddrive.
The other is a tool for positioning and scaling 3 planes of video, creating presets of the scenes you create and then make transitions between the scenes.

the matrix for the rest of us (well, almost)

Google Earth has implemented a new technology called street view, developed by Immersive Media

This could roughly be seen as a mix of very advanced quicktime VR and a film effect from The Matrix movies. Interesting to me as another example of the interrelation between space,time and motion.

From the Google Earth Blog:
“One of the many secrets behind their technology is a patented 11 lens camera system that simultaneously takes photos in 11 directions based on a dodecahedron geometry. They can capture 30 frames a second of high resolution photography. That’s right - we’re talking high resolution video in digital 360. You can stop, start, back up, single-frame, etc”

Cloud

The Aleph installation reminded me of the installation Cloud that David Rokeby recently completed.

“Cloud is a monumental kinetic installation hanging suspended in the Great Hall at the Ontario Science Centre. One hundred identical sculptural elements, arranged in ten by ten grid, are rotated at slightly differing speeds by computer-controlled motors. The elements slowly shift in and out of synchronization. When the motors are just out of sync, huge waves ripple across the space. When completely in sync, the work appears almost solid then suddenly almost invisible. When far out of sync, the sculptural elements float in apparent chaos. “

Synk at Dansstationen in Malmö

Synk, originally uploaded by hc gilje.

On Friday May 4th, I perform the piece Synk with Kreutzerkompani and Justin Bennett on sound. Synk was originally made in 2002 for the Ultima festival, but has been played quite a few times the last five years.

The idea of Synk was that no prerecorded video or audio would be used, only material sampled during the performance was allowed, to investigate live as raw material : to impose a structure on a live situation to allow for unpredictable results within that frame structure.

It creates a dialogue between the physical space on stage and the mediated space from the screen and speakers, and the relation between the memory and the present of a space.

More info on Kreutzerkompani and Justin Bennett

More images from Synk (click on the small images)

 

David Rokeby: Machine for taking time

Another project I wish I had done:

“ A colour surveillance camera has been mounted outside the gallery on a computer controlled pan/tilt mechanism, allowing it to see most of the surrounding gardens. Every day since March 28, 2001, the system has been taking still images from 1079 pre-determined positions along a sweeping path around the garden.

[..] the computer software travels through this accumulating archive of images, wandering through time, but progressing very slowly and smoothly through the successive positions in the original path.

The software does four kinds of wandering. It sometimes moves along the path using images from a single day. Or it might disolve sequentially from day to day as it progresses along the path. Alternatively it might dissolve from date to date randomly. Occasionally it will stop its movement along the path and show all the images taken from that position in rapid succession. The shifting of modes and the choices of dates is a function of a somewhat random process, and so the piece never repeats itself.”

from David Rokeby´s website

Rokeby has a lot of video documentation available through youtube.

lab jan 2007: masking projections

lab jan 2007, originally uploaded by hc gilje.

(Slideshow of more images from the projection sketches here)

The second focus in this lab session was to work with video projections, and masking them to create several projection surfaces from one projector, and to be able to relate to physical shapes in the room, like my projection on a sphere in Iball, or the masking of video to fit the gallery spaces in one of the sleepers installation:

The simplest way to do it is to work with a 2 dimensional mask. I also wanted to try to work with 3D masks: projecting a virtual 3D version of an object back onto itself. In this way I managed to cover 4 sides of a cube with 2 projectors. Another thing I did was to link the individual projection surfaces (from one projector) so focus could move from one area to another, quite like how the nodes in nodio operate.

Finally, I did some simple tests using the projector as a light beam.

 

lab jan 2007: nodio 3rd generation

nodio 2, originally uploaded by hc gilje.

In january 2007 I got the first opportunity to work two intense weeks in a project space, enabling me to test out some new elements in the nodio system.

I set up 9 nodes (macminis+monitors) in a local network, and expanded my nodio composer tool to work with the 9 channel setup. A continuous discussion I have with myself is where to put the control: on the individual nodes or on a master computer talking to the nodes. Usually I end up with a combination, that one of the nodes does slightly more work than the others. I managed in this work period to sync the nodes with an audiosignal, making it easier to put more control on each node.

The results of this period resulted in a more refined way of moving elements between the sources:

I also played with using the soundtrack on videoclips (instead of generating sound from the video which is usually the case), and moving clips between the different nodes:

In general I was able to improve the sequencer aspects of the system a lot, making it possible for more complex patterns and rhythms:

 

nodio composer, 2006

nodio composer, originally uploaded by hc gilje.

nodio composer is a composer/sequencer for realtime multichannel video and audio, developed in maxmsp and jitter.

The nodio composer system consists of 4 modules:
The client is installed on each node, and does realtime processing of image and sound.

The motor talks to the clients and coordinates them, and it sets an saves the state of the whole system, and plays back sequences.

The composer is the gui for the motor, so it allows the user to set and save the state of the system and to program and play sequences. the motor sends feedback to the composer about the current state of the system and the sequencer.

The simulator is a fully working simulation of the system. It contains the three clients, and shows the three screens and pans the sound according to which client generate the sound. It is a slight modification of the node clients, but is made to be easy to replace with updates of the clients. As with the node
clients, simulator communicates with the motor.

The system is intended to be operated in 4 different modes:
composer, standalone, performer and composer_offline.

composer:
each node has a client application, one of the nodes contains the motor, the
composer is on another computer.

standalone installation:
each node has a client application, one of the nodes contains the motor.
performer: each node has a client application. The motor and the composer is on another computer.

performer:
each node has a client application. The motor and the composer is on another computer.

composer offline:
When the network or individual nodes are not available, it is possible to run the composer and motor together with a simulator on the same computer.

Here is a video with documentation from one composition and a brief description of how it works: