Sketch 1 Final Stage: A Fairy Companion

After settling on the children's fairy companion from our stage 2 ideation we set about working on implementing the idea using Max/MSP/Jitter.

We started by planning out exactly how to go about achieving our desired effect. The program would need to detect a person on camera and react to that person's position and actions, drawing the fairy near the figure's location. To detect a person in the frame we chose to use the background subtraction technique we were shown in class in order to isolate any figures entering the frame. We then went through a variety of attempts to reliably track the position of a figure in the frame and make use of the x/y location data. We began by using cv.jit.bounds to get a bounding box around the figure which we could draw the x and y co-ordinates from.

This solution turned out to be sub-optimal however as noise in the camera feed often resulted in multitudes of fluctuating and unpredictable boxes and location updates were jumpy and led to undesirable and jumpy animation of the fairy.

cv.jit.centroids gave us a different set of x/y co-ordinates and we found that tying the fairy to this output resulted in far smoother and more attractive motion.


Another alternative we briefly explored made use of the cv.jit.faces tool to track faces in a black and white video feed. This gave us an x/y point at the centre of any face that entered the frame which made for fairly smooth and easy info to pass to the part of the patch handling fairy animation. The face version also allowed us to have multiple people in the frame. Unfortunately the solution was extremely taxing on the CPU and crashed nearly ever time we tried to test it and had to be abandoned.

The fairy itself began as a simple circle, drawn at a location slightly offset from the x/y position provided by the figure tracking. Its location would shift as the person moved, seeming to follow nearby. This simple circle was replaced later on with a small picture of a fairy, animated in flash and played in the sketch as a looping movie.

Project Ideation Phase 2

In the lab following our team's first brainstorming phase we learned that the camera/projector set-up that the whole class would be using made all our ideas unfeasible in their current configurations. A further brainstorming period was required in order to improve our ideas and adapt them for the classroom's equipment set-up.

Most of the ideas we tossed around were extensions from the Invisible Friend idea from phase 1, and of those ideas there were a few that seemed viable and are detailed below.

An Emotional Fairy
Audience: Children (though adults are allowed to play too)

What is it: Taking inspiration from a line in Peter Pan that said "fairies are so small that they only have room in their heads for one emotion at a time." We have decided to create a digital fairy that will follow, flit about, and interact with people who are visible on camera. The fairy would be depicted as a glowing colored ball of light, with the color depicting its emotion. The fairie's default state is to orbit around and follow the user, maintaining about a 1' distance. If the user stays very still for a while or moves slowly enough the fairy will calm and fly in closer. Potentially landing upon the user for a time.

The fairie is easily startled and its emotions react to speed of movement from the user. A curious fairy will follow a running child moving away from it but fast, abrupt movements towards the fairie when it has drawn close will startle it and send it zipping away and then flying quickly around the user. Waving and swiping at it when its scared will freak it out even more, potentially leading to angry dive-bombings. Slow, calming movements from the user will coax it back down and perhaps once again land on their hand.

Playful Shadow
- 2 people in different rooms interacting with one another’s shadows
Peter Pan's Shadow

Continuing the Peter Pan inspirations, this idea was born from our wish to take the invisible friend through footprints theme up onto the wall where the projector will be aimed.

In this case the goal is to have the camera tracking the user's silhouette and the projector sending that silhouette back onto the screen offset slightly from the user and acting as his/her shadow. After an initial period where it perfectly matches the users movements the shadow will begin to mis-behave, taking on a life of its own and interacting with the user.

Achieving this second shadow seems challenging and the team tossed around possibilities for pre-recorded movements and other solutions before deciding that a dual set of the interactions could work to create the effect. A second camera and projector in a different room could be tracking a different person's silhouette. And each "misbehaving" shadow is in-fact the motions of the person tracked by the other camera. This second set-up makes the idea unfeasible for this project but remained an idea we were pleased with. 

2D --> 3D (Completing the Message)
- Pixar short inspiration
- 3D world only seen through 2D shapes
- Completing the message based on the amount of participants interacting

A piece of film, a message, some sexy typography is displayed onto the screen but only inside the silhouette of a user being tracked by the cameras.

One version of this could have it be such that the more people in the frame the more complete the message/broadcast is.

The second version would have each person blob in the view frame linked to its own video or message, and how the things displayed inside them relate and contrast with those of other users could become part of the focus.  

Iat320 Assignment 1 Ideas:

Our Body Interface design team sat down together to brainstorm ideas for responsive environments. Some of the ideas that we discussed are detailed below.

Number 1:

Camera location placed above the space. The system analyzes people in the space as blobs, tracking the location and size of the blobs, with larger clusters of people comprising larger blobs.

Result Possibilities: Projector (also above) displays balls of light (generated in processing?) that cluster around the x,y position of the largest blobs. Surrounding and illuminating the areas with more people. An addition to this could be tracking motion as well as clusters. If a smaller blob is displaying a lot of motion they can pull light balls away from the other blobs.

Another element that can be part of the result is more sound based. In this alternative speakers are placed around the space being tracked by the camera. Rather than focusing the results on the areas with the largest clusters of people this version focuses on the least used. Dubbed by our team as “The Needy Room”, this result plays soft, lonely sobbing noises in the loneliest corners of the room that just wants to be used and loved.

Number 2: Invisible Friend

The camera is again overhead and the projector projects upon the floor. In this version individual people are tracked by the camera and a pair of glowing footprints follow alongside them when they are within the space. These footprints act as an invisible companion for children in the space, usually following the individual but sometimes running ahead and trying to get the user to chase the footprints and play games of tag.

Number 3: Colored blocks as environment controls.

The third idea tracks colored blocks or pillows in a space instead of tracking people. The analyzing system in this idea reacts to color keys and shapes and produces different environmental effects depending on whether particular color objects are present in a zone. A user in the environment could control the feel of his workspace by carrying in color blocks for head and dim light, or bringing a white-noise block in that tells the system to blot out sounds from outside the environment. Carrying in a black pillow might trigger soothing music, etc. The idea with the project is to take the tangible interface notions of experimental computer interfaces and apply it to the control of rooms and environments.