Tips for Using Kinect (and Other Sensors) in Live Interactive Installations
Avoiding common pitfalls when working with cameras and other sensors in public settings.
Back in 2016, I built Kinectron, a tool that sends volumetric and skeletal data online using the Microsoft Kinect. Since then, Kinectron has become widely used by students at NYU ITP, especially in the end of semester shows. After getting some feedback from students, I thought it would be helpful to write a quick post with some tips for using Kinectron in public installations. I’ve written specifically about my experience working with the Kinect sensor, but the tips are good to keep in mind when using any sensor that will respond to users in a public place.
Tip #1: Use Ethernet, Not Wi-Fi
This first tip is important for any installation that uses a local or public network to transfer sensor data, like Kinectron. Hardwired Ethernet lines guarantee a certain amount of bandwidth at each jack. Wi-Fi on the other hand shares bandwidth between all the users connected to each router. As more people join a Wi-Fi network, each user has access to less bandwidth.
This is a common pitfall because Wi-Fi will usually be stable during set up and testing, but as soon as several people come into a room and join the network, the bandwidth becomes diluted—and then it’s too late to troubleshoot.
The best way to avoid this is to give yourself a hardline to the network. Remember that when you plug into Ethernet, you should also switch your Wi-Fi off. This way you keep your computer from switching between networks while your project is running.
If you’re working with Kinectron, make sure that you hardwire both the computer running your server and the one running your client code. If you only have access to one Ethernet port, prioritize the computer running your Kinectron server.
Tip #2: Control Your Environment
It’s important to take the time to set up your installation to minimize unintended data from entering the range of your sensor. It’s true that there’s no way to have complete control over an installation environment (especially at an ITP Show!), but there are certain ways to minimize impact.
First, ask the show producers for what you need—even draw them a diagram of your ideal space and setup. You never know what is possible until you ask, and I’ve found that you’ll usually be pleasantly surprised at how accommodating show producers can be.
When working with the Kinect make sure you position your installation so the Kinect faces away from a crowd or heavily trafficked area. The Kinect tracks up to six bodies, so unless you’re intentionally looking for chaos, keeping it from facing directly into a crowd is a good way to minimize unexpected results. From here, use whatever is available to you—curtains, screens, tape, signs or other artificial barriers—to help users know how to best move through your installation. It’s amazing how effective a few lines of gaff tape on the floor can be at keeping unwanted users from getting in front of a live camera or sensor.
Tip #3: Be Specific in What You Track (And Don’t Track)
Each sensor has different capabilities and requirements, but there’s always a way to hone in on the data that you need and put aside the data that might get in the way.
When using Kinectron, for example, you can use player tracking IDs to track certain people and ignore others. Let’s say you have an installation made for one person. You can create a function that looks for a player on a key press. Once it finds one player, it locks that tracking ID and ignores anyone else who comes into the scene. On the next key press, the program releases the player and looks for a new one.
Here’s some sample code to track a single player on key press using Kinectron in p5.js.
// create a boolean to indicate if we're tracking a player
let tracking = false;// create a variable to hold ID of current player
let playerID = null;function keyPressed() { // press enter to reset tracking to false
// this will allow you to look for a new player
if (keyCode == ENTER) tracking = false;
}// this function runs each time body data is received
function trackBody(allbodies) { // get all bodies
let bodies = allbodies.bodies; // kinect tracks up to six bodies
// let's loop through each of the six bodies
for (let i = 0; i < bodies.length; i++) {
// if we are not yet tracking a player/body
// and we find a tracked body
// set tracking boolean to true
// and set the playerID to the trackingId of the current body
if (tracking == false && bodies[i].tracked == true) {
tracking = true;
playerID = bodies[i].trackingId;
} // if we are tracking and the current tracked body
// is the same as the player ID
// then draw the joints of the body
if (tracking == true && bodies[i].trackingId == playerID) { // iterate through all of the joints and draw them to canvas
for(let j = 0; j < body.joints.length; j++) {
joint = body.joints[j];
drawJoint(joint);
}
}
}
}
Conclusion
Every time you write a program you make 10s, 100s, maybe 1000s (depending on the size of your program) of decisions that will impact the way that someone interacts with your work.
Although you can never completely control your environment, thinking about the user and the environment as you make programming decisions will help to minimize issues at show time.
I wrote this article as a response to questions from creative coders using the open source software, Kinectron. Kinectron is a realtime peer server for Kinect V2 that makes skeletal and volumetric data available in the browser through an easy to use API.
Kinectron was developed under the Google Experiments in Storytelling (xStory) Grant at New York University’s Interactive Telecommunications Program (ITP). It is currently under development. Please get in touch if you’d like to contribute.
Many thanks to Aarón Montoya-Moraga for all the helpful insights, and for his unwavering commitment to open source creative tools.
