Holodeck – Vicon meets Oculus

Introduction

This project is based on an idea to build a budget-version of the ‘Star Trek Holodeck‘. This is not a particularly novel idea – it’s been the holy-grail of Virtual Reality for the past 30 years! There are a number of people working with a variety of tracking devices and budget virtual reality equipment to produce similar solutions. This version uses external infra-red cameras for tracking and the Oculus Rift for the headset.

This hardware / software combination allows a user to walk around a real, physical space but the physical movement is used to move a corresponding virtual avatar and the output of that is piped back to the Oculus Rift headset. Ultimately, this gives the experience that the user is walking around a virtual space while actually walking around in the real world.

Practical Applications

While their are many practical applications, for example, imagine the benefits of walking around a 3D model generated from architectural plans before a building has been built. I am personally more interested in it’s use either artistically or for entertainment.

One issue is that tracking of user’s body parts. In the initial version only the player’s head is tracked and the rest of the body is implied from this head position. This means that when a user looks down – they will not necessarily see a matching body. One option is to track additional body parts, however, this has the added difficulty in dressing a user in a marker-covered suit beforehand. A simpler solution is to give the user an item to hold – then by tracking the item we could imply the position of the hands.

Development

The prototyping hall has an area of tracked space that is approximately 9m x 7m. The external camera tracking is performed using 20 Vicon Bonito 1.3 infrared cameras. These are the same ones used by motion-tracking studios when creating motion information from actors for Hollywood films. Typically, an actor would wear a full-body suit covered with reflective markers. Our system is set-up slightly differently as it is designed to track locations of individuals – so it can track many simultaneous users but no in as much detail. Instead of wearing a marker suit, users don a baseball cap. This means we have a very accurate location of a users head. This location is used to position a virtual camera.

A 3D Unity model of the prototyping hall has been constructed whi

ch has been scaled to match the real-world movement of the user. Information taken from the Vicon tracking system is used to position the virtual camera. While this provides good quality data – it would be possible to combine this with the positional data from Oculus.

This image shows a user wearing the Oculus and marker covered baseball cap. All the equipment to run the experience is contained in the backpack and connects wireless to the Vicon tracking system in order to receive positional data.

Next Steps

The trackable item that I am planning to provide to the user is a Buckshot plastic gun as developed for the Nintendo Wii system. This is a plastic housing for the Wii controller which is intended to be held in two hands implying the position the both hands.

Ultimately, I think this should be extended to four player collaborative experience – perhaps in the increasing growth of very unique out-of-home experiences that customers are willing to pay a premium for. It would be possible to have the real world environment to roughly correspond to the virtual environment so that walls and doors can be felt by players. The most obvious theme for this would be a Left-For-Dead style zombie game.

Since starting this work I’ve seen a number of people trying to create something like this, for example Project Holodeck which has recently become Survios.