CATEGORIZED AS

Getting Started with Mixed Reality (MR) in 9 Easy Steps

Posted on August 29 2016


Virtual Reality (VR) v.s. Mixed Reality (MR)

Following the release of Virtual Reality (VR) products such as the HTC Vive and Oculus Rift, VR has already become the hottest topic of the year. This has also led to a wave of sales of high-end VR-Ready PCs. As a gaming hardware manufacturer, MSI recommends gamers use an Intel B150 or higher motherboard paired with an NVIDIA GeForce 1070 display card for the best VR experience, or just directly purchase the newest AEGIS TI desktop computer that already has VR Link capability.

We’ve listed all of our VR Ready products on this page for your reference. If your PC meets the necessary performance specs, you can refer to this previous article (MAINSTREAM VR GADGETS: GET READY FOR THE VR REVOLUTION) to figure out which VR gadgets you can consider buying. This other article (THE UPCOMING VR GAMES YOU SHOULD LOOK INTO) is good for learning about the latest games you can purchase. Also, We’ve released MSI Electric City, a VR experience game; those interested can download it from its exclusive webpage.

Now, let us talk about how to create mixed reality videos to create a virtual visual experience. Below are some videos illustrating the concept: (Below content is translated from The CY  blog and agreed by the author.)

 (HTC Vive SteamVR Promotional Video)

 (Tilt Brush Promotional Video)

 (Sound Stage Promotional Video)

 


Step 1: Purchasing Equipment, Introducing Concepts

MR Shopping List

If you already have a Vive and a VR Ready high-end PC, you will probably end up spending roughly US$900 buying everything listed in the MR Shopping List, excluding the video camera. You may be able to find these items at a cheaper price if you shop around for lower-priced distributors.

Concept:

There are a few Unity games that support mixed reality video, such as Space Pirate Trainer, Job Simulator, Zenblade, Tilt Brush, and so on. Mixed reality recording mode will automatically open after a third controller has been connected and a special configuration document has been created. The third controller is used to position the camera to produce the virtual environment. The video feed of the user playing the game in actual reality is processed so that they background is removed. All of these images are compiled in real time by Open Broadcaster Software (OBS) to complete the mixed reality environment.

image007

 (Photo Credit: How to Create Mixed Reality Videos for the Vive - with Two Controllers)

 

Step 2: Setting Up a Green Screen Studio

The purpose of a green screen studio is to allow for the background to be removed and replaced.  You’ll need at least three pieces of green cloth, two for the walls and one for the ground.  After setting up two racks to hang up two pieces of green cloth so that they overlap, you can lay down the final piece on the ground. For the best results, you could pull the hanging cloth taut and tape them to the racks so that they stay as smooth as possible. Even lighting within the studio set-up can ensure a better end result.

SONY DSC

 

Step 3: Install a Capture Card or Box

Capture cards or boxes are used to capture live video into OBS. It is normally a hassle to install a capture card, since you have to open up the computer case. I bought a GC550 LGX capture box because it connects directly to your PC with a USB cable (you don’t have to buy such an expensive model, look to my shopping list for other recommendations).

SONY DSC

 

Step 4: Create a Game Configuration File Folder

Copy the following text into a new Notepad file and save it as “externalcamera.cfg” (be sure to change the file extension to .cfg) or click here to directly download the file. Place this file into the directory where you want to open your MR game in (some supported games include Space Pirate Trainer, Job Simulator, Zenblade, and Tilt Brush).

image013

x=0
y=0
z=0
rx=0
ry=0
rz=0
fov=60
near=0.1
far=100
//m=-0.999059,0.015577,-0.040472,-0.0127,-0.016016,-0.999816,0.010544,0.1799,-0.040301,0.011183,0.999125,-0.0846
sceneResolutionScale=0.5

Step 5: Position the Third Controller Near the Camera

The function of the third controller is to locate the position of the camera, so it needs to be placed in near the camera where it will remain completely untouched. Cameras will normally have a hot shoe mount, which can be used in conjunction with a clip to secure the controller, or you may choose to fasten it directly to the flash unit of your camera. In my trial, I used the Sony HDR-CX405’s flash unit hot shoe mount, added an L bracket and used an iPad clip (something I found while shopping around that was a perfect fit) to secure the controller on top of my camera.

SONY DSC

The most optimal arrangement is to secure the controller to the camera horizontally, shown below, as this will make sync set up a lot easier.

image017

(Photo Credit: ">Mixed Reality Trailers: Behind the Scenes)

 

Step 6: Turning On Controllers in a Specific Order

The Unity plug-in will automatically establish the third controller as the one used to position the camera lens, so it is necessary to turn the controllers on in the following order:

  1. Make sure no controllers are connected to the computer
  2. Turn on SteamVR, then turn off SteamVR
  3. Turn on Controllers 1 and 2 (the two controllers not secured to the camera), in other words, press the system button (the button below the disk)
  4. Turn on SteamVR, and confirm Controllers 1 and 2 have been detected
  5. Connect Controller 3 to your computer by plugging in the USB
  6. Confirm Controller 3 has been detected

The above directions ensure the controller used to position the camera is correctly detected as Controller 3.

 

Step 7: Launch a Game in Mixed Reality Mode

We used Space Pirate Trainer in our example. After making sure externalcamera.cfg is within the game directory, launch your game and follow these instructions:

  1. Confirm all three controllers have been detected by Lighthouse
  2. Hold Shift while double-clicking on the game (Space Pirate Trainer) to launch it
  3. Choose the highest resolution for your screen
  4. Normally at this point, the game will still be in regular view, and one controller will be missing
  5. Put on your headset and press the trigger a few times while looking at Controllers 1 and 2 until they both appear in game
  6. If your computer screen has now split into a 4-way screen, you have successfully turned on MR mode

 

image019

(The four screens are, respectively, Top left: foreground; Bottom left: Camera angle (Controller 3); Top right: Background; Bottom right: first-person viewpoint)

 

Step 8: Setting Up OBS

This example was conducted on a Full HD (1080p) monitor:

  1. Connect your camera’s output to your capture card/box’s input
  2. Download and launch OBS (Click here to download OBS Studio)
  3. Click on “+” → “Video Capture Device” under Source, and turn on your camera
    Device: Choose your capture card (I was using AverMedia GC550)
    Resolution/FPS type: Custom
    Resolution: Choose the resolution of your real world camera (Mine was 1920x1080)
    Press Confirm once your camera’s image appears in OBS
    image021
  1. Right-click on your newly added capture device, choose Filters, and left-click on “+” → “Add a new Chroma Key/Color Key” → “Key Color” → Choose “Green” as the Type, and adjust your Similarity and Smoothness levels until the green is completely erased.
    image023

 

  1. Launch your game in MR mode, then press “+” → Add a “Window Capture,” and select your current gaming window.
    image025

 

  1. Right-click on the newly added window capture screen, select Filters, and add a new Crop filter to the top and right to remove all the screens except for the lower left screen. Once you’ve enlarged and centered the window, you’ll be able to see the background of your game.
    image027

 

  1. Use the same process laid out in Step 6 on the foreground screen, erase the black using the Color Key filter, and align the image with the background. Note that the order should be: 1. Foreground, 2. Camera, 3. Background. Lastly, all you need to do to complete the sequence is to sync the camera and controller.
    image029

 

Step 9: Syncing the Camera and Controller

This is the last and also most troublesome step. The information within externalcamera.cfg includes the distance between the camera and Controller 3, the angle difference between the two, and the actual FOV quality of the camera. Perfect synchronization can be achieved by correctly adjusting the parameters within the document by following these steps:

SONY DSC

(Positioning and direction of the camera and the controller’s preset position should be as illustrated above)

 

  1. Measure the distance between the camera and the controller’s preset position, which includes horizontal axis (X), vertical axis (Y), and front/back (Z), as well as degree of rotation (normally if placed level your rotation should be 0 degrees, but mine was slightly slanted upward 16.5 degrees)
    image033

    SONY DSC

 

  1. Calculate the Vertical FOV. This value isn’t as easy to calculate as it seems, as most FOV values provided within a camera’s general specs is not the Vertical FOV. Kert Gartner has provided a calculator, which may be used by more professional users. I utilized another method to measure my FOV, which will be explained further on. Here I’ve listed some Vertical FOV values for a few cameras:
  1. Fill In Your Values
    Open externalcamera.cfg using Notepad and fill in the values you just measured. The units for X,Y, and Z are meters, and rx,ry, and rz (rotation) are in degress.

    The following is my example: the difference in positioning between the camera and controller measured X with a difference of 2 cm, Y with a difference of 17 cm, Z with a difference of 2 cm, and a front/back degree of rotation of 16.5 degrees. Save your document after entering the values.

x=0.02                      
y=-0.17
z=0.02
rx=16.5    
ry=0
rz=0
fov=42.7
near=0.1
far=100
//m=-0.999059,0.015577,-0.040472,-0.0127,-0.016016,-0.999816,0.010544,0.1799,-0.040301,0.011183,0.999125,-0.0846

sceneResolutionScale=0.5

  1. Launch the game and view it in OBS. Turn the game off and on, making adjustments to your values, until the display looks just right. If you did an accurate job of measuring your values, you will only have a very small amount of fine tuning to do before everything looks perfect. Take your time adjusting the parameters.
  1. If you were unable to calculate your Vertical FOV in Step 3, assume that the values for XYZ or rxyz are already correct and begin making large adjustments to the FOV. After grasping the characteristics of the FOV, slowly adjust values until they seem correct, and then go back and fine tune xyz and rxyz. This may take a lot of time, but there are no shortcuts for this step, so it is imperative that you make the correct measurements in Steps 1 and 2. This will save you a lot of time as you slowly pinpoint your FOV.
  1. You will have finished syncing after completing the aforementioned steps, and are now able to record or live-stream your MR game video through OBS.

 

 (This is a recording I made during my test run; if I spent more time fine-tuning, I would have able to produce better results)

------------

 Thanks to author The CY for the contribution.

BLOG CATEGORIES

RECENT POSTS

RELATED ARTICLES

Article tags

us