Skip to content

Lab 5: Not My Body

The purpose of today's lab is to give you experience with setting up avatars for VR in Unity. We'll learn about how to use rigging and inverse kinematics (IK) to have our avatars move the way we move! We'll also cover how to perform some simple gesture detection to open and close a hand menu.

Part 1: Getting Started

First, download the starter project here: Avatar_Start.zip

Unzip it and open it in Unity Hub. If asked to select a version of Unity, select a version that is 2021.3.29f1 or above. Accept all the prompts the prompts to "migrate" the project to a newer version if needed.

Important: Go to Edit > Project Settings... and then Player > Company Name and replace this with your EID or a similar short name. This will allow multiple people to build on the same headset without clashing with each others' apk.

Open the NotMyBody scene (it should be in the Scenes folder under Assets in your Project view). I've provided you with the environment, a tiny menu, and a mirror to look at. You'll need to set up the menu and the avatar through this lab.

Download an Avatar

First, we'll need to find an avatar that we want. Adobe Mixamo provides a number of different characters and animations which we can use. Go to the Mixamo website. Click "Sign Up for Free" and create an Adobe account (or, I recommend using your Google account or one of the other social log-in options!).

Afterwards, go to the character gallery and find a model that you like.

Once you have a model selected, click the Download button on the right. Under format, set the Format to "FBX for Unity(.fbx)" and Pose to "T-pose". Click Download to download the model.

Once it's downloaded, go back to Unity. In the Project View, navigate to Assets > Avatars. Inside, right-click on the empty-space and then select the Import New Asset... option. In the file picker, find the .fbx file you just downloaded. Alternatively, you can also drag-and-drop the .fbx file into the Project View.

Afterwards, you should see your model in the Project View.

Configuring the Import

Select your model in the Project View, then go over to the Inspector.

First, go to the Model tab. If your character ends up too big, you will need to change the Scale Factor here. Keep this in mind!

Next, go to the Rig tab. Here, set the Animation Type to Humanoid. The other settings can be left as-is. Click the Apply button to apply the settings.

Now, go to the Materials tab. By default, Unity imports the materials all wrong for the model. If your model looks gray, we need to configure some import settings to fix it.

Set the Location drop-down to Use External Materials (Legacy). Click Apply and wait for Unity to extract the textures and materials from the file. If prompted to fix any normal maps, agree to those as well.

Fixing the Materials

Sometimes there are transparent materials that Unity doesn't render properly either. In the Project View, you should see a new Materials folder next to your model. Open it.

Inside, you should see your materials for the model. For each of them, select it and go to the inspector. Switch the Rendering Mode to Cutout.

Now your model should be ready to use! Drag-and-drop it into your scene or hierarchy to add it.

Part 2: Rigging the Avatar

First, reset the avatar's transform. Next, we'll need to move the Camera Offset slightly forward so that it sits right in front of the avatar's face (so that you don't see it from the headset view). Adjust it carefully so that it is as close as possible without being inside the avatar's head.

Next, we need to set the avatar up for inverse kinematics! This would typically require installing the "Animation Rigging" package, but this time, it should be installed already.

First, click on the avatar in the Hierarchy. With it selected, click on Animation Rigging > Bone Renderer Setup in the top menu.

Now, you should be able to see the model's bones!

Each one of these bones also represents a joint in the skeleton and can be rotated to make the model move. This is how animators make characters move for 3D animation! The process of setting angles for each joint is also called forward kinematics. Try clicking on a bone, and then use the rotate tool to rotate it!

But we don't want forward kinematics, so undo your rotations (sorry lol).

We want inverse kinematics! What's the difference? With inverse kinematics, we set a target location for the hand, and algorithms are used to calculate which angles the other joints need to rotate in order to support the hand location. See this animation1 below showing the difference:

In the Hierarchy, expand the avatar's GameObject and click on the mixamorig:Hips object. (Your object may be named something similar, like "mixamorig4:Hips"). With it selected, click on the Animation Rigging > Rig Setup option in the top menu.

This will add a new rig called Rig 1 as a child of the hips.

Configure the Arms for IK

Left Hand

  1. Click on Rig 1, then create an empty child GameObject and call it Left Hand IK.

  2. Select the object, then, in the Inspector, add the Two Bone IK Constraint component. We need to set the Tip to the left hand joint in the skeleton. In the upper-right of the Inspector, there is a little lock. Click it.

    This will make it so that the Left Hand IK object stays focused even if we click on other objects in the Hierarchy.

  3. Now, in the Hierarchy, expand the hips object and go down the spine until you find the left hand:

    Drag the mixamorig:LeftHand object into the Tip property in the Inspector. (Alternatively, you can click the little circle button and search for the left hand.)

    Once you're done, unlock the Inspector!

  4. We also need to set the mid to the forearm and the root to the shoulder. Fortunately, there's a fast way of setting these!

    Right-click the header for the Two Bone IK Constraint component, and then click the Auto Setup from Tip Transform option. This should set the bones AND create two new child objects: a target and a hint (we'll get to these).

  5. Next, in the Hierarchy, expand the Left Hand IK object to see the children. Also expand the XR Interaction Setup until you find the Left Controller object. Drag and move the Left Hand IK_target to make it a child of the Left Controller instead.

  6. Set the position of the Left Hand IK_target to (-0.03, -0.04, -0.2) and its rotation to (0, 90, 90). These position it so that the hand aligns with the controller. The values were determined by trial and error.

Now, run the game and try it out! Move the left controller around and the hand should move to match. If needed, adjust the position of the IK target some more to match. Don't forget to exit play mode before making changes!

You may notice that the wrists turn strangely when the palms face up. This is a limitation of the built-in Unity IK system. There is a way to make it slightly better. Follow these steps:

  1. Select the Left Hand IK object again. In the Inspector, add a Twist Correction component. Important: Drag the Twist Correction component and move it up to appear before Two Bone IK Constraint.

    This component will make the forearm rotate along with the hand.

  2. Expand Twist Nodes and click on the plus to add a new entry. Then set the entry to the mixamorig:LeftForeArm object. I recommend locking the Inspector first.

  3. Set the value on the right to 0.9. This means that, when we rotate the hand, 0.9x of the rotation will be applied to the forearm instead (this is closer to how our hands actually rotate).

  4. Set the Twist Axis to the Y axis. Now, expand the Source section and set the Source to the mixamorig:LeftHand object. Unlock the Inspector when you're done.

  5. Last, but not least, let's improve how the elbows move. Imagine if each of your elbows were attached to a point on the floor with an elastic band. They would tend to bend towards that point. This is what the "IK_hint" object does!

    Select the Left Hand IK_hint object (a child of the Left Hand IK). Move it along the X axis to be parallel with the left shoulder (about (-0.2, 0, 0)).

Right Hand

Repeat these steps (including setting the twist correction and hints) for the right hand.

Set the position of the Right Hand IK_target to (0.03, -0.04, -0.2) and the rotation to (0, -90, -90). Move the Right Hand IK_hint to be parallel with the right shoulder (about (0.2, 0, 0)).

That's it! Now the hands (and arms) are configured to follow the controllers.

Configure the Head to Match

Let's setup the head to track with the headset. First, create another empty child under Rig 1 called Head IK.

In this object, create a Multi-Parent Constraint component and configure the following settings:

  1. Set the Constrained Object to the mixamorig:Head object (a child of the neck).

  2. Add a single source object and set it to the Main Camera (XR Interaction Setup > XR Origin (XR Rig) > Camera Offset > Main Camera).

  3. Expand settings and set Maintain Offset to Position and Rotation. This will make it respect the camera adjustments we made earlier.

You should have something like this:

Make the Body Follow

Great, we now only need to make the rest of the body follow the head. We will do a simple body follow. More complex IK setups can make it so that the legs bend and stuff, to enable crouching. We'll just have the body float around with respect to the head.

Note

If you want to try adding foot IK, it's the same as setting up the arms. You should make the IK targets children of the XR Origin. When later moving the avatar via script, also move the X and Z positions of these targets.

In the Hierarchy, select the main avatar GameObject (it should have the hips as a child). Add a new component/script and name it AvatarController.

Open the script. The goal is to get the position of the head and move the body relative to it.

First, create the following member variables so that we can get access to the head location and also store the offset between the head and the initial body location:

public Transform head;
private Vector3 offset;

We want to store the offset between this object's location and the head's location. We can do this by turning our position into a "local position" (aka a position that's relative to the head). Put this code in the Start() method:

offset = head.InverseTransformPoint(transform.position);

Now, we have to make sure to update the position and rotation of the avatar at each frame. This is a little tricky.

We want to make sure the position of the avatar respects the original offset (so that the body doesn't move in front of the camera). We also need to restrict rotations along the Y axis only (turning left/right) so that we don't have the body leaning in strange directions.

This code should do the trick. It basically fixes the offset so that it only cares about the head rotating left/right, and not up/down. The height is also fixed to the original height. Put the code in the Update() method:

// Get the orientation with respect to the 2D floor plane (XZ)
Vector3 relativeX = Vector3.ProjectOnPlane(head.right, Vector3.up).normalized;
Vector3 relativeZ = Vector3.ProjectOnPlane(head.forward, Vector3.up).normalized;
Vector3 newOffset = relativeX * offset.x + Vector3.up * offset.y + relativeZ * offset.z;

// Update the avatar based on the head position and offset
transform.position = head.position + newOffset;
transform.rotation = Quaternion.Euler(0, head.rotation.eulerAngles.y, 0);

Now, go back to the Unity Editor and set the Head property of your script to the mixamorig:Head GameObject.

Finally, under each controller in the Hierarchy, you will find the LeftHand and RightHand models. Expand each of them and you should find a hands:hands_geom child. This represents the visible geometry for each hand model. Go ahead and disable them.

That's it! Like I mentioned, you could also enable foot IK for a better-behaving avatar. Proper avatars also enable head rotations independent of the body, and have animations for the hands when the controller buttons are pressed. These are more difficult to implement, so we won't dive into them unless you choose to do so for your group project!

Part 3: Making a Hand Menu

Last, let's make the menu that appears when the user faces their palm upwards. I've already created a mini menu with buttons that zoom the mirror and quit the app. You can find the Hand Menu in the Hierarchy.

First, let's attach the menu to the hand. Expand the XR Interaction Setup until you find the Left Controller GameObject. Now, drag the Hand Menu to be a child of the Left Controller. Reset its transform.

Reposition the menu so that it's hanging off the right side of the left hand. This will depend on what your avatar's hands look like.

Now, we just need to make it so that the menu shows and hides based on the hand rotation. There's a few different ways to do this. I'm going to show you how to do it based on vector angles. We need to compare the hand palm orientation with a target orientation. If we look at the vectors of the controller, we can see the palm direction is represented by the X axis:

Add a new component/script to the Hand Menu and name it ShowHideMenu. Open the script.

We will need access to three things, the menu itself, the controller, and a GameObject that represents the target orientation. We also should make the show/hide menu thresholds configurable. Add the following member variables:

public GameObject menu;
public Transform controller;
public Transform target;
public float showMenuThreshold = 30;
public float hideMenuThreshold = 40;

Now, on each frame, we need to compare the angle of the controller with the target and show/hide the menu based on how close the angle is.

Write the following code in the Update() method:

float angle = Vector3.Angle(controller.right, target.right); // Comparing local X axes

if (angle < showMenuThreshold)
{
    menu.SetActive(true);
}
else if (angle > hideMenuThreshold)
{
    menu.SetActive(false);
}

Now, go back to the Unity Editor. Let's set the Menu property to the child Canvas GameObject and the Controller property to the Left Controller GameObject.

For the target, create an empty child GameObject on the Main Camera. Name it Target Pose. Configure the rotation for Target Pose so that the X axis is facing back and up a bit, by setting it to (-60, 0, 90). This will make it so that the user is effectively looking at their palm.

Go back to the Hand Menu object and set the Target property to this Target Pose object. You should have something like this:

Try building it and running on the headset! Follow the instructions here. Don't forget to turn off the XR Simulator first!

Submission

Submit a screenshot of Unity showing your scene view (including avatar) to Canvas for lab credit.

  1. Animation from https://immersive-insiders.com/blog/how-to-animate-your-avatar-for-vr-in-unity ↩