Virtual and Augmented Reality with Unity
PF Villard
My story
Augmented and virtual reality in Medicine
→ Check my webpage for more details
Research
Respiration modeling
Heart valve simulation
Deformation and motion of tongue, diaphragm, liver, ...
outline
Presentation of Unity
Installation of Unity
Basic use of Unity
Presentation
Tool to design games
Features :
3D model import
Terrain modelling
Physic engine
Audio
Network
Script programming (C#)
Rapid game prototyping
Why using Unity
Cross-platform game editor
Available in pay or free versions
the free version offers fewer features such as advanced rendering effects
Allows you to create games
Standalone PC, MAC, Linux
Games in a webGL player
Games for iOS and Android
PS4, Xbox One, PS Vista Games
Facebook
Based on C# (POO)
Differences with Java Games
A lot of programming
Start from scratch
Result not automatically visible
Unity Games
Little programming
Components already available
Very visual
Interactive
Selection of games under Unity
Install Unity
Download and install Unity fromwebsite
Select also:
webGL → project export
Vuforia → to make augmented reality
Graphical user interface
The main interface
Scene : The scene you are currently working on
Game : Overview of your video game in real time
Hierarchy : The hierarchy of the different components of your game
Project : The different assets loaded in the project
Inspector : Area that allows to have details on each component of the game
File : Create, open, close scenes and projects, and compile your game
Edit : Copy/Paste/Cut and different selection and zoom options
Assets : Create or import assets
GameObject : Create the objects/elements of your game (3D, 2D, lights, sound, interface, particle, camera,...)
Components : Add components or properties to your objects/elements
Window : Open/Close the different parts of the Unity interface
Help : Unity Help, About and Updates
Importing an existing asset
Open the asset store: Window → Asset Store
Search the package Altar Ruins
Click on Import then Download asset
Observe structure and data in the part Project
Modification of the layout
Load the scene contained in the scene folder
In the part Hierarchy select Tree
Moving in the scene (arrows + wheel )
In the Inspector modify position, rotation and scale
Upper left part
Change the display of the scene to Shaded wireframe
Try all display modes
Try the button to activate the 2D mode
Try the buttons to activate/deactivate the lights
Try to activate/deactivate effects (fog, sky,...)
The drop-down menu Gizmos allows you to choose which components will be displayed or not in the scene
On the far right, above Inspector from the drop-down menus presented below:
Layers : allows you to select the graphic layers to be displayed
Layout : allows you to view the scene differently
Import the package Standard Assets
Import a First Person Character
Import the package Character
Confirm import
Drag and drop FPSControler in the scene
Delete the Camera
Test the game
Test the game by clicking on the icon play
Browse the scene to see the possibilities
Test the buttons pause and frame by frame
Change the dimensions of the image and the ratio
Mute the sound and display statistics
Click on Maximize on play the game will be displayed in large
To change the keys, go to Preference → Keys
Outline
Notions of fps
Graphics pipeline
Interactivity
6 fps usable but slow
Between 30 fps and 60 fps for video games
It depends on the material
Computer screens generally 60Hz
Try it on your computer!
Graphics pipeline
Sequential operations applied to the data to obtain an image
Graphics pipeline : 1 - Unity
→ Development of an abstract description of the scene
Geometry (shapes, location)
Photometry (colors, materials, textures, opacity, etc.)
atmosphere (lights, sound, fog)
behaviour (animation, interaction)
visualization (type of projection, observation position)
Graphics pipeline : 2 - 2D Projection
→ For each vertex:
Transforms the position 3D into position 2D
Calculates the attributes (ex: color )
Proceed to clipping
→ On the GPU and programmable via the Vertex Shader
Clipping
Clipping
vertex shader → (X,Y,Z,W) Coordonnées clippées
W=0 : vecteur
W=1 : point
Le vertex shader
graphics Pipeline : 3 - Rasterisation
→ Setting up the triangles:
Search for the pixels contained in each triangle
Uses the Bresenham algorithm:
→ On the GPU
Graphics Pipeline : 4 - Pixel colours
→ For each triangle:
Interpret the attributes of the vertices over the whole triangle
Calculates the color of each pixel
→ On the GPU and programmable via the Fragment Shader
The Fragment shader
Graphics Pipeline : 5 - Z-Buffer
→ Tests pixel visibility:
Test the depth
Mix the colors
→ On the GPU
Mysterious depth handling
Outline
3D space
Meshing
Construction
Terrain modeling
Using Blender
Vertices, line and triangles
That's all you need!
No volume
Vertices
Line
Triangle
Space in Unity
Creating and manipulating objects
Create a new 3D project
Add a Cube (GameObject →3D Object )
A cube of size 1x1x1x1 is then created at coordinates (0,0,0,0)
To manipulate it, use the buttons at the top left.
view (shortcut Q ) allows you to move around the scene
Translation (shortcut W ) allows you to move the object
Rotate (shortcut E ) allows you to rotate the object
stop (shortcut R ) allows you to change the scale
2 - Mesh
Polyhedral approximation
Approximation by flat facets from points distributed over the
surface area
Take into account the geometry of the object to preserve the edges
vivid
Rely on a face configuration
Triangulate
Put the triangles together in strips
Pooling vertices (indexed geometry)
Mesh complexity
Increase resolution → Costly
Backface culling
Rendering optimization
The fastest way to display a polygon is not to display it
Can be activated or not
Example with a V-shape
Only one side will be displayed at the end
Vertex order
Used to determine whether front or rear face
Front panel if trigonometric direction
Example
Geometry modelled under 3DS Max
Geometry visualised under Unity
Unity File
Record the scene
Register the project
Exit
Open the project
Add water
Explore the Island
Add invisible walls around the island
Import the package Character
Add a FPSController in the middle of the island
→Run and observe the island
Using Blender
Many free 3D models exist on the internet
Otherwise, the easiest way to use Blender
Training with internet tutorials
Import in FBX format
Mesh optimization
Low Poly / High Poly
Unity 3D announces 64k per object
Play on calculation time / accuracy
It is better to play with textures and other tricks
Depends on equipment (changes over time)
Outline
Colours
Pixel colours computation
Ambient colour
Diffuse colour
Specular colour
Transparency
When should the colour be calculated?
Gamut
=all the colours that a certain type of material can reproduce
Choice of a colour
Create a new material
Choose the colour via the albedo
=proportion of incident light reflected from the surface
How to calculate the colour of each pixel of the object?
Simple lighting model
emissive + ambient + diffuse + specular
Lighting calculation
Couleur en surface =
couleur Emissive +
couleur Ambiante +
Pour chaque source de lumière :
couleur diffuse (source) +
couleur spéculaire (source, point de vue)
C= E + A + ∑ (D (L) + S (L,V))
Diffuse light on a plane
Diffuse light on a sphere
Diffuse light on a sphere
Need for normalization of normals
The dot product
Diffuse materials
C= ambient + colour* ∑ (N⃗.L⃗i)
It's called the Lambertien model
Included in Unity
Use of normals for shadow calculation
How to switch from known normals to geometric normals?
Use of smooth normal
By default (Smooth shading )
Otherwise, write your own shader
Demo
couleur = KA *material+KD *material*(N⃗.L⃗)
Specular materials
Specular materials
specular colour = max(N⃗.H⃗, 0)n
Specular materials
specular colour = max(N⃗.H⃗, 0)n
Lighting effect
Generalisation
Physically Based Rendering (PBR)
Takes into account physical properties e.g. energy conservation or light scattering
By default, Metallic workflow with the properties :
Metallic , usually 0 or 1
Smoothness how light is reflected
Baking
When is it possible to save the pre-calculated shading colours?
Gouraud shading
Mach bands
The human visual system increases the contrast on the contours
Phong shading
Phong shading
Calculation example
Each vertex is shared by 5 triangles
Each triangle covers an average of 60 pixels
→ Phong shading is times slower than Gouraud shading
(help: example for 100 triangles)
Transparency - Introduction
Transparency - caustic
Screen-Door Transparency
Alpha blending
Transparency in Unity
Linked to Albedo
It contains the value of the alpha channel
How Texturing Works
How Texturing Works
Texture mapping
Texture Atlas
Mosaic
UV Coordinates Distortion
Quiz on distortion
What is the source of the distortions?
How to apply a texture
Repeat
Repeat in mirror
Stretch at the edges
Modes of application of a texture
→ Only available through shader writing
Texture transformation
Texture magnification
Texture magnification in Unity
Texture magnification
Magnification Minification
Texels/Pixel < 1
Texels/Pixel > 1
Texture magnification demo
From fuzzy to sharp
Why do edges go from blurry to sharp?
Unzoom on a texture
Magnification Minification
Texels/Pixel < 1
Texels/Pixel > 1
Mipmapping
Magnification Minification
Texels/Pixel < 1
Texels/Pixel > 1
Mipmapping
Mipmaps chain
Quiz on increasing the size of a mipmap
Examples in Unity
Examples in Unity
Anisotropy
Anisotropic sampling
Sampling : retrieving a texture’s colour for a fragment
Anisotropic : having a different value in different directions
Transparency Mapping
Transparency Mapping
The Meaning of Alpha
Premultiplied Alpha
What is the advantage of having a premultiplied Alpha ?
C=αs Cs +(1-αs )Cd
Cp =αs Cs
C=Cp +(1-αs ) Cd
Particles and Billboards
Other examples:
Displacement and normal map
Displacement and normal map
Displacement and normal map
Example in Unity
Displacement and normal map
where does the displacement of the surface happen?
Fire creation in Unity
Create an environment composed of 6 walls (cubes)
Apply any material to these walls
Add a cylinder with a wood texture
Add a particle system (from GameObject )
Parametrization solution:
Start Lifetime to change the flame height:
Start Speed to modify the propagation:
Start Size to change the size of a particle:
The parameters of Shape
Rate over Time of Emission to modify the number of particles
The Size Over Lifetime curve to have a flame shape
Material with a shader Particles/Additive with a flame texture in Renderer (such as this one )
Skybox
→Texture type: cube map
Skybox sur l'île
Go to the asset store to import a (small) package of
skybox (ex: Cope! )
Select Window >>Rendering >Lightning Settings
Apply the skybox previously downloaded
Reflection map
Sphere with Reflection map
Component -> Rendering -> Reflection Probe**
Create new material with Metallic and Smoothness at 1
Prehistory of VR
Simulate the feeling of the presence of a virtual space .
Difficulties :
Simulate movement (walking)
Graphic Realism
Apparatus
→ Little chance
Ex: The Lawnmower Man
6 basic rules of immersion
Guide with light
Take advantage of the background and scale
Create a sound universe
Interact with the eye
Do not neglect the visual
Build on existing projects
Guide with light
The user has complete freedom to look 360°.
Light is very often used to attract the eye
ex: Clickable luminous aura
Take advantage of the background and scale
An overly large environment allows the user to be told that he or she is small
Increase the scale when using doors or ceilings
Create a sound universe
Sound and sound effects help to locate the user in the 3D universe
Use zones of influence
Allows to create an atmosphere
Ex: fire, water, animals,...
Interact with the eye
Do not use controllers, keyboard, mouse
Use of the manhole instead
ex: Turn on the light by looking at the switch
ex: An object becomes luminous when you look at it
Do not neglect the visual
The application must always be beautiful and fluid
The user must be interested in discovering and exploring the universe
Finding a compromise between fluidity and beauty
Build on existing projects
Many projects exist on Unity, Unreal Engine, Google VR, etc....
Free and Open Source
There are many 3D models and free scripts
Which interface for RV?
3D display
Augmented reality
MYO
Leap Motion
Devices
Goal: to improve the visualization of the scene
2 characteristics:
Best 3D printing
Better impression of immersion
Can be combined
Passive glasses
Can be very simple: red/blue
Not very precise
Cheap
No immersion
More complicated: polarized glasses
Accuracy depends on calibration
More expensive, with or without sensor
No immersion
Active glasses
More expensive, with or without sensor
Google Cardboard
Possible to tinker yourself but purchase of lenses necessary
Minimum configuration: 5" screen, full HD, 1.6 GHz 4-cores processor, accelerometer, gyroscope and Android or iOS.
2 to 20€
Oculus Rift/Go/Quest/Half Dome
HTC Vive
Next advances in VR:
Best resolution of the screen (like smartphone)
Improved focus:
Utilisation of eye-tracker
Variable screen position e
Integration with Unity
→ See updated instructions for use on Unity web page
Introduction
Accessible via the Asset Store
To build basic or advanced geometries directly in Unity
Design of complete UV textures
Possible to export models
GUI
Select Tools >ProBuilder >ProBuilder Window
Select New Shape
Select a shape and set it up
Use the edit bar to change the shape:
object, points, edge or surface
Displacement (→ extrusion ), rotation, scale
Can be combined with shift
The Toolbar is color-coded to help:
Orange for Tool Panels
Blue for Selection Tools
Green for Object Actions
Red for Geometry Actions (Vertex, Edge, Face)
Texturing
Open the material editor
Sliding/leaving materials
Apply the material (alt+nb)
or select faces
For UV texture editing, open UV Editor
Tools ->ProBuilder ->Export
Formats:
stl, obj, ply, Unity asset
Exercise
Building a plot of land with a castle
a hill to access it
crenelated walls
a door
towers
a patrol path
Tips
Subdivide the surfaces
Use the extrusion tool
Merge the faces
Select all the faces concerned by an extrusion to have a constant height
Physical Behaviours
Physical objects
Contact management
Rigid objects
Physics → Rigidbody
Defined by:
Position
Speed
Acceleration rate
Forces exerted
Which relationships?
RigidBody
Mass : object mass
Drag : resistance of the object to movement (such as air resistance for example)
Angular Drag : Resistance of the object for rotation
Use Gravity : Does the object undergo gravity or not
Use kinematic : If activated, the object is no longer subject to forces, collisions and others. It is only managed by scripts
Interpolate : Adapts motion calculations to framerate (can be expensive in computation time)
Collision Detection : Adapts collision calculations
Constraints : Allows to prevent movement or rotation on one or more dimensions
Practical
Create a Plan
Tilt the plane 10 degrees
Create a Sphere positioned above the plane
Position the camera to see the two objects
Add a RigidBody to the sphere
Change the parameters and observe the behaviour
Gestion de collision
Detect if there is a collision
By using geometry in space
If there is a collision, measure the distance
Response to the collision
By calculating a force to extract the object
Physics → XXXCollider with XXX that can be :
Boxes
Sphere
Capsule
Mesh
Collision detection
Example with a sphere:
Sphere with centre (xs , ys , zs ) and radius R
Point with position (xu , yu , zu )
Response force to the collision
Plan
Introduction
Augmented reality based on sensor
Augmented reality based on vision
Augmented reality with Unity
Introduction
Augmented reality: complete our perception of the real world by adding fictitious elements that are not naturally perceptible
Most often: Realistically embed 3Dvirtual objects in a image sequence
For purists: interactive, real time
At the confluence of various domains (computer graphics, physics, computer vision, optics, robotics, HMI, etc.)
In the laboratories for 20 years, growing success with the public
Perspective projection
Pinhole camera model :
plan P (grid of the drawing)
3D point M of the scene projected on m belonging to P
projection centre C (pinhole)
Perspective inversion
Find the point of view of the eyes in real time
Different methods
Sensors : robust, not very accurate
Image analysis : accurate, not very robust
Hybrid solutions : accurate and robust
Magnetic sensors
3 orthogonal coils emit a magnetic field picked up by 3 receptive coils
+ Good accuracy (0.2 mm, 0.1 deg)
- Sensitive to magnetic field disturbances
- Small environment (3m × 3m × 3m × 3m × 3m)
Acoustic sensors
3 microphones / speakers, spheres intersection
+ Good accuracy (≅ magnetic)
+ Not sensitive to metal objects
- Small environment (3m × 3m × 3m × 3m × 3m)
Optical sensors
Stereo cameras + targets + triangulation
+ Very accurate (0.1 mm)
- Limited volume
- Risk of target occultation
Inertial sensors
Fusion of data from "autonomous" sensors (gyroscopes, accelerometers, magnetometers)
+ Unrestricted volume (approx. outside ok)
- Rotations only
- Inaccurate
GPS
Satellite navigation system (disk intersection)
+ Outdoor environment
- Positions only - Positions only
- Very inaccurate (10-20 m)
Techniques based on algorithms
Several image-based techniques exist:
Use the Vanishing points (Sketchup...)
Use 3D-2D matches of points (3DS Max...)
Manually , by trial and error (all modeling software that accepts a background image)
Only the 2nd can be applied automatically in video sequences!
Vanishing points method
Find the projection matrix by leak point
One-point perspective projection
Frankfurt International Airport
Pietro Perugino in Sistine Chapel (1481–82)
Example in Sketchup
Use two vanishing points corresponding to horizontal lines of orthogonal directions
The method of invading perspectives
Find the projection matrix by knowledge of the points in the real world
System of equations
Camera inversion = calculation of R, t :
known focal length: 4 points are enough
focal length not known: 6 points
Use of 3D-2D correspondences
Ex : student projects
avec PoLAR
Automatic calculation by fiducial markers
Fiducial = detectable and identifiable
Consists of 4 main steps:
Extraction of points of interest in each image
Robust mapping of points of interest between adjacent images
Projective reconstruction of the paired points of interest
Metric construction (autocalibration)
Example : ARToolkit
Augmented reality with Unity
Solution in Unity: Vuforia
Plugin developed by Qualcomm for
Camera management
Environmental detection
Displaying projected 3D objects
Create an account onVuforia and log in
Create a license in the menu License Manager
Workflow for using Vuforia plugin
Create an account on the Vuforia website
Download and install the plugin
Define images to be interpreted as markers
Constitute the database for Unity (Target Manager )
Import this database into Unity
Les fichiers images correspondants aux marqueurs utilisés en classe se trouvent ici
Using the Target Manager
Go to the menu Target Manager
Add a Database
Select Device (the cloud is not free)
Define a name
Add a Target
Select a Single image
Fill in the fields
Download the corresponding Database
Quality of the marker
Advice for a good marker quality
Have a maximum of details
Have a high contrast
The image must be large enough
The image must be acquired in planar position
The image must be acquired by avoiding reflections
The marker must be easy to handle (e. g. playing cards )
Unity integration
If Vuforia is not installed, go to the plugin site , select Unity and install Unity with Vuforia
In an already created Unity project:
Check that Vuforia is enabled (Edit →Project settings →Player )
In the Project tab, put the license number in Resources →VuforiaConfiguration
Import your Database (Asset →Import package →Custom package... )
Exercise: Creating a scene with RA
Add a ImageTarget (GameObject →Vuforia →Image )
Replace camera with ARcamera (... →*ARcamera )
Select a marker from the Inspector
Add a sphere in ImageTarget of the Hierachy tab
→ Test the different images with more complicated 3D models
→ Test with several markers at the same time
Interaction between markers
In Vuforia configurations, allow tracking of 2 targets**
Example (to be reproduced) with two 3D objects:
Exercise
Create 3 textures
Create a script to attach to a 3D object
Set 2 material variables and drag and drop the materials (public Material mat
)
Retrieve the other 3D object with GameObject.Find("nom_objet3D");
Retrieve the positions of objects with objet3D.transform.position;
Make a material change according to the distance between objects with Vector3.Distance(position1, position2);
Texture animation
Apply a texture to a Material mat1 belonging to a GameObject :this .GetComponent<Renderer>().material = mat1;
Change its offset: rend.material.SetTextureOffset("_MainTex" , new Vector2(timer ,1 ));
With rend define by:public Renderer rend
rend = GetComponent<Renderer>()
timer is a variable to increment for each frame
Expected outcome
Scripts in Unity
Introduction
Scripts linked to GameObject
Debugging
User interface
Script writing
Changes the behaviour of objects
Written in C#
Important functions :
Start
: Called when a script is instantiated
Update
: Called to each frame
OnGUI
: Used to display a GI (score, health,...)
OnCollisionEnter
: Collision detection
OnTriggerEnter
: Collision with a trigger
Important classes
Mathematics
Vector3, Quaternion, Mathf, Ray,
...
Audio
Physique
Rigidbody, Collider, Physics,
...
GUI
Other
GameObject, Input, Application,
...
Script creation
Right click on the tab Project**
Select Create then C#Script
Rename the script with the desired name
Double-click on the script to open it
Un premier script
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
public class NewBehaviourScript : MonoBehaviour {
void Start ( ) {
}
void Update ( ) {
}
}
Code compilation
Is done each time you save and return to Unity: :
Allows Unity to validate the script
If we add an error, it will be reported back up:
void Start ()
{
score=500 ;
}
Execution of the code
Execution: script must be assigned to a GameObject
Do a slide/drop of the script to the gameObject
Can be seen in the Inspector panel of the Object Game
Organization
Use a specific folder for scripts
Avoid long and complex folder trees
Explicitly name folders
Separate scripts related to UI and features
Comment the code
Scripts linked to GameObject
Role of MonoBehavior
public class NewBehaviourScript : MonoBehaviour {... }
Linked to a GameObject
Allows access to many features:
Awake called to the very first instantiation
Start called to creation after Awake
Update called at each frame (ex: 60 times/sec)
onDestroy called for the destruction of the GameObject
Doc : here
Communication between entities
Before communicating with a script, it must be located
List of methods related to the search for a GameObject :
GameObject.Find(string name)
GameObject.FindGameObjectWithTag(string tag)
GameObject.FindGameObjectsWithTag(string tag)
Debugging the code
void Start
{
Debug.Log method");
}
try {
int result =100 /0 ; // Error
}
catch()System .Exception e{
Debug .LogException ("Impossible division" );
}
Debugging graphically
Draw a line between points:
Debug.DrawLine (startpoint, finishpoint, color);
Duration if not in Update
ex :
void Update ()
{
Debug.DrawLine(Vector3.zero,new Vector(10 , 0 ,0 ), Color.green);
}
Using MonoBehavior
The method OnGUI
is called at each frame
Two positioning solutions:
GUI** : Fixed resolution
GUI.Button(new Rect(0 ,0 ,80 ,20 ),"Click" )
GUILayout : Variable resolution
GUILayout .Button("Click" )
GUI Elements (1)
GUI.Label (new Rect(0 ,0 ,80 ,20 ), "Hello !" )
Toggle displays a checkbox
GUI.Label (new Rect(0 ,0 ,80 ,20 ), "Hello !" )
GUI Elements (2)
Toolbar , SelectionGrid control the creation of toolbars
HorizontalSlider, VerticalSlider Value modifier by selection
HorizontalScrollbar, VerticalScrollbar Similar to the previous ones with the addition of a control size
Exercises
A series of exercises is available here
Create a cube in an empty scene
Create the script RotationCube
with the following code : float vitesse=50.0F ;
void Update ()
{
transform.Rotate(Vector3.right*Time.deltaTime*vitesse);
}
Apply this script to the cube . Try it.
Add the function OnGUI()
and in the body of the function put :
GUI.Box(new Rect(Xmin,Ymin,dX,dY),"texte" )
Use the right settings to display the text
Add another line in the function OnGUI()
to have a horizontal slider that will vary the speed with : vitesse=GUI.HorizontalSlider(new Rect(Xmin,Ymin,dX ,dY ),variable,
nombre minimum, nombre maximum)
Use the right parameters to be able to vary the speed
→ Creation of a stopwatch in the upper right-hand corner
Subject
Create a scene with a door
Detect when a FPS is close
Gradually open a door
Combine the two
Preamble
Import the package Activity2.unitypackage
Make a scene with a door and its upright
Adding a floor and walls
Add textures contained in the "Environment " package (See Package Characters )
"FPS detection" script
Recover the position of the FPS
Display it in the console (Debug )
Indicate when the position is close
Help: Help: **
Use FindGameObjectWithTag
Use the attribute transform
Use the attribute position
Use if
Use Vector3.Distance(X,Y)
Door opening script
Duplicate the door and put it in the open position
Uncheck "Mesh Renderer " to avoid seeing it
Declare global variables:
A GameObject for the open door
Two Vector3 for the open and closed position of the door
A float for the opening/closing time
A float for the time when the animation is finished
A bool to know if the door is open or closed
Virtual and Augmented Reality with Unity
PF Villard