LATEST VERSION 1.2 (17.02.2014)

Overview

openni2 (1)

Human body posture detection is an important problem that was recently tackled using various approaches. The most common ones are based either on depth map generation or on the human body parts classification based on a camera image. Dynamic boost of the entertainment technologies related with body posture recognition resulted in the availability of cheap and reliable 3D sensors such as: Microsoft Kinect [http://www.microsoft.com/en-us/kinectforwindows/], Asus Xtion Pro Live [http://www.asus.com/Multimedia/Motion Sensor/Xtion_ PRO_LIVE/], SoftKinetic DepthSense 311 [http://www.softkinetic.com/]. Most of these solutions are based on the structured light method while some others use an alternative technique base on time-of-flight. Currently, Kinect is the most often used sensor in interactive robotic research projects. 

The first generation of Kinect is a low cost device for 3D measurement that provides both 2D color image and a depth map. RGB camera provides VGA (640x480px) resolution while depth sensor's resolution is limited to 300x200px. However, depth image is interpolated inside this device to VGA size. The depth sensor provides accurate data on objects that are 0,4 – 6,5m away. Moreover, Kinect is equipped with a 4-microphone array, 3-axis accelerometer and a motor to control the tilt angle of the sensor head. Communication with the sensor is based on a popular USB interface. The manufacturer of the depth mapping technology used in Kinect is PrimeSense [http://www.primesense.com]). The Kinect for Windows software development kit (SDK) enables developers to use C++, C#, or Visual Basic to create applications that support gesture and voice recognition by using the Kinect for Windows sensor and a computer or an embedded device.

openni2 (2)

OpenNI is an open source API that is publicly available [http://structure.io/openni] (old unofficial copy of OpenNI can also be found here [http://openni.ru]). Currently, OpenNI supports Kinect, Xtion and Carmine sensors. Moreover, the main partner of the OpenNI organization (PrimeSense) provides NiTE - the Natural Interaction Middleware. It allows to perceive the world in 3D, comprehend, translate and respond to human movements without any wearable equipment or controls. NiTE enables human body detection, skeleton extraction and simple gesture recognition. A person can be detected and tracked and the wire-frame of his/her skeleton can be marked on the Kinect image.

OpenNI 2.x is a new version of the well-known library – biggest advantages of the new SDK are:

  • Support for the latest generation of 3D Sensors
  • Easier for development: Clean and improved design and API
  • Simpler distribution: private copy of OpenNI and NiTE for each application
  • More flexible: depth units control, exposure control, etc.
  • More open: Middleware API are separate for each library
  • Better and simplified multi-sensor support
  • Allows event-driven programming
  • Better backwards compatibility
  • Large offer of third party Middleware Libraries: Body Tracking, 3D Reconstruction, Object Recognition, Analytics, and many more.

openni2 (3)

This module implements all of the important functions of OpenNI2 and NiTE2, i.e. color and depth reading, recording streams as .ONI files, human body detection with unlimited number of users  (extraction of position and orientation of 15 joints in human body), simple gesture recognition (three gestures: click, wave, hand raise) and pose detection (PSI pose and crossed-hands). The number of functions provided by the module, can be easily extended thanks to many available middleware libraries.  

 

openni2 (4)

More information about OpenNI and NiTE can be found here:

1. OpenNI main page http://structure.io/openni

2. OpenNI unofficial page http://openni.ru/

3. NiTE 2 API Programmer Tutorial Guide With C++ link[.pdf] 

4. NiTE 2 API non-official http://openni.ru/files/nite/index.html

 

Hardware requirements

  • Kinect XBOX 360 or Kinect for Windows (recommended),
  • 32 bit (x86) or 64 bit (x64) processor,
  • Dual-core 2.66GHz or faster processor,
  • Dedicated USB 2.0 bus,
  • 2 GB RAM.

Software requirements

If only running the module:
  • Windows 7, Windows 8, Windows Embedded Standard 7, or Windows Embedded POSReady 7,
  • Microsoft Kinect SDK (tested with 1.8) - needed for kinect driver.
In order to compile the module, additional libraries are required:
  • OpenNI2 (tested with 2.2.0.33),
  • NiTE2 (tested with 2.2.0.11),
  • OpenCV, best used with Intel® Threading Building Blocks (Intel® TBB),

UKinectOpenNI2 was compiled with all the shared libraries (all included in UKinectOpenNI2.zip package). Copy them to the uobjects folder or set path in system environment variable (PATH).

  • OpenNI2.dll (x86 ver.)
  • NiTE2.dll (x86 ver.)
  • opencv_core231.dll
  • opencv_imgproc231.dll
  • tbb.dll
Additional NiTE files:
  • NiTE\s.dat (file has to be copied from \Program Files(x86)\PrimeSense\NiTE2\Redist\NiTE2 to urbi_engine_folder\NiTE2)

Module functions

Main section

UKinectOpenNI2.Open (kinect_number, color, depth, skeleton, pose, hand) - open the sensor connection with device number (0,1,2,...) and initialize boolean flags:
kinect_number – number of Kinect device, -1 means use any available device
color – use color stream from Kinect
depth – use depth stream from Kinect
skeleton – use skeleton tracking with NITE library
pose – use pose detection with NITE library (pose = true is the same as skeleton = pose = true)
hand – use hand gesture detection with NITE library,
Enabling hand gesture detection without skeleton detection, allows to detect gestures but not the user who performed them.
UKinectOpenNI2.OpenFile (file_name, color, depth, skeleton, pose, hand) - just as described above, but opens a recorded .ONI file.  
Skeleton, pose and hand gestures detection from .ONI files is currently not supported by OpenNI (in version 2.2.0.33).
UKinect.Close() - close connection and release device,
UKinect.Poll(wait) - poll all initialized video streams,
true - wait for new data from video streams,
false - check for new data, always returns immediately; it is recommended to use sleep (10-20ms) function in the main urbiscript loop,
UKinectOpenNI2.isOpen (read only) – information if connection with a device or .ONI file is opened.

Other important functions 

UKinectOpenNI2.devID (read only) – ID of the opened device,
UKinectOpenNI2.registration – enables shifting depth image to cover color image (default: true),
UKinectOpenNI2.synchronization – if true, all data will be refreshed (new frame read) at the same time (default: true),
UKinectOpenNI2.convertToMeters – if true, all distances will be in meters, otherwise in millimeters (default: true),
UKinectOpenNI2.fps (read only) - get computed fps performance,
UKinectOpenNI2.time (read only) - get time performance.

Color camera section

UKinectOpenNI2.colorEnabled (read only) - true if color camera has been started,
UKinectOpenNI2.colorImage (read only) - get camera image,
UKinectOpenNI2.colorResolution - set color camera resolution (default: 1):
0 - 1024x960@12fps              
1 - 640x480@30fps,
2 - 640x480@30fps in YUV (mode is not supported),
UKinectOpenNI2.colorWidth (read only) - image width,
UKinectOpenNI2.colorHeight (read only) - image height,
UKinectOpenNI2.colorAutoExposure - set auto exposure (default: 1), supported only in Kinect for Windows:
0 - OFF,
1 - ON,
UKinectOpenNI2.colorAutoWhiteBalance - set auto white balance (default: 1), supported only in Kinect for Windows:
0 - OFF,
1 - ON,
UKinectOpenNI2.colorExposureTime - set camera exposure time (not supported for Kinect sensor),
UKinectOpenNI2.colorGain - set camera gain (not supported for Kinect sensor),
 
UKinectOpenNI2.colorFps (read only) - get computed fps performance of color stream.

Depth camera section

UKinectOpenNI2.depthEnabled (read only) - true if depth camera has been started,
UKinectOpenNI2.depthImage (read only) - get depth image (in RGB space),
UKinectOpenNI2.depthResolution- set depth camera resolution (default: 0):
0 – 640x480@30fps,
1 - 320x280@30fps,
2 - 80x60@30fps,
UKinectOpenNI2.depthWidth (read only) - image width,
UKinectOpenNI2.depthHeight (read only) - image height,
UKinectOpenNI2.depthFps (read only) - get computed fps performance of depth stream.

Skeleton stream section

UKinectOpenNI2.skeletonEnabled (read only) - true if skeleton detection has been started,
UKinectOpenNI2.skeletonImage (read only) - get image with the user's skeleton drawn in,
UKinectOpenNI2.skeletonOnImage - draw skeleton on color image or black background (default: 1):
0 - draw skeleton on a black background,
1 - draw skeleton on a copy of color image,
UKinectOpenNI2.skeletonChooserMode - set skeleton chooser mode (default: 0):
0 – track all skeletons,
1 - track the closest skeleton,
2 - track two closest skeletons,
3 - track one skeleton and keep it,
4 - track two skeletons and keep them,
5 - track the most active skeleton,
6 - track two most active skeletons,
7 – track two newest users,
[ID0, ID1,...] = UKinectOpenNI2.skeletonIDs (read only) - get a list of detected IDs,
[ID0, ID1,….] = UKinectOpenNI2.skeletonTrackedIDs (read only) - get a list of tracked skeletons IDs,
[x, y, z] = UKinectOpenNI2.skeletonPosition(ID) – get center of mass of skeleton with the given ID in Kinect coordinates,
[x, y, depth] = UKinectOpenNI2.skeletonPositionOnImage(ID) - get center of mass of skeleton with the given ID on color image,
[x, y, z] = UKinectOpenNI2.skeletonJointPosition(ID, joint) - get joint position in Kinect coordinates,
[x, y, depth] = UKinectOpenNI2.skeletonJointPositionOnImage(ID, joint) - get joint position on color image,
confidence = UKinectOpenNI2.skeletonJointPositionConfidence(ID, joint) – get joint position confidence (in scale from 0 to 1),
[x, y, z, w] = UKinectOpenNI2.skeletonJointOrientation(ID, joint) – get joint orientation in quaternions,
confidence = UKinectOpenNI2.skeletonJointOrientationConfidence(ID, joint) - get joint orientation confidence (in scale from 0 to 1),
UKinectOpenNI2.skeletonCalcActivity – if true, activity of all tracked user will be calculated,
UKinectOpenNI2.skeletonMostActive (read only) – ID of most active user, if no user is currently detected or skeletonCalcActivity if false, ID will be -1,
UKinectOpenNI2.skeletonJointPosThreshold – confidence threshold for detecting joint position (from 0 to 1, default: 0.3),
UKinectOpenNI2.skeletonJointOrientThreshold – confidence threshold for detecting joint orientation (from 0 to 1, default: 0.3),
UKinectOpenNI2.skeletonFilter – smoothing factor for calculating joint positions (from 0 to 1, default: 0.3),
UKinectOpenNI2.skeletonFps (read only) - get computed fps performance of detecting skeletons.

Pose section

UKinectOpenNI2.poseEnabled (read only) - true if pose detection has been started,
[pose, phase] = UKinectOpenNI2.poseType(ID) – pose of given ID:
pose: 0 – no pose detected, 1 – PSI pose detected, 2 – crossed hands pose detected,
phase: 0 – pose has just been entered, 1 – pose is held, 2 – pose has just been left.

Hand section

UKinectOpenNI2.handEnabled (read only) - true if hand gesture detection has been started,
UKinectOpenNI2.handImage (read only) - get image with drawn hand gestures,
UKinectOpenNI2.handOnImage - draw hand gestures on color image or black background (default: 1):
0 - draw hand gestures on a black background,
1 - draw hand gestures on a copy of color image,
UKinectOpenNI2.handFilter– smoothing factor for calculating hand positions (from 0 to 1, default: 0.3),
UKinectOpenNI2.handWaveEnable – enable detecting wave gesture (default: true),
UKinectOpenNI2.handClickEnable – enable detecting click gesture (default: true),
UKinectOpenNI2.handRaiseEnable – enable detecting hand raising gesture (default: true),
UKinectOpenNI2.handDetectedGesturesIDs (read only) – vector of user ID for whom hand gestures were detected,
UKinectOpenNI2.handGesturesVector (read only) – vector of vectors [x, y, z, type, phase, ID, hand] describing gestures:
x, y, z – position of the hand performing gesture in Kinect coordinates,
type – type of gesture: 0 – wave, 1 – click, 2 – hand raising,
phase – phase of gesture: 0 – gesture in progress, 1 – gesture completed,
ID – ID of the user performing gesture, -1 if no user was associated with the gesture,
hand – hand performing gesture: 0 – left, 1 – right, -1 if hand (and user) was not determined,
UKinectOpenNI2.handFps (read only) - get computed fps performance of detecting hand gestures,
UKinectOpenNI2.handMaxDist – maximum distance between hand gesture position and user hand position allowing to associate gesture with user,
[x, y, z, type, phase, ID, hand] = UKinectOpenNI2.handGestures(ID, type, hand) – get gesture vector (described for UKinectOpenNI2.handGesturesVector) of given user ID, type of gesture and hand, value of -1 means any user/type/hand,
[x, y, z, type, phase, ID, hand] = UKinectOpenNI2.handGesturesOnImage(ID, type, hand) – get gesture vector with x and y coordinates on image.

Recording section

UKinectOpenNI2.StartRecording(color, depth, fileName, compression) – starts recording of streams in .ONI file:
color – if true records color stream,
depth – if true records depth stream,
fileName – file name of .ONI file,
compression – if true, records compressed streams,
UKinectOpenNI2.PauseRecording() – pause the recording,
UKinectOpenNI2.ResumeRecording() – resume the recording,
UKinectOpenNI2.StopRecording() – stop and save the recording.
 

jointNumbers

        joint   name
       --------------------------
	0	JOINT_HEAD
	1	JOINT_NECK
	2	JOINT_LEFT_SHOULDER
	3	JOINT_RIGHT_SHOULDER
	4	JOINT_LEFT_ELBOW
	5	JOINT_RIGHT_ELBOW
	6	JOINT_LEFT_HAND
	7	JOINT_RIGHT_HAND
	8	JOINT_TORSO
	9	JOINT_LEFT_HIP
	10	JOINT_RIGHT_HIP
	11	JOINT_LEFT_KNEE
	12	JOINT_RIGHT_KNEE
	13	JOINT_LEFT_FOOT
	14	JOINT_RIGHT_FOOT

Urbiscript examples

Example 1

loadModule("UKinectOpenNI2");
var Global.kinect = UKinectOpenNI2.new();
kinect.Open(0, true, true, true, true, true);  //open device number 0 with color, depth, skeleton, 
//pose and hand stream enabled
kinect.skeletonChooserMode = 2;  //track two closest skeletons
kinect.StartRecording(true, true, "new.oni", false);  //start recording color and depth streams without any compression 
tag: loop
{
        kinect.PoolVideo(true);
},      
//or  
tag: loop {
  kinect.PollVideo(false);
  sleep(10ms);
}, 
sleep(10);   
kinect.StopRecording();  //finish recording after 10s
 

Example 2

loadModule("UKinectOpenNI2");
var Global.kinect = UKinectOpenNI2.new();
kinect.OpenFile("new.oni", true, true, false, false, false);	//open connection with .oni file “new.oni” with color 
//and depth streams enabled (NITE functionality is currently not supported for .oni files)
tag: loop
{
	kinect.PollVideo(true);      
},

Example 3

loadModule("UKinectOpenNI2");
var Global.kinect = UKinectOpenNI2.new();
kinect.Open(0, true, true, true, true, true);
kinect.skeletonChooserMode = 0;  //track all users
kinect.skeletonCalcActivity = true;  //enable calculating user activity
tag: loop
{  
  kinect.PollVideo(true);
  if (kinect.skeletonTrackedIDs.size() > 0)  //there are some tracked users
  {
    echo ("Most acrive user is: " + kinect.skeletonMostActive );    //check most active user
    if (kinect.skeletonJointPosition(kinect.skeletonMostActive,8) != [])    //we can read torso position of most active user – 
    //his skeleton was detected
    {
      echo("position of his right hand: " + kinect.skeletonJointPosition(kinect.skeletonMostActive,7)); 
      echo("position of his left hand: " + kinect.skeletonJointPosition(kinect.skeletonMostActive,6));
      //check position of his joints
    }
    else
    {
      echo("Skeleton still not detected");
    };
  }
  else
  {
    echo ("No user detected");
  };
},

Example 4

loadModule("UKinectOpenNI2");
var Global.kinect = UKinectOpenNI2.new();
kinect.Open(0, true, true, true, true, true);
kinect.skeletonChooserMode = 0;
kinect.skeletonCalcActivity = true;
tag: loop
{
  kinect.PollVideo(true);
  if (kinect.skeletonTrackedIDs.size() > 0)
  {
    var pose = kinect.poseType(kinect.skeletonMostActive);  //read the pose of the user
    if (pose[0]== 1)  //if pose type is PSI
    {
      echo("User " + kinect.skeletonMostActive + "is in PSI pose");
      if (pose[1] == 0)  //if pose has just been entered
        {echo ("Pose has just been entered");}
      else if (pose[1] == 1)    //if pose if held
        {echo ("Pose is currently held");}
      else   //pose[1] = 2 – pose has just been left
        {echo ("Pose has just been left");}
    }    
  }
},

Example 5

loadModule("UKinectOpenNI2");
var Global.kinect = UKinectOpenNI2.new();
kinect.Open(0, true, true, true, true, true);
kinect.handRaiseEnable = false;  //disabling hand raise detection – detector gives many false positives 
kinect.skeletonChooserMode = 0;
kinect.skeletonCalcActivity = true;
tag: loop
{
  kinect.PollVideo(true);
  if (kinect.handGesturesVector.size() > 0)  //check if any gestures were detected
  {
    var gestures = kinect.handGestures(-1,0,-1);  //find all clicking gestures, without considering user who performed them and 
    //hand performing the gesture
    if (gestures.size() > 0)  //check if some conditions where met
    {
      for(var i = 0; i< gestures.size();i++)
      {
        echo("Waving detected for user nr: " + gestures[i][5]);  //read user ID
        if(gestures[i][6] == 0)  //check hand performing the gestures
        {echo("Gesture performed by left hand");}
        else if (gestures[i][6] == 1)
        {echo("Gesture performed by right hand");}
      };
    }
    else
    {
      echo("No gestures detected");
    };
  };
},

Download

UObject module LINK

OpenNI2 LINK

NiTE2 LINK

Microsoft Kinect SDK 1.8 LINK

 

 

 

 

 

EMYS and FLASH are Open Source and distributed according to the GPL v2.0 © Rev. 0.9.1, 15.05.2017

FLASH Documentation