This guide will provide an overview of how to use HandEngine to connect to a hand and create an animation.


Requirements



1. Setting up the Glove Wired USB/Wireless Bluetooth Connection


  1. Connect your glove to the PC running Hand Engine using either a USB-MicroUSB wired connection or the StretchSense USB-Bluetooth Dongle.

    a. NOTE 1: If connecting via the USB-MicroUSB cable, you may need to do a one time configuration of
    the COM port settings in Windows Device Manager. See Setting up COM Port for Wired USB Connection (DOC-5002) dongle for further instructions.

    b. N
    OTE 2: Windows will automatically assign a COM port number to the glove and/or USB Bluetooth dongle when they are connected to a USB port. You can find which COM Port has been assigned to a particular glove/dongle by looking at the Ports (COM & LPT) category in Windows Device Manager. When you connect the glove/dongle, the associated COM port will appear in the Ports (COM & LPT) category, and unplugging the glove/dongle will make the associated COM port disappear from the Ports (COM & LPT) category.
  2. Power on the glove by pushing the button on top of the glove electronics once to enter Bluetooth mode. Once the glove is on (blinking blue) press the button twice to enter USB/wired mode. If you want to return back to Bluetooth mode press the button twice again. The glove LED will blink Blue when in Bluetooth mode, and green when in USB mode.

    a. When starting in Bluetooth mode, the glove LED is set to a long blue blink to signify it is searching for an available StretchSense USB-Bluetooth Dongle. If it has previously been paired with a USB-Bluetooth Dongle and it is in range of that dongle it will automatically connect and the glove LED will switch to a short blue blink and the LED on the connected dongle will go solid blue. If this is the first time using the glove with an unpaired dongle (a dongle is unpaired if when plugged into a USB port the dongle LED is solid red), move the glove within <1 ft (<30cm) of the dongle until the red light on the dongle turns first green to indicate pairing has occurred, and then blue to indicate data is streaming from the glove.


2. Getting Started with Hand Engine


  1. Launch HandEngine using the Windows Start Menu or Desktop shortcuts.



  2. Once launched, Hand Engine will open in a web browser tab for you to use.

    a. NOTE: StretchSense recommends running Hand Engine in a Google Chrome tab.

    b. NOTE: To avoid performance degradation, StretchSense recommends you do not have any other tabs open in the web browser window that is running Hand Engine.

    c. NOTE:  If at any point Hand Engine exhibits persistent unexpected behavior, you can save the state of the Hand Engine UI by clicking File→ Save Session to download a configuration *.session file, then you can reset Hand Engine by clicking the Refresh button on your browser tab which will set the Hand Engine UI back to its default state. You can then click File Load Session and navigate to your configuration *.session file to reload the previous state of the UI. This does not affect any calibration that you have done previously. If you have not restored the state of the Hand Engine UI.Image Placeholder
  3. To connect the glove to Hand Engine, expand the Glove 1 dropdown by clicking on it.

    a. NOTE: you can use the following process for the other Gloves to add more gloves as well.
    Image Placeholder
  4. Once expanded it will display the glove controls panel for Glove 1. 

    Image Placeholder

  5. Press Scan to populate the COM Port list and select the COM port of your USB dongle/Glove, then toggle the switch next to the COM PORT box from to On to open a connection.

    Image Placeholder

  6. Inspect the raw data plots in the graphical view at the bottom of StretchSense Device 1 to make sure all 15 (for MoCap Pro Original gloves)/16 (for MoCap Pro SuperSplay gloves) sensors are present and accounted for. Curling your gloved fingers will make the histograms grow taller, and vice versa.

    NOTE: MoCap Pro Original gloves do not have a wrist sensor so you should expect this histogram to be empty.
    Image Placeholder

3. Calibrating the Glove

  1. To calibrate a glove first you will need to create an Actor and Profile which is used to save and recall calibration profiles. Press the Create button next to the Actor field and fill in the details when prompted. 

    NOTE: Calibration profiles relate to a specific hand. StretchSense recommends including the hand (i.e. “L. Hand”/”R. Hand”) in the Profile Description.


    NOTE: If you have previously created an Actor and/or Profile on the same PC you are currently using, select them from the Actor and/or Profile drop down boxes.



    Image Placeholder


    NOTE: If you have previously created an Actor and/or Profile on another PC, you can transfer this data to you current PC by:



    • Export Actor and/or Profiles from the other PC by selecting File → Export Actor or File → Export Calibration to download a JSON file containing the Actor and/or Profile data. 

      - Exporting an Actor exports all calibration profiles associated with that actor

      - Exporting a calibration Exports that Actor but with only the current calibration profile


    • Copy the JSON file to a location your current PC can access.

    • Import the Actor and/or Profile by selecting File → Import Actor or File → Import Calibration


      Image Placeholder


  2. Once an Actor and Profile have been created you can start adding training poses to the calibration profile using the Pose Library Controls. You can select which poses you want to use depending on your particular application, we have provided several pose libraries to help you get started:

    1. Quick: A few simple poses for good general use.

    2. Performance: A more detailed pose set for more nuanced movement.

    3. Expert: An even more detailed pose set for expert hand data output.

    4. ASL: The complete American Sign Language set of poses.

    5. Game_Starter: A compilation of common hand poses in games.




      Image Placeholder

  3. For this example we will use the Quick Pose set. First, ensure that Preview in the Glove controls panel is set to Off. Next, expand the Quick menu in the Pose Library Controls and select the first item on the list Paddle, ensure Auto Advance Pose is On as this will automatically progress to the next pose on the list once the previous pose is captured.

    • NOTE: If you wish to calibrate multiple gloves to the same pose at the same time then toggle Group Calibration to On. When Group Calibration is On, every glove that is connected to Hand Engine and Hand Engine is receiving data will be trained to the selected pose at the same time. Whether you are doing a pair of gloves on a single actor, or calibrating multiple actors simultaneously, data is captured from every glove at the same time so all of the hands/gloves/actors must be in the correct pose at the same time during capturing.

    • NOTE: Even if Auto Advance Pose is On, you can still manually select the next pose to be trained from the Pose Library Controls using your mouse, Auto Advance Pose will not interfere and simply jump to the next pose afterwards.

    • NOTE: If you’ve already calibrated before, make sure the Preview button is turned Off before you try to add more poses from the library to the profile. 



      Image Placeholder



  4. With your connected glove, get the actor to match the pose in the Viewport.

    IMPORTANT! For best results, have the actor initially copy the reference pose with their hand with the wrist straight. It is very important to make sure all of the joints on the actor’s hand match the reference hand pose from the pose library, if the actor cannot achieve the pose, it is better that they do not train to that pose than to train it incorrectly. Furthermore it is highly beneficial to ensure that if a particular joint is in the same position in more than one pose, that for each of these poses the corresponding joint on the actor’s hand is also in the same position. This provides higher quality data to the hand solver that results in optimal calibration performance.  Once the actor is set in the hand pose, ask them to hold the hand pose steady and hold their forearm steady while gently rotating at their wrist (e.g. like the wrist would move when using a skipping rope), and to keep rotating their wrist in this circular motion until the capture is complete. Start the capture process by clicking the Capture button. The capture process takes <2 seconds for each captured pose.


     Image Placeholder



  5. Once the capture is completed, the viewport will automatically progress to the next pose (for example in the case of the Quick library the second pose is the Thumbs Up pose). Repeat Step 4 above using this new pose. Once all poses in the library have been captured the view port will stop updating, you will see the poses saved in the Captured Poses tree in the Glove 1 pane.

    • NOTE: You can delete individual poses from the Captured Poses list by right clicking on the pose and clicking Delete.

    • NOTE: You can delete all of the poses from the Captured Poses list by clicking the RESET button underneath. This empties the calibration profile but does not delete the calibration profile.



      Image Placeholder



  6. Next you can select how to visualise the calibrated glove: 

    • Blend Mode

      - A continuous movement that interpolates between poses 

      - Good for realistic hand movement


    • Pose Detection Mode

      - Snaps between the captured poses selecting the one most suitable

      - Good for keyframing as the poses captured are the only ones that are output


    • Hybrid Mode

      - Lifelike movements afforded by blend mode as well as being able to snap to key poses when handling props for example.

      - Check out our guide on how to setup hybrid mode.


    • Express Mode

      - Reinitialise the Express Mode output form the EXPRESS POSE in Captured poses.


      - This does NOT trigger a recalibration of Express Mode.


  7. To preview the hand animation from your calibrated glove, toggle the Preview button to On. The hand in the Hand Engine Viewport will now show the live output of the MoCap Pro glove hand capture in real time.

    NOTE: If you are using Hand Engine version 1.1.0 or lower and you would like to add more poses from the pose library to your profile, you need to turn the Preview button Off.
    Image Placeholder


  8. If you intend to stream hand data to your animation software, set Streaming to On. The guides on how to configure your animation software to receive this data stream using StretchSense plugins are provided here: https://kb.stretchsense.com/streaming-to-other-software.

    NOTE: Each glove requires a unique TCP port to stream data on. By default TCP port 9000 - 9011 are used for Glove 1 - Glove 12 respectively for Hand Engine Studio, or TCP port 9000-9003 for Glove 1 - Glove 4 respectively for Hand Engine Animator..



  9. Pose Detection Mode uses the same calibration data as Blend Mode. To enable Pose Detection Mode simply press the Use Pose Detection button. In this mode, you are able to specify a Debounce value measured in frames and transition animations using Animate Transition. This Debounce value is the number of consecutive frames that a pose output must be stable before the pose will be applied to the hand asset. The Animate Transition controls enable you to specify the type of animation (from left to right: Smooth in and out, linear, Explode out, Explode in) and the number of frames the animation
    takes.

    Image Placeholder


  10. Repeat the steps above for each glove you are using.

4. Configuring Timecode in Hand Engine


From the Master Timecode section, select the timecode source you wish to use for your glove data from the System TC dropdown box. The options available are:


    • Local System Time (default): use the system clock of the PC running Hand Engine as the timecode source. Hand Engine converts the time from the system clock of your PC into a 30fps timecode value.

    • Glove Clock: use the Glove time from each individual glove as the Hand Engine timecode source when recording data. The glove sends time measured in milliseconds since the glove connected to the PC running Hand Engine. Hand Engine converts the time to a 30fps timecode value.

      NOTE: if you have a properly configured UltraSync One module (see DOC-5008 - StretchSense UltraSync One Module Configuration) physically connected to your glove, the glove will always default to output Glove Timecode (see below) instead of time in milliseconds. 

    • Glove Timecode: Use this option if you have an UltraSync One connected to each glove or have used an UltraSync One module to jam sync each glove.

    • Vicon - Netwok: Use this option to setup the Vicon timecode in Hand Engine (see Vicon DataStream Integration - Timecode (DOC-5026))

    • StretchSense Circuit: Connect an UltraSync One module to a spare glove/circuit and use that single spare glove as the timecode source for all glove data. Requires selecting the COM port associated with the spare glove/circuit.
Image Placeholder



5. Setting the Local Recording Directory and Timer Based Calibration

  1. Changing the directory Hand Engine will save local raw data files to: To change the root directory where files are saved:

    • First create that directory in Windows File Explorer, then copy the full path name of that directory.

    • In Hand Engine, click Settings → Preferences, then paste the path name to the Recording Location text box.

      NOTE: It is important to ensure a trailing backslash (“\”) character is at the end of the path name.
      Image Placeholder

  2. Enabling Automatic Timer Pose: You can automatically run through the poses in a pose library without requiring you to click Capture for each pose by using the Profile Calibration Timer. Set the Profile Calibration Timer option to On, and specify Time between capture in seconds to hold each pose before capture begins.

    • NOTE: StretchSense recommends starting with an 8 second delay to begin with and reducing that time as appropriate.

    • When Capture is clicked, the Capture button changes to Stop (Auto On) and: 

      1. A timer pops up on screen counting down from the time specified in Time between capture and the actor must adopt the displayed pose. 

      2. When the timer reaches 3 seconds to go, the actor will be asked to start rotating their wrist(s).

      3. At 0 seconds the capture process begins. The capture process takes ~1 second to complete. 

      4. Once the capture is complete, if Profile Calibration Timer is On, the viewport will be updated with the next pose in the pose library and the timer will restart.

      5. Steps 2 and 3 repeats until the last pose in the pose library is captured

        NOTE: The Auto Capture process can be stopped by clicking Stop (Auto On)

  3. Browse previous recordings: Select Settings → Browse Recordings to open a Windows File Explorer window at the directory where Hand Engine will save local raw data files to.

    Image Placeholder

6. Saving and Restoring the State of Hand Engine

Once you have setup each glove and setup Hand Engine in the configuration you desire, you can save the state of Hand Engine by clicking File → Save Session to download a *.session file of the state of Hand Engine. To restore a previous state of Hand Engine, click File → Load Session and navigate to the previously saved *.session file. 

Image Placeholder


7. Recording MoCap Pro Glove Data using Hand Engine

This process assumes you have already connected each glove you are using to Hand Engine, created a calibration profile with the desired training poses for each glove, have selected Blend or Pose Detection mode, and have configured your Hand Engine timecode source (if applicable).

  1. In order to record raw capacitance data, joint angle data and timecode data first define a Label for the recording, then toggle the Record Switch to Rec. To Stop the recording toggle the Record Switch to ‘Stop’. The will be a red pulsing light to indicate recording is still going.

    Image Placeholder


  • NOTE: A subfolder is created for each take in C:\Users\<user_id>\Documents\StretchSense Hand  Engine with the naming convention <take name>_<date>_<time>

  • NOTE: Within this subfolder, three files are created for each glove connected to Hand Engine.;

    1x Comma Separated Variable (CSV) text file with the naming convention
    Raw_GLOVE-<x>_<actor name>_<profile name>_Hand_<left/right>_<take name>_<date>_<time>.csv containing (from left to right):

    - Raw capacitance data (16 columns)

    - Raw IMU data (if applicable) (10 columns)

    - Glove time (1 column)

    - Hand Engine time (1 column)

    - Solved joint angles (60 columns)

    1x JSON file with the naming convention
    Cal_GLOVE-<x>_<actor name>_<profile name>_Hand_<left/right>_<take name>_<date>_<time>.json containing a snapshot of the calibration profile used at the time the data was recorded.

    1x FBX animated file.

8. Playing Back a Raw Data File Recorded in Hand Engine

  1. Select File mode, then click Browse and navigate to a raw data CSV file previously recorded in Hand Engine. 

    Image Placeholder


  2. Select the Actor and Profile you wish to use to interpret the raw data file, and then click Train Blend or Train Pose Detection depending on your needs. 

  3. Click Play to start the playback. Click Pause to pause the playback at the current frame and click Play again to resume from that point. Click the Stop button to reset the animation back to the start of the animation file.

    • NOTE: You can reprocess the raw data file using any Actor/Profile and any mode (Blend or Pose Detection). You do not need to use the same that was used for the original recording.

    • NOTE: If you wish to play several recorded raw data files simultaneously, repeat steps 1 and 2 in each of Glove 1-12 in Hand Engine as required, then click the Play button in Global File Playback. This option is useful for triggering the streaming of multiple raw data files to your animation software at the same time and you can set the playback speed.



      Image Placeholder




Disclaimer, Intellectual Property and Licensing



Revision Summary





Revision

Date

Notes

8

2021.05.10

Updated screenshots and extra features for Hand Engine 1.1.1

7

2021.03.16

Updated for Hand Engine 1.0.1

6

2020.12.07

Updated license information and Hand Engine screenshots

5

2020.10.02

Updated Hand Engine screenshots

4

2020.09.21

Updated for Beta3.5

3

2020.08.21

Updated for Beta3

2

2020.07.21

Updated related docs, descriptions, record function and where to find recorded files, note added that informs that user needs to turn of Preview off to add more poses, added disclaimer text

1

2020.07.10

Initial version