The Limb Simulator aims to reduce phantom limb pain for those who have lost a limb. Most phantom limb pain treatments are palliative in nature – they target individual factors of pain without addressing the underlying cause. By simulating the experience of owning and using an intact limb, Limb Simulator hopes to engage and remodel areas of the brain associated with the lost limb to impact both short and long term outcomes of phantom pain.

The Vision

Phantom limb pain is a growing problem as the number of people living with limb loss in the United States increases. Current treatment options are either inadequate or impractical. Limb Simulator envisions to become a home-based, self-administered gamified therapy system that leverages advances in virtual reality to create better pain outcomes. By incorporating a sense of ownership over the simulated limb using new and exciting techniques, Limb Simulator goes beyond simply visualizing an intact limb – it integrates into the user’s body image on a deeper level. Harnessing neuroplasticity in this way helps the brain remodel itself to a “normal” body image and thus disentangle the user from their painful phantom limb.

Current Work

Case studies are being conducted in the Seattle area to guide the development of Limb Simulator. The study team is preparing a fully powered clinical trial to test the effectiveness of the system. Efforts are in progress to improve therapy workflow, to implement new features, and to improve the core technology behind the therapy.

Gallery

https://drive.google.com/file/d/12Jf77bM5Dhy5CeNoIgZHJD84H-1JX1vX/view?usp=sharing

Introduction

We can induce the feeling that a virtual hand is a part of one’s body. One of the major hurdles for understanding this illusion is that it is a phenomenological, subjective feeling. The widely accepted ways of quantifying whether a person felt the illusion - questionnaire, proprioceptive drift, and galvanic skin response to a threat - each come with shortcomings and limitations. Problem 1 - We need a better way to measure when someone is feeling the illusion.

Surprisingly, there has not really been a demonstration that the illusion leads to different motor behavior. Though it may seem obvious that the illusion is compelling and worth inducing, contributing to “presence” (whatever that is supposed to mean,) there has never been an experiment showing a change in movement when the illusion is being felt. Problem 2 - We haven’t shown whether feeling the illusion actually changes the way one acts.

Media

Introduction

Most state-of-the-art prosthetic controllers define modes of operation corresponding to terrain types such as flat-ground walking or stair ascent. But humans don’t think in terms of discrete modes for locomotion. Instead, humans optimize and select complex unique movements based on past experiences of navigating similar conditions. However, given the range of possible scenarios, an all-encompassing controller has been hard to implement.

We are using computer vision, inertial sensors and deep learning to train a system to generate movement controls for a prosthesis based on the movements observed in real scenarios. Such an example-based approach requires a copious amount of data but it provides the capability to generate movement that might be difficult for us to explicitly model.

For the user of a coordinated movement prosthesis controller, the prosthetic limb responds appropriately in response to the rest of the body.

Methods:

Phase 1- Kinematics :

Whole body gait kinematics will be collected using wireless inertial sensors (Xsens Inc). The dataset will include unstructured movements and activities from multiple users. A Long Short-Term Memory (LSTM) network will be trained to estimate instantaneous ankle kinematics.

Phase 2 - Kinematics + Vision :

Scenarios such as obstacle avoidance and transition to stair ascent will need a glimpse of upcoming terrains. Whole body gait kinematics and raw image data from a head-mounted camera will be used to train a Convolutional Neural Network (CNN) + LSTM to estimate instantaneous kinematics, and also to predict upcoming ones.

Introduction

In object grasping, users of wrist-locked prostheses are often compelled to perform compensatory movements due to the lacking range of motion of a wrist locked prosthesis. This can cause overuse of other systems, like the shoulder or intact hand, and causing additional cognitive load. GazeToGrasp leverages deep learning and gaze-centered datasets gathered on human subjects to create predictive control strategies capable of predicting optimal grasps of objects.

Data Collection

We collect gaze-centered vision data from intact human subjects performing tasks of picking up objects in a virtual environment. A deep learning algorithm is trained on this data, learning a relationship between gaze of users and the optimal affordances of an object based on its orientation.

User Study

Implementing this algorithm into a virtual prosthesis, a user study is used to compare performance of this predictive control strategy against a wrist-locked prosthesis. Shoulder compensation and decision time are both reduced for the implementation of the predictive control.

Links

• Full Text
• Video

Introduction

We can design materials to mitigate the effects of external forces and moments on our body due to interaction with daily objects. This is done partly by using geometrical contours like the shapes of handles, and by using smart mechanisms and materials. However, active wearable devices such as powered prostheses, exoskeletons, and haptic interfaces grounded to the human body lack appropriate design of interaction surfaces. The resulting buildup of forces and moments over the interaction surface between the human and the machine leads to decreased comfort and functional fit, and is a contributor to disuse of active wearable devices. This challenge is multi-faceted, and we approach it by addressing 4 major themes:

Themes

Theme 1: Human Measurement We need to quantify the material behavior of the tissues of the body under loads applied at different points and directions.

Theme 2: Design
We need to design the ideal match between an engineered material and the human body to promote comfort and function.

Theme 3: Instantiation
We need to create tangible instances of the designed materials using commercial materials, and custom 3-D printed or cast structures.

Theme 4: User Testing
We need to get user feedback to validate the comfort of the candidate designs, and to quantify the behavior of the interaction between the human and machine system.

Methods

The following must be accomplished

• Indentation experiments for 20 subjects to quantify the tissue biomechanical characteristics
• A design tool to specify material characteristics for the physical human robot interface material
• Jig design to test the effect of the instantiated physical human robot interface material

Haplets are wireless, finger-worn LRAs. They were created to answer the question: “What is the minimum viable haptics that we can add to the fingers to augment hand tracking in AR?”. Our paper covers the hardware design, validation and a user study with Haplets used in conjunction with a pen. Our results suggest that users are more accurate when drawing with Haplets.

Cool facts about Haplets

• They are wireless but have incredibly low latency (1-4 ms).
• Using them with the Quest 2 requires only a dongle plugged into the USB port. No PC required!
• The user study was driven using Bricklayer.
• Visuals are equally important for haptics, so the virtual hands are fully physics-driven and behave semi-realistically.

Links

• 📄 Full Paper (Frontiers in VR)
• 🎥 Video

Overview

Robotic state estimation, wearable motion capture, egocentric camera data, optical hand tracking, wearable gaze tracking, robotic prosthetic limbs, and more! All of these applications generate complex time series data that are difficult to model using traditional techniques. Luckily, there are now outrageously powerful data science methods that learn to represent these high-dimensional and unpredictable datastreams.

Latent embeddings

One of our primary areas of focus is in creating low-dimensional embeddings of high-dimensional time series data, such as body tracking data. We primarily use unsupervised and self-supervised neural networks that are trained to concisely summarize the regularities in the data. One advantage of these methods is that the low-dimensional representations can aid in human understanding of the data. They also provide a low-dimensional space to plan and control in.

Data-driven robotic control

When the environment is uncertain, or when the dynamics of the robot are complex, it becomes difficult to use traidtional, physics-based models for control. We develop new methods for using real recordings of robotic motion and sensors to enable new controllers that leverage the power of deep learning representations and model-predictive optimal control. The applications range from tendon-driven biomechanical robot hands to soft sensor-actuator enabled robots to Model Predictive Policy Improvement using learned dynamics models for flying robots and beyond.

Introduction

Vision-based gesture and pose detecting systems combined with EMG muscle sensing present the opportunity to create extremely capable multimodal human-machine interaction systems.

EMG array armband

We we are trying to estimate hand kinematics and intent from a forearm-worn EMG array. For data collection, Jom Preechayasomboon built a custom 8-channel EMG armband from scratch based on the ADS1299. The electrodes are 3D printed from electrically conductive TPU. The nRF-based wireless communication, based off of technology developed for our haptics projects, enable low latency (1 ms) streaming to a USB dongle. We stream 1000 Hz data directly to a Meta Quest 2 so we can collect hand tracking data and EMG signals simultaneously, all locally on the headset. This is great for deployment for user studies without the hassle of hardware setup, just plug-n-play.

Pornthep also made this sick data visualizer for the collected EMG and kinematic data.

And lastly, a rather unrelated, proof-of-concept of EMG + Physics Hands 🤝

For more information follow Pornthep’s website

Improvement of EMG calibration

Current use of EMG is plagued by the spectre of calibration. A major factor of this is “normalization,” or removing the sensitivity of the signal to scale variation due to factors like differences in skin conductance. The current practice is to normalize the signals by the signal obtained during “maximal voluntary contraction” (MVC). We have demonstrated that MVC is highly senstive to pose, and demonstrate a means for correcting for this effect using the pose-sensing camera (Intel Perceptual Computing.)

Myometric Authentication

Gesture computing could support “movement passwords,” or special combinations of movements and poses used as a password. These would be vulnerable, however, to video recording and mimicry. If the visual modality were coupled with EMG sensing, there could exist a secret channel that augments the gesture password and secures it against mimicry.

See this slide deck for more details.

For more information please go to negshell.github.io!

Details

Targeted Reinnervation (TR) surgery restores sensation by moving the nerves that once led to the missing limb, reattaching them to a new muscle location. This novel surgical intervention allows patients to feel full sensations, emanating from the phantom limb, when the healed surgical site is touched. By working with TR patients, we aim to develop a new way for users of lower limb prostheses to sense what is happening at the bottom of their foot when walking down the stairs, allowing them to regain control and ownership over their body.

The proposed method of sensory feedback will be an Arduino-based haptic device that uses vibration to create a phantom feeling of pressure at the bottom of the missing foot. The qualities and location of the vibration depend on the position of the prosthetic foot in contact with the ground, communicated to the Arduino through an insole array of force-sensitive resistors.

Through the interaction of haptic feedback with native sensory receptors, we aim to better understand the neural principles of sensation and body ownership, helping users of prostheses blur the line between their organic and artificial limbs.

Specific Aims

Aim 1: Create a device that uses a sensorized insole to communicate vibrations to the user

Aim 2: Demonstrate the efficacy of the device in aiding prosthesis users in stair stair descent

Aim 3: Leverage findings of sensory mapping post-TR surgery in order to create and evaluate custom haptic bands

Key Outcomes

This work will explore applications of targeted reinnervation surgery, and inform further work on haptic interfaces with the skin through a greater understanding of the underlying neural principles of sensation and body ownership.

With these outcomes in mind, we see that the assistive technologies we create do more than replace ability: they shape the sense of self and bodily integrity. By including the users of these devices in the development process, we gain a holistic perspective on the intimate relationship between body and prosthesis.

Media

Disrupt Slides

Introduction

Perception is created through the rich interplay of multiple sensory modalities which historically have been difficult to quantify. It is also interwined with internal models of the body and expectations of its movement and the outside world. To make sense of sensation, it is tempting to isolate modalities and study them singly, but this is made difficult by the unexpected ways that the neural and mechanical systems interact. There has been a heroic effort to document and explan how multiple sensory modalities work together, sometimes with contradictory conclusions. Instead of obeserving different sensing modalities under normal circumstances, we intentionally induce conflict between two senses to isolate the contributions of each.

Sensitivity to Conflict between Visual Touch and Tactile Touch

One of the most consistent observations in sensory related research is that visual sensation appears to take perceptual preference over other senses in many contexts. Thus, in our research, we focus on introducing conflict between vision and touch. We hypothesize that in visuotactile perception, if vision is indeed dominant, the smallest noticeable intrapersonal spatial conflict between vision and taction will be different than from a purely tactile one. There have been fewer studies of conflict than general multisensory perception because creating precise errors between senses requires elaborate experimental apparatus. The recent advances in “virtual reality” (VR) provide an opportunity to isolate vision from other senses more easily and accurately than ever before.

Visuotactile Two-Point Discrimination Test (VT Test)

We developed a protocol analogous to the tactile two-point discrimination test that allows for the study of spatial conflict between visual touch and tactile touch. In the original test, subjects are asked to discern whether one or two simuli are present while the experimenter touches the subject with either one or two stimulation instruments. Thoroughout the experiment, the subject’s vision is blocked. In our protocol, however, only one stimulation point is tactile while the other is visual as depicted below.

We seek to use this information to restore sensory feedback in lower limb amputees who received the Targeted Muscle Reinnervation (TMR) procedure. Sensory feedback will be delivered to the user of a sensorized prosthetic limb, providing a “pass-through” for touch. By understanding the spatial resolution of the skin through the VT Test, we can better inform the required density of the tactor array. Understanding visuotactile conflict is also important in the design of haptic hardware for other applications such as VR haptic interfaces.

Soft robotic actuators are now being used in practical applications; however, they are often limited to open-loop control that relies on the inherent compliance of the actuator. Achieving human-like manipulation and grasping with soft robotic actuators requires at least some form of sensing, which often comes at the cost of complex fabrication and purposefully built sensor structures. In this paper, we utilize the actuating fluid itself as a sensing medium to achieve high-fidelity proprioception in a soft actuator. As our sensors are somewhat unstructured, their readings are difficult to interpret using linear models. We therefore present a proof of concept of a method for deriving the pose of the soft actuator using recurrent neural networks. We present the experimental setup and our learned state estimator to show that our method is viable for achieving proprioception and is also robust to common sensor failures.

Links

• 📄 Full Paper
• 🎥 Videos Figures

Introduction

The effects of TR surgery have been studied in the upper limb, but have remained largely unexplored in the lower limb. The characteristics of the stimulus we aim to explore include the threshold at which a sensation in the phantom limb is felt, the quality of the sensation elicited, the density of the distribution of sensation, and the location to which the stimulus maps to. The latter is of particular interest, as identifying which locations of the surgical site correspond to which locations on the phantom limb paves the way for developing prostheses that simulate sensations accurately. Additionally, this mapping can be used to study the patterns of neural regrowth post-surgery.

Aims

Aim 1: Creating a user interface to map perceived referred sensations
Aim 2: Experimentally determining individual sensation mappings
Aim 3: Exploring technologies for applying sensation mappings, such as groups of vibrotactors for feedback

Media