NeuRow

NeuRow: Online Rowing VR for Motor-Imagery Training

Motor-imagery, or motor-imagination (MI), is a mental rehearsal of a movement and is a unique property of the brain as a mental ability strongly related to the body, or ‘embodied’ cognition [1]. Motor-imagery, appears to share the same control mechanisms and neural substrates with actual movement (both in action execution and action observation), providing a unique opportunity to study neural control of movement in either healthy people or patients [2], [3].

The benefits of motor-imagery are also utilized as a technique in neuro-rehabilitation for people with neurological impairments [4]. More recently, motor-imagery offers an essential basis for the development of Brain–Computer Interfaces (BCIs) for physically disabled persons and stroke survivors [5]. BCIs are communication systems capable of establishing an alternative pathway between user’s brain activity and a computer system [6].

More recently, Virtual Reality (VR) feedback has been used in motor-imagery BCI training, offering a more compelling experience to the user through 3D environments [7]. The fusion of BCI and VR (BCI-VR) allows a wide range of experiences where participants can control various aspects of their environment -either in an explicit or implicit manner-, by using mental imagery alone [8]. This direct brain-to-VR communication can produce induced illusions mostly relying on the sensorimotor contingencies between perception and action [9].

For this, and based on previous findings [10], we have developed a novel prototype that makes use of multimodal feedback, in an immersive VR environment delivered through a state-of-the-art Head Mounted Display (HMD), integrated in a MI-BCI motor training task (left | right hand imagery in a multiplatform setup, NeuRow. NeuRow is Immersive VR Environment for Motor-Imagery training with the use of Brain-Computer Interfaces (BCIs). NeuRow is a rowing game with main target to hit as many flags as possible in a fixed time. Moreover, NeuRow is available for PC, Android OS and also with web browser support through the use of Reh@Panel, a midleware between interfaces and VR. Go To Official NeuRow Website Developed by: Athanasios Vourvopoulos

References:

  • [1] T. Hanakawa, “Organizing motor imageries,” Neurosci. Res.
  • [2] C. Neuper, R. Scherer, S. Wriessnegger, and G. Pfurtscheller, “Motor imagery and action observation: Modulation of sensorimotor brain rhythms during mental control of a brain–computer interface,” Clin. Neurophysiol., vol. 120, no. 2, pp. 239–247, Feb. 2009.
  • [3] T. Mulder, “Motor imagery and action observation: cognitive tools for rehabilitation,” J. Neural Transm., vol. 114, no. 10, pp. 1265–1278, Oct. 2007.
  • [4] R. Dickstein and J. E. Deutsch, “Motor Imagery in Physical Therapist Practice,” Phys. Ther., vol. 87, no. 7, pp. 942–953, Jul. 2007.
  • [5] B. H. Dobkin, “Brain-computer interface technology as a tool to augment plasticity and outcomes for neurological rehabilitation,” J. Physiol., vol. 579, no. Pt 3, pp. 637–642, Mar. 2007.
  • [6] J. R. Wolpaw, N. Birbaumer, D. J. McFarland, G. Pfurtscheller, and T. M. Vaughan, “Brain-computer interfaces for communication and control,” Clin. Neurophysiol. Off. J. Int. Fed. Clin. Neurophysiol., vol. 113, no. 6, pp. 767–791, Jun. 2002.
  • [7] F. Lotte, J. Faller, C. Guger, Y. Renard, G. Pfurtscheller, A. Lécuyer, and R. Leeb, “Combining BCI with Virtual Reality: Towards New Applications and Improved BCI,” 2013.
  • [8] D. Friedman, “Brain-Computer Interfacing and Virtual Reality,” in Handbook of Digital Games and Entertainment Technologies, R. Nakatsu, M. Rauterberg, and P. Ciancarini, Eds. Springer Singapore, 2015, pp. 1–22.
  • [9] M. Slater, “Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments,” Philos. Trans. R. Soc. Lond. B Biol. Sci., vol. 364, no. 1535, pp. 3549–3557, Dec. 2009.
  • [10] A. Vourvopoulos, J. E. M. Cardona, and S. B. i Badia, “Optimizing Motor Imagery Neurofeedback through the Use of Multimodal Immersive Virtual Reality and Motor Priming,” presented at the 2015 International Conference on Virtual Rehabilitation (ICVR), 2015.
NeuRow is Immersive VR Environment for Motor-Imagery training with the use of Brain-Computer Interfaces (BCIs). NeuRow is a rowing game with main target to hit as many flags as possible in a fixed time. Moreover, NeuRow is available for PC, Android OS and also with web browser support through the use of Reh@Panel, a midleware between interfaces and VR.

Requirements

If you are using this game in your work or research, please cite NeuRow by using the following reference : A Vourvopoulos, A Ferreira, S Bermúdez i Badia. (2016). NeuRow: An Immersive VR Environment for Motor-Imagery Training with the Use of Brain-Computer Interfaces and Vibrotactile Feedback. Presented at the PhyCS 2016 – 3rd International Conference on Physiological Computing Syst, Lisbon. (CITE) (DOWNLOAD)

Features

FeaturesDesktopAndroidWeb
Logging✓xx
VR✓ (Oculus, HTC Vive)✓ (Google Cardboard)x
Hand Tracking✓ (Leap Motion)xx
Networking✓✓x
Platform Independentxx✓
Vibrotactile Feedback✓ (Arduino)xx

Download Links

Desktop (download) The standalone version for PC, supports high quality graphics for an immersive VR experience, with the support of the Oculus Rift headset (for developer kits), the Leap Motion hand controller and vibrotactile feedback. Moreover, data logging is supported for boat trajectory, target location, score and time.

Mobile (download) (download from google play) The mobile version is build for Android OS devices, receiving data through the RehabNet protocol through the Reh@panel. For phones, the VR feature can be utilized for VR glasses (e.g. Google Cardboard) by applying lens correction for each eye, and uses the phone gyroscope for head tracking, offering an immersive experience similar to the Oculus DK1, DK2 HMDs.

Web (link here) The web version is using the Unity web player (compatible only through Internet Explorer, Firefox or Opera), without the networking component due to security restrictions. Instead, the web NeuRow is acquiring emulated keyboard buttons through the Reh@panel.
For setting up motor-imagery BCI experiments with OpenVibe, a short summary can be found from the OpenVibe tutorial page.
For enhancing the sense of movement in VR, a custom vibrotactile module has been developed with out-of-the-box components including an Arduino board. The vibrotactile module can function as a valuable feedback modality for promoting induced illusions  relying on the sensorimotor contingencies between perception and action, useful in motor-imagery training in BCI. This module is composed by three main components:
  1. Arduino board
  2. Vibrating motors
  3. Cylindrical tubes

1. Arduino board setup:

For connecting the Arduino board with the motors (see figure 1(a)), the following components are needed:
  • 1N4001 Diode
  • 0.1µF ceramic capacitor
  • 1KΩ Resistor
  • 2N2222 NPN Transistor
  • USB Connector

2. Vibrating tubes

– For installing the motors inside the tubes, a 3D printed case has been designed (STL model) in order to accommodate the motor before inserted inside the tube (see figure on the right). – Cylindrical tubes act as grasping objects for simulating the rows movement. For our setup, a pair of carton based tubed had been used with 12cm of length, 3cm diameter and 0.5cm of thickness (see figure 3).

3. Technical specifications

Vibrating Motor Specs:
  • Dimension: 10mm diameter, 2.7mm thick
  • Voltage: 2V – 5V
  • Current Draw:5V current draw: 100mA
    • 4V current draw: 80mA
    • 3V current draw: 60mA
    • 2V current draw: 40mA
  • 11000 RPM at 5V
  • Weight: 0.9 gram
Arduino Mega 2560
  • Microcontroller: ATmega2560
  • Operating Voltage: 5V
  • Input Voltage (recommended): 7-12V
  • Input Voltage (limit): 6-20V
  • Digital I/O Pins: 54 (of which 15 provide PWM output)
  • Analog Input Pins: 16
  • DC Current per I/O Pin: 20 mA
  • DC Current for 3.3V Pin: 50 mA
  • Flash Memory: 256 KB of which 8 KB used by bootloader
  • SRAM: 8 KB
  • EEPROM: 4 KB
  • Clock Speed: 16 MHz
  • Length: 101.52 mm
  • Width: 53.3 mm
  • Weight: 37 g
Finally, upload the following code to the Arduino board for controlling the motors through NeuRow:
String str;
int values[2][6];
int power = 0;
int hand = 0;
int finger = 0;
void setup()
{
Serial.begin(9600);
Serial.flush(); //clears any possible left over information
}
void loop()
{
if(Serial.available() > 0)
{
str = Serial.readStringUntil(';');
power = str.toInt(); str = Serial.readStringUntil(';'); hand = str.toInt(); str = Serial.readStringUntil(';');
finger = str.toInt();
values[hand][finger] = power;
}
for(int i = 0; i <2; i++)
{
for(int j = 0; j < 6; j++)
{
analogWrite(i*6+j+2, values[i][j]);
}
}
}
Go To Forum

Media