From Box Files to the British Science Festival: Part Two

Here is Part 2 of the series on how and why the MIRAGE was developed

If you remember, I was trying to make adaptation to visual shifts (like when wearing prism goggles) more flexible. To do this, I hacked into some motion tracking software (this tracked a sensor on the hand as it moved through space) so that I could read and change the x, y and z coordinates of the cursor on a moment-by-moment basis. This created a system whereby the cursor could either track the location of a subject’s hand, or could be manipulated to move independently of it. Once I could do this, I could show a cursor (a big black dot) to the subject as a representation of their hand location. When I first tried this I glued a projection screen that I found in my loft onto the top of a box, and hung a projector from the ceiling so it shone down onto the screen. The subject’s hand was inside the box with the motion tracking sensor on so wherever the hand moved inside the box, a black dot followed it around on top of the box. This, I suppose, was Mark 1, but I didn’t like it for two reasons: firstly a cursor is not a great representation for the hand and secondly, the projected cursor was not in the right spatial location as it was always above the hand. This was a problem for some people, especially patients with brain damage, as they would always lift their hand up to try to match it to the cursor location above.

Mark 2 was made out of box files, a stolen mirror and some drafting paper (a bit like greaseproof baking paper) and was designed to solve the second of these problems. The projector shone down from the ceiling onto the paper and, because it was a little bit see through, the image of the cursor could be seen on the underside of the paper as well. A big mirror, rescued from a skip, rested precariously on box files so that it lay horizontally exactly half way between the paper and the table. Now, when the subject looked down into the mirror, they could see a reflection of the cursor (above) so that it appeared to be at the same level as their hidden hand on the table (below the mirror). So the cursor was now in the right place, but it still looked like a cursor, not a hand.

Mark 2 was used for Newport and Jackson, 2006 published in Neuropsychologia

Next time: how the ill-fated Mark 3 tried to solve problem 1.

Written by Dr. Roger Newport

From Box Files to the British Science Festival: Part One

This post begins a series written by Dr. Roger Newport about how and why the MIRAGE system was developed.

I am often asked how MIRAGE came about, how I thought of it and how I designed it. The truth is that it all came about by chance - I was actually designing something else. About 10 years ago I was experimenting with prism goggles. Prism goggles shift everything you see over to one side so that when you point at something, you miss it. After a while your brain gets used to the visual shift and you learn to point accurately again, but when you take the goggles off you overcompensate and miss to the other side (because your brain has adapted to the visual shift). While this is very interesting as it helps us to understand how the brain learns and adapts to new tasks, I felt that the fixed visual shift of prism goggles was too simple and that it would be more interesting if we could create a situation where the brain has to learn something more dynamic and flexible. At the same time, I was also messing about with mirrors. You can try this one at home: Fix a mirror upright on a table (I actually had a big mirror wedged vertically between two tables), then put one hand either side and look into the right side of the mirror so that you can see a reflection of your right hand where your left hand should be. Now repeatedly point backwards and forwards (towards and away from your body) either side of the mirror with both hands between two (or more) fixed points while looking at your ‘left’ hand. After about 40 reaches, stop with your hands at the furthest point and peer round the left side of the mirror to see where your left hand really is. You will probably find that it has drifted off either away from the mirror or towards it without you noticing. Sometimes it can drift more than 20 cm and you still won’t notice.

Participant reaching to targets either side of an upright mirror. Instead of seeing the real left hand only a mirror reflection of the right hand (appears like the real left hand) can be seen.

How does this work (assuming it has if you’ve just tried it)? When you look in the right side of the mirror, what you see as your ‘left’ hand is obviously actually just the reflection of your right hand. However, because it happens to be in the right place at the right time, your brain will treat it as if it was your real left hand. Your ‘left’ hand won’t be very accurate at pointing to the targets because what you’re actually seeing is a reflection of the right hand (that you’re not looking at!), so it will appear to miss the target by a small amount each time. Your brain will try to correct for the error of your ‘left’ hand but can’t because it is actually your right hand, and so over the course of many reaches these small corrections all add up and you find that your real left hand has drifted off significantly to the left or right without you noticing. The direction the left hand drifts will depend on which way the right hand missed on the first reach. Here is a series of videos that demonstrate what I mean.

Set-Up as the participant sees it. Note that the 'left' hand appears to be pretty accurate throughout.

This video you shows what the hands are actually doing and you can see the left hand starting to drift off to one side.

In the final video you can see that the participant almost hits the mirror and even changes hand posture without noticing, demonstrating the illusory power of seeing a hand in the right place at the right time doing approximately the right thing.

Next time: How these seemingly unconnected ideas began to fuse into what eventually turned into MIRAGE.


Written by Dr. Roger Newport.