The bizarre variation of a skeleton that is iBot + me, to a Kinect

by Michael S. Kaplan, published on 2010/11/15 07:01 -05:00, original URI: http://blogs.msdn.com/b/michkap/archive/2010/11/15/10090877.aspx

I'll start by saying I think Kinect is, generally speaking, very cool.

Just so you know where I stand, generally. :-)

Others have written about the Kinect and wheelchairs before, like in I Connect with Kinect from August 18th.

I have only interacted with the Kinect twice:

• Once at the end of August (I think the 26th), at an all day meeting talking about the Kinect and accessibility (they took some recordings of the iBot there);
• Once at the end of October (I think the 28th), where someone on the Kinect team got a bunch of recordings of the iBot interacting with the Kinect.

Now when I read reviews and see really cool displays like this one, and the possibilities here are very exciting.

I was not given the opportunity to take pictures or video on either occasion, but maybe if I get a chance I'll get some another day!

That meeting in August was a half day talking about the device, and a half day with us trying it out. Most of the time I was "trying it out" various people on the Kinect team were having me try things out on different devices (that were running at least three different builds of the Kinect firmware/software).

I guess they had already tried out the Kinect with wheelchairs, but the iBot really confused the Kinect in every build I tried. The primary reason for this is that the detection has two core 'tracks" or "modes" that it uses -- one standing, and one sitting.

For the iBot, it couldn't figure out which one was right. Because I seemed to be sitting, yet I also seemed to have legs far enough down to the ground that I appeared to be standing. Not to mention my iBot joystick, especially when it was pushed off to one side a bit, seemed a lot like my right arm.

Skeletal tracking was obviously a little confused in this bizarre variation of a skeleton that is iBot + me.

Now the October build I tried a few months later was pretty much the final or close to it. They had actually done some work to fix a lot of the problems with skeletal tracking as it applies to a guy in an iBot.

I guess I should say they "fixed" it, actually. Sincw the fix was to make sure I was always in the sitting mode (they call it the sitting "pipe") and never the standing mode.

This "fix" actually makes a lot of games not work, since many of require standing mode -- even if you are doing something that clearly implies sitting like driving a car.

I've talked to enough people on the team at this point that  I know this could be considered a limitation with two clear architecting causes:

• The best practices in the Kinect SDK given to game developers that does not push harder on the need to work with the seated pipe, and
• The game developers themselves, who gave so few options here for the seated pipe.

In my mind, these are the immediate problems that will make the Kinect more generally useful for me (in its current state and the state of games most of it is unavailable to me).

So for now, the Kinect took a person who generally does not feel very disabled (me) due to my iBot and made me feel excluded. The same way that I might be if I were not allowed in certain buildings or to certain events.

But in the long term there is more to it.

If you look at the community of people who are in wheelchairs and those who have problems such as missing limbs as one big group for a moment, they fall into two distinct categories:

1. Those who see themselves as whole people with four fully active limbs, and
2. Those who see themselves with the particular limitation their disability gives them.

This can be thought of the way Morpheus described it to Neo, the residual self image, also described here:

Residual self image is the concept that individuals tend to think of themselves as projecting a certain physical appearance. The term was popularized in fiction by the Matrix series, where persons who existed in a digitally created world would subconsciously maintain the physical appearance that they had become accustomed to projecting.

Now I think it is fair to say (once these issues related to best practices and game developers are more widely addressed) that every person who is either missing limb or limbs or who is in a wheelchair see themselves in one of the two ways I mention above, and they would ideally want their Kinect avatar to match that, in large part. Furthermore, I think that if they (by which I also mean me) are asked to have an avatar that denies them their residual self image then they will not enjoy the experience as much.

Once this is addressed, there is remarkable potential to give people back the life they want if the ability to virtually extend themselves into a digital world lets them be who they feel they really are, with their own residual self image, even if circumstances have worked to interfere with that image a bit.

I could spitball (and have spitballed) thoughts on how this might work, i.e. configuration options that would alter the way the Kinect projects the user to the games based on their images of themselves. But I have no idea how all of that will go, how much they will do.

Some of these issues are issues I hinted at in a blog back in February (this blog), though at the time I did not give the option of a digital way out of things -- or a way to move beyond things.

But I think the Kinect has some very real possibilities here, and I look forward to seeing what they come up with....

If I find myself in front of a Kinect again, I'll try to get some pictures and/or video. It really is pretty cool to see it work with something it is clearly not entirely comfortable with!

J on 15 Nov 2010 12:29 PM:

Michael S. Kaplan on 15 Nov 2010 5:32 PM:

fourboyzmom on 17 Nov 2010 10:16 AM:

Michael S. Kaplan on 17 Nov 2010 5:28 PM:

Michael S. Kaplan on 25 Nov 2010 11:24 AM:

referenced by

go to newer or older post, or back to index or month or day