Windows 7 Kinect Control Software Launched

Windows 7 and thousands of applications can be controlled with the natural user interface, WIN&I, released by Evoluce today. WIN&I software replaces the computer mouse by tracking simple gestures from users up to several meters from the screen using the power of the Kinect depth-sensor.

[plb]

Can't wait to see what kind of interesting stuff people come up with.

[nveenio]

Kinect on a PC is much more interesting to me than Kinect on a game console. As someone that spends 16+ hours a day working on the computer, this would get tiresome. Call me when they have pupil tracking and I can just look where the mouse should move.

[Solid_Snake-]

me 2. im gonna go and buy a another 1 for the pc. ive just come into a bit of money, it seems like a good idea. sat back in your sofa using your hand as a mouse to work and watch films.

all we need next is a virtual keyboard like minority report and were "duh" winning.

ps move has minor reports .

[Lindsey]

Everyday Minority Report is becoming more and more true.

[Christopher]

Please note attached image.

I believe we might come close to Minority Report, but not there. The reason is that interpretation logic on what your hands could be doing at all times will always take much longer to process than a point-by-point reference as denoted by the blue points on his finger that could be tracked in real-time and in 3D. At least, that was their intention.

I think surface touch technology is the future for most commercial users since it's cheaper, portable, and most people don't need 3D control, though certain technologies may evolve to using finger-attached nodes combined with Kinect technology for doing precise tracking and maneuvering (such as 3D surgery tools or bomb disarming interfaces).

[B1663r]

I agree that Surface touch technologies play a roll here, but what exactly prevents Kinect or Kinect like technologies from have a role there? I mean from the word go, the Xbox automatically logs in the person who is sitting in front of it.

There is little doubt in my mind that will make it into Windows 8.

How about the touch pad on your laptop? With something like Kinect you no longer need a real touchpad, you could just use a virtual touchpad... or just hold your hand next to your computer like you have a mouse in your hand and tap on the table or desk for mouse clicks.

or you can even put a virtual keyboard on your laptop, and the keys don't get connected anymore, Kinect just processes the input...

Once people are comfortable with the subtleties of that type of interface, you could just put a second screen there like on the DS, but it wouldn't even need to be touch screen, because the Kinect sensor is taking care of all that...

It took Xerox 10 years to come up with the mouse+window user interface that we all use now from the first introduction of the mouse.

And if the Kinect is as revolutionary as the mouse, as I believe, the year is 1974.

[Christopher]

To answer your overall question, there are two massive barriers to Kinect being used widely.

1. Cost. It's still an emerging technology that is not widely used whereas surface technology is not. Surface technology is extremely cheap and will continue to only get cheaper. Multi-camera technology as used in Kinect will continue to be more expensive due to the components involved.

***I mean from the word go, the Xbox automatically logs in the person who is sitting in front of it.***

I quoted this one portion to point out that this is not Kinect technology. My alienware laptop has facial recognition software on it, came out a year before Kinect was even mentioned at E3.

There's a vast difference between software, like the above mentioned item, that can be implement on any device with a camera and Kinect-specific technology that allows hands-free, 3D tracking.

2. Physical Interaction. As for your example of a keyboard, which do you think would perform better, a keyboard that you can recognize and touch or the total lack of any physical interaction? Ask practically anyone and they'll tell you that a real keyboard is much better than touch screen keyboards. Remove the touch surface all together and you're getting further away from improving usability.

***And if the Kinect is as revolutionary as the mouse, as I believe, the year is 1974.***

And I'm glad that there are people that believe in technology. For this one, I just don't see it replacing a physical contact surface. I do see how it can be evolved to truly advance some specific uses in various fields of science and military/police applications.

[Christopher]

@B1663r:

Sorry for the above formating. #1 and #2 should flow together followed by my two quoted responses.

Looks like I did it in a rush earlier and lost track of where I was cutting and pasting in the quoted portions.

[Christopher]

I'm waiting for someone to mod their Kinect with a better RGB camera and then see what they can truly do when working alongside the power of a good PC.

@disagrees: Note that the current limitations of Kinect aren't going to bring to light the "future" of what can be done. The camera has too low of a framerate and resolution to do the necessary finger tracking with the Kinect and 360 processor. The logic is very complicated for interpreting actions, especially with every inch you move away from the device. This sort of thing may be the future (my opinion is that it won't be the commercial future, but the future of specific uses that require a high level of 3D management with external points to improve the response and measurement capabilities), but it will need hardware upgrades to get there. By people modding their Kinects and using it alongside more powerful PCs, we can see how much true potential there is with this technology. Heck, and perhaps even prove me completely wrong.

[B1663r]

The visual camera has very little to do with how Kinect works. It is all in the depth sensor.

Anyhow, this package sucks, watch the video, they just map the mouse to Kinect gestures.

Once Apple, er I mean Microsoft, gets to work on this then we will see Kinect replace the mouse.

[Christopher]

***The visual camera has very little to do with how Kinect works. It is all in the depth sensor. ***

The visual camera does aid tremendously in helping to track movement in the x,y plane. Depth sensing adds in the z plane and enhances the interpretation capabilities of what is being captured on the x,y plane, but on its own it can't properly interpret the exact movements made on the x,y plane.

Improving both leads to more refined interpretation logic. For standing 2 feet away from a Kinect depth sensor, you should have enough to properly interpret the finger/hand movements for depth, but the low res RGB camera greatly hinders aiding in tracking and interpreting movements on the x,y plane.

Most applications put out have only relied on relatively basic interpretation using primarily the depth sensor, but don't mistake that for thinking that for minute tracking as required for finger tracking won't require the full power of the RGB camera as well.

[gamingdroid]

@B1663r:

"The visual camera has very little to do with how Kinect works. It is all in the depth sensor."

Depth sensing really does allow you to outline where things are in x-y plane. It really makes it much more accurate.

If something is let say obscured, the camera can tell because of the depth is different between the two objects. Something that is very difficult to determine with just an RGB camera.

@cgoodno:

"The camera has too low of a framerate and resolution to do the necessary finger tracking with the Kinect and 360 processor."

Why would the framerate matter? At 30fps, it is more than adequate for most uses.

"Improving both leads to more refined interpretation logic. For standing 2 feet away from a Kinect depth sensor, you should have enough to properly interpret the finger/hand movements for depth, but the low res RGB camera greatly hinders aiding in tracking and interpreting movements on the x,y plane."

The monochrome/infrared depth camera actually has a lower resolution than the x-y plane by a quarter.

The difficulty isn't in the hardware, but rather the software. Hand detection is much harder than skeleton tracking where your limbs are usually very distinct in location and where it is attached to your body.

On the other hand, hand detection is easily obscured by the angle of your hand and the location of the fingers.

Now if you just needed to detect specific points on the hand with a generally unique form on your hand i.e. pointing and detecting the tip of the finger then that is definitely do-able.

[Christopher]

***Depth sensing really does allow you to outline where things are in x-y plane. It really makes it much more accurate. ***

By tracking changes in depth and heat (infrared). By it does not track the overlapping elements or the exact orientation of each body part to a precise enough level to allow, for example, exact tracking of sign language hand gestures.

***Why would the framerate matter? At 30fps, it is more than adequate for most uses. ***

Higher framerate means more captured in a short time to aid in the interpretation logic.

***The monochrome/infrared depth camera actually has a lower resolution than the x-y plane by a quarter.***

But the point data sent out tracks each one in real-time. This would need to be greatly improved if the technology were to move to using it from more than 2 feet away for hand tracking. That's why currently it only tracks arms, head, body, and legs at a distance of 6'. But, I'm assuming that with the current technology and using it with the PC, people will be testing it from 2-3 feet away, which means a tighter point-set of data from the ir camera.

***The difficulty isn't in the hardware, but rather the software. ***

Which requires responsive and detailed data being fed to it to improve interpretation. High framerate means more frames to aid in the interpretation. Higher resolution means an easier method of analyzing and determining body area boundaries (this would be interpreted in parallel with IR feedback for outlining body parts and then visual interpretation would take into account skin vs. clothing vs. body obstructions and so on).

***On the other hand, hand detection is easily obscured by the angle of your hand and the location of the fingers. ***

Which is handled by interpretation logic as well. Again, it is the software that will be used to interpret what is happening when signing, but improved return on data as well as the quality of data will lead to improved interpretation and even faster interpretation with better data.

[gamingdroid]

"By tracking changes in depth and heat (infrared). By it does not track the overlapping elements or the exact orientation of each body part to a precise enough level to allow, for example, exact tracking of sign language hand gestures."

Kinect certainly can do that, but the limitation is in the software's ability to detect AND recognize the gestures.

If you have a 320x240 that is about 76,000+ points you can map onto any rectangular surface area assuming the camera point of view allows it. If you make the surface area small enough, it will fit.

Map that resolution onto a 4 sq. ft area (2x2) and you got a 5.26x10^10-5 sq. ft accuracy! What about a 16 sq. ft (4x4), your accuracy drops to 2.10x10^-4 sq. ft. Still pretty darn good, but accuracy dropped by a magnitude (factor of 10) which explains why it is less accurate in skeleton tracking mode.

"Higher framerate means more captured in a short time to aid in the interpretation logic. "

The higher frame rate wouldn't necessarily give you any more useful data. The reason is that one frame from the other doesn't actually change much data. You are just capturing essentially the same data repeatedly.

At best this will only force you to have a faster response time, but if the system can't handle it then you are just cutting your detection algorithm short resulting in less accuracy.

"Which requires responsive and detailed data being fed to it to improve interpretation. High framerate means more frames to aid in the interpretation."

"Detailed" data is only useful if it adds more information FOR the system. In this case, as shown above the resolution is more than adequate.

That is to say the same as measuring temperature every 1 minutes (as opposed to 2 mins) outside. Sure you now know that the temp didn't change every 1 min's, but is it useful in that setting when the temperature changes so slowly in that time frame?

"Higher resolution means an easier method of analyzing and determining body area boundaries (this would be interpreted in parallel with IR feedback for outlining body parts and then visual interpretation would take into account skin vs. clothing vs. body obstructions and so on)."

It would suprise me if Kinect uses a lot of the 2D image for detection if any at all. That is because color differences are harder to detect than depth differences especially in a living room situation. Think about how much open space is around you, then it should be clear that your body shape is pretty unique and stands out easily in the depth field.

So I should say, to correct myself is 2D can help you outline object, but isn't strictly necessary because there is a better more accurate method with the camera. In 2D you are susceptible to color variances from light, for instance if there is a shadow it might make parts of your clothing darker falsely detecting this as a disjoint part of your arm!

The bottom line: Ingenuity in all of this is low price 3D camera, but more importantly the efficient software algorithm able to detect your body relatively accurately.

edit: After you post, you realize how long your text is... sorry about that! :)

[Christopher]

***Kinect certainly can do that, but the limitation is in the software's ability to detect AND recognize the gestures. ***

It can do it, but not well. I'm not saying that Kinect can't do stuff, I'm saying that there needs to be hardware improvements alongside software to make it work for detailed finger tracking.

***but the limitation is in the software's ability to detect AND recognize the gestures.***

Which is greatly hindered by the limited scope of data that the software has to work off of.

***The higher frame rate wouldn't necessarily give you any more useful data.***

In real-world situations, it does. Fluctuations in lighting and movement can be interpreted better with more FPS. It also acts to aid in pre-determining actions to take, which is vital to helping to reduce the interpretation logic needed.

Also, alongside FPS increase is sharpness. Capturing faster means sharper edges rather than blurred edges in most 30fps cameras.

***as shown above the resolution is more than adequate.***

We will agree to disagree here. In addition, the resolution is only adequate at very close distances. We may agree to disagree on that as well, but resolution improvements on both the IR camera and the RGB camera would also vastly improve the capabilities of the device both at close and far ranges.

***That is because color differences are harder to detect than depth differences especially in a living room situation.***

Yet it is the primary means for detecting one via facial recognition...

And, please note that I mentioned the depth/heat detection narrows down the color detection needs.

Process of detection/interpretation as follows:
- Capture IR/depth data
- Capture RBG 2d data (technically in parallel with above)
- Interpret body shape to interpret through IR/depth data
- Enhance interpretation of body detected in previous process with RBG 2d data.

***edit: After you post, you realize how long your text is... sorry about that! :)***

No worries. I'd rather read long and intelligent text than one-liners that add nothing any day of the week.

[gamingdroid]

I think we will just have to agree to disagree as you said, but my point being that at this point in time, it isn't the data fidelity that is the issue, but rather the process of detecting gestures in software i.e. the limitation is in software.

The computer isn't yet able to resolve all the ambiguity. Perhaps a second camera might resolve the issue i.e. two points of view.

Finger movement on your hand is significantly more complex than your body's limbs....

By the way your algorithm:

"Process of detection/interpretation as follows:
- Capture IR/depth data
- Capture RBG 2d data (technically in parallel with above)
- Interpret body shape to interpret through IR/depth data
- Enhance interpretation of body detected in previous process with RBG 2d data."

Seems reasonable, but you might just find out that the RGB data introduces noise causing wrong detection or that it might not be needed at all.

Anyhow, trying to keep it short. ;)

edit: and it still was long... need to be more concise in my writing.

[Bigpappy]

The fadelity ot the infrared sensors are way more important that the RBG properties of the camera. Motion dectection is done mostly by infrared. That is why lighting is not an issue. The RBG only really becomes important for facial recognition and photo taking.

M$ is working on proving you and many other nahsayer wrong. They recently showed a demo of software improvements that help Kienct to identify and follow body parts much more efficiently. The single point decice with always be better for pointing, but Kinect is more than a pointing device, it is ment to copy your full body movement in real-time. It is already doing a good job. They are working to make it even better. Move and Kinect completely different tech. One cound not replace the other.

@dragonelite: Yeh, this title is misleading. This has nothing to do with the sdk M$ has promised.

[HeavenlySnipes]

Is this what they been working on the whole year? I didn't think controlling G4W with kinect was in high demand.

[dragonelite]

How about reading the freaking article they are talking about evoluce selling this stuff for $20.

What a shit i was getting excited hoping microsoft finally released their free kinect sdk.

[HeavenlySnipes]

its about controlling your computer with kinect. O_o