2004: PS2 With 3DV Z-Camera Resembles Natal Breakout Demo

In 2004, EyeToy inventor Richard Marks gave a presentation to discuss what he had learned in his 4+ years of research for the PS2 camera. At one point he and Sony were even experimenting with the Z-Camera from 3DV, the same company Microsoft bought in 2009.

At 1:07:03, Marks shows a Breakout-like game using the Z-Camera and a PS2. Journalists speculated that this could have been the basis for the next-gen EyeToy for PS3, but Sony instead opted for a 2D camera with smaller performance upgrades, and the rest is history.

Other interesting moments:

25:00 - March 2000 demo shows primitive one-to-one tracking with a colored wand.
1:07:45 - Richard Marks admits that the Z Camera alone isn't accurate enough for a game like golf, saying, "You could get a rough interface," but "I don't think (a golf sim) would be a good idea."

[Gue1]

Natal = PS Eye shameless rip-off

[Confused]

What do you expect? It's Microsoft.

[Oner]

Exactly, a company that was FOUNDED on stealing/using others ideas and hasn't stopped since.

[Mike134nl]

Why didn´t sony patent this technology, are they just that stupid.

[sinncross]

Did you not read the part where Richard Marks stats that the 'Z Camera alone isn't accurate enough for a game like golf'

Likewise he said at the E3 conference that the eyetoy showed that they needed a button input device for true interaction.

[Kakkoii]

@sinncross: Did you not read the part where it said "alone"?

Alone yes, it's not good enough, because it only calculates depth, but doesn't see color. If Sony had thought to match it up with their Eyetoy, they would have had an early Natal.

Watch at 1:00:30 into the video, it shows the video stream from the ZCam, you can see how detailed it is and how if it was paired with a color camera like in Natal, you then have a perfect motion tracking system.

[ShabzS]

well think about it man its 2004 ... we are in a time where motion controllers actually are acceptable ... but this was before the wii... i think the 2d eye toy was really required for the current technology like natal to even exist... it makes the technological jump seem smoother... but thats just me though

[myothercar]

@ Kakkoii

The Z Camera does both depth mapping and normal video. The reason the camera alone couldn't do a deep golf game is because the tracking isn't accurate enough to catch the subtle movements in your wrist for slicing and hooking your shot.

A remote with gyros and accelerometers is necessary.

That's the thing Natal fans don't get. Some motion games for 360 are going to have to be simplified compared to what MotionPlus and Sony's remote can do. Sure, Natal can track your whole body, but what is the point, where are the demos, show me that it works and is fun. Otherwise, I'm not convinced.

[Kakkoii]

@myothercar: I didn't say the Z-Cam couldn't do video. I said it couldn't see color. The Z-Cam uses a monochrome sensor.

And again, this is the "alone" point I was trying to make. Alone Z-Cam isn't good enough. But that doesn't mean you can't use it along side with motion sensitive controllers and a color camera.

It all comes down to the fact that's it better to have the Z-Cam, then to not have it. As it can enhance any kind of motion tracking your trying to do. The motion sensing remotes only know how they themselves are moving, not the way the rest of your body is moving. And that's where the Z-Cam would come in. Allowing for increased accuracy and whole body motion mapping. And then combined with a color camera for translating your images into the game.

[myothercar]

I read your first post as if you were saying that Natal or the Z Camera could do golf if they could see color, which wouldn't be accurate. I guess it was a misunderstanding.

But it can see color, watch at 1:03:58.

And actually, PS Eye alone with body recognition software can track the location of your head and torso. So the game where you walk around in a room batting a beach ball would actually be possible with just a PS Eye and 2 remotes. It was only tracking his waist up, so it doesn't need your legs, it can automate the walking based on where your torso and head are.

[Kakkoii]

@myothercar: No, the Z-Cam cannot see color. IT USES A MONOCHROME SENSOR that receives infrared light, it doesn't receive light from other spectrum's that would be needed to produce color. The part in the video your talking about is of them using the Eyetoy along with the Z-Cam. So essential they did have Natal like system back then.

And yes, of course software with the Eyetoy can do guess's based on the images it's seeing. But it's still always going to be a guess based on variable light sources and objects colors/textures. The Z-Cam doesn't see colors, just shading that's created from a single IR light source shining on the objects. This way it is able to accurately see the depth of things without needing to guess using software. Thus bundled with a normal camera like the Eyetoy, it's capable of a lot more, since the system then has 2 different sources to extrapolate information from. A depth source, and a color source.

[myothercar]

That's what I thought everyone assumed, that the product would consist of two cameras being overlapped.

[Kakkoii]

Well, you can't overlap 2 sensors. Partially transparent sensors are still in their research stage. You would have to create a material that only let's a wavelength of 750nm and up to pass through to the monochrome sensor behind. While anything lower than 700nm (visible light spectrum) wouldn't be able to pass through, and thus would be picked up by the first sensor.

So while overlapping sensors should be viable in the future, it's currently not.

[hmmmm]

-

[talltony]

we all know that natal is a blatant rip off of early eyetoy technology. Sony just really snused on this one. They could of got the credit that natal is getting now if they wanted too. What little respect I had for Microsoft as a company is now almost completely gone after seeing this video. I mean dang! Wasnt this made in 1999??

[Kakkoii]

@Ju: Of course you would need a second angle, I thought it would have been implied through common sense that one would turn the object to scan the angles.

And it wouldn't by impossible this way, sure the models wouldn't be PERFECT, but they would be close enough. Software can do a lot with scans that are off angle.

Plus I wasn't implying it could be done well with what we have now. It could be done better if we used high wavelength's that can see through objects, and create a 3D model like that. Or merely create a rotating stand for people to place objects on to be scanned.

It can be done, not perfect, but more than good enough.

[DlocDaBudSmoka]

looks like Christ's second coming was 5 yrs ago. so much for M$ innovative tech, lmao.

[saint_john_paul_ii]

just goes to show you that This guy and Sony has been working on this for a long time... its nice to see that this video was put up on this site so that they can see themselves.

[saint_john_paul_ii]

whats funny about this is that Sony was working on this Home Grown for years, while Nintendo got a patent from Midway about motion controls then took a risk on getting this out on the market first.

If you watch the whole thing you would probably be amazed at what the eyetoy did.

[Godmars290]

more amazed with what Sony didn't do with the Eyetoy Pope. Like make significant games for it. Sell it better.

If Natal is anything close to a success it'll probably have more to do with advertising than any game on it.

Likewise if its a failure, sales as well as the HD-DVD add on, MS will never admit to it. Will be called a success if it only does as "well" as the Zune while they continue to push it.

[Ju]

We all know Sony's marketing is, err, slow at times. And they could have made more out of the EyeToy, and possibly made the PS3 version a 3D camera. But he also mentioned, that their philosophy is always having a dump device and use the machine (console) to make things work. I guess primarily for price reasons. Consumer electronics and especially gaming peripherals need to be cheap.

I would be very surprised if MS will pull this off. They seam to go the same way as Sony's R&D 5 years ago (and without any experience). You never know, though. Who would have thought that the Wii would work before it came out. But the PS-Eye is there because it was build based on the experience they made with those kind of things over the years.

What's funny is, that the EyePet coming out now was basically shown in that video, too. It just had to be developed into a product for today. They have a couple of more of those ideas in their research group, which they can turn into products. Looks like, what they didn't have back then was a product department working with research. That seams to have changed now (maybe especially so because of the pressure from Natal - and Sony realized, that they have all that already in their labs), EyePet looks like to be a real product with a proper production value based on the Eye.

[arsenal55]

ZOMFG

plate spinning game = PLAYSTATION 3 AVATAR

[Henry Cain]

I think a lot of Sony fans have been saying how Sony came out with this first. But the 360 fans just keep on hating on Sony. We all know that Sony pushes the envelope when it come to innovation. M$ is playing catch up still. Eyetoy came out in 2002. Thats why most Sony fans were not impress with Natal at E3.

[saint_john_paul_ii]

simply put, everything he said in the lecture, is happening right now

Everything he showed is happening right now, from PS eye tech, to motion controls, to even Natal tech and marketing, is happening right now. this man is a genius, this guy is a prophet of god!!

[DelbertGrady]

Pushing the envelope you say? Microsoft released their Sidewinder Freestyle Pro controller in 1998. It took Sony 8 years to come up with the SixAxis. And the Sidewinder is probably more accurate too.

http://www.youtube.com/watc...

[D4RkNIKON]

@soda True MS may have done it first, I will give you that. But to say that 1998 tech is more accurate than the sixaxis is a joke, also I watched the video and the response time from the motion in the controller to the effect on screen is a joke.

[sxpacks]

As much as I can root for Sony, the point needs to be stated.
What have they done with the tech?

PS2 had network play, but what has Sony done with online play?
MS has supposedly (don't have a 360) beat them in online play

HW is one step, the final and most crucial step is SW support and it looks like Sony hasn't been giving it.

Dr Marks has been pimping his Eye Toy for almost 10 years and he still is pimping it. Why hasn't Sony given him the marketing support?

[Too_Hyped]

If you bothered listening to the video, you'd know Eyetoy sold more than 2.5 million, so it already worked for Sony.

Natal sold ZERO until now, and has a grand total of ZERO game though.

[myothercar]

Other Natal features people think Microsoft came up with are speech recognition, hand swipes, color recognition, and importing drawings and pictures. These were all done on the PS Eye and other webcams well before Natal, and aren't revolutionary (or even useful, for that matter) at all.

[Kakkoii]

Sure not new, but not useful? O_o

Speech recognition is very much useful, as it can put a whole new level of interaction into games, allowing you to become the character in a game at an even deeper level. Imagine being able to roam through a city and actually ask NPC's questions to help you on your mission, or if your looking to buy something. Or say in a game like Mass Effect, being able to command individual characters in your squad through voice. Voice recognition will have a very prominent role in the future of gaming.

Imaging objects/pictures is also very useful, because it allows a person to bring their own world into a game. Imagine being able to use the Z-Cam to scan the 3D shape of an object, and at the same time a normal camera would be creating a texture for it also. It would be a quick and simple way to create and add your own 3D models into a game. PS3 Home for example would be able to make great use of that.

Hand Swipes.. Meh, that's part of motion tracking as a whole anyway.

Color recognition?...Don't really know what you were getting at that one. If you mean Facial recognition, I think it would be a nice thing to have, not amazingly useful, but still nice.

[myothercar]

Useless was an exaggeration, but...

Speech - People have enough trouble getting automated customer service to understand simple Yes, No, or Agent responses. It's not a feature people want if they have to repeat themselves often.

Image/Object importing - If it had the precision to create smooth 3D models, the possibilities would be endless. But it's not that precise, and the most it's going to be used for is importing textures and faces, both of which have been available on PS3 and 360, and neither of which have set the world on fire.

[Kakkoii]

I wasn't implying these things could be perfectly done right now. But just that they can be quite useful in the future when the technology can provide these services in a proper way.

(p.s: Those automated phone systems are pieces of crap, they're hardly on the forefront of voice recognition technology haha.)

[Dark_king]

Did any of you ever play lifeline for ps2 i had a demo to it years ago the whole game was voice controlled thought that was nice back then.but proved you could control a character with nothing more then your voice.

[Ju]

3D scanning will never work with a (3D) camera which was meant to be a gaming peripheral. You'd need at least a second angle to scan (e.g. back faces), or you will have a very complicated setup to scan each angle at a time. Not practical, and not precise enough for a depth sensor alone.

[Kakkoii]

@Ju: Of course you would need a second angle, I thought it would have been implied through common sense that one would turn the object to scan the angles.

And it wouldn't by impossible this way, sure the models wouldn't be PERFECT, but they would be close enough. Software can do a lot with scans that are off angle.

Plus I wasn't implying it could be done well with what we have now. It could be done better if we used high wavelength's that can see through objects, and create a 3D model like that. Or merely create a rotating stand for people to place objects on to be scanned.

It can be done, not perfect, but more than good enough.

[KingKionic]

oh...interesting.

Well we live in the now.

EDIt

why is this news?

[D4RkNIKON]

In other words, MS only reinvented the wheel and called it new and innovative. The past doesn't matter to you because it only makes MS look bad.

[KingKionic]

"The past doesn't matter to you because it only makes MS look bad."

riiiight and this is news?

I made a blunt statement saying

"we live in the now"

which we do. I wasnt saying anything at all about sony or ms.

Edit

I should have just edited first post...

lol why jump the gun?

[LordMarius]

Your right this isnt news,
it just assure the assumption of MS being the "Me too" company, not that the others don't do it, just that MS does it constantly

[D4RkNIKON]

I am just happy to see that people are realizing that they were blind to think that Natal is something new that we haven't seen before when it is obvious this stuff has been around for YEARS.

EDIT - There goes a phantom disagree as if the video does not prove my point.

[KingKionic]

And yet it doesn't discredit how good or bad natal will be. Now Ms and sony have motion controls. We will have to wait and see what they do with them . It`s all about how much ms or sony will support it.

I wonder if this praiseing of natal have made people mad....

lol

EDIT@ below
Well...cant wait to see it in windows...

:D

[D4RkNIKON]

"I wonder if this praiseing of natal have made people mad...." It is simply undeserved praise. Like you said, we have to wait and see what they do with it, so far they haven't really done anything so they don't deserve praise for tech that is nothing new.

[KingKionic]

Edit @ above

Well actually thinking about it...Natal would be new ,because the tech would be integrated in a game.

Oh the tech will prove itself . Thats a given.

Now support ...thats another story.

In the end i will be happy to see it in computers.

[DevastationEve]

I think it's a game everyone plays. It's why companies have slogans that say "serving you since xxxx" or whatever. They want to trace their business back to the nails they used to put up the 2x4s, lol. They'll say they lay claim to being the original, and then tell you "Well, not us personally. We just later purchased the idea from so and so".

Personally, I agree with the "we live in the now" idea. I don't care about the original friggin KFC documents! Just gimme the goods!

[KingKionic]

The way the Natasl is used in the future will make it new.

[ApocalypseShadow]

"the here and now",we'd never learn from the past and are destined to repeat the same mistakes.

microsoft is copying sony's past tech and are already making mistakes.it's not 1:1 as we have seen onstage and on jimmy fallon's show.sony has shown from past videos like this that you need something that works in tandem with something else for it to work.

nintendo learned and didn't just put out a sensor bar by itself.they made a controller to go with it.then when it wasn't accurate enough,they designed the wii plus adapter.obviously,they knew it was coming and thought about price first for wii owners.let the customers get used to playing before taking them to the next level.

microsoft wants to forgo controllers all together.doesn't make sense when it's so limited in some areas for games for the end user.microsoft WILL come out with a wand eventually.it's inevitable.all microsoft had to do was copy and make their own wii like controller.or sony.either one.but watch as sony releases a game where you can drive with your hands just for kicks to show that it can be done with pseye.

sony is on the right path until there is something better out there.but it just isn't natal.natal is not the next level because it is not accurate.and if they don't release a wand,watch as they make some shooter game that still needs the controller to play.which will go back on their word of controller less gaming.which will make them liers.

the cool thing is that companies can make games for nintendo wii and ps3 at the same time.creating more sales and a bigger base of casual gamers.sad that microsoft will lose out as they don't have a controller in hand to do what the wiimote and the ps3 motion will do.

apocalypse................

[KingKionic]

speculation.
when it comes out of its "demo" stage it will be different story.

I wonder what else this camera can do after watching these videos from 3dv...
http://www.3dvsystems.com/g...

[ApocalypseShadow]

but nintendo and sony's tech will be similar.hense,games can made for both.one game will be higher res and the other on wii will be lower.it's just like games made for 360 and ps3 can't be made for wii.you can't make the same type of game for microsoft's natal.that's just fact.

second,wishing for natal to be accurate later won't make it so.that's speculation on your part.microsoft never stated 1:1 like sony stated and put on show at E3.

i've seen 3DV's work as i have followed it before microsoft took interest in them.it's cool tech they have.like the boxing game.the thumb thing blows.point up and down and such.casuals will not be using thumbs to play.

same with johnny lee.he has great work.you can find it on youtube.but the tech he uses is existing nintendo tech.but he switches the sensor bar and the wiimote around.it's cool.but again,since he is working with natal,there is no sensor bar or controller to do his work.so all they're getting is just his creative mind.

but we'll see how all this turns out.

apocalypse................... .

[D4RkNIKON]

@ apocalypse

Some people just can't stand to hear the truth and you my friend speak truth.

[KingKionic]

lol

I see a lot of people want"Natal" to fail. But dont understand the way its used in the future will make it new. Some much hate for one company "microsoft" is astounding. I guess they get there name right all the time :

Microsoft the "EVIL" company. lol the irony. I find it`s funny how much natal has excite people. There fears and there opinions interact.

It`s pretty amazing in a way...

[Ju]

Sure people want Natal to fail. I mean, it is just such a blatant rip off - especially after seeing the video. Looks like MS did watch that to create their demos - they (the Natal demos) are all in that video, in some form. That's just fine - but they pretend (again ?) to be the "inventors". And what's more, others did that before and based on their experience those things just don't work that way MS wants to make us believe - new version and improvements or not. Its the basic concept which fails (lag in recognition, hidden spots in a pure optical system, you need buttons, people want to have something in their hands and so on).

[Nihilism]

haha it's funny that anyone cares about natal, do 360 kids really want a god damn 360 eye toy, that sh!t is for 8 year olds.... the same 360 peeps that dis wii for it's "gimmicky" motion control are the one's raving about natal, it's a joke, let it pass into the history of gimmicky sh!t that didn't work out

[Sez]

And it's even funnier that sonyfanboys are trying so hard to put down natal. Say MS is trying to pass off natal as something innovative. When Sony copied the wii-mote and are trying to do the samething. Maybe because Natal is getting more exposure than sony's wii-mote copy. That people have been playing with since the wii was released. The only ones playing catch up. Is clearly Sony and not many people are not interested in the wii-mote clone except sonyfanboys

[Sez]

Wow. Your a funny dude. You act like you know what I can afford little boy. First let me clear something up for you. I own all systems including a gaming rig ( that I built). Second all companies steel ideas from each other. Maybe if you was old enough you. Would know that. The only ones weak are people like you crying because MS improved on a idea that Sony was too scared to use. And just because you choose to play games on your $299. ( dude i got a dell ) Dell pc doesn't make you less of a fanboy. Now go back to playing gametap like the child you are. And come back when your old enough and have a better gaming PC to play games on steam.

[TheTruth89]

This 2004 Camera is MORE ACCURATE than Natal

LMAO

[Montrealien]

I find it cool that what Sony showed us this year was something they have been working on for well over 5 years, yet its still an early prototype? MS taking some of this tech and branding it and actually marketing it is not surprising at all though, it is MS after all.

[Miraak82]

i used to have this game and gravitron for the eyetoy, they were really fun, especially when I had people over mixed in with drinking and we had ourselves a grand ol time

[clintos59]

Where as to those who owned a ps2 and know how awesome the ps2 library was would have already seen this already and is why they werent impressed at all. Because the eye toy been doing thing like this for a while now.

I think the potential is there for natal and sonys tech but in my honest opinion of a owner of all 3 systems, I wouldnt worry about sony and nintendo in this area because these 2 companies have always been creative with games, microsoft on the other hand will have to prove they can do the same, till they do so, natal for now is more hype then the actual thing.

[Qui-Gon Jim]

So Sony had access to this tech a few years ago, before Nintendo's Wii created the "casual gamer" and, seeing it from the perspective of what is now called "hard-core gamers," they decided it's not good enough. After the success of the Wii, Microsoft approached this same tech specifically from the direction of the casual gamer and said "Well, it's good enough for THAT, plus it brings in a bunch of other possibilities." I also think that Microsoft has more of a vision that Sony, for some reason, didn't (there is an article on Bitmob.com about how Japanese bureaucracy stifles creativity, perhaps that was a barrier for Sony).

I think this shows that both company's motion control techs have different strengths. Natal removes the final barrier for casual gamers, opens new possibilities with its full body control, and has Microsoft's vision behind it, while Sony's is more accurate and more useful for core games, but still relies on a controller.

[keysy420]

natal is lame im tired of hearing about it

[dogmeat eater]

motion controls came out way back when the NES came out. Remember the power glove?

[Ninjamonkey]

so sony came up with the idea, and MS advanced it...

So why are people calling MS ripp offs when sonys controller blatantly rips off the wiimote...

Companies always copy eachother, and its a good thing to, otherwise technology just couldnt advance properly...

[ApocalypseShadow]

sony copied the wiimote.obviously from the video,sony had wand type ideas way before nintendo released the wiimote that was supposed to come out for the gamecube.

i want links and i want articles where nintendo was showing demos of their work on motion technology and sony stealing it.

microsoft is the actual thief and copier of sony technology.

copying dvd,hard drive(ogps2 had a hard drive bay),dual analogs,rumble built in,usb ports,hdmi,1080p,downloadable content ideas for music-movies-games,high def media,Gt,buzz,singstar,the early cell processor tech,cameras,etc.

and now microsoft is stealing motion camera work and trying to play it off like they created the ideas.no?did you just say no?look at the video again.microsoft's E3 is a clear rip of early sony ideas and what they were doing with 3dv.

i want links to sony copying nintendo's wiimote.but i would guess you have no facts to back it up.

apocalypse..................

[KingKionic]

Microsoft didn't STEAL nothing. They bought a company that made a cameras with there SOFTWARE. People buy companys all the time look up how the eyetoy was made. Then while your doing that look up acquisitions . And while your doing that look sonys hardware.

Theres nothing wrong with buying a company at all. a lot of comoanys were doing bad before major corporations bout the. I see your a little agitated by this news of natal. It`s suppose to be a good thing for games to be made. MS has Natal sony has ps eye and the motion controllers.

Theres nothing wrong with competition.

[Qui-Gon Jim]

So, Sony had already been working with this z-camera five years ago, and Dr. Marks seemed to think that that was the future, but it never materialized from them? It seemed to be working well, even on PS2 hardware, in fact, it seemed to be working better on PS2 than it has been shown on the 360. What happened? It seems clear that Microsoft bought that 3DV company to take access of the tech away from Sony, but i wonder if Sony was working on a combination of the two technologies. That butterfly demo using the balls, but having the player in the z-buffer so the butterflies could move around him was really impressive.