visionOS 2 Object Tracking: open source recipe app demo

Рет қаралды 42,274

robomex

Күн бұрын

Пікірлер: 161

@CodyAvant 5 ай бұрын

It’s unbelievable how quickly it recognized the milk. Nice kitchen, BTW! Let me know if you need an extra beta tester.

@registered_user 5 ай бұрын

of course he has a nice kitchen.. he owns an avp!

@pinguluk1 5 ай бұрын

i think because of the cow

@cooltwittertag 5 ай бұрын

You know what can recognize milk even quicker? You.

@3y3man 5 ай бұрын

Idk, but i think it also remembers where items were placed spatially so it could have been that fast if he did a demo before this final video… maybe

@TR4NS1ST0R 5 ай бұрын

@@cooltwittertag You know who missed the point of this tech demo? You.

@austinlinco 5 ай бұрын

Ah yes… it’s 2034 and I can’t wait to make my cereal in visually-identified pan this morning

@zriley7995 4 ай бұрын

Chill

@Joshpods 5 ай бұрын

Dude don't look at the negative comments. This thing is really cool. Most people dont know where we are going to be in the future and resist change.

@cooltwittertag 5 ай бұрын

I know 20x more than you about tech and programming because I work in the field. This aint going anyhwere. Its clunky, badly made, causes headaches etc. Its a dead end road.

@Joshpods 5 ай бұрын

@@cooltwittertag imagine if those brick phones were thrown out because they were too "clunky"

@filipwassen7947 5 ай бұрын

@@cooltwittertag I absolutely agree that the Apple Vision Pro is not the future at all, but this video is about software and use cases of augmented reality. You’re blaming the hardware, which in the case of this ridiculously expensive product is totally valid, but like phones we will hopefully see more affordable and smaller designs in the future.

@cooltwittertag 5 ай бұрын

@@Joshpods google glasses were more futuristic than this glorified VR headset. What you are comparing this too is ridiculous. Mobile phone were a portable version of a stationary technology. Apple Vision is a portable version of something that is already a portable thing. Portable vr headsets already exist. See through camera systems are already included in those. AR sensors are in every decent phone already, they take up no space. All Apple did was manufacture a clunky vr headset and sell it to fanboys like you as a new technology. There is nothing new or revolutionary about this.

@cooltwittertag 5 ай бұрын

@@filipwassen7947 augmented reality is already part of every half decent phone. Putting a massive sweaty battery driven vr headset on your head isnt gonna revolutionize AR. AR doesnt need VR headsets or really any headset.

@mikailmaqsood818 5 ай бұрын

Thank you for demonstrating it so thoroughly with use cases too

@r.m8146 6 ай бұрын

This is amazing; it must become a cooking app, and it also should be done in conjunction with weighing the ingredients. That'd be a bombastic usecase.

@rybalchenkofamily 5 ай бұрын

AR seems the most useful for people with impairments or compliance and training. Very cool

@AndrewTSq 4 ай бұрын

One thing I have noticed with my bad eyesight, is that the cameras on my Quest 3 makes me see better lol. I have problem reading sms on my phone, but if I have the quest 3 on me doing the same, I can read the text :) I only see good close up irl.

@vitalis 5 ай бұрын

Your open source project is great. Keep it up.

@jassieeeee 5 ай бұрын

Wow, cool! I can see many use cases for this in construction industry and such

@jeffdude8713 5 ай бұрын

Very good explanation of where this is going. I didn’t realize all the use cases. Game changer.

@dosso96 5 ай бұрын

So amazing!!! That is perfect for someone that is approaching to the kitchen art… inside avp you can have all the information you need instead of going to a kitchen class. Love it!

@Peteboi64932 5 ай бұрын

Ah yes, *Breathing captain cereals*

@charles_wren_films 5 ай бұрын

The Vision Pro aims to solve problems that didn’t previously exist.

@PSy84 5 ай бұрын

you mean some apps...not the product. Many apps are also like that in phones...but smartphones have great uses too

@lynxAzur 5 ай бұрын

In 1978 we were living perfectly without computer at home ! 😜

@mohammedfouzan9115 5 ай бұрын

@@lynxAzuryeah booking a cab was a piece of cake huh. Finding your location when your lost must have been a breeze. Send important documents through the postal service was very safe and fast

@lynxAzur 5 ай бұрын

@@mohammedfouzan9115 and so ? We were living perfectly

@lynxAzur 5 ай бұрын

@@mohammedfouzan9115 and we had a better live !

@orihpt 5 ай бұрын

This is crazy on so many levels. Wow

@hellotherenameishere 5 ай бұрын

How does the Vision Pro handle getting steam on it when cooking?

@eyeamwema 5 ай бұрын

I imagine not well tbh. Steam would affect the cameras which may cause problems though I think there’s hydrophobic coating which may help. But the main issue would be with the LiDAR as it would likely scatter the lasers making it much harder to actually track and map rooms. That being said there’s sensor overkill in the AVP so I imagine that adds a level of sensor redundancy that helps mitigate things slightly. Because it’s pre mapping the room and updating live with sensors forward, down etc. I think it would help the R1 realize that there is interferance, use the non anomaly readings and fall back to gyroscopic data with accelerometers to estimate how the space is evolving. That being said, I don’t have one so idk, just thought I’d give some sort of answer since you didn’t get one. That’s how I image it works based on how detail oriented Apple is as well as the extensive sensor array on the APV

@getthissnoop 5 ай бұрын

Amazing! Your creativity is inspiring. Keep it up! 🚀👏

@ninjaasmoke 5 ай бұрын

People who commenting that it’s useless, your thinking is the reason humanity wouldn’t progress. Innovation must not stop at solving problems that we already have, it must experiment with random stuff. We didn’t know we _needed_ know most of the things we take for granted nowadays. as an engineer myself, vision pro is an engineering marvel.

@helloimedden 5 ай бұрын

This is really cool!! I have an idea for a color film development timer kinda app. I was hoping I could track the tank with the film in it and have the UI adapt based on the user when they go to lift/agitate and then have it indicate the chems in the bottles per step for pouring in and out. So this is really cool and useful. 🙏 kinda nervous as this is my first swift app in general but I’ve been doing web dev front and back for a while so hopefully the curve isn’t too steep for me going in blind here.

@robomex 5 ай бұрын

That's a very cool and unique idea! Best of luck building it - please send me a link to the demo/app when it's ready. Given your background I think you'll quickly pick up Swift/SwiftUI, I'm guessing a lot of the patterns will look familiar to you.

@helloimedden 5 ай бұрын

@@robomexthanks will do! Yah I worked at Apple retail out of high school and I’m a big Apple fan so I’m pretty familiar with the platform and the design part. Went to school for web/new media design also. So I feel the curve shouldn’t be too bad.

@HansMilling 5 ай бұрын

For a blind person, they could remove the internal screen and render engine. Would save a lot of power, weight and prolong battery life significantly.

@robomex 5 ай бұрын

That's a very good point - I think we will eventually get a full Apple Vision product lineup, even if it takes 5+ years. I think such a lineup would include a cheaper Apple Vision (maybe it'll be called Apple Vision Air, or just Apple Vision) and perhaps an Apple Vision Ultra "above" the Pro (the Ultra might be focused on fitness & usage while moving vs. the AVP's focus on stationary usage). I personally don't expect a version without internal screens, but who knows. I think versions of headsets you described (i.e. cheaper, fewer features) have a ton of potential for accessibility usage. Given its pricing - and once they add object tracking - the Quest line may be the obvious, short-term answer. More generally, there was an interesting article on AVP + accessibility written last month in Intelligencer: nymag.com/intelligencer/article/apple-vision-pro-disabled-users.html

@AndrewTSq 4 ай бұрын

ah that would be so cool. I saw someone mad a Ipad app with the lidar enabled, to make blind people be able to walk around in unknown areas full with debris

@filmweaver2013 5 ай бұрын

Great work! This is the purpose of these headsets, a new perspective to education. Ps. A horror game in ur home would be an amazing app 😂

@reubenb5088 5 ай бұрын

The kids of tomorrow will not be able to pick up anything that their headset does not specifically highlight

@eweweweweew-e6o 4 ай бұрын

5 years later, "Finding lost McDonald's VR game used for training McDonald's workers"

@Victor-cg4hx 2 ай бұрын

That's perfect, i won't cry anymore when I cutting onion.

@Act_True 5 ай бұрын

this is a very low level scenario but for example oculus keyboard tracking has like 5 keyboards. with this i could possibly add my own keyboard to be tracked by scanning it in or have logitech develop keyboard tracking support for their keyboards and maybe their mice too.

@MisterMotel 5 ай бұрын

2:05 it's cool but I dont think blind people will buy an Apple VISION pro lol.

@stardewvalleybrsl 5 ай бұрын

Vision pro acts like an eye to blind people, and can indicates instructions and information by sounds and voices

@visionpromexico 4 ай бұрын

helllo how can my videos look hd like yours without pixelation?

@XY-km8ds 5 ай бұрын

We as humans in a society and developers have to be so careful about this. This could make the real world without experiencing through VisionPro appear boring and less exciting.

@robomex 5 ай бұрын

Thank you for this comment - I've had similar thoughts in the back of my mind for the 7 years I've been working on AR stuff, but you put the concern so succinctly. Many of my personal projects are focused on the vague/overarching concept of "Enable people to customize their surroundings in new and interesting ways," similar to how people customize their desktops and homescreens. I don't think this product exists, but let's say there's an AR app that works with artists to feature their artwork in people's homes via an AR frame that updates twice a day. I think that concept is interesting and (maybe, to some people) valuable and beneficial. And while I don't think this is possible today in any meaningful way: I think it'd be extremely dystopian for people to use a product that enables them to say "I don't want to look at my friend's face anymore, so whenever I view them through my AR glasses, show me a beautiful model mapped onto my friend's face instead." I think some initial version of the latter will exist in the next few years (also I believe that was vaguely a Black Mirror episode 7+ years ago), and I think we need to question if that's what we really want.

@toniesan1904 5 ай бұрын

Really interesting take, yes it was season 2 of black mirror, the end of white Christmas which vaguely played with the idea of AR blurring out harsh parts of reality. S4, ArkAngel also did this for children (censoring non PG content in reality). The flipside which was mentioned is really interesting, where the real world isn’t good enough compared to vr, kinda like ready player 1 in that sense

@toniesan1904 5 ай бұрын

It would be pretty dystopian to live in a world where real time filters were being applied in AR to make eyes bluer, or sunsets more intense, or change appearances. The pros likely outweigh the cons though I believe

@SongStudios 5 ай бұрын

I guess it's cool it can track a pan, although I don't know what it'd be used for

@retro527 5 ай бұрын

this is incredible! keep up the hard work

@pradeepmax1 5 ай бұрын

Wearable goggles for blind people could help them find things without needing to touch everything. The goggles can use vibrations, sounds, or both to tell the person where to find a specific item, saving a lot of time for them.

@Matt-ym3if 5 ай бұрын

this is the data collection step for robots learning to cook

@twitchster77 4 ай бұрын

$4,000 And now I FINALLY learned the secrets to cooking a bowl of cereal!

@HitEmUpru 5 ай бұрын

Great lightning in the kitchen! I bet this is Philips led stips, yes? p.s. where you buyed this thing for paper towel? Where its standing

@robomex 5 ай бұрын

I believe the paper towel stand was purchased from Target ~10 years ago - sorry I don't have more info!

@mickael450450 5 ай бұрын

What about Quest 3? Will you ever build a release for it?

@robomex 5 ай бұрын

I don't believe Quest has object tracking capabilities at this time, but I expect that to change soon (i.e. within the next 14 months max, likely much less than that). Once object tracking is added to the Quest, my guess is many devs will create many apps that use object tracking in different contexts - I specialize in iOS/visionOS development since I write code in Swift, but other devs will build similar apps/products for the Quest.

@bulls8685 5 ай бұрын

@@robomex Unfortunately you're wrong here, in the V68 they added Meta AI which handles vision detection, and so does the detection of furniture, since it's possible to track furniture and decoration, then it's technically possible to do so with the Quest 3.

@Emerson1 5 ай бұрын

Cool, can it handle tracking moving objects? Like if you had those 3 objects on a lazysusan that was slowly rotating?

@robomex 5 ай бұрын

It can handle moving objects...kinda. When objects are moving slowly the virtual tracking object usually trails the real object by ~1-3 inches. However, I am using the default visionOS 2 object tracking that's available to everyone. Apple has a set of Enterprise APIs for use in internal apps (i.e. apps that won't be distributed on the App Store) that let you track more objects at a higher frequency. Docs for that setting/mode are here: developer.apple.com/documentation/bundleresources/entitlements/com_apple_developer_arkit_object-tracking-parameter-adjustment_allow I'm unfamiliar with other times Apple has enabled "non-App-Store" APIs such as what's at that link. Given the current state of visionOS software and hardware, I assume that means in the future object tracking will get better for everyone (e.g. more simultaneously tracked objects, higher recognition frequency, higher tracking frequency) - it'll likely just require a future release of visionOS and/or a future release of Apple Vision hardware. Totally guessing here: I'm guessing third gen visionOS hardware in the second half of 2026 + visionOS 4 will bring better object tracking across the board.

@grimsk 5 ай бұрын

와 인식 능력 미쳤네요. 마이크로소프트 Azure Object Anchor에 비하면 속도가 아주 그냥 우사인 볼트네. 이걸 대체 다들 퀘스트랑 왜 비교하는거야 ;;;; 정신차려 이 사람들아 ;;

@ninjaasmoke 5 ай бұрын

Please don’t let this die😭 Everyday we are getting closer to being iron man

@RealFinalCionide 4 ай бұрын

Incredible by 2030 i can pick up some sunglasses that will do this

@ovo5326 5 ай бұрын

This is perfect for visually impaired people, however, there might be a problem with setting up eye tracking

@yahya_onal 5 ай бұрын

Video little bit asmr

@lachlanlau 5 ай бұрын

incredible app ! AR really is incredible.

@catluvr8875 5 ай бұрын

how many objects can you track at time?

@robomex 5 ай бұрын

I thought I saw in the docs that the max was somewhere between 8-12, but now I can't find a reference to that. They may have mentioned the max # in a WWDC video. The max I have done so far is 3 different objects being tracked at the same time.

@zlinoliver 6 ай бұрын

Thanks for sharing! Super cool!

@MementoMori_2070 5 ай бұрын

I can just imagine the potential. In 5 years the unpaid version, will constantly inundate you with ads while searching in the fridge for OJ. This use case is a corporations wet dream.

@BrysonTheTomato 5 ай бұрын

its good they are developing the software now because eventually this is gonna be in glasses or something

@brianclear363 5 ай бұрын

interesting. how long did it take to train all those objects. I saw another demo where it took ages for just one object captured with photogrammetry though they only had an m1

@robomex 5 ай бұрын

I've got a maxed-out M2 Ultra and of the ~10-15 models I've trained so far the range is 4-16 hours of training in Create ML for each .referenceObject

@brianclear363 5 ай бұрын

@@robomex wow I suppose the cereal box could be done with ARKit Image tracking to save time.

@robomex 5 ай бұрын

@@brianclear363 yeah - I used image tracking on a map in this demo and I think you're right, for many use cases image tracking is just fine: kzbin.info/www/bejne/mYTCqq2bobJqrsk

@RiccardoMerloVegan 5 ай бұрын

It is quite impressive

@LeeGoGo 5 ай бұрын

this is incredible

@laden6675 5 ай бұрын

Would be really cool if you could prepare a workspace (desk or maybe a whole warehouse), then put on a Vision Pro headset, and start annotating and planning right there in the headset, zero programming skills required. Similar to tutorials in games.

@robomex 5 ай бұрын

If there aren't teams already working on what you outlined, I think you just described a startup that will be worth a lot of money in ~1-5 years.

@KyleBevis-u7j 5 ай бұрын

This is amazing!

@Kat-wk6zt 5 ай бұрын

This looks rlly cool

@thisnameisnotavailable 5 ай бұрын

Cool thing. I can foresee some useful apps. BUT FIRST, please fix the lamp near by your door 😁))

@robomex 5 ай бұрын

:-) I had no idea how many things in my home have different refresh rates/flicker frequencies until I got the Vision Pro (e.g. entryway bulb, microwave clock, office bulb, etc.). visionOS usually accounts for mismatched lighting flicker frequencies and adjusts the Vision Pro display updates to eliminate/reduce flickering within ~2 seconds - I just didn't look at the entryway long enough. You can see an example of the real-time adjustment to the fridge light when I open up the fridge at 1:23. In real life/sans headset when I open my fridge there is no flickering. When I open up my fridge while wearing my Vision Pro, visionOS takes ~2 seconds to recognize a flicker frequency "mismatch" between the environment (i.e. my fridge LED light) and the Vision Pro's internal displays, and then adjusts the Vision Pro's display updates to reduce visible flickering in-headset.

@Leyverse 6 ай бұрын

Very cool!

@XNiFo 4 ай бұрын

So basically black mirror tv show will be true

@Symbiotix_Gamer 5 ай бұрын

Love my Vision Pro

@petermarin 5 ай бұрын

What’s your background and interests? Wanna build something together?

@robomex 5 ай бұрын

I've got my plate full with existing projects at the moment - but if you ever want to chat AR/tech in general, feel free to reach out (same username on IG, plus I've written about existing/future projects and have my contact info on my website: vision.engineer)

@pascalmeger7814 5 ай бұрын

Unfortunately not scalable because of the model training. Otherwise it is great

@robomex 5 ай бұрын

Agreed - I'm guessing many object tracking apps in the near future will be for the internal use of companies with a lot of resources. Once an object can be trained in minutes on a mobile device (that's likely several/many years away) things will get very interesting.

@cesarbattistini 5 ай бұрын

Imagine am assembly manual. It can be pre trained.

@link1797 5 ай бұрын

Cooking simulator in an actual kitchen..

@ElmoPlayss 5 ай бұрын

I can do this with my phone so why would i need a 3000 dollar device (still cool tho)

@imdaniel_5872 5 ай бұрын

This is awesome

@samajier2566 5 ай бұрын

Thanks for sharing

@Jake-wl2ol 5 ай бұрын

Mark Zuckerberg says fuck you apple

@snapdrive7675 5 ай бұрын

Great!!

@christianmccauley7340 5 ай бұрын

As a consumer, this feels silly and useless. But, as a developer, this feels like a Pandora’s box of opportunity. Mostly for useless shit, but about 30 percent of the box’s contents _aren’t_ useless. And that’s more than usual.