Using an iPhone as a tracker for Virtual Production

[@abrahamlimarga4015]

some guys use word "indie virtual prodution" while it's still costs thousands of dollars... but this video is the real indie... thank you Bro !! really help for someone from 3rd world country where the resources are very limited

[@zawadzki_rezyser]

It's quite unvalivable because many people will leave this knowledge for themselves and You decide to share it. Big BIG THANKS MAN

[@EpicxNL]

Can confirm that this also works with an Iphone 7. Thanks for making this tutorial!

[@zoombapup]

Green bits are caused by lighting on the green screen. Fix it by altering the vignette value in the colour keyer transform pass.

[@taser42]

Thanks this helped with the edges!!!

[@button9]

Finally. After watching all kinds of virtual setups with trackers and vr kits for tracking and making a virtual cam with the iphone, I kept thinking "can't we just ... use an iphone to do this" but no one ever did, but I was sure there was a way. Sure enough your video finally shows up in recommendations after watching all these other ones (the algorithm finally pays off) and you deliver what I assumed could already be done. Thanks

[@hyperbolicfilms]

You should consider that this wasn’t far from the limit of where I got with this approach, and I don’t know that it is really viable for production of any kind. In some of the comments, I’ve listed some of the companies making plugins to make the phone tracking approach a little more reliable, but I don’t know that any of those are production-ready over a year later.

[@button9]

@@hyperbolicfilms yeah, no doubt, I was just looking for simple stuff around the house - mainly to have the knowledge of it. I also didn't feel like spending $600+ on sensors and such just to mess with it a bit when I have an iphone and an ipad and a dslr laying around. It's a good start to bounce off of.

[@WhitewashStudio]

Very cool. Definitely not an easy setup. Hope it gets more fluid the more people get used to filming this way. Thanks for making the video!

[@armondtanz]

Not arf!!!

[@videojournal2236]

I bought hitfilm pro yesterday and am getting and iphone 12 today. I have been working on a large scale unreal scene and have been looking into getting in virtual production. Thanks for posting this. Its motivating that others are shooting for same or similar goals.

[@hyperbolicfilms]

Be sure to check out CamTrackAR from Hit Film.

[@anointed4artistry160]

Virtual hand claps for you sir! Bravo, well done, and thanks x1million.

[@latentspacex]

Wow this was such epic tutorial, completely revolutionary!!!

[@hyperbolicfilms]

As I’ve said in some other comments here, this method might not be the best approach going forward. It seems like there’s at least one app being developed to handle the tracking of the camera with a phone much better (filmengineers dot com had something in beta which seems like it could be the perfect solution).

[@georges8408]

@@hyperbolicfilms there is no video or something about these "filmengineer . com " solution in this field as far as I search there is nothing else but only their page

[@anirudhsays1534]

@@hyperbolicfilms Thanks a lot for this, its been one year now, do you know if new latest tracking apps have released for this yet?

[@hyperbolicfilms]

@@anirudhsays1534 I have not been following closely. There were a few promising apps like byOwls and Filmengineers, but I have not seen anyone’s result using those. I have been thinking of trying this again a year later to see what’s possible now. Hopefully I’ll have time before Christmas.

[@JheredStern]

Great video man. Helped me out a lot. I am doing similar work with my Canon T5I and T6S and getting decent results.

[@leomaverick]

Loud and clear!! It was such inspiring, thanks!!

[@DOCTORFANPRODUCTIONS]

Great video man! i have a GH4, my question is what cable should i use ? should be the same as GH5? Could you please tell more info? Thanks a lot!!🎥📸

[@bei-david]

I'm from Germany and I must to say thank you for your help now I will make Livestream with this

[@hyperbolicfilms]

Blind Gamer Good luck!

[@1978rnio]

I reached 13:20 min And I have not "virtualcamera game mode" in my list in Game mode override. I made all your steps before. What I do wrong?

[@deeeny_]

Hey man, same here. Did you figure out any solution?

[@1978rnio]

@@deeeny_ No . I enabled all plugins and nothing happens

[@deeeny_]

@@1978rnio It's for the unreal version, I downgraded to 4.23 version and it worked.

[@1978rnio]

@@deeeny_ Thanks , try to downgrade

[@steempunk]

Thanks for illustrating this overall concept, and in such good detail. Very inspiring!

[@goetzvideo-filmmaker]

Great tutorial! This is basically all the kit I got at home, I will give this a try :) thanks!

[@CosmicOutpost]

This is such a great video and has saved me probably about a year or mores worth of research! Please try outputting the unreal feed to a data projector to skip the green screen? And in the unreal vid today, Matt Workman said he had to genlock the camera to the video feed. Didn't quite follow why then you tube suggested this video!

[@hyperbolicfilms]

I'm not sure I'll get to the projection or LED wall stage of things, just based on the limitation of the space I have in my apartment. I just moved to a larger space, but still there's no real good space for a projection based on the layout. If I can get into someone's studio for a while, I'd definitely be interested in playing with it, but I've moved more towards animation in Unreal rather than live action/CG background integration because it's doable for less money. I'd suggest looking into Ian Fursa's work (kzbin.info/www/bejne/lXemYp-sndSDiMU). He posts a lot of his experiments on Matt Workman's Facebook group and he's definitely shown that you can have some success with a low cost projector and a Vive tracker, so probably around $1000 for the hardware.

[@CosmicOutpost]

@@hyperbolicfilms Thank you for the link and not sure either way will be easy on our pockets!

[@KoenigbergProduction]

Hi, very nice Tutorial. How did you get access to the "VirtualCameraGameMode" in the World Settings? :) i can't find it in Unreal 4.27 or in 5.1.

[@arjanjua.official840]

same issue.

[@mikeho520]

Thanks so much for the video tutorial, its really helps me as a entry level.

[@hyperbolicfilms]

A lot of this information may not apply in more recent versions of Unreal Engine, as a lot has changed in almost 2 years.

[@OscarHernandez-js3pk]

Can't wait to get home from work and watch this

[@yoyo78]

You just opened my mind... THANK YOU!

[@thejetshowlive]

I think there is a frame rate offset...and if you type how many frames you want to offset it by 4 - 6 it usually fixes the drift. Don't quote me but it's something along those lines.

[@hyperbolicfilms]

Thank! I haven’t been able to find that, but i have heard that there should be something to compensate.

[@rohitkhanvilkr1636]

thanks for such a detailed video, but still a question.. does it track dolly movement too, or just a fix trapped movement? please answer.

[@hyperbolicfilms]

It should record any movement, except Dutch angles. Because of the iPhone’s inability to lock into landscape mode, tilting into a Dutch angle causes the Unreal Remote app some problems.

[@rohitkhanvilkr1636]

@@hyperbolicfilms thanks for this valuable information.

[@MichaelKarius]

Thanks for this very helpful introduction. Answered a lot of my questions!

[@hyperbolicfilms]

Great to hear! If you experiment with this, I'd love to hear your findings.

[@vivektyagi6848]

This is so valuable. Many Thanks.

[@josenevarez8079]

Thanks for sharing this video. Much appretiated!

[@toxichatter5853]

What’s the app that you use for tracking on iPhone?

[@oulyn]

Very nice tutorial

[@nikilragav]

This was awesome! Thanks for the detail!

[@Hencekevin]

Esto es todo lo que busco !!!! Oh porfavor esto es oro para filmakers

[@hyperbolicfilms]

probablemente no sea bueno para la producción. solo un experimento

[@fawadfarooq]

very Excellent for beginners, one question How did you connect iphone cam with unreal?

[@MANOJDINARI]

I think the green flickering is due to the auto vignetting down to the chroma keying module. And the synching issue is something to do with genlock on camera. Please correct me if i am wrong.

[@legemimiao3416]

Hello, My iphone XR always shown "Connecting to *****" Unreal Remote 2 was not found in the phone Settings, so there was no way to set permissions.

[@hyperbolicfilms]

I’m not sure. You may want to contact Epic Games support

[@quicksolphoto]

the same problem. Could not connect...

[@cipheroth]

I require a good camera and a good green background. but THANK YOU SO MUCH SIR !

[@3Dgiants]

Awesome . This is so far the most clearly explained video I have watched about virtual production . I only have one question .. I was wondering if its possible to connect live video from my camera to Unreal engine wirelessly using 7D Mark 11 . Because 7D mk 11 has an inbuilt Wi-Fi feature which works when you purchase a card for it that you place in an SD slot of the 7d mk 11 camera ? Would that work without having to connect the camera via Camlink device ?

[@zoombapup]

By the way, you might get different delays between the tracker (phone in this case) and the render. Remember that there's a delay in capturing the image and this might well be different than the framerate of the transform captured from the phone. So usually with a vive tracker, we add a buffer of N values, where N is some tweaked values that are eyeballed to give the approximate correct delay between tracker update and video display update. Anyway, keep it it, might be a useful thing to be able to do!

[@hyperbolicfilms]

zoombapup I think there are too many limitations to this approach. Shortly after I recorded this, I realized that doing any sort of severe Dutch angle on the camera would cause the iPhone to switch into portrait mode, which makes the traffic unreliable. My best results were in capturing the camera movement data, recording a take of the live action plate in camera, and combining the live action and CG element in After Effects. It’s definitely not nearly as accurate as a Vive tracker would be, but I think it can work for some shots.

[@zoombapup]

@@hyperbolicfilms Good point. No way to lock the camera in landscape mode? I'm using a vive tracker and a single base station and that approach works pretty well. I think these virtual cameras are probably best for doing entirely virtual shots though. I've hooked up a vive tracker based shoulder rig with a virtual camera setup and that works well (made some custom controls for an xbox PC controller). I'm also doing some work on controlling take recorder via an app on my surface pro so I can do different virtual setups and pull focus and play with the virtual cameras.

[@hyperbolicfilms]

@@zoombapup I've mostly moved on to playing with mocap and animated characters for my own work. I think Epic could make the iPhone do what I was hoping it could with some adjustments to the Unreal Remote app, but considering the situation now with Apple, I doubt it would happen. It's frustrating because it is close to being a good solution for mixed live/virtual productions, but just not enough in the app.

[@Jsfilmz]

hyperbolic films the virtual camera spawns a new camera instead of using yours.

[@mikkepalvanen]

How should I go about adding the buffer? Just using an iphone but the tracking is coming later than my video feed

[@mineshg3437]

I followed this video for virtual production, thank u faculty at GBC

[@mcdk4716]

You are amazing… technology is amazing… this is amazing…. UE5 is amazing… KZbin is amazing thank you man. Please do more green screen UE5 tutorial

[@deadmikehun]

Awesome video! Did you actually move the video camera itself while rendering the 15sec video? I didn't understand that part. If i had a greenscreen that is infinite around, i could possibly orbit around the subject with my camera and the sequencer would record my movement? Thank you!

[@hyperbolicfilms]

In theory yes, but this method has broken since Epic changed things in more recent versions of UE. Their new mobile app doesn't work the same way and doesn't seem to work with many iPhones. I've tested it on my iPhone 11 and a new iPad and they won't connect with Unreal.

[@deadmikehun]

@@hyperbolicfilms that's sad. Thank you for your reply!

[@georges8408]

this is a nice tutorial !!!! thank you... I will try to follow the steps

[@KarimJovianGames]

Wow I have never seen this before. Pretty awesome

[@rasmusolsengroup8969]

Hi, im currently working on trying to implement a setup like this at my university, is there an updated video I should use as reference instead? This does seem like something that would work, but it’s also 2 years old :) Thanks for the great video!

[@hyperbolicfilms]

Hi, a teacher at the college where I took a virtual production course this summer tried to recreate it with the latest version of the software, and he determined that there was no good way to get sync. I still haven't found a good, reliable, affordable way to do virtual production. I had tried Assimilate LiveFX, which is a very good live compositing program that says it supports tracking from phones or the affordable Intel RealSense camera, but even using two fairly powerful computers with multiple SDI I/O cards and timecode, it was still out of sync. I could record the Unreal background and the video in the camera and sync the two and use LiveFX for a video preview though. There is a new very affordable tracker that hit the market now (has similar packaging/marketing images to the ReTracker Bliss, but I can't recall its name). It might be the game changing device, but still doesn't have a lot of software support. The fastest/most affordable way to do any of this is using the CamTrackAR app on iOs, but it has its limitations and isn't real time.

[@rasmusolsengroup8969]

@@hyperbolicfilms thanks for the very detailed answer! I really appreciate it. 🙏🏼

[@benz.1730]

You can get rid of the green corners if you change the material to "M_SinglePassDiffColorKeyer". Question: Do you know better methods or plugins for a very good greenscreen keying? Or would you stick just to the in-built tools? Thanks

[@hyperbolicfilms]

Thanks for the info. I have not had the opportunity to dive into this much since I made this tutorial. I’m going to start using Aximmetry in the near future to see how well that does, partially because of the keyer and because it already has the input and output side of things handled.

[@benz.1730]

@@hyperbolicfilms Would be great to share your experience again :)

[@judgeworks3687]

This is a great tutorial. Thanks! Any chance you have an HD version so we can see the drop down selections better?

[@hyperbolicfilms]

I uploaded a 4K file, so I'm not sure if KZbin is still crunching that or what.

[@AmitRai1818]

Hello, at 13:23, on world settings dropdown menu, VIRTUALCAMERAGAMEMODE option is not visible there? plz help

[@hyperbolicfilms]

Hi, my apologies, I didn't see this update. I think that mode was removed from Unreal Engine 4.25 and later. In my experience, the best way to do some kind of virtual production now with an iPhone is to do it with the CamTrackAR app using the iPhone camera too. The method I described in this video has been broken by successive updates.

[@AmitRai1818]

@@hyperbolicfilms thank you!!

[@佳瑶曾]

Thank you for the video. Is it necessary to have a camera? Can I control it with only iPhone12?

[@hyperbolicfilms]

I have not scene a way to do it live entirely on an iPhone. But the app CamtrackAR may be a better way for an iPhone only approach.

[@佳瑶曾]

@@hyperbolicfilms Well,thank you very much!!

[@Ragin.Official]

At 8:18 my video tab isn’t showing my cam link. It’s just empty. My cam link is installed in my computer correctly since it’s working on other apps. Any help?

[@johngabriel1067]

What are the details on the iphone mount? Looks like you're using an adjustable mount and an adapter to the camera body? Links or details about both would be appreciated!

[@hyperbolicfilms]

John Gabriel The mount is a simple tension-based phone holder attached to the camera by a cold shoe to 1/4 20 mount. This is a Manfrotto phone holder, so was about $20, but I have a few $2 ones that would do just as well.

[@starwarsfananimations]

Great tutorial!!! Any idea how to turn your TV into a makeshift LED Wall? Not for like an official production but for leaning and practicing without 1000s of dollars

[@hyperbolicfilms]

I'm trying to figure that out myself, if I get it going in the next few weeks I'll post a step-by-step as well. I assume it has something to do with nDisplay still. I was thinking that using a green screen would be good for wide shots, but maybe getting an 80 inch TV would work for close-ups and like toy commercial/product shot type things.

[@adamvarner986]

Especially inspiring because I just bought a new computer, a lumix g7, a zhiyun crane, and a new phone because my moto z2 had lost it's mic from sanitizing liquid.

[@aldermediaproductions695]

Any chance of redoing with 5.4? Hopefully they’ve simplified the process. There are like 75 steps to put a thing in front of a camera. I’ll never get Unreal.

[@hyperbolicfilms]

I haven't yet tried the new Vcam app to see if it has potential in this old style way of doing things. I did just release a new tutorial using Aximmetry, which you can now very easily use an iPhone for both the camera and the tracking. I'm still testing using a DSLR/Mirrorless camera with the iPhone as a tracker, but my initial tests looked promising. I'll have a tutorial of that as soon as I am sure of the workflow. Aximmetry uses Unreal Engine to work, but really takes care of the camera input and keying aspects. There is also a new subscription-based app called JetSet from Lightcraft Pro that lets you use an iPhone for virtual production.

[@aldermediaproductions695]

@@hyperbolicfilms Thanks for the detailed reply. I saw the JetSet video and it looks pretty promising although I don't know if the price is right. May be worth a look though. I'd tried their older app which was basically Facetime AR and wasn't very good but hopefully they've ironed it out. I hope that some day there's a simplified method and also maybe some Unreal slimming down, i.e. a "beginner mode" that works but maybe offers only a small percentage of the options because it's pretty overwhelming.

[@leomaverick]

Use the iPhone 8 instead of HTC seems to be too much easier and works perfect...What are your thoughts about it?

[@duncanweakley7525]

Great video! One thing though... my camera shows up in media player, but now video shows... but when i open camera in windows, the video works fine... any solution? Thank you!

[@danielenstrom1873]

Looks like the offset of the phone and the camera? At this close range the angle is changed a lot from the field of view?

[@mxt1201]

Great tutorial! Thank you!

[@debbiedeerprods]

Very helpful video, thanks for this!

[@shreifelsaqa8053]

thank you for this tut but i want to ask you if i want to make out put from this to ndi

[@tmaintv]

Hey there, great video. It has been massively helpful. Can I ask can you still use the uremote camera control fully this way? I have found I can control the camera with the uremote, a keyboard or a controller all the way up to the point I change the composite output to player viewport then I seem to lose control of the touch controls and mouse/keyboard inputs once it plays the comp. Any idea how to keep the inputs and the comp?

[@hyperbolicfilms]

I haven’t had much of a chance to revisit this workflow since the summer as I’ve been back at work the last few months l, so I don’t have a good answer for you. The URemote does seem to pick its starting point and there’s no way to change much. It’s a very blunt tool, and I hope they add in more control at some point.

[@tmaintv]

@@hyperbolicfilms thanks. No worries. If I work it out I’ll let u know. Cheers

[@faizalgunawan5271]

if we use sensor on iphone for camera movement in unreal engine , is it only for rotation movement ??? or we can use it also for position movement ?? please answer , sorry for my bad english.

[@joeeastman]

That’s been answered a few times already, it does both 👌🏼

[@benz.1730]

I wonder that my iPhone connected, but did not drive the camera. I did not use an ThirdPersonCharacter. Why didn't you use VirtualCamera2Actor as in the official information? (I tested also a VirtualCamera2Actor with LiveLink without success)

[@hyperbolicfilms]

I made this process up based on the documentation and information I could find at the time about a year ago. I never saw anything in regards to VirtualCamera2Actor.

[@benz.1730]

@@hyperbolicfilms Thanks for reply. It's under the link you have posted, right here the second page: docs.unrealengine.com/4.26/en-US/AnimatingObjects/VirtualCamera/VirtualCameraActorQuickStart/

[@benz.1730]

@@hyperbolicfilms After many tests I found out, it only runs until 4.25. In 4.26 it is connecting but not working properly. Unbelievable that there is no information from UE provided.

[@ivancuetowicq]

hello, I already set the video camera feed and the iphone as a tracker. I havent done the compusure yet, but I was wondering ,after setting all these, then what? I mean do you know how to record this and export a video? Can the camera movements tracked by the iphone be recorded and possibly exported as an fbx? I know you will tell me yes with sequencer, but how? by the way for now im talking about just the camera tracked with the iphone, no video feed necesary. Im doing lot of test focusing mainly in tracking the camera.

[@jacobzudal6957]

Is there a way to stabilize footage in post if I record video on an iPhone while shooting video on my camera in a rig similar to what is shown in the video? I guess a better question would be is there a way to utilize the iPhone sensor suite to stabilize footage in the same way go pros or iPhone work? Im thinking Sony released a post processing software that can stabilize footage in post using sensor data so i feel like this is possible.

[@hyperbolicfilms]

No idea. I think this would take Epic making a big update to their app, which seems unlikely.

[@G.I_BRO_SHOW]

Legend 👍🏼

[@hyperbolicfilms]

You can now do this much easier than I was trying in Aximmetry with their new Eye app

[@TheXarxus]

This is a very nice tutorial! But i have few questions: 1, do you need to match your real camera's lens info to the cinecamera in UE? i saw when you moving the camera the two layers of the image are not aligned. 2, i didnt see you make any offest in unreal to compensate the difference between your iphone and you real camera's physical position

[@hyperbolicfilms]

These are the places that I'm stuck, and I wonder if it's possible to get past that hurdle. Seeing others worth with the Vive tracker (like Greg Corson's great project and tutorials on his KZbin channel), there are specific steps to follow to do the alignment to get everything accurate. I don't know that it's possible to get perfect alignment of lens and offset with this iPhone method using the virtual camera. There are a lot of aspects of how the virtual camera works that are a mystery to me, such as what direction it will start when I begin my scene. It will start in the position of the Cine Camera in the scene, but not in the direction that the Cine Camera is pointed. I don't know if this is a limitation of my knowledge, or a limitation of the virtual camera and remote control app. There's a fair bit of documentation on how to use the virtual camera, but not a lot on how it works.

[@hyperbolicfilms]

Also sorry for the slow reply, I haven't been seeing notifications for new comments!

[@tmaintv]

hyperbolic films did u ever work out how to solve the direction the iPhone/iPad view starts at? Even if I set waypoints in the app my camera goes to the point but again at an annoying offset angle

[@hyperbolicfilms]

@@tmaintv No, it seems to be a limitation of the app. I think it creates a camera specifically for the virtual camera. I really think the Byowls Android app will surpass all this because they are actively developing it and adding new features. Epic hasn't done much to update the app, and the documentation is so minimal that I was guessing at a lot of things when figuring this tutorial out.

[@mariorodriguez8627]

thanks! got it working with my iphone 11 pro

[@drsjamesserra]

This is awesome, I will try this!

[@ValicsLehel]

I have done it almost the same :-) I have an iPhone 12. Connected, I can see the screen, but all I can interact if I touch the screen, the actor will jump. The camera movement is not working :-( (YET).

[@Will_Huff]

I am looking to combine live action from Camlink 4K (DLSR) and a matching virtual camera background from Unreal using the built in chromakeyer in OBS. Skipping the composure elements all together. I want to use ipad or iphone for the tracking, but I can't tell from the video if you get world translation with the iphone or just rotational movement of the camera. I.E can I move freely in the virtual environment?

[@hyperbolicfilms]

Will Huff You do get both translation and rotation with this technique. There are a number of problems with the accuracy though that I haven’t sorted out. The Unreal Remote app seems to be adding some smoothing that cant be removed, so it’s not a perfect match the way a Vive tracker can be. That is definitely a more tried and true method with far more documentation on it than this approach (which only a handful of people have ever told me they’ve tried).

[@Will_Huff]

hyperbolic films ahh ok I see. Thanks for that. I’ll prolly do a test then get to ordering the vive trackers and go thru those videos on setting up. Thanks!

[@rodcaballero6780]

thanks for this! definitely wanna give this a try.

[@johncrisoctaviano]

Is this need any Vive Tracker or HTC? Or only Plug-ins and Iphone to connect and camera track?

[@AllanRowland-i6p]

this is blowing my mind

[@ahmedsheeraz]

very nice and quick info, thanks

[@ivancuetowicq]

do you know if this would work with an ipad? I think an ipad would be better just to having a bigger display to see better

[@hyperbolicfilms]

I have only tested it with an old iPad Air, which did not work. Newer generation iPads may work.

[@ivancuetowicq]

@@hyperbolicfilms what about an ipad 10.2? also called ipad 8 generation. I see its specs and it doesnt have proximity sensor. Would this lack of sensor wont allow for walking around?

[@hyperbolicfilms]

@@ivancuetowicq The best way to check is to download the Unreal Virtual Camera app from the App store and see if you can move around in your Unreal scene. Walk around with the iPad and see what translates.

[@hyperbolicfilms]

@@ivancuetowicq The best way to check is to download the Unreal Virtual Camera app from the App store and see if you can move around in your Unreal scene. Walk around with the iPad and see what translates.

[@HuseenMurad]

Thanks for the excellent explanation

[@dedead76]

this video is old now but still impressive ! I've tried to do the same but outdoor and offline using BlendARtrack (i'm on android, i didnt tried on iphone yet) I was close to a good result but even if i turn off the stabilization of my differents cameras, it's not perfect :( I think the problem is even with IBIS Off the sensor still move a bit and make the tracking a little bit wrong... Did you tried to do an offline/outdoor tracking ? on iphone there is also CamtrackAR app. See you !

[@ildadomaniasli338]

hi! I have a question, how can i connect an intel realsense tracker camera in unreal engine? to make something like the step two. I should do that magic that you did with the Command Prompt? or what is the ideal way to do it? thanks for your help and have a nice day.

[@hyperbolicfilms]

Hi, I’m not sure how that would be done, I haven’t used realsense.

[@jibi_jibs]

do you think it is possible instead an iphone, we use android smartphone as the tracker??? please kindly answer my question, thanks.

[@hyperbolicfilms]

Search for something called ByOwls. They have an Android app for Unreal.

[@johncrisoctaviano]

I just want to clarify if only iphone that make a camera tracker . And not using any HTC tracker ?

[@hyperbolicfilms]

This uses only an iPhone. It is not nearly as stable or flexible as using a Vive tracker. There are some companies pursuing this approach, but I’m not sure if they’ve got it more stable.

[@johncrisoctaviano]

@@hyperbolicfilms so you mean is surely possible to use an Iphone to be a Camera tracker even if dont have a Vive Tracker . Because i dont have vive tracker to use, Im so amazed your content to use an Iphone to be a camera Tracker . And i want to try to my future Project by only using Iphone

[@johncrisoctaviano]

@@hyperbolicfilms do you have tutorial how to completely using an Iphone to be Camera Tracker instead of HTC / VIVE TRACKER . BECAUSE VIVE TRACKER IS SO EXPENSIVE

[@hyperbolicfilms]

@@johncrisoctaviano There is software being developed by Filmengineers to use just an iPhone and one from byOwls for Android. I don’t know whether they have been released yet.

[@ibrahimertuc24]

thanks for your share actually not easy setup

[@JerSquash]

when I connect unreal remote 2 and connect I can see the scene in the phone but get no camera control?

[@hyperbolicfilms]

I haven’t tested the new version of unreal remote, so I can’t say for sure.

[@harikrishnan-zw5rr]

Is the phone sending the gyroscopic motion ? Can we go forward, backward and sidewise in this setup !! Also with tilt and pan

[@hyperbolicfilms]

Yes, you can move in all directions and the virtual camera follows.

[@mkp8176]

Is there also a way to record the iPhone tracking data for 'later use'?

[@hyperbolicfilms]

Yes. Unreal will record the camera position you’ve done with the iPhone. So for instance you can render your background out of Unreal and then composite a cleaner version of the green screen footage onto it in a compositing software.

[@mkp8176]

@@hyperbolicfilms Thank you! What I meant was more: Filming something on set and import video footage and iPhone tracking data later on into Unreal. Or in the virtual camera in AE for that matter etc.

[@mkp8176]

@@hyperbolicfilms Just found this one: nofilmschool.com/free-software-tracks-smartphone-camera-movements This is more or less what I meant. But for AE (full version). And apparently it also works with UE too now. So question solved:). Exciting times!

[@hyperbolicfilms]

@@mkp8176 That’s a great program. I’ve had it since it came out and they constantly release updates with more functionality.

[@AdrianPruett]

Incredible work!

[@hyperbolicfilms]

Thanks a lot!

[@kehao]

and when you click this and tada, we're losing our comp... hahahahah but really cool setup!

[@ivancuetowicq]

Sorry to bother you again with so many questions, but do you get latency with the cam link? I can see at 4:25 that you move the tripod and on the iphone there is some latency.I just want to know to see if a go for the elgatocam link or for a blackmagic decklink which is like 4 times more expensive

[@hyperbolicfilms]

I never solved the latency problem with this approach. I am sure there is a way to do it within Unreal, but I did not come up with a way. From my experience doing corporate streaming for the last year and using both Blackmagic and Elgato setups, the Camlink is very reliable and I’ve never noticed much difference in the latency compared to my Blackmagic Intensity card. The $5 to $10 HDMI no name capture cards seem to offer more of a delay.

[@ivancuetowicq]

@@hyperbolicfilms oh so you are using a blackmagic decklink and getting delay anyways? perhaps might be also the computer itself, maybe a computer with faster cpu, an expensive rtx videos card, tons of memory I dont know. There is guy Matt workman from from cinematography database youtube channel, I saw a long video interview with 3 unreal engine guys and he, and he said repeatedly that the only tools he has found that there is no delay is the blackmagic ursa mini 4.6 g2 connected via sdi to a blackmagick 8k decklink card. He stressed that so many time.

[@hyperbolicfilms]

@@ivancuetowicq At the time I made this video, I was using a 5 year old tower with a Nvidia 960. I now have a 3080, so I’m hoping to get some free time soon to retry this whole thing with that new system and the latest version of Unreal to see if it’s any better now.

[@ivancuetowicq]

@@hyperbolicfilms cool man thanks, ill wait for it.

[@deeeny_]

Hey man, thanks for this amazing tutorial. Just one question, I don't know why the virtualcameragamemode option doesn't appear in the gamemode override options, even after have installed all the pluggins. Thanks very much C:

[@hyperbolicfilms]

I think things may have changed in more recent versions of Unreal. This was made in 4.23 I believe. There is a new version of the virtual camera app, but I have not been able to get Unreal LiveLink to see my phone with it.

[@deeeny_]

Thanks very much again for this tutorial!

[@josephsgameroom7845]

When i export the sequence, the rendered video or image sequence is 4 times faster than it should be.

[@CELTEKHUNTER]

Are you able to move the camera forwards and back in real space or does it only track the rotation?

[@hyperbolicfilms]

It tracks movement and rotation, but a Dutch roll will trigger the iPhone’s orientation sensors at some point.

[@cheyennethomas2498]

Excellent!, is the camera track exportable??

[@hyperbolicfilms]

I don’t know. The approach that I was taking after this was to record the green screen in camera, render my CG background in Unreal, and composite in After Effects. As of three days ago, there is an app in beta testing for Android and IOS that makes this approach moot. It apparently does the phone tracking and allows you to offset the position of the data to account for where your camera is. I’m not a beta tester unfortunately so I can’t say how good it works, but it seems like it’ll be a great tool for indie filmmakers if it works.

[@teddykillah1379]

Spoiler alert. Deckard is alive! :) Thank you for a great tutorial!

[@hyperbolicfilms]

Glad you like it!

[@thaolinh2637]

Great tutorial ! 👍👍

[@toxichatter5853]

Does it matter if the capture card is 1080p or 4k?

[@maxdanieldaiber]

Do you think it would be possible with an ipad pro for tracking AND viewing??

[@hyperbolicfilms]

I don’t think so, the Unreal Remote app doesn’t allow for any video and I don’t think iOS allows you to run two apps at once.

[@maxdanieldaiber]

@@hyperbolicfilms Thats a shame, it would be a magic combo for virtual production

[@anime4063]

Thanks alot for the great tutorial, I've been looking everywhere for this. Do you have any updates tweaking this?

[@preachtv3721]

Improved lighting, 25 fps, RTX 2080, and a better camera will fix this.

[@ivancuetowicq]

Hello. So with this method can the compositing(iphone and video camera feed) be recorded? I have tried everything and it cant just record anything. When trying with sequencer and take recorder it creates another cam that doesnt record anything cause it´s a "ghost" cam that disappears as soon as you stop playing. Can you please tell me how to record sth after do all this set up?

[@Jsfilmz]

Guys I just tested this with a person greenscreen. The playback after export will be in superspeed since UE4 only likes image sequences. Just a warning this method is not reliable at the moment. Save your money and get a decklink so you can output to a recorder. :)

[@hyperbolicfilms]

I think that has to do with the way Unreal Engine processes the two elements. It's taking more than realtime to render the CG element, but is recording the video in real time. So the final video becomes fastforward. In my later experiments, I recorded a take to get the virtual camera motion recorded, rendered out the CG element, recorded the live action in the physical camera, and then composited in After Effects. I think there's a good chance that the Byowls Android plugin will be better for this eventually because they are actively developing features still. Epic put out the Remote, and haven't added anything new since the initial release.

[@蛋炒饭-n3n]

Thanks for the video. But I encountered a problem during the operation: After connecting the UE4 with iPhone, the iPhone can display the screen but the screen cannot be rotated (the phone moves, and the screen also moves). Looking forward to your reply--from someone who doesn't speak English well Thank you

[@hyperbolicfilms]

蛋炒饭 What model of iPhone are you using? I have only successfully tested it with an iPhone 8. It should work with any model after this, but I’m not sure.

[@蛋炒饭-n3n]

@@hyperbolicfilms Thank you for your reply. I use ipad air 2. The screen can be transferred but the ipad animation surface cannot move like you did at 14:40.

[@蛋炒饭-n3n]

@@hyperbolicfilms Is it the problem that you use the ios personal certificate? I am not very clear

[@ShhheilaASMR]

can you use unreal engine to make shorts on a monetised youtube channel legally with the free version of Unreal Engine??

[@hyperbolicfilms]

As far as I know, yes. I think you can earn up to $100,000 based on the license. It says when you download it.

[@ShhheilaASMR]

@@hyperbolicfilms awesome thank you!! :) I wasn't sure!

[@CaptainSnackbar]

The sync issue is a common problem. most people i know they delay the trackers because its frame rate is far higher than the video source

[@bentheremedia3011]

So I made it to the part where you type your IP address into the app on your phone and when I hit play, I don't have control over the virtual camera. Just wondering what I'm doing wrong.

[@bentheremedia3011]

Okay I got it figured out! Unfortunately mine doesn't work as easily as yours does but I really appreciate you providing links in the description because after following the Unreal Virtual Camera link, I was able to figure it out! I have to open the Live Link tab and select my phone every time.