I don't have it anymore, but there wasn't much we did to make it work better. Maybe check your Metashape settings and/or make sure your room is well-lit, camera is not moving too fast, etc.

Thank you, I will try that@@NickRewkowski

No, it's just a basic AR app with 3 long rectangular prisms at the origin

@@NickRewkowski thanks for answering even after some years

no, it's just the basic ARFoundation app they provide

Thanks a lot!

Not sure I can do much without more info. This tutorial is for the HL2 so it probably won't work at all on a HL1, especially since the HL1 is ARM and the HL2 is x64. As for the marker part, this tutorial doesn't cover the marker part in detail, so please go to their site and the MissionAR tutorial by Microsoft to get more info. Thanks for watching!

you can use DrawGizmo or what I did was simulate a red line by placing a cylinder halfway between the eye and 10 meters in the forward vector of the eye and making the cylinder's rotation = eye rotation + 90 degrees along one of the axes (since the length of the cylinder is along the up axis). You can do all of this with the built-in vector/rotation functions like RotateAround, Quaternion.LookRotation, etc

it's realtime and it's being controlled by hand. the cams are attached to laparoscopic grippers. Each cam is used by Vuforia in independent Unity instances and a separate UE4 instance controls HoloLens & ViveTracker communication to align the grippers & cams in HL space for the wearer. More info at www.nickvr.me/ARmulticamlaparo

this is from the built-in recording software which doesn't show the HL2's lens distortions. I have other videos on the HL2 lens problems which I guess haven't been fixed yet (e.g. kzbin.info/www/bejne/sGebd6WDi7Glers)

I think I compiled & initialized from VS and you'll need to right-click the .uproject & open it in VS Code or notepad to see which UE4 version it uses

Funny enough I found the step I missed because of it. you have to switch the starting orientation AND THEN you have to flip it so the bones can go the correct way. thats a basic way of putting it but thats why

Thanks!

A couple of things could help, but some scenes are just too hard to reconstruct (e.g. lots of foliage, glass, symmetry, etc.). You can try it with better pictures, higher quality settings, manually removing pictures that have a lot of motion blur (especially if using video), etc. But if none of those work, it may just be a really difficult scene...

I'm pretty sure they're these: www.amazon.com/Camera-Glasses-1080P-Video-Memory/dp/B07KWLHJN7

@@NickRewkowski thank you

When you get so close to something that it disappears (like the walls turn grey), double click the part of the object that you want to focus on and it should become the center of rotations. If you click the solid lines on the rotation gizmo, you can also rotate specific axes instead of the arbitrary axis you get when you drag the ball part of the gizmo around. Apart from that, I'm not sure there are too many other ways to get better movement.... since Metashape isn't really an editor like Blender, I guess they didn't put much effort into it. For Blender, I like CTRL+F to fly (with scroll wheel to change speed). Also, NumpadPeriod will focus on the object you have selected

@@NickRewkowski Thnx! Yes I think what I was doing wrong: I clicked on eg a wall, then that becomes the center of rotation. But if you then rotate around 180 to view the room, the view still has the tendency to get occluded. I now click on the wall, then move away from it a little bit, and then rotate. That works better. Also making the field of view wider helps. But yes I think and option to move the ball perpendicular to the screen would be nice...

@@josvanriswick7817 You should be able to move the sphere parallel to the camera plane and perpendicular.... Zooming in (mouse wheel) will move it perpendicular to the camera plane. Holding middle mouse button or right mouse + drag moves it along the camera plane. Left click handles rotation. Hope this helps...

I don't think the VR templates ever had a C++ default option. You use C++ by just adding a new C++ class. It's probably not worth their time since most VR devs use C# or BP...

It's just the ARFoundation samples project github.com/Unity-Technologies/arfoundation-samples

@@NickRewkowski Thanks

It's using the HoloLens 2 hand tracking, which mainly uses the short-throw mode of its depth sensor for hand-tracking. I'm guessing it uses more or less the same algorithm that the Kinect used for skeletal tracking. Look at these resources for the specs: docs.microsoft.com/en-us/hololens/hololens2-hardware docs.microsoft.com/en-us/windows/mixed-reality/develop/platform-capabilities-and-apis/research-mode

In our case, we were generating synthetic data for reconstruction research that reconstructs windows/glass, so the mesh from UE4 would be the ground truth, Metashape reconstructs the non-glass parts of the scene, and our system adds in the glass. The .txt file of camera poses can be used to make a reconstruction program with MATLAB, python, etc. b/c you can skip the entire camera pose estimation part of photogrammetry. I was just giving examples of how the synthetic data can be used in the video.

I think there are more than enough videos about face-tracking around YT...

1. It sounds like you're talking about Holographic Remoting, which does work in general (it lets the HL work like a VR headset, so you can test stuff out without building the project every time) and you can find some tutorials online easily. However, the video capture on the right is done with the HL's built-in recording tool since you can't use Holographic Remoting with the HL camera, which I needed to track the blue dragon marker for calibration. The result is that this video capture is offset from what the user sees (since the HL built-in recording tool uses the front camera, which is not between the user's eyes, to represent the user viewpoint). I have a different video showing the view when I put a webcam directly behind the HL eye lens. The video on the left is directly from Unity, but it's not a view from the HoloLens, but from one of those small endoscopes on the sides. 2. The markers on the cube and the pegboard were ~1.5cm. The large blue dragon was about 7cm, I think. More details of the project and all of the demos are available on www.nickvr.me/ARmulticamlaparo.

@@NickRewkowski thank you really much!!! your information is very helpful!!

It sounds like you're talking about the environment mesh itself rather than the drawn lines I'm talking about in the video. In either case, it seems like some kind of limitation with far clip range... probably a small amount of VRAM that forces it to offload parts of the mesh (it may be dependent on the underlying environment mesh rather than the drawn lines). It just seems like that process could be done more efficiently/continuously than the sudden corrections we see in the video. I don't have an ML1 on hand (they're generally not very useful for research) but I can't imagine that it will do better than the HL2 given that the ML1 is only slightly stronger than the HL1...

@@NickRewkowski could you please expand why is not useful for research?

Off the top of my head: 1. ML is a small company with a device with a unique API. Research involving really niche, unreliable companies like this often ends up in disappointment when the company gets bought out or bankrupt. There are tons of AR/VR companies that made headsets that were already barely usable, then the company failed and the API is suddenly no longer supported as game engines update. AR/VR researchers are more hesitant to spend money on small companies like this for that reason. Since MS is a big company dedicated to expanding AR, they're a more reliable choice for research that might go on for a few years. If we are going to spend a couple thousand $$$ on an HMD, it better have guaranteed support for a few years. 2. The HL API is more standardized across other MS devices and services. MRTK works with the Windows MR headsets and is very easily compatible with Azure, Kinect, etc. It makes it much easier to work with from a developer POV. It also means you can more easily write code that works the same way between AR and VR 3. The ML doesn't have nearly as good eye or hand tracking 4. The ML hardware is significantly weaker and the sensors are not as high-res (in particular, depth) 5. The battery pack makes the ML much more annoying to work with, ergonomically 6. The ML needs special lenses; you can't wear normal glasses while using it 7. The ML uses a custom version of Android that does not necessarily have any ARCore support. Most mobile AR devs who want to use an Android device really want ARCore b/c its support is more reliable, it has tons of example code, it can track almost better than the ML even without depth sensors, etc. 8. HL2 FOV and picture quality is a lot better, which matters a lot for any kind if UI/UX research we want to do 9. The ML1 user interface is not as friendly (I constantly ran into bugs with it when I was using it ~2 years ago). The remote thing that the ML1 uses is not as easy to use as the HL2 (or even HL1) gestures. It also crashed a lot

@@NickRewkowski thanks for the info 😊

I have experience alot of the ones you mentioned myself and that explains why I found the device for 4rd of the price. And HL1 super expensive at least I have something to play with and stuff to consider for the next time.

Yeah, but it's all about managing expectations. The HL1 and HL2 were designed with rooms in mind... and this is obviously a much bigger tracking task than a room. In an enclosed space, these devices are much more reliable. I do think they could fix the tracking system to work with more general tracking tasks like ARCore, but I'm guessing that's not really their focus right now.

the only 3d objects in the demo are the hands and the red lines... I was able to more or less see the red line but yeah it definitely wasn't opaque like in this recording. I published a video a while ago that showed the difference between recording though the lens and the normal front camera recording we see here. But I can't really record through the lens while walking around outside...

@@NickRewkowski The hand tracking in the video is excellent, did you use another hand tracking logic other than the mrtk's?

No. I just downloaded the Graffiti 3D app from the MS Store for the HL2

don't think so; the indoors demos did not have this problem and I drew longer continuous lines. I also don't think that would explain the huge shift of the previous line... it seems clearly like tracking failure to me

It's not so much the device itself, but the tracking method. The HoloLens relies very heavily on its IR-based depth sensing to tracking stuff, which doesn't work for things that don't have consistent depth (like foliage) or that don't reflect light in a way that represents the surface (like glass/metal). Mobile AR like ARCore/ARKit use image (RGB) recognition + IMU + Kalman filter (similar to how Oculus Quest tracks) + temporally-smoothed depth maps, which isn't as sensitive to failure cases for depth sensors but also probably not as accurate in the HoloLens' ideal scenarios. The way the HoloLens does things is fine for the insides of rooms & closed surfaces, which is what it was meant for, but is not as great in general. I think a mix of the methods would work great for most cases.

Dark surfaces aren't as bad, I would imagine, since it uses a much higher res depth sensor (equivalent of 2 Kinect Azures embedded in it, I think). I don't recall if they were ever particularly bad on the HL2. Reflective and transparent surfaces are almost fundamentally unsolvable with video alone, I think.... ARCore uses temporal smoothing which sort of cancels out reflections, but the result is that it has fewer "apparently static" objects in the image frame to track with. We did some work in GAMMA on using audio reflections to figure out where transparent surfaces are, but we're still working on publishing it. Fortunately, I didn't have any overheating problems despite it being like 95 degrees here in MD :)

@@NickRewkowski it's great that thermals are improved on HL2 I couldn't even use HL1 in India without an AC.

No problem; there's a playlist with the 5 videos here by the way: kzbin.info/www/bejne/hGi3hnZmj9uofa8

@@NickRewkowski , THANK YOU SO MUCH!!!You deserve more views,because your videos are very valuable for the Unreal Engine community and your explanations are wonderful. I will definitely leave a like and subscribe!))

@@NickRewkowski,hello.Sorry for the late question!!! I have a problem,that my bullets are spawning,but i can't see them during the playing,and they also don't effect on the cubes.I can see that my bullets are spawning in the World Outliner only. I did the same as you,but i see the bullets only in the World Outliner. But despite of this problem i have made a line trace for my gun with the 'LeftCone',so it is working,but without the actors...I can use my line trace,but it was interesting for me to use actors in order to learn something new. And i have one more question,sorry for the disturbing...Can i use kilograms for line tracing?Because i want to have an impulse effect,when a bullet hits the object...Is it possible to do this? Thank you for you attention and sorry!!!

Sorry for the delay; I didn't see this comment. Where does the world outliner say that the bullets are (what is their transform?). It sounds like they might be spawning under the map or something. I'm not sure what you mean by "kilograms for line tracing"... line tracing has no concept of weight. You can convert from kilogram-force*seconds to Newton*seconds and put that measurement in AddImpulse (which might be in the next part of the video if not in this one)

In the descriptions: kzbin.info/www/bejne/aIu9qZ2ljaqJadU

Nothing. Just ARCore 6dof tracking in Unity. It uses phone IMU+visual tracking (I believe it falls back on IMU when there are not enough visual features in the camera image).

It was the version we were using for an AR project at the time and I was having some issues getting our project to compile on 2020. We have since ported to Unity 2020 and ARFoundation.

no. maybe someone online has one