Indoor Navigation with Augmented Reality in a hospital

[@joshuadrewlow]

Tutorial playlist in progress: kzbin.info/aero/PLcYgptwJHxUE7Q9ymWEX0zDs3SDgMarxu The paper in German is now available on ResearchGate: www.researchgate.net/publication/355402560_Navigation_with_Augmented_Reality_AR_in_a_hospital Our bachelor thesis was nominated 3rd place in category bachelor by DMEA (biggest medical informatics conference in Europe). Check out our video submission here (German): kzbin.info/www/bejne/r2mXipJoq7p-oZY

[@AgenticAI]

Didn't think that many area targets were feasible. Amazing.

[@joshuadrewlow]

It needed a bit of tweaking, because it wasn't described in the docs back then as it is now. And it is interesting to see that we solved it differently. Would you be interested in a tutorial? I actually planned long time ago to create one. Cheers

[@elijahs9295]

@@joshuadrewlow a tutorial would be great! :) I'm struggling trying to make a simple indoor nav project.

[@joshuadrewlow]

@@elijahs9295 Ok, so I will get to it! :)

[@joshuadrewlow]

@@elijahs9295 are you the student from Egypt right? If yes, you have the link for the tutorial already (sent via LinkedIn) 🤓

[@elijahs9295]

@@joshuadrewlow I'm from singapore :) and yes I have received it. Thank you!

[@Saikrishna-xi7wh]

What app used for it

[@joshuadrewlow]

We used Unity to create an Android and iOS app. For the Augmented Reality part we used the Vuforia Engine plugin for Unity.

[@MD.ISMAIL-ph4lm]

did you use any pathfinding algorithm such as A* etc.?

[@joshuadrewlow]

Yes, the Unity internal path finding system called NavMesh.

[@minhchang6009]

Hi, we're working on a similar project and tbh, we have no experience with AR before. So can I ask how difficult do you rate this project and do you think we can complete in 10 weeks? I think the area is around 200m2. Besides, it would be grateful if you could publish the code. Thanks

[@thuangg5582]

:))))

[@joshuadrewlow]

Hello Minh Chang, it was my second AR project and I would rate it a very complex project. There are many different components and factors to consider like scanning the environment, data model, positioning, navigation, occlusion, UI, and after all: good usability with navigation is hard. One thing to consider with Unity: simple UI elements like lists, buttons, etc. can take loads of time, so spend it wisely. A simple prototype should be possible in 10 weeks when you use a Matterport subscription with API access, which is expensive. Because there you can have a point cloud with size up to 400 square meter. This means you could solve positioning with one point cloud which is quite easy with Vuforia Area Target (an optimized point cloud). As soon as you have to do positioning in multiple Area Targets it gets much more complex. For prototyping I recommend to use Area Target Generator app for iOS devices with LiDAR-support, because you can create 20 Area Targets for free (disadvantage is that they are quite small, around 50 square meters), so you would have to figure out on how to do positioning with multiple Area Targets. There is loads of help in the Vuforia docs, much more then when we did the project a year ago. We used Vuforia 9.6 so it was also using old version. We decided not to publish code until now because we were working on a startup until now, but this could also change. I might publish a starter template to Unity in the future. I hope this information helps you at least to get a bit of an idea of the project. Cheers :)

[@joshuadrewlow]

@thuan lengoc May I ask you why are you so happy? :))

[@minhchang6009]

@@joshuadrewlow Thank you for sharing. We intended to use Area Target Generator app in this project and also remove some of the features such as rerouting, count down clock,... and the app only cover 1 floor of the campus. Do you think it would be more possible to complete it in 10 weeks. Btw, since we're just noobs so we might encounter many difficulties in the process of building this app, would you mind if I can contact you somewhere else more convenient such as mail?

[@joshuadrewlow]

@@minhchang6009 sorry I didn't see your comment until now! Long time past by... Have you been able to figure things out?

[@eshanverma1496]

Hi I need help to build the same kind of project, please help

[@joshuadrewlow]

Can you please be more precise with what you need help with?

[@eshanverma1496]

@@joshuadrewlow hello sir, I am doing my thesis in indoor navigation using holo lens 2, so i came across your project over KZbin and found it quite interesting, I went through your both part 1 and part2 but would request some detailed guidance can be provided. I require mentorship from you in this field and would love to give a mention of you in my thesis also. Also request if your contact can shared for better understanding. Regards Eshan

[@joshuadrewlow]

@@eshanverma1496 are you on LinkedIn? Then please contact me there. If not I will post an email here.

[@thumbnasiuminc4373]

Splendid work. How did you mask the green navigation paths around corner obstacles?

[@joshuadrewlow]

Like you have already seen the answer, I commented on your comment in the tutorial video. Cheers

[@thumbnasiuminc4373]

@@joshuadrewlow sorry about the double post! When I originally posted on this video, my comment didn’t show up until several hours later for some reason.

[@funandgameswithzayn]

Is there any workflow video of your project?

[@joshuadrewlow]

Not yet, I'm working on one! 😉 Do you work on a similar project? Cheers

[@funandgameswithzayn]

@@joshuadrewlow yes i m doing it my university

[@joshuadrewlow]

@@funandgameswithzayn Hello there, how is your progress on you project? I published today the first part of a tutorial on the channel. Check it out and let me know if I can help: kzbin.info/www/bejne/fGbaopmPgLuVrJI

[@daddymuphasa]

Pls how did you build this.... What softwares did you use to achieve this

[@joshuadrewlow]

Unity and Vuforia. Check out other videos on my channel. I created some tutorial videos. Thanks for your comment.

[@daddymuphasa]

I appreciate a billion bro

[@manpreettaak3409]

Is there we can see the coding and script?

[@joshuadrewlow]

No I haven't published any code or research paper. I hope to publish the paper somewhere soon. I'm thinking about publishing the code. But I'm not really sure if I like others just being able to copy this project. For sure I would need to remove some copyrighted content first. On the other side it would help some people out there to make some really cool projects even better than this! What project are you up to? Cheers 🤙

[@marlexpagaran2906]

@@joshuadrewlow Your project is awesome 🔥🔥🔥 We students here from the philippines want to do the same project for our school. Do you think you can publish your codes for us to be use as a guide. Or can you do some basic tutorial for us. Thankyou very much. It is for our capstone project and your work will really help us big time. We already have the 3d model of the school scanned by an iphone with lidar. Many thanks and God bless you

[@joshuadrewlow]

@@marlexpagaran2906 I am working on a tutorial yes and I will also publish some parts of code. That is already very nice that you already have scanned all the area of the school! How many square meter is it? Did you import already all the data into Unity? Cheers

[@joshuadrewlow]

@@marlexpagaran2906 Hi there, a first video of a tutorial is now available on the channel: kzbin.info/www/bejne/fGbaopmPgLuVrJI I also published our paper, link is in description, but unfortunately it is only in German. How is your project doing?

[@funandgameswithzayn]

Hii can i do it by using the 3D map of the area

[@joshuadrewlow]

How did you create your 3D map? It should somehow be compatible with the Vuforia Engine (or other AR Frameworks) to create and AR Navigation like this. We used Matterport to create the 3D data of a map, because it can be imported to Unity really easy.

[@funandgameswithzayn]

@@joshuadrewlow i create a scale model of a building on blender...

[@joshuadrewlow]

Hello there, sorry for my late response. A scale model of a building in blender cannot serve as a point cloud for detecting the device camera's location. It can be used of course as augmentation object. A point cloud has to be created with some image camera to have a visual reference during the localisation process (which is running all the time in AR application). Does this make sense to you?

[@funandgameswithzayn]

What did you use for user device tracking within the environment?

[@joshuadrewlow]

Area Targets from Vuforia Engine

[@funandgameswithzayn]

@@joshuadrewlow is there any solution for ar foundation and one more thing does it redirect the position of target device or does it rearrange the virtual part of the application.

[@funandgameswithzayn]

@@joshuadrewlow and what should i do if i had to implement it on a larger scale and recommendations?

[@joshuadrewlow]

Sorry for my late reply. KZbin doesn’t show notifications for all replies… Have you find answers to your questions? I didn’t find a way yet on how to do it with AR foundation. It does redirect the visual parts and places it relative to the real world. If you are still looking for help with Vuforia this is a great page to start: library.vuforia.com/develop-area-targets/area-targets-unity

[@shashankmali7725]

Hey! Is it possible to scan the environment with android phone. if yes which app to use?

[@joshuadrewlow]

Unfortunately this not possible no.

[@amirfatemi7361]

Hello, I scanned my environment using the iPhone using the Matterport program. Is it possible to use your environment in Unity? Or do I have to scan with a Pro scanner?

[@joshuadrewlow]

Hello Amir, awesome you tried out scanning your environment! 👏🏻 Yes you can use the environment in Unity. That's what we did for our application as well. To create a Vuforia Area Target with a Matterport-Scan that you can import into Unity, you need a Matterport account and the Matter Pack of that scan. The process is very well documented in the Vuforia docs (link below). There you will see that there is also the possibility to create a scan with an app from Vuforia. The scan with the Vuforia app can only be 5min long and has less detail in texture, but it makes trying out something much easier, because you don't need a Matterport account. Does that help you or did you already find a solution? All the best! library.vuforia.com/features/environments/area-targets/scanning-guide-for-area-targets-using-matterport.html

[@amirfatemi7361]

@@joshuadrewlow Hi, thank you for replying. I did it according to the Vuforia document where it said you can scan through Matterport. And we used the iPhone X to scan through the Matterport app. In the document, Vuforia did not explain whether the environment scanned by the iPhone X in Unity could be used for indoor navigation.Thanks for your guidance

[@joshuadrewlow]

@@amirfatemi7361 Of course it can be used. Any scanned environment with any tool can be used in Unity for navigation, as long as you can create a Vuforia Area Target. To prepare environment for navigation you have to create a NavMesh, implement how to show navigation and then present it all in some way to the user.

[@Fw8888]

Ho big is the photogrammetry file size for the complete hospital building? Btw can it fragmented into smaller size and become asset bundle? If yes, can the map only be downloaded asynchronously when user go from point A to B in multiple area targets? It would be very funny if user downloads the app and then being told to download another 2.4GB of map data...

[@joshuadrewlow]

Hi there :) In our prototype we covered only 690 m² from the approximately 14'000 m² walkable area in the hospital. The iOS App is 447 MB big and for Android 147 MB. We didn't have time to implement dynamically loading assets, but with Unity and Vuforia it would be possible yes. In the Android app we left away some media, for example 100 MB videos and reduced the meshes for occlusion to the minimum. The point cloud of Vuforia of about 450 m² is only 37 MB big and in the environment it has quite a lot of details. So loading additional "maps" on the go would not be a big problem. A bigger problem are the meshes that are used for occlusion. Currently the mesh of the scans are used which have very great detail and therefore take a lot of file size. We found that occlusion with simple manually created walls is sufficient for the most walkable areas (especially corridors). How additional data will be downloaded when supporting big areas is a good question. Probably common used areas will be part of the installation and rare areas will be downloaded as a user chooses a destination. Does that answer your question? Do you have another idea how to handle file size the best?

[@Fw8888]

@@joshuadrewlow Thank you so much for your unselfish act of your knowledge sharing. Here's my found out to add to your reply: Yes mesh occlusion is working well, noticeably better than Unity MARS at the moment. My guess is because it's based on the cloud point mesh (which Unity hasn't released this feature yet). On downloading additional "map", It can be achieved by using Asset Bundle, and add them into the MultiAreas. When you have many areas in chunks, In the hierarchy of MultiArea, Vuforia says the "un-used" map can be automatically put as "inactive", but not enough information at the moment in their doc, so far I just tested on my bedroom and living room, on camera it works; in editor for debug, can't really see anything or gizmo in it unless I think I have more areas, but I'm going to extend the areas until my condo corridor, way to my parking lot and update here. One question: how do you display the navigation link (line renderer) in runtime flawlessly? Mine it works well in editor but unfortunately not well in runtime because of the world centre mode is camera kinda stuff? Do you create line renderer instance in runtime before drawing it? p.s: Vuforia has documentation to add target only on object and image in runtime, cross finger that extra areas can be added directly by using similar methods. In Vuforia docs, one app can have scanned area max files up to 255 (correct me if the number is wrong, but I read it somewhere in the doc). p.s: I can see you're using the camera scan to get the point cloud, I wonder it will double the device performance job to recognise the space, because you have extra millions of dots? Regardless it definitely gives more accurate detail. I'm using iPad LiDAR scan and it works well too. Notice that the room light in scanned area is also one of the big factors to let the VuforiaBehavior to recognise the space easily, probably because of the iPad LiDAR doesn't get enough point cloud info.

[@joshuadrewlow]

​@@Fw8888 Thanks for your detailed answer. The point cloud of with the Matterport Pro2 camera is quite big yes. But after generating an Area Target the size is reduced drastically. I guess Vuforia is filtering out points that are not relevant for recognition. In our app we used Area Targets generated with both pro camera and iPad LiDAR. For recognition it worked about the same good. About the line renderer: As far as I remember we calculate and draw the line 10 times a second (it works well for updating while user is walking, but it is not the best solution, you will see why). The problem you are experiencing is due to the World Center Mode, yes. When using multiple Area Targets it should be 'Device' (I think it says somewhere in the docs). You experience this problem probably because the Area Targets are placed in the Unity scene where it is recognised in the real environment, relative to the AR starting point. So looking at x,y the position of the recognised Area Target is always different and even changes, when Vuforia has to do correction because of drifting. So by drawing it 10 times a second it always is correct according to the current position and Area Target recognition, but it is still not 100% flawless, because it also has "hick ups", you see it a bit in the video. I think the best would be to have the Line Renderer child of the current Area Target, so it is always at the correct place. Then update the drawn line only to the next point of the renderer (from the user) and the whole line, when there is a bigger change in direction. I used this script here: gamedev.stackexchange.com/questions/67839/is-there-a-way-to-display-navmesh-agent-path-in-unity/86255#86255 And this is the method to tell Unity to calculate the path: docs.unity3d.com/ScriptReference/AI.NavMesh.CalculatePath.html Oh and very important for multiple Area Targets: you have to use NavMeshComponents (we told Vuforia they should add this to their docs, because the standard NavMesh is static in the scene and when having World Center Mode = Device the Area Targets are dynamically placed in the Scene as explained above): docs.unity3d.com/Manual/NavMesh-BuildingComponents.html Let me know if this helps and if you have any other ideas/questions. Unfortunately I didn't have more time yet to spend on the project for improvements, etc.. Did you manage to extend your "map"? Cheers

[@Fw8888]

@@joshuadrewlow Thank you so much and appreciate for your information. I get a lot of knowledge to achieve it. I'm here to update my work: Multi area is working even better with the new script posted on the Vuforia site. I have achieved drawing navigation line in multi areas, the only problem is: In multi area, av meshhaveNav Mesh and regenerate the nav path when I see the Nav Mesh is drifting or broken, but what is the best way to trigger re-bake nav mesh function? I really try my best to avoid baking the nav mesh continuously. In your video, is your map scanned in full piece, so you have your Nav Mesh also in full piece? or, you also rebake and re-calculate the path continuously? X'D Thank you.

[@joshuadrewlow]

@@Fw8888 Why do you rebake the Nav Mesh? The areas should more or less stay the same, aren't they? You should use NavMesh from NavMeshComponents to have a NavMesh that can be child of current localized AreaTarget. This helps to always have the NavMesh at the right place. We have a full piece yes and we re-calculate path continuously.

[@eeww4510]

Hi, I like your video and I want to try build AR outdoor navigation but I have no idea how to do this. Can you teach me how to build?

[@joshuadrewlow]

Hi there, thanks for dropping by. For outdoor you have to consider a couple of different factors: light plays a big role in point cloud creation and GPS data. You probably should check out other AR frameworks, like the one recently published from Google and there are also others that are specialised for outdoors. Vuforia doesn't recommend to use Area Targets outdoors. You should also narrow down the use cases: AR at specific spots or a bigger area? How big are the areas? etc.

[@eeww4510]

@@joshuadrewlow Thank for your replied!! May I know how to get the GPS data?

[@joshuadrewlow]

You have to search the docs of the framework you will use. And otherwise Google is your friend.

[@eeww4510]

​@@joshuadrewlow Thank you so much!! Do you why I run my unity ar project my phone and laptop just show black screen. I don't know how fix this 😢 my unity version is 2022.1.20f1

[@eeww4510]

erm... How to get GPS data. I Google searched but still can't get it :(

[@JackSparrow-dz2tu]

Man! This is good work and complex as well. In my University Fest, I want to create similar AR navigation throughout the campus for different locations available on Menu. After all this time, can u suggest less expensive or smart way to achieve it?

[@joshuadrewlow]

You could try use the product of a company that have produced something similar to my concept (they were much faster and professional than I): www.treedis.com/

[@muhammedshaaran5971]

Is there any tutorial available?

[@joshuadrewlow]

We described the process in our bachelor thesis report. But it is not like a step by step tutorial. Would you be interested in a tutorial video? (I was thinking about making one but haven't had the time & energy yet) On the Vuforia Webpage you can already find a lot of information on how to proceed: library.vuforia.com/features/environments/area-targets/area-targets-in-unity.html

[@JakeSteinerman]

Yup kzbin.info/www/bejne/kIqXZ3aNo8RnnJY

[@joshuadrewlow]

I will start this week working on a explanation series. First I will create an overview video to explain the key components to it. If you need immediate help, in the comment section of this video another user asked some detailed questions about the video. There you might already find lots of hints on how to do things. Cheers, and I hope to see you around

[@joshuadrewlow]

Sorry I totally didn't continue working on that... It's so hard to keep up with new family life, life after studies, etc. Currently I'm working on an Indie-Game so probably this tutorial will never happen 🙈

[@joshuadrewlow]

Hi Muhammed, are you still working on an indoor navigation project? I finally managed to publish the first part of a tutorial on the channel: kzbin.info/www/bejne/fGbaopmPgLuVrJI

[@hafizvj7939]

Are you using any vuforia liecence ?

[@joshuadrewlow]

Hello Hafiz. Yes I used a vuforia license for developers which is free. The downside is you cannot publish to app stores with it. But for trying out and creating a prototype I can very well recommend it! Did you already try out the vuforia engine? Cheers!

[@hafizvj7939]

@@joshuadrewlow I'm also learning with that. I'm working on a similar project, and I'm getting too many doubts. Can you share your project files??? It will be really helpfull.

[@joshuadrewlow]

@@hafizvj7939 What are your doubts or problems about? I don't want to share project files because we invested more than 900 hours in this project. If you seek any architectural explanations there is another comment thread with much detailed information. I hope you understand. Cheers. If you are interested I might create a tutorial how to get started with the basics for navigation.

[@joshuadrewlow]

@@hafizvj7939 Hi Hafiz, until now we published the paper for public download on ResearchGate (link in description) and today I published a first video of a tutorial for this project: kzbin.info/www/bejne/fGbaopmPgLuVrJI The more I think about it, I probably am going to publish the code as well, but I'm not sure yet when and in what format. Cheers and happy coding 🤓

[@hafizvj7939]

@@joshuadrewlow Thanks a lot for the tutorial, had been waiting too long for this. ❤️❤️

[@topanime7563]

This looks good, I'm also trying to do this one in my capstone project

[@joshuadrewlow]

That sounds nice! What is a capstone project? How is it going with the development? Cheers and good coding ;)

[@anthonyuribearce3290]

How can I contact you to develop an application using this technology for a hospital in Peru.

[@joshuadrewlow]

Hi there, thanks for reaching out! Are you on LinkedIn? www.linkedin.com/in/joshua-drewlow/

[@JakeSteinerman]

Amazing use of Area Targets!

[@joshuadrewlow]

Thanks 😊 What was the tutorial you wanted to link to? (because it looks like you deleted the comment again...) 🤔

[@JakeSteinerman]

@@joshuadrewlow kzbin.info/www/bejne/kIqXZ3aNo8RnnJY Will share as a top level comment again. Not sure why it got deleted

[@joshuadrewlow]

@@JakeSteinerman I have seen your video already, well done 👍🏻 Although it doesn't have detailed information about NavMesh, how you show the way, occlusion problems you might run into with Matterport-Scans, localisation with multiple Area Targets, etc. For an overview the video is great, but it is rather a tutorial for Area Targets than a tutorial for indoor navigation. I am currently working on a tutorial, but it will take a while to publish it, due to new child, post bachelor thesis fatigue, etc. 😆 But so cool that you are reaching out from Vuforia/PTC You have a great team over there! (Had great conversations over mail with the support 👍🏻👍🏻 )

[@joshuadrewlow]

@@JakeSteinerman Why the comment is not visible I have no idea... it still shows up the notification but I can't even see it in KZbin Studio. Weird 🤔

[@JakeSteinerman]

@@joshuadrewlow We just published some documentation on authoring a NavMesh with an area target over on our Developer Portal library.vuforia.com/features/environments/area-targets/area-targets-in-unity/using-unity-navmesh-for-navigation-with-area-targets.html

[@cocgaming5682]

from where i dowload this aap

[@joshuadrewlow]

Thanks for your interest. The app is only beta and makes only sense to use inside the Bern University of Applied Sciences building in Biel/Bienne, Switzerland because this is where we scanned the environment of the prototype. Get a plane ticket and have a visit! 😆😉

[@anemone5264]

Excuse me, I'm interested in your works. Do you have the paper or sort of that explain the work? Kindly I need to study for my research. Thank you in advance.

[@joshuadrewlow]

Hello there, thank you very much for your interest. I'm sorry for the late response. Our paper was not published in a journal or so, but you can download it from ResearchGate (but I have to warn you, it is in German): www.researchgate.net/publication/355402560_Navigation_with_Augmented_Reality_AR_in_a_hospital Feel free to contact me for any other questions. Cheers :)

[@anemone5264]

@@joshuadrewlow Great! It's alright, it's really helpful, thanks 🙏

[@joshuadrewlow]

@@anemone5264 Hopefully it's helpful 🤓 Do you work on a similar project?

[@anemone5264]

@@joshuadrewlow I'm kinda researching AR for daily use, I'm looking for some method of markerless works, and area target is one of them. Your project is awesome, kindly this help me to learn it because I can't afford the equipments for now.

[@joshuadrewlow]

@@anemone5264 Ah interesting. Indeed markerless AR is really cool! This opens so many use cases for AR. Last year Facebook bought a really interesting startup called Scape Technologies that was scanning entire cities. Google of course has loads of data from Google Street View cameras, but this kind of data is so valuable no company really wants to share it. Luckily we had the means at my college to try out the different things. A really kind CEO even let me lend their camera for free to do the scans, because it was a student project. Have you tried out the Vuforia Area Target App for iPhone or iPad with LiDAR? (I'm sure you can find a friend who let you use it for one or two hours.) 10 Area Targets are included in 1 Vuforia developer license.

[@clifflin7149]

Great work! Is Vuforia area target free?

[@joshuadrewlow]

With a developer license you can create and use 10 Area Targets for free. The down side: you cannot publish to App Store with this license.

[@adiwijayakusuma2288]

Amazing !

[@joshuadrewlow]

You are outstanding! 🎉