OpenKneeboard: Mixed Reality Sim Done Right!

[@seattime4075]

Brilliant, did not know this worked in MSFS. I know what I will be doing this afternoon. Thanks for sharing.

[@andy.puempel]

Thanks for this video, Russ. The speed at which this technology is improving is incredible. Appreciate your information on this and always look forward to your work. Great presentation style.

[@jamessoutar4354]

Great video Russ! Your XP12 looks awesome. Would be great to see a video on how you’ve set that up to look so good.

[@JamesTyranski]

I've used OpenKneeboard, but I had no idea it could do mixed reality pass through. Awesome tool.

[@gagagu01]

You have to use Virtual Desktop and activate chroma key passthrough

[@whiterooksme]

I am ever so much more impressed with mixed reality. Thanks for sharing.

[@RogerBorgesGrilo]

Thank you so so much for finding a solution for mixed reality in XP12!!

[@kristianjorg418]

Thanks! Looks like the absolutely best solution yet. Just in time for MSFS 2024 as well! :) I've been into MR with MSFS 2020 since I saw your first video on the subject. A real eye opener that was!

[@seattime4075]

Just tried this after watching the how too video and it works brilliantly with no noticeable load. I used 0R, 100G, 51B for the colour (a type of green), which for some reason works perfectly and is not so jaring on the eyes.

[@rbarlow]

@@seattime4075 interesting. I was afraid blue and green might produce a color evident in natural scenery( trees oceans etc) and felt pink was likely safe. It does accentuate.the passthrough edge. I experiment.

[@seattime4075]

@@rbarlow So did I, but maybe its just my setup? No bleed that I can see...kzbin.info/www/bejne/r17afJqEppKqnpI

[@TBooneFisher6931]

Thanks for a very nice video. I am a retired Eagle ATR Capt with lots of real sim time. IMHO when really good hand tracking is finally added(allowed) in Xplane/MSFS which we know is definitely possible, there will be no need for physical panels, etc. and sims like these will be taken seriously by real pilots. When folks like Austin Meyer( owner/developer) of Xplane have enough incentive($$$) to do this it will get done.😉😉😉😉

[@rbarlow]

@@TBooneFisher6931 the only negative with hand tracking is lack of tactile feedback but I have seen gloves that simulate touch but so far they are pretty clunky. I have seen use of dummy panels that only provide knobs and buttons and are mapped to the VR world so the hand tracking has something to touch and feel.

[@SimHangerFS]

Leading the way as always, thank you 😊

[@svoigt2622]

Excellent video again sir.

[@HomesickforAlaska]

Excellent content. I can always count on you to do the "Heavy lifting" for me and find what I need.

[@vitmedia]

I just invested in a Quest 3 on the strength of your video, Russ. As always, informative and inspiring and thanks for making the effort. I'm amazed at how the pass through video appears locked to the underlying real world view - or are you keeping your head very, very still lol?! I assumed the view would keep moving to wherever your head was pointed. I'm now much more inspired to get into VR as I can use my Honeycomb and arduino powered instrument panel (and Knobster, of course!). Thanks again.

[@rbarlow]

@@vitmedia to get the best passthrough video be sure to have plenty of lighting and turn any LCD screens down to a very low level so they don’t bloom. You will need a wifi6 router which I connected to my existing home network via Ethernet. Buy Virtual Desktop from the Meta store for your Quest3 and install the VD Streamer on your desktop Sim computer.

[@vitmedia]

@@rbarlow yep all done! Good advice about the screen brightness. Cheers 🙂

[@andy70d35]

Russ, another amazing video.

[@Ron-f8v]

Hi Russ thanks for all the great flight sim videos. I have followed all the steps in the Create with Tech and all is good. I have two touch screens and using Air Manager for PFD and MFD which work fine but when I enter mixed reality (Quest 3 with windows 11 MSFS 2020] and the pass through window comes up both my touch screens freeze!!! With all your experience I was wondering whether you could offer any explanation for this and point me in the right direction. Keep producing the videos which I am sure helps so many of your followers. Hope to hear from you soon. Regards Ron

[@DaFrE81]

Hey Mr Barlow, even though i don't play flight simulators, i keep watching and enjoying your video's. I play Euro and American truck simulator, mostly in triple screen, once in a while i play them in VR Quest 2. I have a diy trucking dashboard. Would this MR tool OpenKneeboard also work with those trucking simulator? So i can play them in VR but still see my diy dashboard? Keep up the good work ;-)

[@rbarlow]

As long as you have Virtual Desktop and it runs as an OpenXR VR game it should work. Of course the passthrough would be black & White. Let me know if you try it and if it works or not.

[@rcbruce841]

hey Russ, follow you religiously, great recommendations. question on open kneeboard. almost have it working well. one issue i seem not to be able to solve. using 2 wimxit 15"mon, 2 kneeboards showing, airmanager/knobster. Jump into vr and the 2 monitors with G-1000 do not function. any ideas. Are you not using airmanager and 2 popouts

[@WaltKingLee]

Great video as usual Russ! I have finally successfully set up the new MS24 Air Manager, and Open Kneeboard . One problem I am wondering about a workaround...When I go Mixed Reality, my G1000 popouts on my touchscreens disappear. I would love to know how to prevent this. Any advice would be appreciated.

[@VRFlightSimGuy]

Very nice find Russ, thankyou for sharing this with us all. Can't believe this is free too! I wonder if this might work with other mixed reality headsets. Best wishes.

[@rbarlow]

@@VRFlightSimGuy it would require the use of Virtual Desktop’s Chroma Key for Mixed Reality but the kneeboard function displaying documents should work with any OpenXR VR game. I saw a moving map that can be displayed with it. If you explore this it might be interesting content. I haven’t had much time to create content lately as my 95 year old mom is having serious health challenges and I have lost most of my free time attending to her needs.

[@VRFlightSimGuy]

@@rbarlow My best wishes to your Mother Russ and thankyou for the reply. Take care.

[@Back2TheBike]

Naive question from one who doesn't own a VR headset but develops scenery for XP12 ... Hi Russ, presumably this means you need to have built a hard, physical cockpit for each aircraft? So your example is a 172, but say I also wanted to fly my dad's Lancaster bomber, would I need a separate physical replica for that too?

[@SiwertJonasson]

Hi Russ! What size (inches) is the touchscreen you use in the video? Is there a particular touchscreen you recommend?

[@vitmedia]

Russ, may I ask who made the parking brake, gear and flaps levers? I'm gradually trying to retire my Bravo throttle as I have everything else covered with other gear. These are the last pieces I need for my (almost) 172 instrument panel. Thanks.

[@rbarlow]

Those were DIY using an Arduino Nano and running with Air Manager. You could also use a Leo Bodnar board or a board from Desktop Aviator. If you want read built modules try Desktop Pilot . If money is no object the best options are switcho products from VirtualFly. Etsy has some sim products too.

[@blaisepoemape3596]

Hey Russ! Do you have a video where you share your settings to have the closest experience to real flying when using Xplane11? Thank you for your help! Great videos 👏🏻

[@EmesiS]

Thanks for the insight Russ. MR is getting better every day. Have to ask, why were you using the touch screen flaps instead of the pyshical one in your cockpit?🤔 Cheers!

[@rbarlow]

@@EmesiS I was rushed and didn’t have time to figure out which Arduino pins were attached to which switches. I use Air Manager to control those. I figured it didn’t really matter to demo the MR with OpenKneeboard. You might have noticed I haven’t posted in a couple months. My 95 year old mom has been quite sick and we have been busy helping her but I really wanted to get this video out. I was trying to squeeze it in.

[@EmesiS]

@@rbarlow Understandable. Wishing your mum the best.

[@melb00m21]

Another great video, I always enjoy your thorough and thoughtful content. I am currently on a Varjo Aero and was thinking about the Quest 3. I do not really need Mixed Reality passthrough as shown in this video (I don’t have a home cockpit), but would like to have the option to switch between VR and passthrough quickly. From your experience, as you seem to own both, how much of a visual downgrade is the Quest when coming from an Aero? Is it something you would recommend considering?

[@rbarlow]

@@melb00m21 you may be disappointed as I would describe the resolution as only slightly better than the HP G2 was but with an enormous sweet spot. Reading the small text on the G1000 can require a lean in. Also the Aero is well built compared to the Q3 but at $500 Meta offers quite a value. If you want to try decent MR now, I would consider it as an additional headset and not a replacement.

[@melb00m21]

@@rbarlow Thank you for sharing your perspective. I owned the Reverb G2 before the Aero, so if the Q3 is on a similar level, then I'll probably pass.

[@rbarlow]

@@melb00m21 yes I understand. The Q3 has the best lenses I have seen with edge to edge clarity so it seems way better than the G2 but still it is only about 25 ppd vs 35 for the Aero

[@thewarthogproject]

Great work!

[@bkpickell]

Do you have to use a contrasting color for the mask or could you use black? I'm afraid I would be too distracted with the halo.

[@Viking8888]

Thank you so much for showing this to us Russ. I now have a Quest 3 and was thinking about getting the other program you featured that does passthrough areas (I can't remember what it's called, but it costs $10). I'm going to give openkneeboard a shot first,

[@rbarlow]

SimXR is okay but requires lots of work each time you use it…good plan to try this first.

[@claudiufarcas]

Curious to see how it works with motion platforms like DOF reality.

[@shaneastwood6364]

Hey russ. Ive got the crystal light. Is there anyway to make the pass through view any clearer. Kindvof looks like a 70s tv with no tv arial

[@stvcolwill]

is the pass through even color on the crystal light? I'm still waiting on mine to ship (been a month so far since ordering).

[@robrights5520]

Another informative video on sim evolution possibilities ! I noticed the Xplane video was displayed at full 1920 width, but the MSFS video had a decreased width. Is this an actual visual difference between sims, or just a video capture/render difference . Thanks

[@rbarlow]

@@robrights5520 no the Quest3 camera capture can be set 1920x 1080( landscape or portrait) or 1024 x 1024 ( which displays the entire field of view in the headset). I just wanted to convey that you can glance with your eyes and see a much wider vertical field of view. The only way to capture mixed reality is through the headset because the pass-through is not shown on the mirror image on the computer monitor attached to the computer running the Sim.

[@likes2fly]

Great video. Do you know of a work around using vr and pop out panels of instruments like the garmin 530 and g1000? They freeze now as soon as you turn on VR. I use realsimgear.

[@rbarlow]

Yes that is a shortcoming of MSFS in VR. XP12 doesn’t have a similar problem so Asobo could fix it if desired

[@Backcountrysimpilot]

Is the quest 3 good enough on a 3080 12gb card to read all the guages easily on msfs? I was looking at going quest 3 for mixed reality or going pimax crystal lite for full vr.

[@MI325A]

Hi, Russ. I'm presuming you're using the Air Manager as I see the Knobster. I'm wondering have you also tried this interesting program with Ron's Altimeter Motives panels? Thank you. Have a great safe day. -- Matthew

[@rbarlow]

Well not this program but a similar one. kzbin.info/www/bejne/bp7NgYuXgatnaLMsi=PaisFzZv9824uTVz

[@MI325A]

@@rbarlow Hi, Russ. I think you may misunderstood me. I have one of his panels. Not exactly one you shared in the link, I call it the "mini" panel where it is just the 6-pack. Still all in all, a great panel and awesome how it works. So my question is, will OpenKnee Board work with this? Also, I'm looking to get the Meta3, but the version of 3s. It's more in my budget but seems to be similar except just a little less high definition in pixels. We will have to see, especially with possibility of it being reduced for Black Friday. Thank you for taking time to reply Enjoy your channel. -- Matthew

[@rbarlow]

@@MI325AI strongly encourage you to go with the Quest3 vs, the Quest3s. It is worth the extra money. If you intend to try MR the passthrough clarity in the 3s will disappoint you. The 3s has fresnel lenses with a smaller sweet spot while the 3 has some of the best pancake lenses I have seen with edge to edge clarity.

[@MI325A]

@@rbarlow Awesome feedback, Russ. Thank you and have a great and safe week ahead. - Matthew

[@rbarlow]

@@MI325A Sure. And regarding using your mini panel with the Quest 3 just be sure to reduce the brightness on the LCD panel to a low level and it will look better on the passthrough. Goold Luck!

[@yokoh691]

reality mixer wroked ok for msf with city stutters and great for dcs and warthunder, stoked to have something to try with msf and no stutter finally , i hope lol

[@gascrich]

I may be late in my comment, but… I just installed OpenKneeBoard and use it with Air Manager. Works very well, but one problem. The pop out window of a glass instrument (GNS530 or G1000) gives a static image or a black image. I don’t know if anyone can provide a solution…

[@rbarlow]

Yes that is an Asobo/MS issue. The window doesn’t get updated in VR or when the FS window loses focus. It should work in X-Plane since the display is streamed to the AM instrument I think. If not in XP you can run your AM panel on a networked second computer and stream the displays over the network

[@ammarsadaka320]

Thanks for sharing

[@Buddy308]

Gee, Professor Russ, if you mentioned any significant shortcoming of this app. I missed it. Is there anything, in your opinion, that needs further development, or is it actually near-perfect already? I've been putting off VR until, hopefully, Pimax adds high-quality passthrough cameras to the Crystal Light, but if this app is fully ready for prime time, instead of pinning everything on something that might never even happen, it's probably time to throw for a Quest 3.

[@rbarlow]

@@Buddy308 it could have some optical or tracker way to perfectly align the passthrough areas with the physical hardware like Varjo does. The UI could be streamlined for MR only to simplify it. I’d love to get rid of the magenta line between Passthrough and VR but that is a Virtual Desktop issue. Many people complain about the passthrough quality but good bright light and dimmed LCD screens are my tricks. TBH this is just so much better than all the alternatives these all seem minor and not worth mentioning in an initial review.

[@737sst]

Great find, Russ. Now if somebody could come up with an idea to get rid of the pink line between VR and the real cockpit that would be the icing on the cake. BTW, does this also work with P3D?

[@rbarlow]

Good call! I need to experiment on different colors or darker shades for the Chroma Key mask. I chose R 255, G 0, B 255 since that is an extreme color that seldom is seen in everyday life and nature. There may be less bright options that aren’t so obvious. Of course we don’t want passthrough areas appearing in the scenery.

[@ArtFlyer01]

Thanks Russ, subscribed : )

[@adilahmed5072]

hey man, im new to the world of VR flight simming. could you tell me what are the specs of your pc? cheers

[@Serpico360]

I've been using Openkneeboard and while I like it, It doesn't seem to work in MSFS using DX12. I've only gotten it working in DX11 and in DX11 games. Everytime I've used it in MSFS using DX12, it crashes the sim. I believe it's a problem with DX12 and Pop up windows. Are you using MSFS in DX11?

[@AndyTulenko]

Does VR reduce the overall amount of computer pwer required to run it well, or does it eat up even more CPU time?

[@rbarlow]

@@AndyTulenko it all comes down to pixels. Quest 3 has 2064×2208 per eye so both eyes have over 9 million pixels. A 4K monitor (3840 x 2160) has less at just over 8 million. If your computer can run MSFS on a full resolution 4K monitor it should run almost as well in VR. But yes VR needs a strong CPU and GPU.

[@AndyTulenko]

@@rbarlow thanx!

[@joshinspace3903]

Hi! Question: when running x plane or msfs... Do either of these mirror the vr display to the monitor and if so can you turn it off? I noticed that i can run some vr games that don't mirror the display. But any that do, crash miserably. I have a quest 3 and a gaming laptop. U fortunately the laptop is a 2021 asus tuf... Which means it struggles to run VR even though it has a 3060 rtx. (Because of dual graphics card conflicts)

[@rbarlow]

Not sure but they are present for me. Try Googling it. It is surprising how many things I find that way.

[@RogerBorgesGrilo]

Does this work in MSFS 2024?

[@AviNerd]

Can you hook up two headsets to one machine and have a dual pilot cockpit?

[@ErikFigueroa]

VR is rendered by having two displays that are positioned from the viewpoint of two spots that are as wide apart as your eyes. In fact, it's like running two versions of the game at the same time, which is why VR is taxing on the GPU. Running a second VR headset on the same GPU would not be very practical, because it would be like rendering four versions of the game at once. In essence you can have dual VR in the same cockpit but each player would need to be running the sim on their own PC. A second player would need to connect to the first player's sim using a program like Your Controls, which requires a stable internet connection if you're both going to be using the same network. This would allow for mixed play, since you can both use VR headsets or one could use VR and the other a 2D monitor.

[@AviNerd]

@@ErikFigueroa So you’re saying there’s a chance, get a second GPU and I should be fine! Kingpin is coming back soon. The recommended settings only calls for 12gb of vRam. These graphics cards are becoming more and more powerful within the next iteration it’s actually surprising. Just from my 3090 (should have waited as they released the 3090 ti) but the 4090 was like 50% better (something that I heard. Besides the point I suppose, the point that I’m trying to make is it would be very useful if they were able to make it work as a flight instructor could sit in with a student, or a pilot could sit in with their family or friends and show the what if scenarios boosting safety by a lot. This is of course as long as they have a dual cockpit sim. Everyone who uses VR knows how useful it is and sure you could use monitors but with what Russ Barlow just put together just sent a lightbulb off and I’m thinking of how the safety problem in the GA community could be helped with that implementation. Will it fix it 100%? No. Flying is as expensive as it is however, with the right investments it can reduce the excess flights or being abused by pilots just looking to build hours (only heard of this haven’t experienced it).

[@RandamTheFurious]

How do you stabilise your vr videos for recordings

[@rbarlow]

@@RandamTheFurious not exactly sure what you mean but for MSFS turn off headshake or g effects and got the Quest camera settings be sure image stabilization is OFF …

[@podwor]

Can I use it with link cable ?

[@rbarlow]

@@podwor no Virtual Desktop only works via WiFi and Wifi6 is best. I keep the link attached as it slows the battery drain and it is the best way to get captured video off the headset

[@podwor]

@@rbarlow , thanks… will be great a software that works with link cable.

[@georgefaulk2528]

@@podwor He just said it doesn't work with the link cable. Can you read?

[@sigbauer9782]

Doesn't quest need a fb account? no effin way if it does.

[@rbarlow]

@@sigbauer9782 not anymore

[@georgefaulk2528]

You need a Quest account not FB.

[@sigbauer9782]

@@georgefaulk2528 oh right- fb makes oculus.

[@sigbauer9782]

@@georgefaulk2528 Quest is made by meta, aka fb

[@trailblazer7108]

We love you Russ

[@VRFlightSimGuy]

Btw.. I'll share this on my Discord if that's OK.

[@rbarlow]

Of course

[@JohnVanderbeck]

MR cockpits remain out of reach for most simmers no matter how good the software, because by its nature you need to essentially have a while replacement cockpit in the real world.

[@Fromega001]

I, ok this seem nice BUT i can find where is the MIXED realityu here, the only think i see is just VR

[@ColeRees]

His entire dashboard is real, everything outside of that is the VR headset.