Mixed Reality Vtubing: Hand Cams and Green Screens and VR, OH MY

Hiya, I’m Iiji, and I’m a professional 3D modeler and illustrator, and today I wanted to chat a little about mixed-reality because I feel like it's a really great medium that we could be getting more use out of as vtubers!
I hope you find this video helpful, and if there are any other vtubing or art concepts you'd like to see me discuss, leave me a comment!
If you'd like to see more of my content:
- Twitter: / squiiji_3d
- Tiktok: / squiiji_3d
- Tumblr: / squiiji-3d
- Booth: squiiji.booth.pm/
- VGen Commissions: vgen.co/squiiji_3d
- Throne: throne.com/squiiji_3d
Credits:
Free to use music: Noiseless World noiselessworld.net/en
Background, Editing, and 3D Model: Me!

[@wikiprion]

that face replacement software would be fantastic for people who already like cosplaying and would be able to make their models outfit irl!!!

[@squiiji3D]

Oo you're right, it'd be really great for cosplay!

[@Vieus]

I believe there is so much potential in mixed reality vtubing that has so much room for us to explore. I'm currently working on a video inspired by Azumalim where she goes to real places and animates in herself so it looks like she was there. I think this is an area that so few have tried and it looks really cool! :) The ideas in your video have helped me understand some more options so thank you

[@squiiji3D]

Ooo that's such a cool concept, I'll have to look into her! I'm always hunting for more people who are doing cool, outside the box things with vtubing! And I'm glad I could help!

[@zumuvtuber]

Mixed reality definitely has a lot of potential, but only for those who are actually doing things outside of their computers (cooking, sewing, painting, playing an instrument, etc). Otherwise I don't really see a point in doing it (unless the person really wants to, of course!)

[@squiiji3D]

I mean yea that's true, but I feel like vtubers could benefit from branching out from the computer-centered activities mostly being done rn!

[@apolloisnotashirt]

OniGiri is a great example of this. I hope this gains traction though, would love to see other vtubers interact irl and not just in 3D

[@M3D_ALIV3LOX]

This is why I'm mostly interested in 3d vtubers cuz there's so much fun stream ideas that can be done here that can be mixed with irl and 3d (if they're not comfortable to show themselves fully). Also, that face tracking app looks really dope cuz for a second there, I really thought you were just wearing a helmet made out of your character xD

[@squiiji3D]

Lmao a helmet of my model would be CRAZY! But I agree, I feel like 3D models offer more flexibility for things like this, for animations, mocap, and so much more!

[@gone23398]

As someone who is still in the planning process of becoming a content creator with a vtuber model, this was very helpful! I didn’t want to be limited on what I could do so having this kind of info for future planning has definitely increased the possibilities of content to make! Thank you very much!

[@squiiji3D]

Ah I'm glad I could help!

[@apolloisnotashirt]

wish you luck on your content creator journey! one of the things I notice is that there is a perfect mix of passion and corporate strategizing. AKA post shit you love but make sure your staying on some form of trend or niche to garner those views! But this is just my observation as a content consumer so I might be wrong! again wish you luck!

[@gone23398]

@@apolloisnotashirt appreciate the feedback, thank you! I’ll take it into consideration and add it to my notes!

[@rooley123]

Your enthusiasm captivated me for the entire video! Really got those cogs turning and demonstrated everything nicely

[@squiiji3D]

Ah thank you so much!! 🧡

[@HandleToBeDetermined]

Omega Sisters have been doing this sort of mixed reality VODs for about a year now. From what I can see, the way they do it feels closer to a virtual animegao kigurumi. Bigger corpos seem to employ some sorta object tracking for 3D lives similar to what Disney Research Hub has been researching about 6 years ago. Mixed reality is definitely an interesting area for development with tons of potential!

[@squiiji3D]

Oo I'll have to look into Omega Sisters!

[@chaosfire321]

I think VR might be something really helpful for this in the near future. Standalone headsets like the Quest 3 are having a bit of a renaissance with passthrough MR letting you merge digital characters and props with real environments in increasingly diverse ways. Brad Lynch/SadlyItsBradly has a ton of showcases on twitter, one of which had him wearing his VRchat avatar and hanging out with his VRC friends in his real room.

[@Kreativedragonling]

Yes! This! More mixed reality VTubers please!!! I actually plan on doing more hand cam stuff cause I'm primarily a traditional artist and have loads of cool craft stuff, but cameras, lights, and other set up things have been EXPENSIVE. My advice would be to not immediately buy all of the equipment needed for mixed reality stuff until you've really thought about ehat you're going to do with it. If you don't actually like doing handcams or mixed reality, do not force yourself to do this even if it looks cool. Investing in a simple camera and lighting set up was worth it for me because some of my biggest skills and are non-digital so I can get a lot of use out of my set up. I also don't really care if people see my hands or arms, though I'm probably gonna get some gloves at some point to avoid any weirdos But please let me know if you do mixed reality or handcam stuff, VTuber or not, cause I wanna hear everyone's stories!

[@LenoraVT]

Yeah! I feel this so much! (mostly b/c I suck so bad at digital art it's embarrassing) But also because I cook/bake, and that's a super applicable mixed reality skill that I want to make the most of... But since I'm still so new, as in not yet debuted, I honestly still haven't figured everything out, so it's gonna be just hand cams from me for now!

[@neonmoonVT]

As someone who wants to do irl type streams, virtual face looks like a really good idea! But I feel like it may be a bit harder for me to pull off since I still have to wait a bit to get a model and my avatar is a furry 😅

[@squiiji3D]

Honestly if you wear a turtleneck it REALLY resolves the transition between your virtual head and neck, so don't give up! Good luck!

[@TadpoleArtz]

As the tech gets better and more accessible this is going to be a cool space

[@pawsvtuber]

Really enjoyed the video and breakdowns shown. Thanks!

[@DungeonSlime]

I do hand stuff, but I think having hand stuff that could replace my hands with slimey tendrils would be next level.

[@_lei_c_8138]

How did i not see this video sooner 💀 anyways, seeing a real person body with your characters face literally broke my brain cuz I still hadn't processed what the title and thumbnail meant XD yeah I'm slow 😅 great video as usual! Can't wait for the next! 😆

[@sparksparkle]

I know nothing abt vtubers, and I was watching another one of your earlier videos that was recommended to me and I had a thought "do vtubers ever show their hands or such? (To paint or cook or just show stuff like plushies etc) and I went into your channel only to find you had already posted this video with the answer! Neat !!

[@TheVicarstownSentinel]

Oh now this is cool; yeah I remember Kson doing a bit of that, and ofc we have Giri's cooking streams, and ofc sometimes Vtubers do handcam streams for like, unboxing things or making arts and crafts, etc etc. But this is really enlightening. I hope more Vtubers try playing around with mixed reality stuff in future. Also, first time viewer here! This was in my recommendations and surprisingly not one of those broken algorithm suggestions (lately the machine keeps trying to feed me sub-1000 view videos that are unrelated or only adjacently related to my interests- there's one in there rn about the Meta Quest w 16 views) and I have NO idea why YT keeps doing this, but it's great that I still have plenty of GOOD recommendations with a decent view count that are actually relevant to my interests. Anyhoo- rant aside, your model design is so cool and very adorable!

[@NinetyUnderScore]

ive found this technique called using a "camera" to capture my movement and then using the "video" that it recorded to directly be sent to the stream so that the viewers can visualize the 3d physical world around me projected onto a 2d "video"

[@peachenelle]

ANOTHER BANGER The way you present all this info is so interesting (on top of being actually helpful and informative). Like I'm no vtuber, heck I don't even follow the space. The most I've done is create pngtubers for others, but you bet I'm 100% here and listening whenever a new Iiji vid is up 😤

[@squiiji3D]

Ahh thank you so much! I feel honored that you're so invested despite not being a vtuber yourself!!

[@dwwynn]

Glad i stumbled on your videos recently, your videos give off a very fun vibe

[@stevoisiak]

The real time face replacement app seems like such a neat idea! Thanks for introducing me to these ideas I'd never considered

[@darkshadowsx5949]

i just like watching the model move around with the cute voice. it was more entertaining than the video subject. i did play around with Vtuber maker on steam but my modeling skills are a quite rusty and basic so making a quality custom model would take tremendous effort.

[@FranKapella]

Looking forward to your tutorial on Virtual Face! Been dabbling with half my body playing guitar and my Vroid Model as my face. Stopped since I felt really limited to making my body and face match up and I couldn't move as much, but after seeing you demonstrate it I'm hyped! I'm hoping to use it for live show recordings in the future!

[@Ottertuber]

I've been sharing this around and lots of people are getting inspired to do this now!

[@apolloisnotashirt]

This has always been an idea in mind too! Like corpo vtubers could be vlogging themselves irl, but then edit on top their avatars. Would love to see TakaMori shenanigans in japan!

[@radianttakanuva8388]

I'm learning 3d modeling with the intention of mixed reality content. There's obviously a lot of stuff that still needs to be modeled, but "Hey, make this helmet/character head model in blender" is a much simpler proposition than modeling a full body rigged model and tracking it appropriately

[@CoffeeIsDead]

off topic compared to the video, but I absolutely adore your vtuber, inkweed mushrooms are my absolute favorite mushroom and your color pallete reminded me of a mobile game oc I created that had the exact same colors. ❤❤

[@mikejett2733]

I agree with u this is pretty cool hope many vtubers come around

[@blacksheep1924]

I would love to hear more about this.

[@GabrielOlivares123]

So many different things to do as a vtuber, interesting stuff! 😁👌

[@vombine9321]

so cool to see! thank you for showing :))

[@NoitHub]

your video is pretty informative, thank you 😄

[@zephyrartch]

Whoa, this is the first time I've seen Mixed Reality Vtubing! Interesting!

[@AttacRacc]

Any other vtubers here who also go on cam? I enjoy being on cam and having that connection to my community that they know my face. Sometimes for fun I will pop my cam on while my model is up! It's fun and shows how tracking works in conjunction with my face expressions.

[@Chichi-ov5pd]

I would love to try this! I’ve created a model and trying to rig it (well) haha so any tutorials on this would be a godsend 🎉

[@MenheraMaji]

Mixed reality vtubing is at the limits of what can call vtubing. It is vtubing and it's not at the same time depends on how you stretch the concept. For me personally it is contextual, Onigiri is the best example of using the medium correctly. If I had an example on how not to use mixed reality I would say SquChan.

[@inkhexes]

i'd love to see people throw concerts with the mixed reality!!!!

[@dissonanceparadiddle]

That's really cool

[@NihongoWakannai]

3:30 oh god no, I hate looking at stuff like this because it just goes into uncanny valley and makes me uncomfortable. If people are gonna wear gloves, I'd much rather just see regular ones not trying to look like you're wearing fake skin. Also, I like the goopy drip physics on your model

[@mythrando]

The wheels are definitely turning now!

[@unicornclutter]

hmm I keep coming back to this, I wonder if it would be possible to prerecord a video just normally, then take the video and use it as imput for your models face tracker to animate your expressions and such, cut off the head and have that follow where your head is in the video. Idk would prob be insane amounts of effort but I know that with my bright blue hair it could be possible to use some sort of program to pin one moving impage on top of another. Mostly trying to think of workarounds since the face replacement software looks so cool but am too poor for an iphone and the idea of doing house rennovation videos with an avatar is very appealing now I finally have the ball rolling on making my house cute

[@Incinda]

Great video Iji

[@squiiji3D]

Thank you!!

[@kosoricluka]

If you really don't want your face to be revealed just put on a green screen like balaclava and map onto that. Worst case you'll give some children nightmares for a few days if it stops working.

[@squiiji3D]

That could work if you don't want/need face tracking!

[@Eibarwoman]

I see some potential in it, but have no idea if I can utilize it properly as I'm not fond of the 3D model technology.

[@alixsprallix]

great video

[@nanobotkemo]

Pretty cool you learned how to make a model yourself. I'm curious how long it took to learn how to do that? Along with the animation? And did you have a sort of background anywhere close to that?

[@squiiji3D]

I spent about 5 years off and on learning to use blender, for people w/ more time to put into it, they could prolly get to where I am in a year or two haha and for animation I've been learning over the last year. I'm getting a master's in fine art rn not in a subject that's related to 3D modeling, but I'd say the art skills I have definitely helped some (:

[@SilentVinyl]

Looks kinda hilarious

[@JohnnyYeTaecanUktena]

Mixed reality has potential, mostly if you wanna overlay your models head on yours so you don't have to design clothes you can just wear whatever. Other than that it is just a untapped market as it could also allow for cheaper models you know just the head and or neck part with the head fully rigged if the person don't wanna draw it themselves.

[@squiiji3D]

Ohhh I hadn't considered people getting head-only models as a result of something like this, that'd be an interesting change

[@JohnnyYeTaecanUktena]

@@squiiji3D There has been some experiments of overlaying an entire model over a persons body for mixed reality. In-fact i think Calli did something similar when she appeared on Trash Taste a few times as she was there in person But major downside to head only is skin color as you are removing a lot of the fantasy aspect meaning the models have to be designed off of your actual skin tones. So unless you are getting concealer or other stuff you can't be a big green Orc And even then people don't like wearing makeup which is why they put on a model in the first place. Wait why am i de selling this for me? Anyways it really is a cool idea for those getting into it or for those that wanna be humanoid anyways

[@squiiji3D]

Ohhh I hadn't even thought of that w/ Cali! You're right! Virtual Face has a color changing option for the neck, but it's a paid feature that I don't have haha but I also found that wearing a turtleneck REALLY helps bridge that weird kind of skin tone clash that happens in the neck.

[@JohnnyYeTaecanUktena]

@@squiiji3D Well if you are going to wear full long sleeves, gloves, and a turtle neck might wanna crank up the AC to max. It might get hot fast lol

[@chickenbreast7016]

Mixed reality would be so good for cosplay

[@LenoraVT]

I love mixed reality because it offers a way to make non-digital skills showcaseable (not a word ik) as a vtuber. However, I will admit that my immediate thought as a Vtuber with a non-humanoid model was how SILLY it would look to have hands or arms sticking out of a body like mine (which is just a floating green dewdrop shaped thing with a hat and some hair). (≧▽≦)

[@Sadeyo1]

I'm all game for the evolutionary growth for vtubers. Get out of the mold and make a name for yourself and etc. I saw the stream of the woman working on her AC a while back but I definitely see the 2.5D being helpful. Shame it's locked behind an iphone and their apps. I can easily hop over and afford it but I'm used to the simplicity and accessibility of an android after all these years. Thanks for sharing an extra ideas.

[@squiiji3D]

Everything's locked on iphone rn because androids don't have IR or Lidar yet and quality camera based tracking is VERY reliant on those factors RIP

[@ShiaNeko]

I don't really see why it would be the future of VTubing, but if it's something you want to do, then you should do it!

[@Deteng]

Shoot If I can find a good way to do it I would love to do a knife throwing stream that would just be basically a just chatting while I throw my knifes

[@joeojeda4651]

This would be great for fitness vtubing lol

[@helloww_]

very cool

[@patrickpatrick3935]

ight idgaf about vtubing but this is cool asf

[@BleedingRayne]

I want to be a mixed reality KZbinr. Not exactly a vtuber relying heavily on mixed reality to hide my identity

[@thewhisperingsky]

amazing :D

[@vxicepickxv]

Augmented Reality is the technical term. It's just kind of backwards from regular AR.

[@squiiji3D]

I called it mixed reality because that's what I've been hearing in association with the Quest 3 that just came out, but you're probably right that AR is likely the same as mixed reality!

[@Hualiama]

amazing video, i'll def be looking into some of these options. only issue i see here is the hp printer in the background of some of the shots lol

[@squiiji3D]

LMAO not the printer

[@KynneloVyskenon]

4:47 but what happens if there are more than one people on the screen in VirtualFace? can the tracking go from one person to another? does it track both people? does it black out if it detects more than one person?

[@squiiji3D]

I actually had that happen! My friend who helped me record came into the frame and it blacked the screen out!

[@InueShizaki]

I want to be a V-tuber and a real life KZbinr/streaming at the same time Like, switching between the 2 I dont really have a problem showing my face, but at the moment I don't really have the confidence for doing that But I plan to do that sometime Like, starting as a full v-tuber and them do a face reveal and then do mixed reality content

[@Anim8ionManiac]

what happens with something like virtual face, when there is more than one face on frame, does it track multiple targets or does the tracking stop?

[@squiiji3D]

From what I can tell the tracking stops and goes to the black out screen

[@Speed001]

Just a remember that Hololive was a tech company first. And this reminds me of AR. Perhaps combining that tech with this idea would provide decent results for 3D model hand cams. Maybe not.

[@TenshiKisai]

IMO, it's "not" the future, it's just a fun way to interact with real world objects in a controlled environment (eg a room especially prepared for it, not outside.) - Giri's cooking streams would be impossible to do without a fancy 3D model and a 3D set, and 3D food in all the various stages of being cooked. It's just not feasible (look at FFXV and various other Japanese games that have cooking minigames that put a lot of detail into the food.) Half of food is presentation, and that's just not possible with virtual food gaining physics properties to the food texture. Even "cooking" simulator games take too many shortcuts and don't really approximate a cooking experience. - Virtual Face would be too distracting and highly dependent on camera quality and location to use in a live performance. You could also accidently reveal other things about yourself (eg mirrors, windows, reflective surfaces, etc) plus if the camera malfunctions it could accidently reveal things, like text messages and 2FA tokens from the device. The benefit of ARKit, over -everything- else is that you can send JUST the mocap data to the recording/streaming software so that there is zero possibility of revealing anything. Webcams carry the risk that your face/body will be revealed by accident by software that accesses the camera (eg discord, zoom, OBS), things like SlimeVR/Mocopi require a phone, but don't necessarily require a specific phone, to send body movement. Full hand and body capture kits transfer the 3D motion data in real time. However performing with 3D mocap suit/trackers is limited to very short times due to battery life, limited range, and even pain when a HMD is involved. Camera-based mocap of any type is even more limited due to viewing angles of the camera. We have a ways to go before we have an "ARKit invisibility suit" where the software in the phone digitally erases the actor using depth cameras and draws the 3D model in place. It's just most of the alternatives to ARKit, exist only in the robotics space, and getting them into the Virtual Performer space is presently still challenging.

[@semiengima]

This is interesting and i feel like this will only make Vtubing even more interesting

[@squiiji3D]

I agree!

[@UhhOlivia]

Girl I got hand eczema too, hand streams are a no go T o T

[@squiiji3D]

It's so rough I'm sorry you're stuck suffering with me too RIP

[@casualrayla1893]

Another negative with Virtual Face that I don't think a lot of people think about is that having a model with long hair will always look goofy. If you let the hair be as long as it normally is on your model it'll look like you're a squid (since it covers your body). If you decide to use the in app cropping you'll have a very sharp, weird cutoff line. If possible I suggest porting a version of your character with a ponytail or bun to the app to avoid having to deal with it ^v^

[@squiiji3D]

Oo that's a very fair point, I haven't tried it with a model that has hair that reaches the shoulders, although I feel like it wouldn't be so bad if there was hair that falls in front of the shoulders on the model. I'll have to do some testing, but you're probably right that a short hair or hair up alternative model to a long hair model might be better

[@casualrayla1893]

@@squiiji3D That's exactly what I'm thinking! Luckily I just got a ponytail hairstyle with my newer 3D model, so I'm gonna see if it works better! Maybe someday, as tech gets smarter, it'll be able to cut out whatever hair it tracks behind the shoulders?

[@Crystalkitzoku]

Virtual face sounds so fun to use, sadly I'm an android user

[@lanceolson5988]

Full immersion or XR guitar playthroughs. Uhhhhhh, I need to think about this…

[@Bobogdan258]

4:45 What would happen if it accidentally detects another face and your model jumps over to that face, would it leave you exposed?

[@squiiji3D]

My friend who helped me film accidentally helped test that, when she came in view it went to "no face detected" and blacked out, but that would be something good to probably test out more

[@MenheraMaji]

Hey squiiji have you watched the Mushroom song by Enna Alouette ? any opinion ? that could be a good idea for a video, vtuber music videos and how they should be done.

[@squiiji3D]

I have seen it! I think the style of it is really fun and unique, and Enna has a lovely voice. I'd have to poll some friends as I'm not super well-versed on standards for motion graphics like you see in vtuber music videos, but it could be a cool concept! Thank you for the suggestion!

[@OnosakaKoike]

what software youre using for the entire video? i mean while youre talking and moving

[@squiiji3D]

For the non-mixed reality I use a couple things! ifacialmocap for my face tracking, a leapmotion for my hand tracking, and vNyan to put it all together and stream all of it into OBS!

[@OnosakaKoike]

@@squiiji3D too bad i dont have an iphone

[@catinacafe7105]

not even a vtuber but this was still interesting.

[@YourRyeBread]

Your own setup is really what I’m looking for so if you've shared what you use anywhere please help direct me to it 🙏🏽 (like I can move a lot but still "in place" ya know? gotta account for the adhd and am willing to save money) oh and I never said it cuz I forgot to but I love your model!

[@purrplepaws_]

Is there any alternatives of virtual face for android?

[@squiiji3D]

Sadly not that I know of ): Most rely on the IR and Lidar features iphones have but androids lack

[@rinlidia]

Did you have second channel earliel? I think I know your voice :)))

[@rinlidia]

Anyway, I'm really happy to found your content again...

[@Pattyrikku]

Are you still on vgen? I don't know if my message ever got to you. ;_;

[@damnyiffers]

problem being this augmented reality stuff has a tendency to disappear randomly for a few frames. The risk to avtubers anonymity is to great Edit* Nevermind she covered this

[@AYZNESR34]

a smol mushroom girl ahmm... IDK can eat to her maybe lol

[@73greenz]

what if... I tried mixed reality vtubing..? :O

[@squiiji3D]

The way I still haven't rigged shmorp for vtubing yet whoops

[@geher4073]

Bros muscular

[@PandamoniumBruu]

4:53 NOSE REVEAL! (?)

[@squiiji3D]

LMAO yea guess you aren't safe from getting so close to the camera that your model clips, what can ya do 🤷🏻‍♀️

[@sookendestroy1]

Finally... vtubers can go outside and touch grass

[@costapasmore3440]

☀️ 'Promo SM'

[@Solizeus]

That is... (censured), i personally don't like it, it gets that uncanny valley feeling with a head that is not yours but it moves, nope, some people might like it though, if it is the entire body then it would be ok

[@RosurinVTOfficial]

I personally notna fan of it. It looks too odd and semi uncanny valley

[@hightouch3986]

I'm going to puke.

[@mr-x7689]

Would be nice to see V-tubers stopp using chest physics. It adds nothing to the streams, other than soft porn for 13 year olds to drool at.

[@Spwookie]

Check Mashiro spooky exploration streams. They are great. Nice vid!

[@squiiji3D]

Oo I'll check that out okay! And thank you, I'm glad you liked it!