【Vtuber Tutorial】How to use HANA_tool blendshapes for high quality Vtuber face tracking on VUP

What to download:
Unity (Personal use) : store.unity.com/#plans-indivi...
VUP (Steam) : store.steampowered.com/app/12...
UniVRM (unity plugin) : www.github.com/vrm-c/UniVRM/r...
(i downloaded UniVRM-0.69.1_0ece.unitypackage )
HANA_tool (unity plugin - about 7 € ) : kuniyan.booth.pm/items/2437978
Tutorial i used with extra steps you might need : www.hinzka.hatenablog.com/ent...
__________________________________________________________________________
Hello there! first tutorial even made so please give me any feedback to improve for next one!
if you have any questions or difficulties about the tutorial itself, let me know in comments or in my discord!
sorry if my english is bad i only speak ringlish :)
_____________________________________________________________________
Follow me on twitch and hope to see you there during my live streams! you can find the schedule in my twitch bio !
---------------------------------------------------------------
Erina's Discord: / discord
Webtoon : www.webtoons.com/en/challenge...
Twitter : / erinavtuber​​​
Twitch : / erinachandesu​​​
Instagram : / erina.vtuber​​​
Facebook : / erinachandesu​​

[@jmqt13]

New Vtuber here and I have to say, your tutorial tremendously helped me figure HANA_TOOL out! Thank you for making this video!

[@Kishiboi]

Your seriously underrated in my opinion, Your model looks great, your steps through the tutorial, including zooming on on important stuff wich"A LOT of people DON.T do", was great, and the tutorial helped me out a ton. THANK YOOU !

[@ErinaVtuber]

HHHAA thanks for taking the time to compliment me 😫 i very much appreciate it and I'm happy i helped ❤️

[@oct0nami]

Your tutorial was perfectly thought out, well paced, and extremely thorough. Thank you so much for posting!

[@ErinaVtuber]

Very happy to be helpful ☺️

[@elliemouse545]

Best, quickest, easiest to use tutorial on this subject. Thank you

[@ErinaVtuber]

Very welcome ☺️

[@activemotionpictures]

Marvelous! I really like it how you zoom into the options. Thanks! Great video!

[@ErinaVtuber]

I'm glad to hear that 🥰 i was very new to tutorials lol

[@naru_kami4149]

Thank you for this concise and simple tutorial 👍😁

[@jarretticeearthguard8614]

This is a pretty good tutorial Erina! Well done!

[@ErinaVtuber]

thanks Ice! :D

[@ryder1508]

A note for people who might run into this issue as well. For whatever reason when I import my VRM into the program I had to first create a folder in assets (from File Explorer) and drag the VRM there. Only then did it convert automatically to a prefab, if I just dragged the VRM file directly into the assets window I was met with "NotImplementedException: The method or operation is not implemented."

[@AnupetMunuth]

Thank you so much for this! I had no idea what I was doing and now I'm spending just a few minutes to get it done!

[@ErinaVtuber]

Glad to be of use :)

[@soulzu]

this is so easy and simple thank you!!

[@sydneybeaa]

Thank you! So excited to find something that doesn’t need an iPhone

[@petit-pedestre2334]

Hi, sorry to bother you! Just wanted to say that this is by far the best video on the subject, so thank you for the quick, to-the-point and easy to understand tutorial! I've checked your channel it's really great, keep up the amazing job! You got yourself a new sub!

[@ErinaVtuber]

thanks for taking your time writing such sweet feedback, it's very appreciated 🥰 glad i helped

[@petit-pedestre2334]

@@ErinaVtuber I should be the one thanking you! Seeing your great Vtuber model inspired me to make some of my own, and thanks to this tutorial I'll combine that to Blender to take it to the next level! So thanks again for the motivation, you're the best ☺️

[@Sillybean_Rela]

Thanks for making this! It’s very helpful :)

[@ErinaVtuber]

Glad to hear that :D !

[@KaruHart]

thanks for sharing! cant wait to try it out

[@IraChYT]

Good lord that's some fire BGM 👌

[@heraldo_u_u2901]

Your explanation is very good, Mama!

[@ErinaVtuber]

thanks heraldo :D

[@JanePlayz]

I'm your 400 sub! :) Also thank you very helpful

[@ErinaVtuber]

KYAA THANKS FOR SUBBING ! glad to help!

[@anzui]

thank you !!!!! best tutorials!

[@HoshiiHiro_Vt]

thank u very much its the best tutorial

[@ErinaVtuber]

Seems Like the latest Version of HANA_tool doesn't work with old Unity versions, so try to update it if HANA doesn't work. For info, In the video i'm using HANA_ tool v.5e and Unity 2020.3.0f1.

[@Cosmic_Commentary]

Actually it's the UniVRM that interferes with it working! I had the pre-release version .78 instead of the officail release, 0.66!

[@anon12352]

Can you make a comparission vid? Like how the model looks and tracks side by side before these tweaks? Thanks a lot.

[@caeshura]

Thank you!!

[@Twiggefer]

hello! Thank you for this tutorial ^^ I have an issue I've been trying to fix. When I go to Clip Builder - I click on my model and then Clip build but I get an error saying: Calling apply or revert methods on an object which is not a prefab instance is not supported I also have options for "Perfect Sync" and "Sranipal" above my Vroid and Other options? I haven't seen it in any video tutorials but it shows up on mine lol

[@casualdoomer8277]

Erina i love watching your streams. Didnt know you did tutorials! How are you good at everything?

[@ErinaVtuber]

haha im not good at everything, just able to do a little bit of many things :)

[@manuelmateo3392]

It'll never be that good as it is.

[@angelicstrawberry666]

So, I'm having trouble matching up the blendshapes on vup because some of them won't even load right. Idk what to do

[@MultiLegocrazy]

Does VSeeFace or VUP support poi shaders? Also do I still need to use their own dynamic bones or could I still use the original dynamic bones script as seen in VRChat models?

[@SabielaYT]

Merci t'es douce

[@SNKGM]

Geat tutorial. Simple but very direct and complete. So, any idea about what we can do since VUP is now death?

[@ErinaVtuber]

I transitioned to Warudo on stream! It's a still new so a bit hard to setup but the results are similar to VUP!

[@mauricioinigo7750]

the eye movements (look up, look down, etc) aren't generating properly, any ideas why that is happening?

[@biibeebs]

Hey Erina, I was hoping you could help! I tried using the HANA tool, but no matter which option I choose (be it female or male blendshapes), Unity tells me "The number of vertices in the file does not match the number of vertices in the avatar, it is likely not to work properly" and doesn't let me go further than this. I have also installed UniVRM as instructed. I'm currently using Unity 2020.3.26f1 Personal, HANA version 3.0.3 EN, and my VRoid model was made in the Stable Version (not beta). I fortunately have a VRoid beta model, and doing the steps, it added just fine without this issue. Is there anyway to do this on a Stable Version model or would I have to look for another program?

[@ErinaVtuber]

hello! when exporting your VRM model, please untick everything and keep everything to 0, also make sure you updated your HANA tool version! i made a really quick tutorial for you and since i had the same problem, i figured other people might need it too!

[@Hidden_G3ms]

By any change do you have a step by step video on how to download the univrm from that webside?

[@bunny_noctis]

I have one vrm model I use but only have body as shape key and when I try to press read on the Hana Tool I got "the number of vertices does not match"

[@Whimmery]

I am getting the same issue, supposedly there is explanations saying: "Make sure you don't have "Delete Transparent Meshes" checked when exporting from Vroid. That will cause this error." Though I do not sure where to even find that option when exporting

[@feet-chan594]

I wonder what would happen if I drop my homework folder in my vtuber model??

[@Amyzinq]

Question, can these blendshapes only be used with VUP, or does it work with other Capturing Programs - like VSeeFace, VMagicMirror? o: Great tutorial - didn't even know custom blendshape templates existed!

[@ErinaVtuber]

as far as i know, only VUP allowed me to bind specific face movements to the blendshapes, but they should work with other programs too since the blendshapes are part of the model after you add them, so they should be available everywhere

[@VelvetCake423]

i think for vseeface yooud need an iphone. this looks pretty simmilar to ARkit which also uses iphone tracking

[@leehanyang]

Hi, thank you for this tutorial! But I have a problem where the eye gaze doesn't work when I apply the hanatool. Before using the hanatool, iris moved in the direction of my gaze but now it doesn't.. what would be the problem?

[@ErinaVtuber]

Hi! Are you using VUP? If so, make sure : - you didn't turn on "look at camera" in the broadcast settings - all of your rigging related to eyes are set to "NULL"

[@Dipperdog_RPGirl]

Hi Erina. I followed your steps but the facial expressions are not active on my model in VUP. any advice?

[@ErinaVtuber]

Hello! Problem might come from unity, so suggest you try the version 2018.4.36f1. when you install the blenshapes from Hana, make sure to check the inspector to verify that the new blend shapes are there. Also don't forget to clipbuild! If this doesn't work, let me know !

[@Dipperdog_RPGirl]

@@ErinaVtuber thank you!

[@anongirlfrompoland]

can you make a tutorial on how to do this with mmd pmx models?

[@Ravenride13]

I can not for the life of me gt this to work, and Im doing every step right :c once it goes into VUP, half of the expressions dont have a matching movement... for example, I have no options to raise eyelids or make my eyes look left or right.

[@ErinaVtuber]

When you get to the inspector tab (min 2:44) do you see all those new blend shapes like on my screen?

[@Ravenride13]

Mine actually has more than yours, and had some before hand if that makes a difference

[@ErinaVtuber]

@@Ravenride13 i see, i'll check out on the VUP discord, i'll let you know if i find anything :)

[@LinRARVtuber]

Hi erina! I for some reason don't show clip builder in my hana_tool unity package. Can you help? Thank you!!

[@ErinaVtuber]

Try using Unity version 2018.4.36f1 instead! i heard it's more compatible with HANA! if it still doesnt work i'll look into it more!

[@uliu1612]

I know this tutorial is super old but, VUP is no longer on steam, is there any other place I could get it??

[@kawaiizombiegal]

the hana tool doesnt show up in the tool bar

[@Dlx_Violet]

I'm not too sure on how to download univrm. Like get it in my files so I can import the custom package

[@ErinaVtuber]

Oh, to download uniVRM, click on the link on description, then scroll down to assets then download the "source code zip) file. Don't forget to unzip it ofc before importing it like in 0:50

[@calebkirkman8739]

Hello, when I go to Booth it blocks my payment access to purchase Hana Tool. I am from the United States, and my card was issued here it that the issue?

[@ErinaVtuber]

Hey, not sure if it's the card. Iirc i had a bug message once too but it was just a bug after all and I found my download link in my booth purchases. If you still buy, I'd suggest using PayPal cause it might be a conversion problem that your bank doesn't support (which could be solved by contacting them). Hope you find a solution :)

[@murosaki1236]

Hello. I did everything as you said but when i click hana tool>reader there is only PerfectSync_VRoid_v1_0_0_Female and PerfectSync_VRoid_v1_0_0_Male. Which means its the first version. I could not find the reason. How can I fix this? (first version does not have eye movement and I paid for the latest... so :( (I contacted niku屋 and waiting for the response)

[@murosaki1236]

I also tried with other unity and uniVRM versions combined. No hopes... Pls help :(

[@santonopoulou]

When I try to import Hana tool it doesn’t show up in the bar like yours. VRM does but not hana tool. Any ideas or advice? I’m so exhausted trying to get this to work :/

[@Cosmic_Commentary]

same for me

[@ErinaVtuber]

@@Cosmic_Commentary are u guys using unity version 2020.3.0f1 ? im afraid they updated Hana tool since i'm using version 5e and they are selling version 9b now, so maybe use the latest version of Unity and not the one i'm using in the video

[@iamzuru8769]

Viewers: Maybe one day

[@blueberrylu5964]

I'm having some trouble with the clipbuild step, i have an error that says "calling apply or revert methods on an object which is not part of a prefab instance is not supported", what does this mean?

[@ErinaVtuber]

Did you create your avatar on vroid? I had a problem with polygons when reduced them on vroid

[@blueberrylu5964]

@@ErinaVtuber yeah, its from vroid

[@MAMAvsGOD]

@@ErinaVtuber So should we not reduce polygons? Because mine is saying the number of vertices in the file does not match the number in the Avatar.

[@livipup]

Have you made or do you know of a demonstration comparing different models using Hana Tool to show if it works well for a wide variety of face shapes?

[@ErinaVtuber]

I'm not sure what you mean by variety of faces, i just know Hana works on vroid models only, not blender and other custom models

[@livipup]

@@ErinaVtuber What I mean is different designs. For example, some people have rounder eyes on their models while others go for more defined shapes. Another example, some people might make the mouths on their models bigger or smaller. I can imagine that with great variations there may be some cases where things are a bit awkward-looking when using pre-determined morphs on the model. The degree to which that may be a problem depends on how Hana Tool works. If the changes are relative and take into account variations in the models that the tool is working with then strange results should be outliers. If instead the changes are always identical then strange results may be common. If Hana Tool includes a way to adjust the blendshapes in Unity or if it is possible to return the models to Blender after using Hana Tool without losing the changes it made then it should be easy to correct and strange results. I hope I did a good job clearing up what I meant.

[@ErinaVtuber]

@@livipup i see now. If I remember correctly there's in unity a day to calibrate how big the expression is gonna be but i didn't use it because they were looking good no matter what model i edited. Now something I can say is that in VUP you can also adjust how large the mouth eyes and eyebrows move when performing the expression. So i use it when I think a certain expression is too exaggerated or not expressive enough.

[@livipup]

@@ErinaVtuber That's good to know :) I'm going to try to make my own blendshapes, but if I'm not satisfied with the results I'll consider using Hana Tool instead.

[@volbeatowns]

Can I use this method if i make a model in Blender? Mine do not have the face option :/

[@ErinaVtuber]

I think Hana tool is made for vroid models only :(

[@volbeatowns]

@@ErinaVtuberoh thats okay I went ahead and made them all by hand haha thanks for the amazing tutorial ^^

[@towakona]

Does the hana tool improve the blinking of the model too?

[@ErinaVtuber]

Depends on the kind of improvement you're looking for, but i don't think the blinking is changes in VseeFace, it does change in VUP though, the one I use, since you can manually change the bindings in there

[@LeahKinsXx]

Thank you so much for your video!! Not sure what's happening, but I'm getting an issue where the eye tracking isn't working properly. EyeLookInLeft and EyeLookInRight are moving the eyelid rather than the eye itself? Not sure what's going on there. Can't find a way to fix it :(

[@ErinaVtuber]

HI! do you mean in VUP? if so, in VUP rigging window, eyeballs dont move left or right, but during face tracking they should move correctly. :) let me know if i got you wrong

[@LeahKinsXx]

@@ErinaVtuber thanks for responding so fast! Yes, in VUP. I can see that my eyes are being tracked whenever I click broadcast because you can see it on their little avatar, but it's not actually tracking my eyes on my own vrm. When I was doing the rigging, the eyes don't move at all, just the eyelids! Super weird.

[@ErinaVtuber]

@@LeahKinsXx alright then, i have three possible solutions : - in VUP, replace all eye tracking related rigs with "NULL", dont use anything and see if it works. - if it still doesnt, maybe those movements arent just in the model anymore? when you install blendshapes, do you do "exchange" or "add" blendshapes? make sure the original blendshapes from vroid studio are still there. - if nothing works, maybe your model needs someone expert in blender that can add those eye movements. hope something of these work out

[@LeahKinsXx]

@@ErinaVtuber I went through and did the whole process again to see if I missed anything, but I'm still having the same issues!! Not sure what's going on. I opened VSeeFace to see if the eye tracking works through there, and it does, so it has to be something weird going on with VUP? Maybe? I can find the corresponding eye movements for VUP but it won't actually move the eyeball itself still, just the eyelid. Works perfectly in VSeeFace though. :(

[@LeahKinsXx]

@@ErinaVtuber NEVERMIND! I'm just silly. Turns out there was an option that was on by default for me that was causing the avatar to focus on the camera! Under broadcast > broadcast > look camera

[@ngomez2551]

I've tried importing the Hana tool package but then the menu doesn't show up at all. I am using the same version of unity as you so I am not sure what's wrong.

[@ErinaVtuber]

it might be showing up on a second screen or somewhere on the left of unity, i remember looking for the tab for 10 minutes too and it was appearing somewhere weird on unity but i wasnt paying enough attention lol

[@Cosmic_Commentary]

@@ErinaVtuber where did you find it? cuz I'm still looking after 2 days... it doesn't seem to be anywhere

[@thinksi7715]

Hey Erina, thanks a lot for this simple tutorial! Is there much of a difference between Vseeface and VUP? Also, can you do all the customizations in VUP that you can do in unity?

[@livipup]

There are some significant differences between VSeeFace and VUP. VSeeFace is pretty barebones, so it takes much less of your computer's resources to run, but it doesn't have as many useful features. I personally find that VUP has better tracking too. That may simply just be preferences, though. The things Erina does in this tutorial require Unity. It is necessary for setting up custom blendshapes. VUP cannot do this on its own. What VUP does is interpret the input from your camera to scale the blendshapes based on the settings you set up. It cannot edit the model in way directly. As blendshapes are, simply put, dynamic modifications to a model's mesh, it is not possible add them in VUP. Even Unity can only do it because of custom plugins.

[@thinksi7715]

@@livipup Ah ok. that makes sense. Much appreciated!

[@happythekatt8419]

Have you found a fix for this error when trying to use the clip build? "ArgumentException: Calling apply or revert methods on an object which is not part of a Prefab instance is not supported. Parameter name: prefabInstanceObject" I have the latest version of unity, hanatool, and VRM0 as far as I'm aware. I'm really stumped on this one. Do I need an older version of VRM or something?

[@ErinaVtuber]

Are you using a vroid model from the beta or live version of vroid? And are you sure you're adding the blend shape to "face" in the .prefab file model unity?

[@happythekatt8419]

@@ErinaVtuber Model was created on stable version of Vroid and when I go to Hana tool/reader i go to the circle and under “scene” I clicked face and perfectsync_Vroid_v1_0_0_Female.txt and exchange and then read blendshapes. It says it worked. Then when I go to clip builder and select my now modified model and clip build it doesn’t do anything and I get the error I mentioned in my first comment.

[@happythekatt8419]

@@ErinaVtuber Question: What does the clip builder do? Could I theoretically go without that step, or does it fix something?

[@ErinaVtuber]

@@happythekatt8419 iirc they removed clip builder in the latest versions of Hana tool. So you should consider downloading the latest version from the link where you bought it from (should be free if you use the same account). :) Maybe the old one isn't compatible With the new vroid versions

[@nox_rayne]

Can you use Hanatool in vseeface?

[@ErinaVtuber]

Yes but I think the blend shapes will be visible only if you use arkit+iphone tracking. Please watch videos about this combo because I'm not familiar with them ☺️

[@anaarisu7239]

How i download univrm?

[@millaxc2414]

I'm having the save issue, lmk if you figure it out please

[@blushbunni3]

It's in the description

[@samuelarman9693]

I have a question, does HANA tool work in model thats not created in vroid?

[@ErinaVtuber]

Hello! i think they are originally created for vroid models, couldn't find any info on using it on other 3d models :/

[@samuelarman9693]

@@ErinaVtuber i see. Do you know fast way to create face expression blendshapes for non-vroid models?

[@CheshireCat478]

@@samuelarman9693 I dunno if you already figured this out but you can use blender + the Cats addon to make blendshapes(shape keys in blender)

[@samuelarman9693]

@@CheshireCat478 can you link me video tutorial that shows how to do that? I know cats plugin, it used to convert fbx file to vrm easier, create visemes, etc. But i dont know cats can be used to create face expression blend shapes.

[@inory5460]

My model doesnt even detect my smile in V seeFace :(

[@ErinaVtuber]

I suggest you to switch to warudo and join the discord server, the program is Insane good

[@inory5460]

@@ErinaVtuber I looked at it but its so complicated. I cant even get my model to move naturally or move at all. xx only the eye move and its creepy af >

[@sleepylunaart]

so you dont need an I phone for this right?

[@ErinaVtuber]

No IPhone, everything on PC :) the only difficulty is to sync your model mimics to the VUP reference but it's easy to understand

[@sleepylunaart]

@@ErinaVtuber well I kinda have trozbles with that are there basic emotions that can be triggered by your face without hotkeys like on vseeface?

[@ErinaVtuber]

@@sleepylunaart yes! it should be able to trigger specific expressions that you chose on the face channel mapping

[@0ceanBlossom]

Would anyone be able to do this for me? I’m lazy ngl

[@pixi_meat]

lmaoo would you pay?