Machine Learning Models in Unity with Barracuda: Image Classification

In Unity Tutorial Matthew Hallberg goes over running machine learning models in Unity with their barracuda inference engine. This allows you to run most ML models in .onnyx format inside the Unity video game engine for cross platform support on Android or iOS.
Image Classification Github Repo:
github.com/Thi...
Stay in Touch with Third Aurora
--------------------------------------------------------
Website: thirdaurora.com/
Facebook: / third-aurora
Instagram: / thirdaurora

[@tgw3356]

Coming from a ML Scientist here, a little note: Since you converted the RendererTexture directly to byte array before converting to tensor for usage as real time inference data. The byte array variable list input the image values from left to right, bottom to upwards in your method. Neural network models are typically trained with Tensor data which are formatted typically by describing an image Left to right, Top to downwards. And when you convert the byte array directly to tensor such as in your method using that 'for loop', your image's bottom left pixel are projected into the Tensor variable as the Top left pixels, resulting your input image flipped. May not be a big deal if your inference data quality is not affected even if your inference data are flipped such as your case (a flipped water bottle is still an image of a water bottle), but for some neural network model like a number recognizer, flipping the image means a different input data. Example for number recognizer neural network: Flipping input image containing 6 will give you a 9. I used your sample code to run an Eye Tracking neural network model and realised I couldnt infer anything meaningful for a long time, since the eyes orientation of a flipped facial image doesnt make any sense or has any relatable features from the original training data used which are proper orientated facial images.

[@AmardeepRM]

How can I work with SSD mobilenet V2 model in unity3D for detection bounding boxes?

[@StatsOverTime]

Hey, can you please make a tutorial on how to create an AR app that gets 3D models from the users to generate AR of the given 3D model. This will help product designers/architectures every time to view and check their 3D designs in 3D world before building it.

[@user-sl1eu1ez8g]

Great Video! I'm working on a project right now with AR Foundation, how can I apply use the AR Camera Manager instead of WebCam? Thanks in advance!

[@weirdough6254]

those glitches should become signature move of this channel. thatd be cool.

[@Giga23kabl]

Hey! I really like the way you present topics. Concise and on point while also leaving me to actually do my own research. Thank you :) Oh, and yeah, if you could make Compute Shader tutorial about preprocessing/postprocessing for ML models I would be forever thankful :D

[@sachchitkholkute7180]

thanks ! this video is really useful !!

[@65square_official]

Cool! Thanks for this. You still developing?

[@TheSararomano]

Hello! Thank you so much for the tutorial, it has been very useful! :) Do you think that the same process might be implemented with a Stereo Camera?? In that case, do you have any suggestion?

[@JadenGoter]

This is super cool. I wonder if I could get a text generation ONNX model running (GPT-2?).... and i wonder how it would impact performance in an actual game build

[@user-qv1ix1dr4c]

Hey, I want to do this but not with the web camera I want to use the mobile camera can you please help.

[@rexgatling4891]

Is there any way to run the model on the phone camera besides the Webcam?

[@lakpatamang2866]

Can you do another tutorial on loading tensorflow object detection model in onnx format.

[@pecke86]

22 fps in unity editor ? whats the performance gonna be once its deployed on a mobile phone ?

[@haraldgundersen7303]

Think ML Agents beta now run ONNX? Don't really get the diff between Baracuda and ML Agents? Kinda of got the impression : Barracuda out . ML Agents inn. BTW also think ML Agents now recommends pytorch?

[@monojit104]

How can I run pre-trained model of multi-level classifier(trained from python) in unity? Any suggestions?

[@houssemelhadj6881]

Hey! Thanks for your Tutorial ! I like your easy way of presenting things, awesome!! I'm working on a unity project to do hand gesture detection (to control the player movement) and emotion recognition (to change game difficulty based on the predicted emotion) using the webcam image. So I'm wondering is it possible to inference multiple onnx models in parallel and grab their results asynchronously. (Assuming I have the perfect models) If yes, how would you suggest doing it: Like here in this tutorial can we inference both of these two models that you tested (classification and emotion) at the same time using the same input image? if you can do a tutorial about this, I would be forever thankful to you :)

[@pooryapiroozan5265]

This is super helpful! thank you. 😍 I noticed that on Android, the image is rotated 90 degrees, so the cropped image doesn't get the proper frame. I tried some code to read the pixels considering the rotation but I didn't succeed. Do you know how to address it?

[@roveee6992]

Good job Poorya!

[@retrohead]

@@roveee6992 andreaaa 😁

[@zafirahzaini8227]

I'm using XR origin camera but my project freezes whenever I run it for both iPhone and android. Can u advise on how to solve this issue?

[@ariezflores]

Okay, this time i am first in the comment, and this helps me a lot , thank you!

[@MatthewHallberg]

Awesome Glad to help!

[@corriedotdev]

god damn.. this is helpful thanks. Im about to give this a whirl.. if i get it working, move to my own synthetic dataset and try train using the yolo detector on lidar scans. fucking scared lmao