How To Customize the Unity Render Streaming Web Client

Unity Render Streaming over WebRTC is a fairly complicated topic and project. In our previous videos, we took a look at how we can Dockerize the Web Server and how we can Get Started with the Render Streaming examples. In this video, we will be diving a bit deeper into how you can customize the actual Unity Web Client project and make change accordingly within Unity in order to make sure the Unity client is communicating correctly with the Browser.
Relevant Links:
WebServer Docker : hub.docker.com/repository/docker/fuseman/renderserver
Unity Render Streaming : docs.unity3d.com/Packages/com.unity.renderstreaming@3.0/manual/index.html
Render Streaming Github : github.com/Unity-Technologies/UnityRenderStreaming
Github Open Source WebRTC : github.com/Unity-Technologies/com.unity.webrtc
Coturn Server Github : github.com/coturn/coturn
Trickle ICE Server Testing : webrtc.github.io/samples/src/content/peerconnection/trickle-ice/
Chapters:
0:00 Intro
0:55 Open Source
1:38 NodeJS Web Server Overview
4:34 Modifying the Example
6:46 Modifying Unity WebRTC
7:46 Sending Data to the Browser
9:48 Conclusion
Some Affiliate Links that help support the channel :
Order Oculus Quest 2 ; amzn.to/2G3NwXq
Check out our book on Amazon : amzn.to/2lmry7H
Leap Motion : amzn.to/2VDTcur
Looking to get into VR? Try Oculus Quest : amzn.to/2otkjfe
Or for PC VR try Vive Cosmos: amzn.to/2ILa6lo
Google Cloud Referral Link: gcpsignup.page.link/F1Fi
Want to Donate?
Bitcoin Address : 3QzAxbMBVr9nrq64AtcwjUoBqVgyjyHQcR
Ethereum Address : 0xB2B92DDAC0c8D976ee22Daf58Ff67AeBD194dE0d
----------------------------------------------------------------------------------
Welcome to FusedVR! These videos/streams are a chance not only for anyone to learn more about AR/VR development, but also a chance for anyone to ask questions about LITERALLY anything! Whether it be about your own AR/VR development, your future projects, or just AR/VR in general, we would love to talk with you about it!
Interested in keeping up to date with the channel? Follow us on social media or subscribe to the channel!
Twitter : FusedVR
Subscribe on KZbin : bit.ly/1SZXwtn
#rendering #unity #webrtc

[@logo8566]

I just recently started getting into Unity Render Streaming and struggled with modifying the WebApp example. This helped me a lot. Keep it up! As suggestion: Since my Unity+WebApp is now working I experienced issues while multiple users accessing the stream. I've read about SFU broadcast servers and will try to set it up the next days. Maybe some more people are interested in multi device/user support.

[@FusedVR]

Very cool! Good point on the multi user topic. I have a video coming up on how to set that up so that different users can control different cameras. But, probably makes sense to dive into the SFU Broadcast server

[@stefmartens7444]

Hi! I'm working on building a React dashboard where I want to display multiple streams simultaneously. Any advice on how I can achieve this? I've got my own Node.js Express backend with a basic WebSocket server configured, and Unity connects to it using the Unity Render Streaming asset. Do you think it's feasible to implement this setup? Any help would be much appreciated!

[@krakenchan]

You do realize your videos are gold... haha yes you do! thank you for this valuable knowledge!

[@chienthanhcong4729]

Hello, I am very grateful for the service effort render streaming, I am facing some problem and hope you can explain, i have imported this package into my unity project, also i have deployed webapp streaming to AWS, when the steaming server and unity client are used together sharing the same network, the data is displayed, but when another computer connects to the computer network other than the unity client, the web cannot display the data from unity i don't understand what the mechanism is here, hope you can explain it I?

[@patmw]

Major value as always! Thanks mate

[@FusedVR]

you bet! this rendering streaming stuff has me really excited!

[@Yarwad]

FuseMan, if I had a multiplayer mobile game, but I also wanted desktop players to be able to play through a browser could this be a use case for the webRTC methods you’re unpacking? Or is it better to accomplish my scenario in a different manner?

[@FusedVR]

@@Yarwad hmm, that's interesting. You could stream frames from the mobile game to the desktop via webRTC and yeah that could work to create a multiplayer game. alternatively, you could setup a seperate server that streams the game to both mobile and desktop

[@poojagarg9758]

Starting the webapp through npm commands does not make the connection. It shows black screen on videoPlayer play in browser. while if I do ".\webserver -w" and downloading the webApp exe from unity does make the connection. Please help me on this, I need to customize the webapp by streaming the customized data. Could you also please share the github link for metamask repo where you have done the changes shown in the video above.

[@FusedVR]

are there any errors you are seeing / are you passing the same parameters to npm?

[@tarawilde4507]

Excellent video, I'm definitely going to use some of the ideas you've had. I was struggling to understand the Unity side of the sending/receiving. Question: How can I configure my Unity game to use Private mode? I'd like one browser web-client to one Unity game and in the signaling server it looks like it expects both peers to share a connection ID? I can't seem to figure out where that connection ID is defined on the Unity side, nor do I even see a connection ID present on the requests coming from the Unity peer, unlike the web-client peer. Help is greatly appreciated!

[@FusedVR]

i thought there was a sample that shows how to use the private mode. but you can also change the code on the nodejs server code too

[@tarawilde4507]

@@FusedVR Not that I could find. Looked like only the bidirectional sample supported private. I ended up writing my own handlers for the signaling server and built connection pairs differently than vanilla but that seemed to do the trick. Thanks for taking the time to reply! :)

[@Zuzu7570]

I connected a camera device to a websocket server . When i tested it with a local server it works fine with me and i see the video . I tried next to connect this websocket server with UNITY 3D and i used a nativewebsocket packages , a message in the console appears "connection open" but nothing was shown meaning i don't get a jpg byte displayed . When i close the websocket server in Cmd , a message appears "connection close" . Any help please, despite there is a connection between the server and client but nothing was shown to me in the Unity 3D display .

[@FusedVR]

when you say nativewebsocket package, what do you mean? are you using the render streaming package from unity?

[@Zuzu7570]

@@FusedVR i used two libreries to create a web soket client in unity : nativewebsoket and websocketsahrp the nativewebsocket makes it easier to create websocket client based on websoket-sharp . can i contact u by messages please to send you my codes which i have written in my project.

[@FusedVR]

@@Zuzu7570 sorry dont have bandwidth to review code

[@БямбасүрэнГанболд-т7ь]

Nice explanation. Can i stream user camera to unity application?

[@FusedVR]

yeah should be possible as there is bi-directional support

[@Mrsithu86]

Hi dude, thanks for the nice explanation. I'm just wondering, how can we interact with Unity Ui from the browser? Is it possible?

[@FusedVR]

should be possible. the browser is sending the mouse input to Unity and so the UI should be able to react to that

[@Mrsithu86]

​@@FusedVR Hi dude, I did some tests on Renderstream with unity UI, and it works. Unfortunately, UI interacts only with Unity Editor gameplay and not working on Runtime (.exe). Do you have any idea?

[@FusedVR]

@@Mrsithu86 intresting...do you see any errors in the logs? seems like it could be an issue with how the unity input system is setup

[@lyndaphone]

Hi thanks for the video, it really helps a lot, do you have tutorial about the Http signaling setting for the render streaming in Unity as well?

[@FusedVR]

same process should just work actually if you just switch the switch to Http signaling

[@xavierpuigfernandez6142]

Has anyone managed to run this on Linux? Whenever I try, I get: error code ELIFECYCLE

[@FusedVR]

funny timing, just succeeded in getting it working in ubuntu last evening...lots of trial and error but will most a video in the near future

[@xavierpuigfernandez6142]

@@FusedVR This sounds fantastic, if you have any leads, please let me know. I am using this for research purposes and your videos have been extremely helpful in setting renderstreaming. Keep up the good work! Looking forward to your video.

[@FusedVR]

@@xavierpuigfernandez6142 some of it might depend on your graphics card, but I was looking at using Ubuntu with a Tesla T4 on AWS this image helped a lot with finding the right drivers : aws.amazon.com/marketplace/pp/prodview-p64lyuknl2pys?ref_=beagle&applicationId=AWSMPContessa and then make sure you have both OpenGL and Vulkan installed succesfully finally also make sure you install these packages listed for WebRTC docs.unity3d.com/Packages/com.unity.webrtc@2.3/manual/index.html

[@xavierpuigfernandez6142]

@@FusedVR thanks a lot for the info! Is that to run a unity executable on linux or also to run the node webapp? I have been able to run unity on ubuntu in the past using openGL, but the issue seems to be at running the web app with node.js now

[@FusedVR]

@@xavierpuigfernandez6142 yeah that is for running a standalone on Linux. hmm have not tried from a web client in ubuntu, but i have tried from within a quest (android) and it works smoothly the web server i know can run in linux though as i have been able to successfully dockerize that - so seems like it could be a browser issue?