MMO Architecture overview with Unreal Engine

[@misiek08]

You want to use Kafka for live game? Lags and latency will kill all of the joy. Maybe for some card games it can fit well, but nothing really dynamic will fit it.

[@unreal-dev-with-yaz]

hey :) kafka is used here to distribute the messages across different nodes, its used for an MVP as its easy to integrate, there are some better alternatives now i believe i expect it to increase latency and as such don't think it will be suitable for very fast paced games, e.g. real time shooter games in terms of how much latency it adds, what info are you basing this on? the compute instances would ultimately be hosted close together in terms of load, the largest would be motion information if i check kafka performance benchmarks: developer.confluent.io/learn/kafka-performance/ lets say it adds 10ms overhead latency, which from graph there would be comfortably within 99 percentile for load of 200mb/s (which is quite a lot), is that a dealbreaker for the added flexibility? in my opinion that's acceptable, but i understand some would consider that too high. bear in mind, there will be other optimizations put in place, such as UDP leveraged to push motion data

[@rohansingh1057]

@@unreal-dev-with-yazActually Kafka is inefficient for this as the avg latency of Kafka goes up to 100ms+ as soon as your CPU usage hits 50%. However it does not crash even if it hits 100% CPU util. Instead latency keeps increasing linearly till a crash. The most wasteful part about Kafka is the fact that it stores all messages in disk as soon as it receives it. Do we really need to do that? I think an in memory system will work just fine. It’s ok if we miss a few messages. In case of a crash. That is what packet loss is in a game Instead, use something like rabbitMQ/redis in a smart way. RabbitMQ latency and CPU usage is much smaller compared to Kafka. Its typical latency is in microseconds (not ms). And keeps up with performance all the way up to 100% CPU usage. (After which it crashes) So although we still have to use a broker/stream, rabbitMQ would probably fit the PS better. Even though kafka will perform decently when not in load.

[@pzeub]

I wish setting those up is a easy as to understand your tutorials. Amazing content YET I cannot deliver such eventough I'd wish too

[@0x_Anakin]

Im rly curious about server authority, validating players movement and latency compensation with this approach

[@unreal-dev-with-yaz]

Hey :) So when it comes to players, the custom server will need to have authority. There would be 2 main scenarios: Client says: my new motion is Server responds: OK, passes this motion to listening players Or Client says: my new motion is Server says: NO, your new motion is , passes this motion to listening clients if different (either pure reject (last good motion) or acceptable motion relative to the requested motion) This validation is not yet implemented by the way. In terms of latency compensation This can be applied to motion as they are bound values. i.e. players cannot move faster than X units per second (configurable) - we know the timestamp of last (accepted) update, hence we know what’s acceptable range with new update, even if there’s frames dropped or latency. Bear in mind, skills that affect motion (e.g. teleport) will undergo separate, independent validations. Since this is something that’s not yet implemented, I’m also open to ideas, but I think this is a relatively easy and scalable way of handling motion updates

[@0x_Anakin]

@unreal-dev-with-yaz This sounds promising and I'm very curious about how it performs once all that is done. I come from a similar dev background, you could swap Kafka with redis too just saying. I'm wondering tho if with this approach the latency is more than expected compared to a traditional setup. Have you tested latency even in a local ideal setup?

[@unreal-dev-with-yaz]

@@0x_Anakin Sounds good :) Btw redis not quite optimal choice against Kafka as you’d like a solid queue implementation (SQS queue for example should be better, I used Redis as queue before, its not bad but not quite as rich in features :)) I have in fact added redis as a shared cache in my latest PR though, so will be utilising it more in near future :) As I am implementing combat logic, I need to cross validate against other actors’ motion and redis will be great for sharing that context In terms of latency, of course it has to be higher than if you didn’t split the load with other servers, after all, there are more steps involved (you refer to Kafka overhead I assume?) However it doesn’t add much (benefits outweigh this significantly), perhaps I will create a video providing the latencies between msg sent on client1 to update received on client2 👍

[@deluroth-eg1tg]

Great content, how did you go about learning all these concepts? Any tutorial suggestions?

[@unreal-dev-with-yaz]

hey! :) well, for this one in particular, I think would be difficult to suggest 1 thing for me personally, it came from years of experience working in several different companies and getting involved with software architecture quite a few of my friends are also developers and we generally just chat about the new and upcoming technologies software architecture changes quite quickly so going and viewing tech conferences could be a great way of getting exposed to some of the new techniques and staying up to date there was a nice book on tech architecture that I read in the past, I believe it was this one: Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems but it's not specifically for game development then its about looking at topic specific things and seeing how it can all fit together, for instance see Networking for Physics Programmers: www.gdcvault.com/play/1022195/Physics-for-Game-Programmers-Networking this suggests that for motion networking I should use UDP, however for inventory management, I may still consider TCP for its reliability

[@deluroth-eg1tg]

​@@unreal-dev-with-yaz you are the man! sorry for the late reply but you went quite in depth! That sounds like the main thing I am lacking is experience I guess I'll start by just prototyping co-op networking games to understand and successfully implement this on a smaller scale before I fiddle around with Kafka etc.. I'll give myself a project and see how far i can get. UDC for motion and TCP for inventory definetly makes sense.. I didn't even think about different application of network protocols yet Anyways thanks again and I'll check out the link next

[@theonlyponguin]

this a high level question, i’m still learning about UE replication and serverless is new to me. but how is the message queue (kafka) faster than the built in replication? in the case where there are thousands of players would kafka not start to slow down to send & receive all of the data in order? or are cpu’s actually just that fast now? or is this just a problem that exists in both solutions but less so with serverless

[@unreal-dev-with-yaz]

Hey :) This approach is not actually serverless, that’s something else too :D This is a horizontally scaled system (i.e. multiple server nodes can be spun up to cater for load) Vs vertical scaling, where you add more resources to single server instances Kafka is a key to connecting relevant nodes When you have more players join, you will have the option to scale Kafka by adding additional brokers to split the load and cater for it, it scales very well. I drew Kafka as one line, but underneath, its broken into multiple brokers If interested, you can learn more on Kafka cluster scaling here for example: developer.confluent.io/courses/architecture/cluster-elasticity/

[@kaito5790]

Allowes this approach to sync gameplay specific logic from UE-servers? For example, the Gameplay Ability System?

[@unreal-dev-with-yaz]

so unfortunately not out of the box. it would be possible to manually handle this and it can work, but it would require a bit of effort. potentially a feature that could be done in future :)

[@kaito5790]

@@unreal-dev-with-yaz I'm looking forward for it. The GAS is so flexible and powerful, it would be insane replacing it with own code.

[@MaiuOnline]

Hey, Regarding architecture... I have small question about http vs websockets for not realtime stuff like creating characters, authenticating, even picking drop items. Is it good idea to use there http and for game state, like movements, combat etc use websockets? Or just sticking to websockets for both type of operations is better choice?

[@unreal-dev-with-yaz]

in long term I actually plan to introduce UDP also So the game will be split into several communication channels 1. before game session begins, i.e. login, character create/select screen. These will all be handled via HTTP this can also be useful when displaying stats on website for example as you can leverage same endpoints 2. most in-game logic I will leverage using websockets, so inventory control, starting skills and interactions, etc. 3. actor motion is currently using websockets, but it will become a bit heavy. I plan to extract and put it into its own service running using UDP for better performance inventory using HTTP/Websockets - the reason websockets are better is because I'd be leveraging the sessions notify functionality, rather than client polling for updates. (for example when item is dropped on map, or when you equip item and you need to notify other users to update the mesh) this is much more performant

[@palto567]

Epic is releasing a new network model called Iris (still experimental) that will improve the number of clients that can connect to a UE dedicated server comfortably, however I have not seen official numbers as to how much of an improvement.

[@LuizGamingYT]

hello, what u thing, how many players can handle one server with 10 map, and how many UE servers i can run (with different ports like 7777, 7778 etc) per one instance xeon 3.7ghz, 64gb ram, NET 1gb/s? I wonder how many people can be on 1 map and on 1 server, and on 1 dedicated machine. Game project - very low poly style without complicated game mechanics. I also wonder if I am able to host, for example, 200 players on 1 EU server, scattered over several maps so that they do not see each other too much, to avoid unnecessary replication. I know that in the case of Fortnite the server can carry 100 players, but on 1 map. Do u have any experience? Greetings!

[@unreal-dev-with-yaz]

hey :) there's some complexities in evaluating these things and will highly depend on your game I'm assuming you're referring to native UE replication each map can be run on an independent server, you can also split map into zones which can be run on other UE servers for more flexibility (e.g. check OWS: kzbin.info/www/bejne/r4O7q5WXg7lplc0si=GITiGXNxtg9D5BaZ) the server does not load UI, so low poly etc won't have impact on the network load I'm not sure how you deploy but you can check cost calculators to get an idea of requirements and costs: playfab.com/mps-calculator/

[@LuizGamingYT]

@@unreal-dev-with-yaz Thanks!

[@Shangri-la95]

Hey , do you have a video of how to do a dedicated server vor mmo games? Like to run it on playfab or aws. I cant rlly finde helpful videos online 😅

[@unreal-dev-with-yaz]

hey :) not yet, perhaps I will look to push something simple to AWS/Playfab - it can be quite complex subject :D

[@PHaRTnONu]

only comment, would have LOVED to have hardware references when your talking about Kafka & the websockets and all aspects, EG server set up hypotheticals, bottle necks due to what circumstances in what configuration and so on. I get that wasnt your goal here, im just saying i ran across this video looking for such information and it never seems to be broken down as cleanly as you did here for the service side

[@unreal-dev-with-yaz]

Thanks! It is definitely something that I’d like to cover particularly when looking at actual deployment (which may take some time to get to though :D) It’s quite common to try draw up the architecture diagram referencing those components to help understand which connections need to be made and it can also help with ‘seeing’ where the potential bugs may come from :)

[@naruine7631]

great stuff, quick question ue servers that control mobs is that small enough to run in docker images? Is it possible to make a design where every type of instances/workers are deployed with kubernetes and docker images?

[@unreal-dev-with-yaz]

I'm not sure about the UE servers running on docker and kubernetes, but all the other services will be able to :) definitely would be great if it was possible, may have to check that up. for the UE deployment in general, this will likely have its own separate pipeline and again will depend on what tools you use to deploy it, e.g. with PlayFab or AWS etc

[@naruine7631]

@@unreal-dev-with-yaz I see thanks, do you think using docker would be even helpful or probably same effort than just using PlayFab and the other solutions?

[@aerocodes]

@@unreal-dev-with-yaz Thanks, How would playFab play a role in this architecture? I'm currently working on the systems design of a MMO architecture and it's been interesting. Do you have a discord group of the likes?

[@unreal-dev-with-yaz]

@@aerocodes Hey :) PlayFab can be used to host the UE server, such as the ones controlling mobs I may also consider using PlayFab authentication too, to handle player login, rather than building that layer in my custom server. They have other nice services & APIs that I will potentially explore

[@BBeeR7]

hi, have you every tested your solution in live during huge amount of players or event perf testing? it does not look like that can handle even 100 players It looks like you're just trying to apply web app microservices architecture into the game - isnt good idea. It make sense to add such logic into Communication component (chat, leaderboard etc) or fill some statistic data etc but not gameplay stuff

[@unreal-dev-with-yaz]

hey :) I'm not going to have a complex debate in comments section, however if you think this approach will struggle with 100 players, then you've probably misunderstood the approach :D perhaps you will want to check this doc about how such an approach has been scaled to work with millions of concurrent users, albeit its not an 'mmo game' environment, but those are slow to adapt to new web technologies (this doc is referenced in the post also). www.confluent.io/blog/real-time-gaming-infrastructure-kafka-ksqldb-websockets/

[@brianmorin]

​@@unreal-dev-with-yaz for what it's worth, I think the solution will scale to a few thousand depending on details around gameplay and how clustered players are. There are a lot of similarities to how Rift was set-up. However, Kafka is a peculiar routing choice and the JVM is going to have challenges with GC causing server hitches. My concern would be the smoothness and response of the simulation given all the work we had to do in Rift to reduce and speed up RPCs. With a huge pile of custom C++ a similar architecture got to the low thousands in 2010, with better hardware I would expect it to do quite a bit better depending on how you reign in N^2 from entitles within vision of each other. Also with current hardware you probably could manage to put more processes on the same machine to get traffic off the network card trading off performance for a larger blast radius when hardware fails.

[@cryptogam3r69]

I really appreciate this video, I have always dreamed of creating an Oldschool Runescape like multiplayer rpg and hoping to have servers with at least 500-1000s players per world and this seems like a step in the right direction to be able to handle these workloads

[@unreal-dev-with-yaz]

yes I actually keep Runescape in mind when developing the server :D as well as some other mmo's such as WoW

[@cryptogam3r69]

IK you already have a blog but have you considered opening a discord server?? would love to chat with you since yyou already have a lot of resources i will most likely be following in the near future! i am trying to build something similar to oldschool runescape as far as point and click, many skills to train, and player freedom. but the setting and story is completely different. i already made a bunch of 3d assets, and have a musician friend who made some banger tracks, and we even have 2d concept art as well! but as we have no lead dev im gonna try and put it together haha@@unreal-dev-with-yaz

[@moritz164]

Can you do a tutorial on how to set up a best practice setup for this mmo architecture?

[@unreal-dev-with-yaz]

Hey :) Do you mean setting this up in deployment? I was planning to do this closer to when an MVP is done (which is still quite far :D) But perhaps I can try do this after finishing the basics of the combat feature, as I expect I added most of the necessary components already