Attempting To Recreate Hypixel's Server System - Gearflux Devlog 1.5

[@magical_whale]

This is looking really cool! One thing i do notice is how the proxy server is also the server manager (for spinning up the minigame/lobby servers). For me it would make more sense to have the manager run separately from the proxy servers, purely so you can eventually just spin up more proxy servers. It would also mean the manager isn't going to perform worse when the proxy server comes under (great) load.

[@pixobit5882]

One big optimization would be to always have a few servers of each minigame running so players can connect directly instead of waiting for the container to create and the server to spin up.

[@samuelhulme8347]

Or just the most popular mini games

[@mirokko]

... And you are waisting a lot of resources because someone cant wait for 15-20 seconds.

[@DeepfriedChips]

​@@mirokkonot to mention it usually takes more resources to fire up servers, that should only be done as a worst case scenario so there should be at least some always on servers

[@deidarakisame9203]

@@mirokko if you have a massive server this is money well spent of course

[@ShafterPlay]

​@@mirokko 15-20 seconds is a lot for someone to wait just for a minigame to load. "53% of mobile site visits are abandoned if pages take longer than 3 seconds to load" - you can apply the same logic here. For me, there should always be a server available for each minigame.

[@BigVirusBoi]

Man, I cant wait to see how this will turn out! Such high quality videos as well!

[@Lemonix0]

This man is gonna blow up. And Im gonna solidify myself as a semi-og. Honestly love the whole technical explanations.

[@gormdenorm678]

Cant wait to see what you end up creating :) I would probably consider having a buffer of servers ready for start. So when players wanna play: "Void Rush", they get instantly sent into the lobby. And when that gamemode starts the RabbitMQ will then initialize more servers so the buffer always remains the same. This way players dont have to wait to get response from clicking the NPC. A smart way is to make the buffer variable a setting so that you can edit how many servers RabbitMQ should have ready, and how many it should initialize to fulfill the buffer.

[@fleap]

Loving this series! How big minecraft servers works always interested me. As a student of computer science, i already tried re-creating Hypixels “MasterControl” to learn how it kinda works. This is a great distributed systems challenge and I hope you have the determination to do it! 👏❤️

[@hhhhhhhhhhhhhhhhhhhhhh]

Really cool project, I was planning to do something similar in the past but Kubernetes was a bit too scary for my taste, so I never really committed to it. However, here's some tips on what you might want to improve: I would think that Hypixel does not spin up lobby servers for every proxy. Instead they spin up proxies based on some factor of the current player count, since not all players will be in the lobby at the same time (though in your case that might happen when releasing the server to the public). That way all the proxies would be able to make use of all the lobby servers instead of having their own set of lobby servers. Also, as for minigames, I'd urge you to look into either Minestom (1.19), or Minestom-CE (fork of Minestom for 1.20). Minestom is basically Spigot/Bukkit without all the vanilla gameplay features, making it much more lightweight to run, which would probably be beneficial for minigames since in most cases you won't need vanilla gameplay features anyway, but if you do, you could always add them back yourself. Hope this helps, looking forward to what you are building

[@chonkydog6262]

hello fellow minestom user

[@onecrazygenius]

Awesome project idea. Remeber commenters, working before optimise. This man has almost recreated a system that can spin up servers and play minigame. Omce he has done that can look at having holding servers etc im sure

[@Torbikini]

This is neat, I just subbed. I’ve always thought about doing something like this, but it just seemed like a lot for just one dev

[@melom578]

this is such a high quality interesting video, im a big minecraft player but ive never tried writing a plugin, webdev only lol

[@rosebudcat]

awesome work and great commentary, you deserve much more attention for these types of videos

[@Slimyslimy101]

Nice video, the tech stuff is super interesting to me. Always wondered how these huge minecraft servers worked. I took a distributed systems class at my uni so was cool to see those ideas in practice.

[@joaoteixeira2254]

Great job man, keep bringing tech stuff that we love. I'm in my third year of software engineering and taking an AWS course, and i would love to be a part of that. Congrats for your accomplishments so far.

[@ctrlcheeb]

I just found this channel, I'm really interested and following along look forward to more

[@verssedd]

I can see you but a bunch of effort into this video, respect that. New sub

[@scoliosissy]

h

[@devwckd]

Great job, nice video! You might wanna separate the matchmaking/pod creation systems into it's own applications because if for some reason one day you need to run multiple proxies your system would not work properly since it's running as a (stateful) plugin.

[@p-pizza]

fire video botherman

[@slbubuzamazal5518]

cool serise please continue :)

[@alexdrum2873]

So a quick suggestion for the UUID system. Instead of doing them in the name, doing it as a label, means you can fetch the labels, and there's the UUID. Like you do the app label, adding a "uuid" label, could do exactly what you're looking for. I'm not great at Kuberneties but I know in Docker it works fine.

[@Sleepdeprivedddd]

Bro this is so sick

[@TrueKingRp]

These videos are so fucking interesting, good job dude

[@canable]

keep up the good work!

[@Berkshire-Hathaway]

Thank you for your really informative videos!

[@lample8284]

I can tell that this is already going to be epic

[@voigondev]

Very interesting video, very similar to something I am working on right now, but I noticed what seems like a bug In CreateLobbyServer it seems that you create the api client each time you create a lobby server, but also set it as the default one in the Configuration class, I guess you meant to create it just once for the entire instance of the plugin then re-use it wherver creating a server

[@TacticalTerere]

Again great Video

[@legally4946]

i just know that i will love this series! i just have couple of questions like : are you using a vps or dedicated to create the server?

[@ominousdev]

I will most likely host it on something like AWS, Google Cloud, or Digital Ocean. Hope you enjoy the series!

[@felixbemme7257]

​@@ominousdevI did thought about cloud services. But I think this might get kinda expensive overtime or what do you think?

[@jonmjauu6435]

@@felixbemme7257that would only be in the long run. It really depends on what type of cloud services he uses (price etc)

[@unfamousthomas]

Yeah, generally speaking a cheap VPS or Dedicated Server is always cheaper than Cloud for Minecraft. Minecraft relies on single core performance and differently from web applications you can't really spin up a Minecraft server in seconds usually. The pricing also goes very high and the performance to price ratio is quite bad. You can read more if you search for something along the lines of "cloud hosting spigot". There are a lot of threads on why not to do it.

[@pitpot2]

nice vid, didnt know that hypixel's servers were made on the go, but it makes sense now

[@RektOrNah]

Really cool stuff, for the lobby naming, I'm assuming you're caching it as a list, to save (even the most minor performance) use the numbered index prefixed by lobby E.G; LOBBY_0, LOBBY_1, LOBBY_2 etc instead of a uuid?

[@Kellan03]

Great job!

[@JustThiemo]

Pascal Case in Java 😢

[@unfamousthomas]

Yeaaah I also noticed that, to be fair class names pascal case is fine, recommended even but for method names and variables it is a bit... bad? Weird? i dont know

[@TechnicallyJava]

i think hypixel allows u to join games so fast is they will probably have some servers just open, when a server starts, it creates a new one, im not sure but this seems like a way to reduce wait time

[@bedlessdeus]

Hey, just recently saw this, seems like you've got this figure out pretty well, but, id like to make a few suggestions 1. For the Lobby Names/IDs, you could do a format like L125M, the L would be for Lobby, the 3 Numbers (could be expanded if more are needed) would be randomly generated, and the M would tell you the size of the Lobby, so L for large M for Medium and S for small, you could also use that for different gamemodes that require less players, to partition the amount of resources more efficeintly 2. Id recommend makig a fork of paper, and implementing you're extra player data logic and that kind of stuff directly into the server, to not have another layer and have it in a plugin! 2.2 It would be best, but a bit complicated and unnecessary for right now, but you could get a spigot 1.7 version, scrape everything that isnt need it off, and have an interface that loads each minecrafts versions assets, for best performance!

[@venomxl1655]

LOVE THIS VID!!! you should make a video on how to run this on k3s on a pi cluster!!

[@GTDWN]

Very good video, however would you consider turning down the background music a bit in the future?

[@ominousdev]

For sure! Thanks for the feedback!

[@xerathgaming1915]

i have an idea on a minigame you could implement. id love to see your spin on the evolution gamemode from mineplex

[@unfamousthomas]

There's a few things I would like to mention and clarify here. 1. Any specific reason you went with a MQ instead of Redis? Redis pub-sub is the standard for these type of things in the MC server space. 2. You might want to hold either a database or local server cache of the servers, and then query that when searching for a game server, that should make it significantly faster. 3. You might want to see if it would be worth to create custom CRDs instead of using straight containers. pods. If you do want to keep using straight pods, try to use Deployment instead, it links pods together easily and generates pods names. 4. Consider DevOps as well, right now the image name is hardcoded, which means if it has an update you have to manually go to code to change it to latest version. 5. As for naming logic, I don't think UUID is too bad, but I would recommend "-" instead of "." and also as server scales you might want to add a check if that UUID exists, otherwise Kubernetes will error. That is all for my feedback, still, very well done.

[@unfamousthomas]

Also, pod ip should stay the same while we keep the pod online, however, the standard way would be to create ClusterIP services.

[@matsv2]

Hi i watched these two parts fully for what u want I recommend making somting like a discord server to share or update ppl on ur code / server (can olso ask someon else to make the server thats up to u)

[@jasper1061]

What’s that shader??? :o

[@alexvernonpls]

Two questions: Do you ever plan on releasing this server that you're developing? And can later down the line can you deeply explain how minigames instances and managing those instances works? Good stuff

[@Hm-sv7pd]

Have you thought about making a discord for people to join to ask questions about this project? I could see it being quite useful

[@CalledCracki]

An extremely interesting project :D I'm looking forward to seeing what comes out of it. If you need support in programming unique game modes / systems, my team and I are at your disposal! 😇

[@pirateplayz3011]

just an idea for a minigame maybe like mob wars where you need to get iron and gold from like a mine and other drops you can get at map and who ever has least team deaths or if they kill every member on others team with the mobs they send over (respawn time 30-90 seconds probly) and you can also buy gear from the shops and tokens to get upgrade like x2 mobs or fast gens or item mult or prot and you need reasources for tier upgrade system as well as buying mobs or short time team effects such as 2min of streghth you can edit time or stuff but this just suggetion i would think would be cool

[@ominousdev]

That's a super interesting idea so kind of like a MOBA in Minecraft.

[@pirateplayz3011]

yeah but the only way you can get mobs on your side is ethier random game event, time event or if other team spawns em@@ominousdev

[@dertypmitnemvogel]

Hey Can you put the code in description or something please :)

[@jaxeli1208]

I thought of making the same thing but the server stuff was too hard I would love to help with the map and the minigame systems I also know how to make texture packs with custom items is there any way I can help.

[@jackdavenport5011]

What cloud service are you planning to run the server on? (e.g. AWS, Gcloud, Azure)

[@Tacobite]

How can someone get involved with this? I have previous experience and love this project!

[@beastbomber2316]

How do you handle Load Balancing on the main server. Do you got Round Robin DNS with multiple servers instead? Or is it the proxy you mentioned? Im just a bit confused lol

[@Katzen48]

I really would extract the server management from the velocity proxy. You are limiting yourself to one proxy at a time and whenever your proxy (which can and will happen) crashes, you get a huge problem.

[@CranFused]

6:00 why not assign your PODS a custom name? like the actual server's name and then till number x where x ∈ [1, ∞)? inshort, _

[@CranFused]

sorry if i got too much in detail 💀i feel like idk how to explain so i try to explain step by step- lmao

[@Mar_Ten]

What happens when the proxy dies? Seems like the whole application (server) needs to be rebooted. Basically seems like you made one big monolith? I think an external 'manager' application that can send commands to the servers might be more fault-resilient. It could restart on crashes and then stitch back the existing network under it's manager.

[@felipeitalian]

i would love to have a tutorial on this! i cant really understand any docket or k8s 😅😅

[@PolishKrowa]

Why wouldn't you use something like Cloudnet? Wouldn't it be faster and a definitely easier solution for the server networking? I haven't really used it a lot, but it does seem quite powerful.

[@hhhhhhhhhhhhhhhhhhhhhh]

Cloudnet is basically Kubernetes but specifically tailored for Minecraft server networks. It'd technically work but doing this with Kubernetes and Docker will make things more applicable for other situations in life because those are both professional software and get used by all the major tech companies. Besides, Kubernetes should be far better at scaling things in the long run compared to Cloudnet

[@jst]

Do you think you can ever make a tutorial series on how to set up such a system?

[@felixbemme7257]

I wouldnt really recommend setting up a system like this If you dont understand it. You will run into problems with kubernetes and docker sooner or later and If you you dont understand them you wont be able to fix them either.

[@unfamousthomas]

@@felixbemme7257 With that said, it could be somewhat of a gateway to get started with systems like this. I got started with Kubernetes with experimenting and breaking stuff too. Definitely don't use in production if you do not understand it, but if you just want to try for practice, then that might be worthwhile.

[@thehuggz-i9k]

Dude... fr making me want to try to do something like this.

[@cubedetre]

do you think the server will have more custom mini games, or mini games like skywars that already exist?

[@ominousdev]

I am aiming to have entirely custom mini-games!

[@cubedetre]

@@ominousdev smart! minigame servers right now feel very stale, and i don’t play them like i used to. excited to see what comes out of this

[@themirespl3896]

Do you have any public github repo?

[@CoggySings]

When can we play on the server???

[@mirokko]

I dont think that you need to manage lobby servers directly with plugins. Just autoscale them based on number of people that on those servers and call it a day for now. One client can't connect to multiple lobby servers anyway, so... No problems on that end, i suppose.

[@LowReso1ution]

Will there be a Discord or something similar we can join? super excited for this!

[@ominousdev]

Currently working on a Discord.

[@LowReso1ution]

@@ominousdev awesome

[@LittleCowMoo]

1:48 highlight: "Subscribe"

[@MikkelHebel]

I currently have my own small minecraft server with a big lobby a small build team and I have been wanting to expand into minigames for some time. This just made me want to build minigames servers even more. I have my own VPS server which I use to host everything myself. I would be more than happy to work together since you seem like a super interested guy with lots of experience and a creative mind. Let me know if you are interested in creating stuff together.

[@tommyinaboat8458]

I would to help you test everything out if possible 😄

[@ominousdev]

I am going to make a discord soon for people that want to get more involved in the project!

[@alexvernonpls]

Will you be releasing the source code for this whole system?

[@BedrockScorchAI]

This is actually sick. As an engineer this is very interesting. Imagine running a minecraft server on something like AWS and only using resources that the server actually needs. Im really interested in seeing where this goes. Good luck! ps. Do you have a discord server where we can see smaller updates or playtest?

[@unfamousthomas]

From what I have heard AWS (and other cloud) stuff is not really worth it in Minecraft. This is mainly because Minecraft server relies mostly on single core clock speed, which is possible to get good on cloud but usually a dedicated server is just cheaper and economically more worthwhile. There are a bunch of discussion threads about this if you search for stuff like "cloud hosting spigot". Possible it has changed since then but last I checked it was still better to get a dedi within your budget from OVH or Hetzner than to have a cloud system setup. Of course, you can leverage cloud systems for stuff like "oh no we dont have enough resources due to sudden surge, lets temporarily order a server" but thats about it as far as I know. I do also want the discord though.

[@samuelhulme8347]

@@unfamousthomasfolia is a multicore minecraft server made by the developers of Paper so that might work better on AWS

[@unfamousthomas]

@@samuelhulme8347 Possible but probably will still end up being cheaper to have dedicated servers or VPSes. On another note, I don't think Folia is production ready fully either, for example scoreboards do not work.

[@unfamousthomas]

Yeah, I did some quick research. Unfortunately it looks like Folia basically balances the cores out, but that means that you still need a beefy single core performance which is cheaper as a dedicated server or VPS.

[@Dumbity]

once your done you should release the server

[@DroopThatSnoot]

Can I help with your server?

[@davidscandurra560]

Baited with the Kotlin hashtag

[@Berkshire-Hathaway]

I'm so hyped. Could you maybe provide us the yaml files for kubernetes?

[@unfamousthomas]

To me (looking at the code) looks like there is no direct .yaml file, instead he specifies each attribute in code.

[@Berkshire-Hathaway]

I'm talking about the kubernetes yaml files ;)@@unfamousthomas

[@Berkshire-Hathaway]

It could also be that I'm misunderstanding your statement. Can you maybe elaborate?@@unfamousthomas

[@unfamousthomas]

@Berkshire-Hathaway Sure. Basically instead of having a yaml file and doing something like kubectl apply through code, he instead sets all the values you would usually set in a yaml file through code, e.g image, labels, ports etc. You can see this at around 4:30. He never references a yaml file and instead uses the java kubernetes library. You can also think of this as the imperative approach (as opposed to declarative that yaml files are) that you usually so via something like "kubectl run mypod --image=IMAGEHERE" Hope that helped.

[@Berkshire-Hathaway]

@@unfamousthomas oh, okay. So basically he's just doing the whole stuff the imperative approach instead of declarative. Sounds interesting. Thank you for sharing the info :)

[@ZolareaFN]

if ur gonna make it like playable server im gonna play it more than hypixel

[@fleap]

Will you open-source the code? 👀

[@Skaskey]

party games and tnt tag are the one of the best gamemodes against cheaters Is it possible to add 1.8 and. 1.9 gamemodes? we gotta bring the community together. id spend hundreds of hours for free to make this server more fun then hypixel, please let me help with the gamemodes/ building

[@phofers]

This is really cool! I've got an idea for a minigame. It's a Capture the Flag/wool style thing but is persistent so it lasts a while. Maybe just one or two games running at a time, because it's on a big map or even naturally generated small world that players can build on in an attempt to protect their flag, so a game could last for a couple hours or 1+ day. Basically, capture the flag but on a big map with lots of players, and it's hard to just rush in and win because players can build defenses.

[@MisterKattsu]

Discord Server?

[@Berkshire-Hathaway]

You cannot use pod IPs! Always use services! (Kubernetes: at minute 7:10)

[@Berkshire-Hathaway]

These IPs always change and you cannot rely on them. Use Services (e.g. ClusterIP Service)

[@unfamousthomas]

@@Berkshire-Hathaway While it is definitely not a good idea, I think it is fine in this case? IPs stay the same as LONG as the pod is running, presuming he is doing the usual minigame design pattern where they are deleted after use, using IPs is actually not too bad, of course, not ideal but not a completely horrible issue.

[@mc_lecraft80]

Hi, can you make a full guide on how to make a server? You seem really experienced and i have never seen something like this on youtube. Most people do like aternos tutorials or something like that, but you show how to make a professional high performance server. I have been trying to make a server for the past months but i never managed to make a good one, so i would be really thankfull if you could make a complete guide on this topic. From the start (buying a server or something) to the full running server.

[@Squawkykaka]

If your wanting an easy way to host a server I’d recommend self hosting pterodactyl panel(a way to create servers from a n if e gui) on something like oracle cloud (it gives you 24 gigabytes and 4 cores of cpi)

[@dgxo]

I've used pterodactyl and I think it might be too advanced for the majority of server admins, including people with years of experience. A good alternative that allows much easier setup and server management is Crafty Controller, they have a fantastic community as well.

[@LucHeartVR_]

your not gonna run into issues with a random uuid, thats fine.

[@Wvryn]

Hey, if you need a builder, I'd love to help.

[@realmazurex]

ok

[@aisflai8277]

U need all of that for a minecraft server network? im fcked

[@Bossy_Dawg]

make it cracked please

[@ForikiTheRat]

gl making an anticheat lol

[@ominousdev]

Definitely going to be a challenge lmao.

[@sinscript]

@@ominousdev AI TIME

[@codglitches1337]

why recreate plugins that are already made? huge waste of time

[@Kellan03]

It’s called, ** a learning experience **

[@codglitches1337]

its called just a waste of time. there is over 500+ plugins that do these same things. if u want people to be impressed create something challenging and new not hating on u lmao just saying this is pointless for your time and your watchers. Make something new @@Kellan03

[@ipigtaiwan]

​@@codglitches1337why u have to hate on him, at least he puts efforts to his video, unlike you just comments just to hate him

[@alejandroooow]

@@codglitches1337 it's called wanting to have total control over your own software and using your own structure for the code. I don't want to deal with someone else's code and have to work my ass off to have it work for my purpose. Hypixel doesn't even do that

[@IMC-zx6xl]

I wish you could go into more technical depth. Like, what service are you using for your Kubernetes cluster?